A national vocal dialect prosody intelligent correction method and system

Abstract

This invention proposes an intelligent error correction method and system for ethnic vocal music dialect prosody. The method includes: collecting multimodal data, extracting features to form a training dataset, constructing and updating a dynamic dialect prosody map, and performing cross-modal comparative analysis to obtain a joint representation of cross-modal error features. A neural vocoder direct-connect correction model is constructed, inputting the joint representation to generate a corrected Mel spectrum. A cross-modal generative adversarial network is constructed, including a generator and three discriminators. The generator generates a joint latent representation based on high-fidelity corrected audio and three features. The three discriminators respectively judge the naturalness of the audio, the compliance of the text tone, and the rhythmic coordination of the musical score. This invention solves the problem of manual error correction by constructing a prosody map, overcomes the bottleneck of manual detection by utilizing cross-modal comparative analysis, improves the generalization ability of the model by combining meta-learning algorithms, and optimizes the correction results with the help of adversarial networks, thereby reducing manual costs and improving the error correction efficiency of ethnic vocal music works.

Description

Technical Field

[0001] This invention relates to the field of intelligent error correction technology, and in particular to an intelligent error correction method and system for the prosody of ethnic vocal dialects. Background Technology

[0002] In the field of ethnic vocal performance and inheritance, the accuracy of dialectal rhythm is a core element in reflecting the regional characteristics and emotional expression of a work. Current technologies routinely rely on manual intervention to correct dialectal rhythm errors: vocal teachers or music producers repeatedly listen to recordings, combining notated scores with dialectal lyrics, comparing the pitch, rhythm, and tone in the audio frame-by-frame with the text annotations. For example, when correcting vowel co-pronunciation errors in Minnan songs, it is necessary to manually check whether the duration of nasalized vowels in the audio matches the notation and manually adjust the pitch curve to match the dialectal tonal characteristics. Such methods highly depend on the experience and auditory sensitivity of professionals and require cross-validation of the consistency of multimodal information (audio, score, text), resulting in a time-consuming, labor-intensive, and inefficient error correction process, making large-scale application difficult.

[0003] However, manual error correction has significant limitations: First, it is highly subjective, with different experts potentially making different judgments on prosodic errors, resulting in a lack of unified standards for correction results; second, it suffers from significant efficiency bottlenecks, with the cost of manual annotation and correction increasing exponentially when faced with massive amounts of ethnic vocal music data; third, it lacks the ability to integrate multimodal information, making it difficult for manual methods to systematically capture the dynamic relationship between audio features (such as fundamental frequency trajectory), musical score structure (such as rhythmic patterns), and text tone (such as dialect tone values), easily leading to problems such as "disconnect between corrected audio and musical score rhythm" and "contradiction between tone annotation and actual pronunciation"; fourth, it has limited ability to detect subtle prosodic errors, such as the weakening of vowels or tone fine-tuning in coarse articulation, which are often overlooked due to the threshold limitations of manual listening. Summary of the Invention

[0004] This invention aims to at least solve the problems of high subjectivity and low efficiency in existing technologies, and innovatively proposes an intelligent error correction method and system for ethnic vocal dialect prosody.

[0005] To achieve the above-mentioned objectives of this invention, this invention provides an intelligent error correction method for the prosody of ethnic vocal dialects, the method comprising: S1. Collect multimodal data and extract features from the multimodal data to obtain audio features, musical score features and text features. Based on the audio features, musical score features and text features, form a structured training dataset. The multimodal data includes ethnic vocal music performance audio, corresponding symbolic musical score and dialect lyrics text. S2. Construct a dynamic dialect prosodic map based on the structured training dataset, and update the dynamic dialect prosodic map; S3. Perform cross-modal comparative analysis on the structured training dataset to obtain a joint representation of cross-modal error features; S4. Construct a neural vocoder direct connection correction model using the updated dynamic dialect prosodic map as a constraint condition. Input the joint representation of the cross-modal error features into the neural vocoder direct connection correction model. Generate the corrected Mel spectrum through transposed convolution upsampling. Reconstruct the high-fidelity corrected audio based on the Mel spectrum using the vocoder of the neural vocoder direct connection correction model. S5. By introducing a small number of erroneous samples, the model-independent meta-learning algorithm is used to iteratively optimize the neural vocoder direct-connection correction model. S6. Construct a cross-modal generative adversarial network including a generator and three discriminators. The generator uses the high-fidelity corrected audio as a base and combines the audio features, musical score features, and text features to generate a joint latent representation through a Transformer encoder. The decoder then outputs the corrected audio, musical score, and text trimodal data. The three discriminators are used to perform adversarial discrimination on the naturalness of the corrected audio, the tonal compliance of the text, and the rhythmic coordination of the musical score, respectively, to obtain the discrimination results of the corrected audio, musical score, and text trimodal data.

[0006] On the other hand, the present invention also provides an intelligent error correction system for the prosody of ethnic vocal dialects, the system including the aforementioned intelligent error correction method for the prosody of ethnic vocal dialects; the system further includes: The acquisition module is used to acquire multimodal data and input the multimodal data into the intelligent error correction method; The processing module is used to execute the steps of the intelligent error correction method, process the input audio, music score and text data, and generate corrected audio, music score and text three-modal data and discrimination results; The storage module is used to store the raw data, intermediate data during processing, and the final corrected audio, musical score, and text three-modal data and discrimination results; The output module is used to visualize the corrected audio, musical score, and text three-modal data and the discrimination results.

[0007] The beneficial effects of this invention are as follows: First, it constructs a dynamic dialect prosodic map, achieving collaborative training of cross-regional dialect data through federated learning. Combined with incremental learning and Bayesian optimization of dynamically expanded map rules, it effectively solves the problems of data silos and rule rigidity in traditional manual error correction. Second, based on the error joint representation extracted through cross-modal contrastive analysis, it can systematically capture the dynamic correlation between audio fundamental frequency, musical rhythm, and text tone, overcoming the bottleneck of insufficient multimodal information integration in manual detection. Third, the neural vocoder direct-connection correction model combined with meta-learning algorithms requires only a small number of samples to quickly adapt to different dialect types or error types, significantly improving the model's generalization ability. Finally, the cross-modal generative adversarial network optimizes audio naturalness, text tone compliance, and musical rhythm coordination through three sets of discriminators, ensuring that the correction results conform to dialect pronunciation habits and provide a high-fidelity auditory experience. This method automates the entire process from error detection and model correction to quality assessment, significantly reducing the cost of manual intervention and improving the error correction efficiency of ethnic vocal works.

[0008] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0009] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of an intelligent error correction method for the prosody of ethnic vocal dialects according to the present invention. Detailed Implementation

[0010] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0011] Example 1 like Figure 1 As shown, a method for intelligent error correction of the prosody of ethnic vocal dialects is disclosed, the method comprising: S1. Collect multimodal data and extract features from the multimodal data to obtain audio features, musical score features and text features. Based on the audio features, musical score features and text features, form a structured training dataset. The multimodal data includes ethnic vocal music performance audio, corresponding symbolic musical score and dialect lyrics text. In this embodiment, the method for extracting audio features is as follows: First, the collected ethnic vocal music audio is pre-emphasized, framed, and windowed. The pre-emphasis coefficient is set to 0.97, the frame length is 25ms, the frame shift is 10ms, and a Hamming window is used to suppress spectral leakage. Then, Mel frequency cepstral coefficients (MFCC) and their first and second order differences are extracted, totaling 39 features to capture the spectral envelope and dynamic changes. At the same time, the fundamental frequency trajectory (F0), fundamental frequency jitter, amplitude jitter, and harmonic-to-noise ratio (H) are calculated. NR reflects the pitch stability and purity of pronunciation; additionally, for dialect prosodic characteristics, tone contour features are extracted (by aligning the fundamental frequency sequence with the text tone annotation through dynamic time warping to generate tone curve parameters for each syllable), dialect-specific nasalization degree of vowels (calculated based on the energy proportion of nasal formants in the spectrum), and initial consonant duration ratio (the ratio of the initial consonant segment to the total duration of the corresponding syllable); finally, all extracted audio features are Z-score normalized to eliminate dimensional differences and form a standardized audio feature vector.

[0012] In this embodiment, the method for extracting musical score features is as follows: First, the MusicXML format data of the musical score is parsed using the music21 library to extract basic structural information such as note sequences, rest positions, time signatures (e.g., 4 / 4 time, 3 / 8 time), key signatures (e.g., C major, G minor), and tempo signatures (e.g., ♩=120). Next, core attributes are calculated for each note: pitch is converted to MIDI encoded values ​​(range 0-127) to quantify pitch, relative time values ​​are calculated using quarter notes as the base unit (e.g., 0.5 for an eighth note, 3 for a dotted half note), and absolute time stamps (accurate to milliseconds) are generated based on the accumulated time signatures to reflect the note's temporal position. To meet the adaptation requirements of ethnic vocal dialects and rhythms, three additional types of distinctive features are extracted: one is note- The system employs three main techniques: first, text syllable alignment features (aligning the musical note sequence with the dialect lyric syllable sequence using a dynamic time warping algorithm, marking the syllable index and duration covered by each note); second, mode-specific pitch features (mapping pitches to the scale degrees of the current mode, such as C being I and E being III in C major, capturing the mode tendency of the melody); and third, rhythmic pattern association features (statistically calculating the frequency of rhythmic pattern changes, the probability of ligatures, and the proportion of cross-measure syncopation within each measure, quantifying the matching degree between the melody rhythm and the dialect recitation rhythm); finally, one-hot encoding is performed on discrete features (such as key signature and meter type), and Min-Max normalization is performed on continuous features (such as relative time values ​​and absolute time sequences), forming a structured musical score feature matrix with dimensions unified with audio and text features.

[0013] In this embodiment, the method for extracting text features is as follows: First, a dialect-specific word segmentation tool (such as LTP-Cantonese for Cantonese or MinNLP for Minnan dialect) is used to segment the original dialect lyric text to solve the problem of differences between dialect words and Mandarin words. Then, a syllable segmentation algorithm (combined with a dialect phonetic dictionary, such as the syllable table in the Great Dictionary of Chinese Dialects) is used to split each segmented unit into three parts: initial, final, and tone. For example, the Minnan dialect word "Gulangyu" is split into "gǔ (initial g, final ǔ, tone rising tone)-lǎng (initial l, final ǎng, tone rising tone)-yù (initial y, final ù, tone falling tone)". Then, each syllable is labeled with the tone type unique to the dialect (such as the 9 types of high level, high rising, high falling, low level, low rising, low falling, high entering, mid entering, and low entering in Cantonese, represented by digital codes 1-9), and the accuracy of the labeling is verified based on the dialect tone dictionary.

[0014] Secondly, extract the phonetic and prosodic features unique to the dialect: For the initials, calculate the degree of voicing (count the proportion of syllables with voiced initials in the total number of syllables, such as the initial of the word "Wu" in the Wu dialect is the voiced sound / ŋ / ); for the finals, mark the nasalized finals (such as the final / ã / of the Minnan dialect word "shān"), stop codas (such as the coda / p / of the Cantonese word "yā"), and other dialect-specific phonemes, and calculate their occurrence frequencies; for the tones, extract the connected tone sandhi features (based on the dialect tone sandhi rule library, such as when two rising tones in Mandarin are pronounced in succession, the first character changes to a rising tone, while in the Minnan dialect word "Xiamen", the tone of "Xia" changes from rising tone to rising tone), and generate a tone mapping table before and after tone sandhi; in addition, combine semantic units to divide the prosodic pause boundaries of the text (based on the dependency parsing results, mark the pauses between the subject-predicate and verb-object structures as first-level pauses, and those between the modifier-head structures as second-level pauses), and count the syllable number distribution in each pause interval.

[0015] Finally, feature encoding and normalization: Use a pre-trained dialect language model (such as fine-tuned BERT-Cantonese or RoBERTa-MinNan) to extract context semantic features from the segmented text sequence to obtain a 768-dimensional semantic vector; perform one-hot encoding on discrete features (such as tone types and coda types) to convert them into binary vectors; perform Z-score normalization on continuous features (such as the voicing ratio and the number of syllables in the pause interval); finally, concatenate the semantic vector, one-hot encoded vector, and normalized continuous features to form a standardized text feature matrix that matches the dimensions of the audio features and score features.

[0016] S2. Construct a dynamic dialect prosody map based on the structured training dataset and update the dynamic dialect prosody map; S3. Conduct cross-modal contrast analysis on the structured training dataset to obtain a joint representation of cross-modal error features; S4. Construct a neural vocoder direct connection correction model using the updated dynamic dialect prosodic map as a constraint condition. Input the joint representation of the cross-modal error features into the neural vocoder direct connection correction model. Generate the corrected Mel spectrum through transposed convolution upsampling. Reconstruct the high-fidelity corrected audio based on the Mel spectrum using the vocoder of the neural vocoder direct connection correction model. S5. By introducing a small number of erroneous samples, the model-independent meta-learning algorithm is used to iteratively optimize the neural vocoder direct-connection correction model. S6. Construct a cross-modal generative adversarial network including a generator and three discriminators. The generator uses the high-fidelity corrected audio as a base and combines the audio features, musical score features, and text features to generate a joint latent representation through a Transformer encoder. The decoder then outputs the corrected audio, musical score, and text trimodal data. The three discriminators are used to perform adversarial discrimination on the naturalness of the corrected audio, the tonal compliance of the text, and the rhythmic coordination of the musical score, respectively, to obtain the discrimination results of the corrected audio, musical score, and text trimodal data.

[0017] The principle of the intelligent error correction method for ethnic vocal dialect prosody in this embodiment is as follows: Firstly, by constructing a dynamic dialect prosodic atlas, this invention achieves accurate capture and representation of the prosodic features of ethnic vocal music. This atlas not only encompasses data features from three modalities—audio, score, and text—but is also continuously and dynamically updated through federated learning and incremental learning algorithms, ensuring comprehensive coverage and timely updates of dialect prosodic features. In the cross-modal comparative analysis stage, this invention utilizes deep neural networks to jointly analyze audio, score, and text features, effectively extracting a joint representation of cross-modal error features. This representation not only includes the dynamic correlation information between audio fundamental frequency, score rhythm, and text tone, but also reveals the potential synergistic patterns among them. By introducing a direct-connection correction model for the neural vocoder, this invention achieves accurate localization and correction of error features. This model uses transposed convolutional upsampling technology to generate a corrected Mel spectrum, which is then reconstructed into high-fidelity corrected audio through a vocoder, thereby significantly improving the audio correction effect.

[0018] In constructing the direct-connection correction model for the neural vocoder, this invention uses the updated dynamic dialect prosodic map as a constraint to ensure that the correction model can generate corrected audio that conforms to the prosodic features of the dialect. Simultaneously, by introducing a model-independent meta-learning algorithm, this invention further enhances the generalization ability and adaptability of the correction model, enabling it to rapidly iterate and optimize with a small number of error samples, thereby improving error correction efficiency and accuracy.

[0019] In constructing a cross-modal generative adversarial network, this invention innovatively introduces a Transformer encoder, combining high-fidelity corrected audio with audio features, musical score features, and text features to generate a joint latent representation. This design improves the efficiency and accuracy of information fusion. Simultaneously, three discriminators are configured to perform adversarial discrimination based on audio naturalness, text tone compliance, and musical score rhythm consistency, ensuring the comprehensiveness and high quality of the corrected results.

[0020] In summary, this invention achieves intelligent error correction of dialect prosody in ethnic vocal music through a series of innovative designs, including constructing a dynamic dialect prosodic map, performing cross-modal comparative analysis, building a direct-connection correction model for neural vocoders, and using a cross-modal generative adversarial network. This method improves the efficiency and accuracy of error correction.

[0021] As an optional embodiment of the present invention, optionally, constructing a dynamic dialect prosodic map based on the structured training dataset in step S2, and updating the dynamic dialect prosodic map includes: S201. Initialize the dynamic dialect prosodic graph based on the structured training dataset, define tone adjustment points, vowel nodes and rhythm nodes, construct co-pronunciation edges and musical structure edges through cross-modal comparative analysis, and assign initial weights to the edges based on the frequency of occurrence of co-pronunciation rules. In step S201, it is necessary to explain in detail that when initializing the dynamic dialect prosodic map, this invention not only considers the basic structure of tone points, vowel nodes, and rhythm nodes, but also constructs co-pronunciation edges and musical structure edges by deeply analyzing the correlation between audio, musical score, and text features. Co-pronunciation edges connect tone points and vowel nodes that influence or co-change with each other during pronunciation, reflecting the unique co-pronunciation rules in dialects. Musical structure edges connect note nodes in the musical score, reflecting the rhythmic and melodic relationships between notes. By assigning initial weights to these edges, this invention can quantify the correlation strength and importance between different features.

[0022] S202. The dynamic dialect prosodic map based on local training of institutions in various regions is jointly trained across regions using a federated learning framework to obtain the global dynamic dialect prosodic map. In step S202, it is necessary to explain in detail that this invention effectively integrates the local training results of institutions in different regions through a federated learning framework. This process not only protects the privacy data of each region but also fully utilizes the advantages of distributed data, improving the generalization ability and accuracy of the dynamic dialect prosodic atlas. The federated learning framework allows regions to share model parameters or gradient information without sharing the original data, thereby achieving cross-regional knowledge sharing and collaborative training. In this way, this invention can construct a global dynamic dialect prosodic atlas that integrates the prosodic features of various regions, making it more representative and applicable.

[0023] S203. When new dialect data is accessed, incremental learning is used to add nodes and edges to the global dynamic dialect prosodic graph, and Bayesian optimization is used to balance the weights of the global dynamic dialect prosodic graph and the newly added nodes and edges to achieve dynamic updates.

[0024] In step S203, it is necessary to explain in detail that this invention designs an incremental learning mechanism to handle the incorporation of new dialect data. When new dialect data is introduced, this mechanism can flexibly add nodes and edges to the global dynamic dialect prosodic graph, thereby expanding the graph's coverage. To maintain the consistency and stability of the graph, this invention employs a Bayesian optimization method to balance the weights between the global graph and the newly added nodes and edges. Bayesian optimization constructs a probabilistic model of the objective function and selects the optimal parameter combination based on this model to achieve dynamic adjustment of the weights. This process ensures the effective fusion of new dialect data while maintaining the accuracy and reliability of the global graph.

[0025] The Bayesian optimization used in this embodiment is an existing technology that can find near-optimal parameter combinations within a limited number of iterations, thereby improving the update efficiency and accuracy of dynamic dialect prosodic maps.

[0026] As an optional embodiment of the present invention, optionally, the mathematical expression of the federated learning framework in step S202 is: in, Indicates the first The client is on the local machine. Model parameters after training round Indicates the first The client is on the local machine. Model parameters after training round Indicates the local learning rate. Indicated on the client Local dataset Above, the loss function with respect to parameters gradient vector, Indicates client loss function, Indicates client Local dataset, This represents the total number of clients participating in federated learning. Indicates client encryption function, Indicates Gaussian noise. This represents the variance of Gaussian noise. Indicates the sensitivity of model parameters. Indicates privacy budget, Indicates the first The global model parameters after one round of training. This refers to the decryption function of the central server. Indicates client Aggregate weights, This indicates a parameter concatenation operation. This represents the frozen layer in the global model. Representing a subset The number of clients in the middle, This indicates the subset of clients participating in the current round of updates. This indicates the layer that the client needs to update.

[0027] As an optional embodiment of the present invention, optionally, performing cross-modal contrastive analysis on the structured training dataset in step S3 to obtain a joint representation of cross-modal error features includes: S301. Based on the structured training dataset, the timelines of audio frames, musical notes, and text characters are aligned using a dynamic time warping algorithm to generate synchronous trimodal data containing timestamps and event markers. In step S301, it is important to explain in detail that this invention achieves precise alignment of audio frames, musical notes, and text characters on the timeline using a dynamic time warping algorithm. This step is fundamental to cross-modal comparative analysis, ensuring synchronization between different modal data. The dynamic time warping algorithm achieves non-linear alignment of the timeline by calculating the minimum distance between different sequences, effectively solving the problem of temporal mismatch caused by differences in speech rate and rhythm. The generated synchronized trimodal data includes detailed timestamps and event markers.

[0028] The dynamic time warping algorithm used in this embodiment is existing technology.

[0029] S302. Compare the fundamental frequency of the audio in the synchronous trimodal data with the MIDI pitch of the musical score, and calculate the pitch error matrix; compare the duration of the audio phonemes in the synchronous trimodal data with the duration of the musical notes, and calculate the duration deviation rate. The expression for calculating the pitch error matrix in step S302 is as follows: in, Indicates the aligned first The audio frame and the first The pitch difference of each musical note Indicates the first The fundamental frequency value of each audio frame is obtained by performing fundamental frequency detection on the singing audio. Indicates the first The frequency value corresponding to the MIDI pitch of each musical note is derived from the MIDI pitch number. ∈{1,2,…, }, ∈{1,2,…, }, This indicates the total number of audio frames (determined by the audio sampling rate and frame length). It indicates the total number of notes in the musical score (determined by the number of notes in the symbolic score).

[0030] The expression for calculating the duration deviation rate in step S302 is as follows: in, Indicates the first The duration deviation rate of each note, Indicates the first audio segment The actual duration of each note is obtained through audio phoneme segmentation and annotation. Indicates the first in the musical score The duration of a note is derived from the note value in the notated musical score (such as a whole note or a half note). This indicates the total number of aligned notes (take the smaller value between the number of audio frames and the number of notes in the musical score to ensure a one-to-one correspondence). Indicates the total number of audio frames. It indicates the total number of notes in the musical score.

[0031] In step S302, it is necessary to explain in detail that by calculating the pitch error matrix and the duration deviation rate, the present invention can quantify the pitch and duration differences between the audio and the musical score, thereby identifying potential rhythmic errors. The pitch error matrix visually displays the pitch difference between each audio frame and the corresponding musical note, while the duration deviation rate reflects the degree of deviation between the actual duration of the notes in the audio and the duration marked on the musical score.

[0032] S303. Compare the audio tone trajectory in the synchronous trimodal data with the dialect tone in the text annotation, and calculate the tone matching degree sequence using the dynamic time warping algorithm; detect whether the audio vowel pronunciation in the synchronous trimodal data and the vowel structure in the text annotation are co-pronounced, and obtain co-pronunciation error markers; In step S303, it is necessary to explain in detail that this invention further refines the identification of dialect prosodic errors by comparing the audio tone trajectory with the dialect tones annotated in the text. Using a dynamic time warping algorithm, a tone matching degree sequence is calculated. This sequence reflects the consistency and difference between the tone trajectory in the audio and the tone annotated in the text. By analyzing the tone matching degree sequence, tone errors can be accurately identified, such as tones that are too high, too low, or incorrect tone types.

[0033] Furthermore, this invention also detects whether the pronunciation of vowels in audio and the vowel structure annotated in text are co-pronounced. In dialects, some vowels may be affected by preceding and following phonemes during pronunciation, resulting in co-pronunciation. By comparing the pronunciation of vowels in audio with the vowel structure annotated in text, it is possible to identify whether there are co-pronunciation errors, such as incomplete vowel pronunciation, vowel substitution, or incorrect vowel pronunciation order.

[0034] The dynamic time warping algorithm used in this embodiment is existing technology.

[0035] S304. Verify whether the musical rhythm pattern and text tone fluctuation in the synchronous three-modal data conform to the coordination rules of the dynamic dialect prosodic graph, and mark the rhythm-tone association error that violates the rules. In step S304, it is important to explain in detail that this embodiment further improves the accuracy of dialect prosodic error correction by verifying the conformity between the rhythmic patterns of the musical score and the tonal fluctuations of the text. The rhythmic patterns in the musical score reflect the pulse and dynamic changes of the music, while the tonal fluctuations of the text reflect the unique intonation characteristics of the dialect. By comparing these two with the coordination rules in the dynamic dialect prosodic map, this invention can accurately identify rhythm-tone association errors that violate the rules, such as rhythm and tone mismatch, or tonal fluctuations conflicting with the rhythmic pattern. This process not only enhances the comprehensiveness of error correction but also improves the depth of understanding of dialect prosodic features.

[0036] S305. Based on the pitch error matrix, duration deviation, tone matching degree sequence, coprophony error marker, and rhythm-tone association error, a structured error feature containing error type, location, and confidence is generated through threshold determination and error type classification. In step S305, it is necessary to explain in detail that this invention analyzes and judges pitch error matrices, duration deviations, tone matching sequences, coarticulation error markers, and rhythm-tone association errors by setting reasonable thresholds. When a certain feature value exceeds a preset threshold, it is judged to have an error. Simultaneously, errors are finely classified according to their nature and manifestation, such as pitch errors, duration errors, tone errors, coarticulation errors, and rhythm-tone association errors. Furthermore, this invention calculates the confidence level of each error feature to reflect the reliability of the judgment. The structured error features contain multi-dimensional information such as error type, location, and confidence level.

[0037] S306. Based on the structured error features, a multi-head attention mechanism is used to fuse the error features of audio, musical score, and text, and the joint representation of cross-modal error features is generated by combining the rule weights of dynamic dialect prosodic graph.

[0038] In step S306, it is necessary to explain in detail that the present invention employs a multi-head attention mechanism to fuse error features from audio, musical score, and text. The multi-head attention mechanism, by processing multiple attention heads in parallel, can capture the complex relationships between different features, thereby improving the accuracy and robustness of the fusion. During the fusion process, each attention head, according to its own weight allocation, focuses on the importance of different features and integrates this information to form a more comprehensive and accurate representation of error features.

[0039] Furthermore, this invention incorporates rule weights from a dynamic dialect prosodic atlas to further optimize and adjust cross-modal error features. The dynamic dialect prosodic atlas contains rich dialect prosodic rules and feature information. By introducing these rule weights, this invention can better understand and utilize cross-modal error features, improving the accuracy and efficiency of error correction. Finally, through multi-head attention mechanism fusion and adjustment of the dynamic dialect prosodic atlas rule weights, this invention generates a joint representation of cross-modal error features.

[0040] As an optional embodiment of the present invention, optionally, in step S4, the modified Mel spectrum is generated by transposed convolution upsampling, and the high-fidelity modified audio is reconstructed based on the Mel spectrum using the vocoder of the neural vocoder direct-connection correction model. S401. Based on the joint representation of the dynamic dialect prosodic graph and cross-modal error features, a neural vocoder direct connection correction model is constructed. The neural vocoder direct connection correction model includes an encoder, a transposed convolution, and a vocoder. A graph rule penalty term is embedded in the encoder loss function. In step S401, it is necessary to explain in detail that the encoder is used to extract audio features, the transposed convolution is used for upsampling to generate a higher resolution Mel spectrum, and the vocoder is used to reconstruct the Mel spectrum into an audio signal. By embedding a spectrogram rule penalty term in the encoder loss function, this invention can guide the model to better follow the rules of the dynamic dialect prosodic spectrogram during feature extraction, thereby improving the prosodic accuracy and naturalness of the corrected audio. This design allows the neural vocoder directly connected to the correction model to fully consider dialect prosodic features during the correction process, generating corrected audio that is more in line with dialect expression habits.

[0041] Specifically, the atlas rule penalty term is constructed based on the prosodic rules and feature information in the dynamic dialect prosodic atlas and is used to constrain and optimize the encoder's output. During training, when the encoder's output deviates from the rules of the dynamic dialect prosodic atlas, the atlas rule penalty term applies a corresponding penalty, thereby guiding the encoder to output audio features that better conform to the dialect's prosodic characteristics. This mechanism ensures that the neural vocoder direct-connect correction model can accurately capture and preserve the prosodic features of the dialect when correcting audio, improving the naturalness and accuracy of the corrected audio.

[0042] S402. Map the joint representation of the cross-modal error features to the input channel of the transposed convolution, use 4 layers of transposed convolution to upsample layer by layer to generate the corrected Mel spectrum, and dynamically adjust the spectrum amplitude according to the error confidence. In step S402, it is necessary to explain in detail that the joint representation of cross-modal error features includes error information from audio, musical score, and text, which is used to guide the correction of the Mel spectrum. By mapping the joint representation to the input channel of the transposed convolution, this invention achieves the effective utilization of error features in the spectrum correction process. The layer-by-layer upsampling design of the 4-layer transposed convolution not only improves the resolution of the Mel spectrum but also ensures the accurate mapping and correction of error features in the spectrum. At the same time, dynamically adjusting the spectral amplitude according to the error confidence further enhances the targeting and accuracy of the correction. This step, through fine-tuning the spectral features, effectively reduces the impact of error features on the corrected audio and improves the quality of the corrected audio.

[0043] S403. The corrected Mel spectrum is converted into a time-domain waveform using the vocoder, and the time-domain waveform is optimized. The fundamental frequency trajectory is adjusted in conjunction with the text tone marking to obtain high-fidelity corrected audio.

[0044] In step S403, it is necessary to describe in detail the process by which the vocoder converts the corrected Mel spectrum back to a time-domain waveform. This process is a crucial step in audio signal reconstruction. To ensure that the corrected audio signal retains the sonic characteristics of the original audio while accurately correcting prosodic errors, this invention further optimizes the time-domain waveform. The optimization process includes removing unnecessary noise components, enhancing the coherence and smoothness of the audio signal, and adjusting the dynamic range of the audio signal. These operations collectively improve the overall quality of the corrected audio.

[0045] Meanwhile, adjusting the fundamental frequency trajectory by combining text tone annotation is another innovation of this invention in correcting audio prosody. The fundamental frequency trajectory reflects the tonal changes of the audio signal and is an important component of dialect prosodic features. By comparing the tone information of the text annotation with the fundamental frequency trajectory of the audio signal, this invention can accurately identify errors or deviations in the fundamental frequency trajectory and make targeted adjustments. This process not only ensures the accuracy of the corrected audio in terms of tone but also enhances the prosodic expressiveness and naturalness of the audio signal, making the corrected audio more in line with the expression habits of the dialect.

[0046] As an optional embodiment of the present invention, optionally, in step S401, the direct-connect correction model of the neural vocoder is specifically an encoder-decoder model based on the Transformer architecture, wherein the encoder adopts a 6-layer convolutional neural network (CNN) structure, each layer containing a 3×3 convolutional kernel, a ReLU activation function and a batch normalization layer, used to extract local texture and global context features of the Mel spectrum; the transposed convolutional module consists of 4 stacked layers, each layer having a convolutional kernel size of 4×4 and a stride of 2, which maps low-resolution error features to high-resolution Mel spectrum correction features aligned with the original audio frame through layer-by-layer upsampling; the vocoder adopts an improved WaveRNN structure, introducing causal convolutional layers and gated recurrent units (GRU) to enhance the continuity and naturalness of temporal waveform generation. During model training, a hybrid loss function is employed, including Mel spectrum reconstruction loss, fundamental frequency trajectory matching loss, and a spectrogram rule penalty term. The Mel spectrum reconstruction loss uses L1 loss to calculate the difference between the corrected and target Mel spectrum. The fundamental frequency trajectory matching loss is achieved by calculating the mean square error between the generated audio fundamental frequency and the labeled fundamental frequency. The spectrogram rule penalty term is calculated using the cosine distance between the encoder output features and the regular features in the dynamic dialect prosodic graph, with a weight coefficient set to 0.3 to balance prosodic rule constraints and spectrum reconstruction accuracy. During training, the Adam optimizer is used with an initial learning rate of 1e-4, decaying by 50% every 10 epochs, and a training batch size of 32, until the loss function converges to below a preset threshold.

[0047] In this embodiment, the dynamic dialect prosodic atlas is stored in a multi-level structured database, comprising three core modules: The first module is a dialect tone template library, which stores the fundamental frequency trajectory curve, duration distribution range, and variant features of standard tones in typical contexts for different tone types (such as high level tone, rising tone, falling-rising tone, falling tone, etc.) of the target ethnic dialect; the second module is a prosodic rhythm coordination matrix, which records the matching weights of musical rhythm units (such as quarter note and eighth note combinations) and dialect tone fluctuation patterns in a two-dimensional matrix form. The larger the matrix element value, the more the rhythm-tone combination conforms to the dialect prosodic habits; the third module is a vowel co-pronunciation rule table, which lists the legal co-pronunciation variants of common vowels in the target dialect under the influence of preceding and following phonemes and the prohibited error patterns. Each rule entry includes the triggering condition, standard variant form, and error judgment threshold. In addition, the atlas supports dynamic updates. Based on newly collected ethnic vocal corpora and expert-annotated data, the parameters and rules of each module can be optimized through incremental learning algorithms to adapt to the differences in singing styles of different singers and the characteristics of regional dialect variations.

[0048] In step S403, the method for obtaining high-fidelity corrected audio is as follows: First, the causal convolutional layer of the WaveRNN vocoder is used to extract temporal features from the corrected Mel spectrum to capture the local dependencies between adjacent spectral frames; then, a gated recurrent unit (GRU) is used to model the global temporal context to generate a continuous initial temporal waveform. To address potential inter-frame abrupt changes in the initial waveform, the amplitude difference between adjacent frames is calculated, and regions exceeding a preset abrupt change threshold are smoothed using linear interpolation to eliminate waveform discontinuities. Secondly, fine-tuning of the fundamental frequency trajectory is performed in conjunction with text tone annotation: the YIN algorithm is used to extract the fundamental frequency trajectory of the initial waveform, and dynamic time warping (DTW) is performed to align it with the standard fundamental frequency template of the corresponding tone in the dynamic dialect prosodic map; based on the aligned fundamental frequency deviation sequence and error confidence, the fundamental frequency value is weighted and corrected. For high-confidence error regions with a confidence level ≥ 0.8, the fundamental frequency value of the standard template is directly replaced; for medium-confidence regions with a confidence level ≤ 0.5 and a confidence level < 0.8, a weighted average method is used to fine-tune the fundamental frequency; for low-confidence regions with a confidence level < 0.5, the original fundamental frequency trend is preserved to maintain the singing style. Next, loudness equalization is performed according to the musical dynamic markings and dialect accent rules: the loudness value of each frame of the corrected audio is calculated, the gain is adjusted for regions that deviate from the target loudness range, and the loudness of the dialect accent positions is enhanced to highlight the prosodic features. Finally, residual high-frequency noise is removed by adaptive Gaussian filtering, and dynamic range compression technology is used to control the audio dynamic range within 80dB, ensuring that the corrected audio can present a clear and natural listening experience on different playback devices, and finally outputting high-fidelity corrected audio that conforms to the dialect prosody norms.

[0049] As an optional embodiment of the present invention, optionally, step S5 includes iteratively optimizing the neural vocoder direct connection correction model by introducing a small number of error samples and using a model-independent meta-learning algorithm, comprising: S501. Collect and preprocess a small number of error samples, filter high-confidence error samples through manual annotation, label the error type and correction target, and generate an error sample set after data augmentation and format unification. In step S501, it is necessary to explain in detail that, to ensure the representativeness and accuracy of the samples, this invention first manually selects high-confidence error samples from practical applications. These samples cover various error types, such as pitch errors, duration errors, and tone errors, and each sample is clearly labeled with the error type and correction target. To improve the diversity and robustness of the samples, this invention also performs data augmentation processing on these error samples, such as adding noise and adjusting the speech rate, to simulate different application scenarios. Finally, all error samples are standardized in format to generate a complete error sample set.

[0050] S502. Based on the error sample set and using the model-independent meta-learning algorithm, fix the encoder and transposed convolution parameters of the neural vocoder direct connection correction model, only open the vocoder parameters, and divide the meta-tasks according to dialect type or error type. In step S502, it is necessary to explain in detail that the model-independent meta-learning algorithm is an effective machine learning method that can quickly adapt and optimize the model with limited samples. In this invention, this algorithm is used to iteratively optimize the vocoder part of the neural vocoder direct-connection correction model. Specifically, the parameters of the encoder and transposed convolution are fixed, and only the parameters of the vocoder are opened for training. The advantage of doing so is that fine-tuning can be performed for specific problems while keeping most of the model structure unchanged, thereby improving the model's adaptability and generalization ability.

[0051] Furthermore, to further improve the efficiency and relevance of optimization, the error sample set is divided into multiple meta-tasks based on dialect type or error type. Each meta-task focuses on solving a specific type of problem, such as pitch adjustment or duration correction in a particular dialect. This division allows for the customization of different training strategies and objective functions for each meta-task, thereby more precisely guiding the model optimization process.

[0052] During training, the model-independent meta-learning algorithm continuously samples mini-batch data from each meta-task and updates the gradients based on this data. Since the parameters of the encoder and transposed convolution are fixed, the vocoder can be optimized independently without interfering with other parts. This local optimization strategy not only improves training stability but also accelerates model convergence.

[0053] After multiple iterations, an optimized neural vocoder direct-connect correction model can be obtained. While maintaining the original sound quality characteristics, this model can more accurately correct errors in dialect prosody and generate more natural and dialect-compliant corrected audio.

[0054] S503. Based on the model-independent meta-learning algorithm, perform inner loop task adaptation, input a single meta-task into the neural vocoder, generate corrected audio through forward propagation, calculate task-specific loss and fine-tune neural vocoder parameters. In step S503, it is necessary to explain in detail that, in the embodiments of the present invention, a single meta-task is input into the neural vocoder direct-connect correction model, and the corrected audio is generated through forward propagation. Subsequently, a specific loss on this meta-task is calculated, which reflects the model's performance on the current meta-task. Based on this loss, the parameters of the neural vocoder are fine-tuned to improve the model's accuracy on the current meta-task.

[0055] The fine-tuning process is implemented through gradient descent or other optimization algorithms, adjusting the parameters of the neural vocoder based on the calculated loss value to reduce prediction errors. Since model-independent meta-learning algorithms allow for rapid adaptation on a small number of samples, this process can significantly improve the model's performance on specific meta-tasks within a limited number of iterations.

[0056] During the inner loop task adaptation, this invention also employs an early stopping strategy to avoid overfitting. Specifically, when performance no longer improves on the validation set, the fine-tuning process is stopped, and the current optimal model parameters are retained. This strategy helps ensure that the model maximizes its accuracy on specific meta-tasks while maintaining its generalization ability.

[0057] After adaptation through inner loop tasks, the neural vocoder direct-connect correction model is better able to handle prosodic errors of specific dialect types or error types, generating more natural and accurate corrected audio. This process improves the model's adaptability.

[0058] S504. Based on the model-independent meta-learning algorithm, perform outer loop parameter aggregation, aggregate the gradients of all meta-tasks, update the global neural vocoder parameters, and balance the cross-task optimization direction. In step S504, it is important to explain in detail that the outer loop parameter aggregation step of the model-independent meta-learning algorithm is crucial to ensuring that the neural vocoder direct-connect correction model maintains balance and consistency across multiple meta-tasks. In this step, the algorithm collects gradient information generated by all meta-tasks during fine-tuning and aggregates these gradients. The aggregated gradients reflect the overall performance of the model across all meta-tasks and can therefore be used to update the global neural vocoder parameters.

[0059] The process of updating global parameters is achieved through gradient descent or other optimization algorithms. It adjusts the parameters of the neural vocoder based on the aggregated gradient to improve the model's performance across all meta-tasks. This step ensures that the model remains adaptable to each meta-task while maintaining consistency and stability across tasks.

[0060] By alternately executing inner loop task adaptation and outer loop parameter aggregation, the model-independent meta-learning algorithm can progressively optimize the parameters of the neural vocoder direct connection correction model, enabling it to quickly adapt to and correct errors in dialect prosody with limited samples. This process not only improves the model's accuracy and generalization ability but also makes the corrected audio more natural and in line with dialect expression habits.

[0061] S505. Verify the compliance of the corrected audio rules based on the updated global neural vocoder parameters and dynamic dialect prosodic graph. If a spectrum segment that violates the cooperation rules is found, adjust the loss function of the model irrelevant meta-learning algorithm through regularization term. In step S505, it is necessary to explain in detail that the verification process mainly uses the updated global neural vocoder parameters and dynamic dialect prosodic graph to check the rule compliance of the corrected audio. This step ensures that the corrected audio is not only technically accurate, but also conforms to the prosodic rules and expression habits of the dialect.

[0062] During the verification process, if any spectral segments violating the collaborative rules are found in the corrected audio—that is, certain parts of the audio are inconsistent with the rules in the dialect prosodic map—this invention will take further measures to adjust the model. Specifically, a regularization term will be introduced to adjust the loss function of the model's irrelevant meta-learning algorithm to increase the penalty for violating spectral segments. This will encourage the model to pay more attention to these violations during subsequent training, thereby gradually reducing or eliminating them and improving the overall compliance and naturalness of the corrected audio.

[0063] S506. Iterate through S503 to S505 until the error rate of the corrected audio is lower than the threshold or the maximum number of iterations is reached, generating the final optimized neural vocoder direct-connect correction model.

[0064] In step S506, it is necessary to explain in detail that by iteratively executing steps S503 to S505, the present invention gradually optimizes the performance of the neural vocoder direct-connection correction model. In each iteration, the vocoder parameters are adjusted based on the current model performance to improve the accuracy and naturalness of the corrected audio. Simultaneously, through continuous rule compliance checks and regularization adjustments, it is ensured that the model consistently conforms to the prosodic rules and expression habits of the dialect during the correction process.

[0065] The iterative process continues until the error rate of the corrected audio drops below a preset threshold or the preset maximum number of iterations is reached. At this point, the model can be considered sufficiently trained and optimized to generate high-quality corrected audio. Finally, by directly connecting this optimized neural vocoder to the correction model, various dialect prosodic errors can be efficiently processed, generating natural, accurate, and dialect-compliant corrected audio.

[0066] As an optional embodiment of the present invention, the model-independent meta-learning algorithm is specifically as follows: First, initialize the global initial parameter θ0 of the neural vocoder to the parameter after convergence in the pre-training stage; then, randomly sample 8 meta-tasks from the divided meta-task set (balancing computational efficiency and task diversity), and perform the following operations on each meta-task Tj: ① Sample the support set Sj (containing 5 mislabeled samples) and the query set Qj (containing 5 mislabeled samples) from Tj; ② Calculate the task loss Lj(θ0) using the support set Sj, and update the parameters through a 1-step gradient descent to obtain θj=θ0-α·∇θLj(θ0), where the inner learning rate α is fixed at 5e-5; ③ Calculate the fine-tuned loss L'j(θj) using the query set Qj, and record the gradient ∇θL'j(θj). After completing the inner-layer computation of the eight meta-tasks, the gradients of all meta-tasks are aggregated to obtain the average gradient ∇θavg. Then, the Adam optimizer is used to perform an outer-layer update on the global parameter θ0: θ0←θ0-β·∇θavg, where the outer-layer learning rate β remains consistent with the initial learning rate during the training phase (1e-4) and decays synchronously with each epoch (decreasing by 50% every 10 meta-epochs). This meta-training process is repeated until the decrease in the outer-layer loss function over five consecutive meta-epochs is less than 1e-6. The resulting global parameters are the initial parameters of the neural vocoder after MAML optimization, which can be used for rapid adaptation to specific dialects or error types.

[0067] The method for calculating the task-specific loss and fine-tuning the neural vocoder parameters in step S503 is as follows: The task-specific loss consists of three weighted parts: ① Prosodic rule loss: Based on the vowel copronouncing rule table and tone standard fundamental frequency template in the dynamic dialect prosodic atlas, the dynamic time warping (DTW) distance between the spectral frame sequence of the corrected audio and the rule template is calculated. Spectral segments that violate the copronouncing rules are assigned a weight of 1.5 times, resulting in the prosodic loss Lrhy; ② Sound quality naturalness loss: The sound quality difference between the corrected audio and the original singing audio is calculated using the Perceptual Speech Quality Evaluation (PESQ) index, and converted into a numerical loss Lqual (i.e., 4.5 - PESQ score, with a value range of 0-4.5); ③ Tone fundamental frequency loss: The fundamental frequency trajectory of the corrected audio is extracted, and the mean square error is calculated with the standard fundamental frequency template of the corresponding text tone, resulting in Lpitch. The final task-specific loss Ltask = 0.4. Lrhy+0.3 Lqual+0.3 Lpitch, the weighting factor can be dynamically adjusted according to the type of meta-task (e.g., for meta-tasks dominated by tone errors, the Lpitch weight will be increased to 0.5).

[0068] The process of fine-tuning the neural vocoder parameters is as follows: Based on the support set samples, the gradient of the Ltask with respect to the vocoder's core parameters (including the hidden state weights of the GRU layer and the kernel parameters of the causal convolutional layer) is calculated. The parameters are updated using the stochastic gradient descent (SGD) algorithm. The inner layer learning rate α is adaptively adjusted according to the meta-task type: α is set to 6e-5 for dialect accent tasks and α is set to 4e-5 for vowel collocation error tasks. After every two gradient updates, the loss change is verified using query set samples. If the loss increases in two consecutive verifications, early stopping is triggered, and the previous parameter state is retained; otherwise, fine-tuning continues until five preset inner layer updates are completed or the early stopping condition is triggered, resulting in vocoder fine-tuning parameters adapted to the current meta-task.

[0069] As an optional embodiment of the present invention, the discrimination result of obtaining the corrected audio, musical score, and text trimodal data in step S6 may include: S601. Construct a generator input layer, taking the high-fidelity corrected audio as the basic input, and combining the audio features, musical score features and text features to generate a spatiotemporally aligned feature tensor through a cross-modal attention fusion module. In step S601, it is important to explain in detail that the cross-modal attention fusion module is one of the core components of this invention, responsible for efficiently fusing information from different modalities. Specifically, this module first preprocesses the high-fidelity corrected audio, audio features, musical score features, and text features to extract their respective key information. Then, using an attention mechanism, the module calculates the correlation weights between different features to determine which features should be given higher importance during the fusion process.

[0070] During the fusion process, the module performs a weighted summation of the features based on the calculated weights, thereby generating a spatiotemporally aligned feature tensor. This feature tensor not only contains rich information from each modality, but this information is also aligned in both time and space.

[0071] By constructing a generator input layer and using the spatiotemporally aligned feature tensor generated by the cross-modal attention fusion module as input, this invention can achieve comprehensive discrimination of corrected audio, musical score and text trimodal data.

[0072] S602. Construct the Transformer encoder of the generator, and use a multi-head self-attention mechanism to perform global dependency modeling on the fused spatiotemporally aligned feature tensor to generate a joint latent representation containing audio-musical score-text collaborative relationships. Then, embed the updated dynamic dialect prosodic graph into the joint latent representation through a graph attention network. In step S602, it is important to explain in detail that the Transformer encoder is another key component of the generator. It is responsible for deep processing of the fused spatiotemporally aligned feature tensor to capture the complex collaborative relationships between audio, sheet music, and text. Specifically, the encoder employs a multi-head self-attention mechanism, which allows the model to consider information from all other features when processing each feature, thereby achieving modeling of global dependencies.

[0073] Through a multi-head self-attention mechanism, the Transformer encoder can generate a joint latent representation that incorporates the collaborative relationships between audio, sheet music, and text. This joint latent representation is a high-dimensional vector that integrates information from different modalities and reflects the intrinsic connections between these information.

[0074] To further improve model performance, this invention also embeds the updated dynamic dialect prosodic atlas into the joint hidden representation. This step is implemented using a graph attention network, which can further adjust and optimize the joint hidden representation based on the rules and structural information in the dialect prosodic atlas. Through this step, the model can better understand and process the complex rules in dialect prosody, thereby generating more natural and accurate corrected audio.

[0075] S603, Configure the decoder branch of the generator, wherein the audio decoder upsamples the joint implicit representation through 4 layers of transposed convolution to generate a corrected Mel spectrum and convert it into a time-domain waveform; the music score decoder adjusts the note timing and pitch based on the joint implicit representation to generate a symbolic music score; the text decoder combines the joint implicit representation to correct tone markings to ensure that the tone is consistent with the audio pronunciation. In step S603, it is necessary to explain in detail that the decoder branch of the generator is the key part responsible for converting the joint implicit representation into the final output. It contains three main decoders: an audio decoder, a music score decoder, and a text decoder.

[0076] The primary task of the audio decoder is to convert the joint latent representation into the corrected audio. To achieve this, the audio decoder employs a four-layer transposed convolutional network. A transposed convolutional network is a special type of convolutional network that upsamples the input features to generate a higher-resolution output. Through the processing of these four transposed convolutional layers, the audio decoder can convert the joint latent representation into a corrected Mel spectrum. Subsequently, using inverse Mel transform or other audio synthesis techniques, the Mel spectrum can be converted into a time-domain waveform, thus obtaining the final corrected audio.

[0077] The primary task of a music notation decoder is to adjust the timing and pitch of notes based on joint implicit representations to generate symbolic notation. During processing, the decoder analyzes the audio-musical notation co-relationships within the joint implicit representations and adjusts the timing and pitch of notes accordingly. This step generates symbolic notation corresponding to the corrected audio, facilitating music composition and performance.

[0078] The primary task of the text decoder is to correct tone annotations using joint implicit representations to ensure that the tones match the audio pronunciation. During processing, the text decoder analyzes the audio-text co-relationships in the joint implicit representations and adjusts the tone annotations accordingly. This step generates text annotations corresponding to the corrected audio, providing accurate information for speech recognition and text processing.

[0079] By configuring the decoder branch of the generator, this invention enables comprehensive processing and output of corrected audio, musical score, and text trimodal data. This step not only improves the quality and accuracy of the corrected audio but also provides users with rich output formats to meet the needs of different application scenarios.

[0080] S604. Construct the first set of discriminators, use a multi-layer convolutional neural network to extract the spectral features of the corrected audio, and use binary classification to determine whether the audio is a real singing recording, so that the generator can improve the naturalness of the audio. In step S604, it is necessary to explain in detail that the first set of discriminators is responsible for evaluating the quality of the corrected audio. To achieve this goal, the discriminator employs a multi-layer convolutional neural network structure. This network structure has powerful feature extraction capabilities, enabling it to extract rich spectral features from the audio data.

[0081] During the discrimination process, a multi-layer convolutional neural network processes the input corrected audio layer by layer. Through convolution operations and activation functions, it gradually extracts high-level features of the audio. These features not only include basic audio attributes such as pitch, timbre, and rhythm, but also reflect the naturalness and realism of the audio.

[0082] After extracting the spectral features, the discriminator uses these features to perform binary classification. Specifically, it compares the feature vector with a preset threshold to determine whether the audio is a genuine recording. This process not only helps the generator understand the current quality level of the corrected audio but also provides directions for improvement.

[0083] Through adversarial training against the generator, the first set of discriminators enables the generator to continuously improve the naturalness and realism of the audio. During training, the discriminator provides ongoing feedback, guiding the generator to adjust and optimize the corrected audio. As training progresses, the generator's performance gradually improves, and the corrected audio becomes increasingly closer to the effect of a real vocal recording.

[0084] S605. Construct a second set of discriminators, align the audio fundamental frequency curve with the text tone annotation based on the dynamic time warping algorithm, verify the tone coordination rules by referring to the dynamic dialect prosodic graph, and determine whether the corrected text conforms to the dialect pronunciation habits. In step S605, it is necessary to explain in detail that the core of the second discriminator is that it combines dynamic time warping algorithm and dynamic dialect prosodic graph to achieve accurate alignment and verification of audio fundamental frequency curve and text tone annotation.

[0085] Dynamic time warping (a prior art method) is a classic time series alignment method that calculates the similarity between two sequences, even if they differ in length. In this invention, this algorithm is used to align and correct the fundamental frequency curve of audio and the tone markings in text. Through alignment, the discriminator can accurately compare the pronunciation in the audio and the tone markings in the text, thereby determining their consistency.

[0086] To further verify the tone coordination rules, this invention introduces a dynamic dialect prosodic atlas. This atlas contains rules and structural information on various tone coordination relationships in the dialect. During the discrimination process, the discriminator references the rules in the atlas to verify the audio fundamental frequency curve and text tone annotations to determine whether the corrected text conforms to the pronunciation habits of the dialect.

[0087] Specifically, the discriminator analyzes the tone variations in the audio fundamental frequency curve and the tone information in the text tone annotations, comparing them with the rules in the dialect prosodic map. If the tone variations in the audio and the tone annotations in the text are consistent with the rules in the map, the discriminator considers the corrected text to conform to the pronunciation habits of the dialect; otherwise, if they are inconsistent, the discriminator considers the corrected text to contain dialect prosodic errors.

[0088] By constructing a second set of discriminators, this invention enables the accurate detection and correction of dialect prosodic errors in the corrected text. This step improves the quality and accuracy of the corrected text.

[0089] S606. Construct a third discriminator, analyze the matching degree between the musical score rhythm pattern and the audio onset detection result, use the dynamic dialect prosodic graph to verify the rhythm-tone association rule, and determine whether the corrected musical score is consistent with the text tone and audio pronunciation. In step S606, it is necessary to explain in detail that the third discriminator integrates information from the musical score rhythm pattern, audio onset detection results, and dynamic dialect prosodic graph to achieve an accurate assessment of the coordination between the corrected musical score and the text tone and audio pronunciation. The musical score rhythm pattern reflects the temporal arrangement and dynamic pattern of notes in a musical work and is an important component of musical expressiveness. In this invention, the third discriminator first performs rhythm pattern analysis on the corrected musical score to extract key rhythmic information from the score.

[0090] The audio onset detection results are obtained through audio signal processing technology, identifying the start time of each note in the audio. By comparing the musical score rhythm pattern with the audio onset detection results, the discriminator can assess the degree of matching between the musical score rhythm and the audio pronunciation.

[0091] To further verify the rhythm-tone association rules, this invention also introduces a dynamic dialect prosodic atlas. This atlas not only contains rules governing tone coordination relationships in dialects but also covers the association information between rhythm and tone. During the discrimination process, the discriminator references the rules in the atlas to comprehensively analyze the musical score rhythm pattern, audio onset detection results, and text tone annotations.

[0092] Specifically, the discriminator checks whether the rhythmic changes in the musical score match the onset detection results in the audio, and verifies whether these rhythmic changes are consistent with the tone markings in the text and the rhythm-tone association rules in the dialect prosodic graph. If there is a good coordination between the musical score rhythm, audio pronunciation, and text tone, and it conforms to the rules in the dialect prosodic graph, then the discriminator considers the corrected musical score to be accurate; otherwise, if there are inconsistencies or discrepancies, the discriminator will point out the problems in the corrected musical score.

[0093] By constructing a third set of discriminators, this invention enables precise detection and correction of rhythmic errors in corrected musical scores. This step not only improves the quality and accuracy of the corrected scores but also ensures the coordination and consistency of rhythm and tone in the corrected score, audio, and text trimodal data.

[0094] S607. Execute the cross-modal adversarial training process. First, pre-train the generator to generate reasonable trimodal data through reconstruction loss. Based on the reasonable trimodal data, train the discriminator and generator alternately. Optimize jointly through adversarial loss and rule penalty terms until the error rate of the corrected audio is lower than the threshold or the maximum number of iterations is reached. Finally, output the corrected audio, sheet music, and text trimodal data and the discrimination result.

[0095] In step S607, it is necessary to explain in detail that in the cross-modal adversarial training process, the generator first needs to be pre-trained to ensure that it can generate reasonable trimodal data. During the pre-training process, the generator will use the reconstruction loss function, combined with the neural vocoder directly connected to correct the forward propagation function of the model, to gradually adjust and optimize its internal parameters in order to generate trimodal data that is as close as possible to the real data.

[0096] After pre-training, the next stage involves alternating training of the discriminator and generator. In this stage, the discriminator and generator compete against each other and improve together. Specifically, the discriminator uses its powerful feature extraction and classification capabilities to accurately evaluate the trimodal data output by the generator, determining whether it conforms to the distribution and patterns of real data. Meanwhile, the generator continuously adjusts its output based on feedback from the discriminator, striving to deceive it as much as possible and make it difficult for the discriminator to distinguish between generated and real data.

[0097] During alternating training, this invention also introduces adversarial loss and rule penalty terms to jointly optimize the model. The adversarial loss measures the degree of adversarial interaction between the discriminator and the generator; it gradually decreases as training progresses, indicating that the generator's output data is becoming increasingly closer to real data. The rule penalty term, on the other hand, constrains the generator's output, ensuring it conforms to the rules and structural information of dialect prosody, thereby further improving the quality and accuracy of the corrected trimodal data.

[0098] The training process continues until the error rate of the corrected audio falls below a preset threshold or the maximum number of iterations is reached. During this process, the generator and discriminator continuously learn from and adapt to each other, forming a dynamic balance. Finally, when training ends, this invention can output corrected audio, sheet music, and text trimodal data along with their discrimination results. This data is not only of high quality and accuracy, but also retains the style and characteristics of the original data well.

[0099] Example 2 A system for intelligent error correction of the prosody of ethnic vocal dialects, the system comprising the aforementioned method for intelligent error correction of the prosody of ethnic vocal dialects; the system further comprising: The acquisition module is used to acquire multimodal data and input the multimodal data into the intelligent error correction method; The processing module is used to execute the steps of the intelligent error correction method, process the input audio, music score and text data, and generate corrected audio, music score and text three-modal data and discrimination results; The storage module is used to store the raw data, intermediate data during processing, and the final corrected audio, musical score, and text three-modal data and discrimination results; The output module is used to visualize the corrected audio, musical score, and text three-modal data and the discrimination results.

[0100] This embodiment of the intelligent error correction system for ethnic vocal dialect prosody is used to implement the intelligent error correction method for ethnic vocal dialect prosody in Embodiment 1. Specifically, this system achieves intelligent detection and correction of dialect prosody errors in ethnic vocal music through the collaborative work of various modules. The acquisition module, as the input end of the system, is responsible for acquiring multimodal data including audio, sheet music, and text. This data may come from various sources, such as live performance recordings, scanned sheet music, or text input. After integrating this data, the acquisition module inputs it into the processing module.

[0101] The processing module executes the intelligent error correction method described in Example 1 to process the input audio, sheet music, and text data. During processing, the system first encodes the multimodal data using an encoder to generate a joint implicit representation containing the audio-sheet music-text collaborative relationships. Then, by configuring the decoder branch of the generator, the joint implicit representation is converted into corrected audio, sheet music, and text trimodal data. Simultaneously, the system constructs multiple discriminators to evaluate and verify the quality of the corrected data, ensuring it conforms to the rules and structural information of dialect prosody. The storage module is responsible for storing the original data, intermediate data during processing, and the final corrected audio, sheet music, and text trimodal data and discrimination results. This data not only provides the foundation for subsequent processing but can also be used for subsequent analysis and research. Through the storage module, users can easily view and manage various types of data in the system. The output module visualizes the corrected audio, sheet music, and text trimodal data and discrimination results. Users can intuitively see the comparison effect before and after error correction, as well as the discriminator's evaluation results of the corrected data, through the interface. This feature not only improves the ease of use of the system, but also provides users with more intuitive and accurate information feedback.

[0102] In summary, this embodiment of the intelligent error correction system for ethnic vocal dialect prosody achieves intelligent detection and correction of prosodic errors in ethnic vocal music through the collaborative work of various modules. This system not only improves the quality and accuracy of the corrected data but also provides users with rich output formats and visualization functions, meeting the needs of different application scenarios.

[0103] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for intelligent error correction of the prosody of ethnic vocal dialects, characterized in that, The method includes: S1. Collect multimodal data and extract features from the multimodal data to obtain audio features, musical score features and text features. Based on the audio features, musical score features and text features, form a structured training dataset. The multimodal data includes ethnic vocal music performance audio, corresponding symbolic musical score and dialect lyrics text. S2. Construct a dynamic dialect prosodic map based on the structured training dataset, and update the dynamic dialect prosodic map; S3. Perform cross-modal comparative analysis on the structured training dataset to obtain a joint representation of cross-modal error features; S4. Construct a neural vocoder direct connection correction model using the updated dynamic dialect prosodic map as a constraint condition. Input the joint representation of the cross-modal error features into the neural vocoder direct connection correction model. Generate the corrected Mel spectrum through transposed convolution upsampling. Reconstruct the high-fidelity corrected audio based on the Mel spectrum using the vocoder of the neural vocoder direct connection correction model. S5. By introducing a small number of erroneous samples, the model-independent meta-learning algorithm is used to iteratively optimize the neural vocoder direct-connection correction model. S6. Construct a cross-modal generative adversarial network including a generator and three discriminators. The generator uses the high-fidelity corrected audio as a base and combines the audio features, musical score features, and text features to generate a joint latent representation through a Transformer encoder. The decoder then outputs the corrected audio, musical score, and text trimodal data. The three discriminators are used to perform adversarial discrimination on the naturalness of the corrected audio, the tonal compliance of the text, and the rhythmic coordination of the musical score, respectively, to obtain the discrimination results of the corrected audio, musical score, and text trimodal data.

2. The intelligent error correction method for the prosody of ethnic vocal dialects as described in claim 1, characterized in that, In step S2, constructing a dynamic dialect prosodic map based on the structured training dataset and updating the dynamic dialect prosodic map includes: S201. Initialize the dynamic dialect prosodic graph based on the structured training dataset, define tone adjustment points, vowel nodes and rhythm nodes, construct co-pronunciation edges and musical structure edges through cross-modal comparative analysis, and assign initial weights to the edges based on the frequency of occurrence of co-pronunciation rules. S202. The dynamic dialect prosodic map based on local training of institutions in various regions is jointly trained across regions using a federated learning framework to obtain the global dynamic dialect prosodic map. S203. When new dialect data is accessed, incremental learning is used to add nodes and edges to the global dynamic dialect prosodic graph, and Bayesian optimization is used to balance the weights of the global dynamic dialect prosodic graph and the newly added nodes and edges to achieve dynamic updates.

3. The intelligent error correction method for the prosody of ethnic vocal dialects as described in claim 2, characterized in that, The mathematical expression for the federated learning framework in step S202 is: in, Indicates the first The client is on the local machine. Model parameters after training round Indicates the first The client is on the local machine. Model parameters after training round Indicates the local learning rate. Indicated on the client Local dataset Above, the loss function with respect to parameters gradient vector, Indicates client loss function, Indicates client Local dataset, This represents the total number of clients participating in federated learning. Indicates client encryption function, Indicates Gaussian noise. This represents the variance of Gaussian noise. Indicates the sensitivity of model parameters. Indicates privacy budget, Indicates the first The global model parameters after one round of training. This refers to the decryption function of the central server. Indicates client Aggregate weights, This indicates a parameter concatenation operation. This represents the frozen layer in the global model. Representing a subset The number of clients in the middle, This indicates the subset of clients participating in the current round of updates. This indicates the layer that the client needs to update.

4. The intelligent error correction method for the prosody of ethnic vocal dialects as described in claim 1, characterized in that, In step S3, cross-modal contrastive analysis is performed on the structured training dataset to obtain a joint representation of the cross-modal error features, including: S301. Based on the structured training dataset, the timelines of audio frames, musical notes, and text characters are aligned using a dynamic time warping algorithm to generate synchronous trimodal data containing timestamps and event markers. S302. Compare the fundamental frequency of the audio in the synchronous trimodal data with the MIDI pitch of the musical score, and calculate the pitch error matrix; compare the duration of the audio phonemes in the synchronous trimodal data with the duration of the musical notes, and calculate the duration deviation rate. S303. Compare the audio tone trajectory in the synchronous trimodal data with the dialect tone in the text annotation, and calculate the tone matching degree sequence using the dynamic time warping algorithm; detect whether the audio vowel pronunciation in the synchronous trimodal data and the vowel structure in the text annotation are co-pronounced, and obtain co-pronunciation error markers; S304. Verify whether the musical rhythm pattern and text tone fluctuation in the synchronous three-modal data conform to the coordination rules of the dynamic dialect prosodic graph, and mark the rhythm-tone association error that violates the rules. S305. Based on the pitch error matrix, duration deviation, tone matching degree sequence, coprophony error marker, and rhythm-tone association error, a structured error feature containing error type, location, and confidence is generated through threshold determination and error type classification. S306. Based on the structured error features, a multi-head attention mechanism is used to fuse the error features of audio, musical score, and text, and the joint representation of cross-modal error features is generated by combining the rule weights of dynamic dialect prosodic graph.

5. The intelligent error correction method for the prosody of ethnic vocal dialects as described in claim 1, characterized in that, In step S4, the corrected Mel spectrum is generated by transposed convolution upsampling. Based on the Mel spectrum, the vocoder of the neural vocoder direct-connect correction model is reconstructed into high-fidelity corrected audio, including: S401. Based on the joint representation of the dynamic dialect prosodic graph and cross-modal error features, a neural vocoder direct connection correction model is constructed. The neural vocoder direct connection correction model includes an encoder, a transposed convolution, and a vocoder. A graph rule penalty term is embedded in the encoder loss function. S402. Map the joint representation of the cross-modal error features to the input channel of the transposed convolution, use 4 layers of transposed convolution to upsample layer by layer to generate the corrected Mel spectrum, and dynamically adjust the spectrum amplitude according to the error confidence. S403. The corrected Mel spectrum is converted into a time-domain waveform using the vocoder, and the time-domain waveform is optimized. The fundamental frequency trajectory is adjusted in conjunction with the text tone marking to obtain high-fidelity corrected audio.

6. The intelligent error correction method for the prosody of ethnic vocal dialects as described in claim 1, characterized in that, In step S5, the neural vocoder direct-connect correction model is iteratively optimized by introducing a small number of erroneous samples and using a model-independent meta-learning algorithm, including: S501. Collect and preprocess a small number of error samples, filter high-confidence error samples through manual annotation, label the error type and correction target, and generate an error sample set after data augmentation and format unification. S502. Based on the error sample set and using the model-independent meta-learning algorithm, fix the encoder and transposed convolution parameters of the neural vocoder direct connection correction model, only open the vocoder parameters, and divide the meta-tasks according to dialect type or error type. S503. Based on the model-independent meta-learning algorithm, perform inner loop task adaptation, input a single meta-task into the neural vocoder, generate corrected audio through forward propagation, calculate task-specific loss and fine-tune neural vocoder parameters. S504. Based on the model-independent meta-learning algorithm, perform outer loop parameter aggregation, aggregate the gradients of all meta-tasks, update the global neural vocoder parameters, and balance the cross-task optimization direction. S505. Verify the compliance of the corrected audio rules based on the updated global neural vocoder parameters and dynamic dialect prosodic graph. If a spectrum segment that violates the cooperation rules is found, adjust the loss function of the model irrelevant meta-learning algorithm through regularization term. S506. Iterate through S503 to S505 until the error rate of the corrected audio is lower than the threshold or the maximum number of iterations is reached, generating the final optimized neural vocoder direct-connect correction model.

7. The intelligent error correction method for the prosody of ethnic vocal dialects as described in claim 1, characterized in that, The discrimination results obtained in step S6 for the corrected audio, musical score, and text trimodal data include: S601. Construct a generator input layer, taking the high-fidelity corrected audio as the basic input, and combining the audio features, musical score features and text features to generate a spatiotemporally aligned feature tensor through a cross-modal attention fusion module. S602. Construct the Transformer encoder of the generator, and use a multi-head self-attention mechanism to perform global dependency modeling on the fused spatiotemporally aligned feature tensor to generate a joint latent representation containing audio-musical score-text collaborative relationships. Then, embed the updated dynamic dialect prosodic graph into the joint latent representation through a graph attention network. S603, Configure the decoder branch of the generator, wherein the audio decoder upsamples the joint implicit representation through 4 layers of transposed convolution to generate a corrected Mel spectrum and convert it into a time-domain waveform; the music score decoder adjusts the note timing and pitch based on the joint implicit representation to generate a symbolic music score; the text decoder combines the joint implicit representation to correct tone markings to ensure that the tone is consistent with the audio pronunciation. S604. Construct the first set of discriminators, use a multi-layer convolutional neural network to extract the spectral features of the corrected audio, and use binary classification to determine whether the audio is a real singing recording, so that the generator can improve the naturalness of the audio. S605. Construct a second set of discriminators, align the audio fundamental frequency curve with the text tone annotation based on the dynamic time warping algorithm, verify the tone coordination rules by referring to the dynamic dialect prosodic graph, and determine whether the corrected text conforms to the dialect pronunciation habits. S606. Construct a third discriminator, analyze the matching degree between the musical score rhythm pattern and the audio onset detection result, use the dynamic dialect prosodic graph to verify the rhythm-tone association rule, and determine whether the corrected musical score is consistent with the text tone and audio pronunciation. S607. Execute the cross-modal adversarial training process. First, pre-train the generator to generate reasonable trimodal data through reconstruction loss. Based on the reasonable trimodal data, train the discriminator and generator alternately. Optimize jointly through adversarial loss and rule penalty terms until the error rate of the corrected audio is lower than the threshold or the maximum number of iterations is reached. Finally, output the corrected audio, sheet music, and text trimodal data and the discrimination result.

8. A system for intelligent error correction of the prosody of ethnic vocal dialects, characterized in that, The system includes a method for intelligent error correction of the prosody of ethnic vocal dialects as described in any one of claims 1 to 7; the system further includes: The acquisition module is used to acquire multimodal data and input the multimodal data into the intelligent error correction method; The processing module is used to execute the steps of the intelligent error correction method, process the input audio, music score and text data, and generate corrected audio, music score and text three-modal data and discrimination results; The storage module is used to store the raw data, intermediate data during processing, and the final corrected audio, musical score, and text three-modal data and discrimination results; The output module is used to visualize the corrected audio, musical score, and text three-modal data and the discrimination results.

Images