Chinese opera cavity audio notation method and system, electronic equipment and storage medium

Abstract

The invention discloses a traditional opera cavity audio notation recording method and system, electronic equipment and a storage medium, and relates to the field of voiceprint splicing, and the method comprises the steps of multi-modal feature extraction, adversarial training of a music score, hierarchical transfer learning adaptation, streaming processing and multi-modal verification, and symbol-parameter decoupling output. The multi-modal feature extraction comprises the steps of attention weighted fusion and dynamic adjustment of time-frequency feature contribution degree; the step of adversarial training of the music score comprises verifying a symbol sequence in the music score through a Chinese opera music knowledge graph; the step of hierarchical transfer learning adaptation comprises adaptation training of different theaters; the streaming processing and multi-modal verification comprises the following steps: performing multi-modal alignment, and dynamically adjusting model output and the weight of a rule template according to a voting result; the symbol-parameter decoupling output step comprises symbol-parameter decoupling and outputting of music score data in a target format.

Description

Technical Field

[0001] This application relates to the field of voiceprint splicing, and particularly to a method for notating opera singing audio, an opera singing audio notation system, an electronic device, a storage medium, and a music score generation platform. Background Art

[0002] Traditional opera has the particularity of performance. In addition to voiceprint identification, it particularly involves body movement coordination with singing, and non-linear voiceprint expressions such as portamento and vibrato. Therefore, at present, music score generation mainly relies on manual listening and notation, with low efficiency; secondly, limited by the experience of inheritors and non-standard expressions, it is difficult to carry out music score generation work efficiently and on a large scale.

[0003] Currently, the technical solutions closest to "audio generating music score" mainly fall into two categories:

[0004] 1. Traditional signal processing + machine learning hybrid solution:

[0005] (1) Extract features based on audio spectrum analysis (such as Mel spectrum, MPCC);

[0006] (2) Use Hidden Markov Model (HMM) or random forest for pitch / rhythm recognition;

[0007] (3) Post-process to generate music scores in MIDI or XML format.

[0008] 2. End-to-end deep learning solution:

[0009] (1) Use CNN + BiLSTM network to process audio time-frequency map;

[0010] (2) Achieve time alignment through attention mechanism;

[0011] (3) Output note sequence (such as represented by piano roll);

[0012] (4) Typical representative: Onset and Frames model of GoogleMagenta project;

[0013] Using existing voiceprint recognition technology to solve the limitations in traditional opera music score generation: 1. Pitch recognition error problem: Portamento and vibrato in opera lead to discretization errors; 2. Rhythm quantization problem: Free rhythm singing is difficult to match the standard beat system; 3. Lack of special techniques problem: It is unable to recognize special techniques in opera such as "douyin" and "suyin"; 4. Polyphonic processing problem: Traditional solutions are difficult to separate the human voice and accompaniment parts.

[0014] Therefore, a voice recognition solution for notating opera singing audio is needed to adapt to the performance characteristics of traditional opera and generate music scores for traditional opera efficiently and accurately. Summary of the Invention

[0015] The purpose of the present invention is to provide an audio notation method for opera singing, an audio notation system for opera singing, an electronic device, a storage medium and a music score generation platform, which at least solves the problem of music score generation based on automated voiceprint recognition adapting to the performance characteristics of traditional opera, and solves one technical problem in the problem of efficient and accurate music score generation for traditional opera.

[0016] The present invention provides the following solutions:

[0017] According to one aspect of the present invention, a method for notating audio of opera singing is provided, the method comprising: a step of multimodal feature extraction, a step of adversarial training of music scores, a step of layered transfer learning adaptation, a step of streaming processing and multimodal verification, and a step of symbol-parameter decoupling output;

[0018] The steps of multimodal feature extraction include attention weighted fusion and dynamic adjustment of the contribution of time-frequency features;

[0019] The steps of adversarial training of music scores include verifying the symbol sequence in the score through the opera music knowledge graph;

[0020] The steps of layered transfer learning adaptation include adaptive training for different drama genres;

[0021] The steps of stream processing and multimodal verification include multimodal alignment and dynamic adjustment of model output and rule template weights based on voting results;

[0022] The step of symbol-parameter decoupling and outputting includes symbol-parameter decoupling and outputting music score data in a target format.

[0023] Furthermore, the multimodal feature extraction step includes a time domain branch and a frequency domain branch processed in parallel;

[0024] Extract transient feature information based on time domain branches;

[0025] The steps of multimodal feature extraction also include: extraction of motion features and extraction of audio features;

[0026] Transient features include,transient features of action features;

[0027] Extract steady-state characteristic information based on frequency domain branches;

[0028] Steady-state features include, steady-state features of audio features;

[0029] The steps of multimodal feature extraction also include,feature fusion;

[0030] Feature fusion includes using an attention-weighted fusion layer to dynamically adjust the contribution of time-frequency features.

[0031] Furthermore, the step of adversarial training music score includes:

[0032] Input feature sequence after feature fusion;

[0033] Output the music score and parameter matrix according to the feature sequence;

[0034] Among them, the continuity of symbol sequences in musical scores is analyzed;

[0035] According to the analysis, the continuity of the symbol sequence in the music score is verified through the opera music knowledge graph.

[0036] Furthermore, the step of layered transfer learning adaptation includes: sharing the coding layer;

[0037] The shared encoding layer is a Transformer network, pre-trained on a mixed dataset across genres;

[0038] Also included is a drama adapter;

[0039] The drama adapter uses a low-rank matrix fine-tuning module (LoRA) and inserts the FFN of each layer of Transformer;

[0040] It also includes training strategies: freezing the shared encoder and updating the genre adapter parameters;

[0041] Load the drama adapter in parallel, switch the drama genre by the drama ID and adapt it.

[0042] Furthermore, the steps of streaming processing and multimodal verification include: a causal convolutional layer;

[0043] Causal convolution layer, which constrains the convolution kernel to only access the current input and historical input data;

[0044] Also includes dynamic buffer window;

[0045] The dynamic buffer window expands dynamically with the beat intensity;

[0046] It also includes multimodal alignment: matching keyframes of motion features with keypoints of audio features through dynamic time warping;

[0047] It also includes the implementation of a correction mechanism: when a conflict between the score and the action is detected, a confidence-weighted vote is used;

[0048] Dynamically adjust the weights of model output and rule templates based on voting results.

[0049] Furthermore, the step of symbol-parameter decoupling output includes: symbol-parameter decoupling output layer;

[0050] The symbol-parameter decoupling output layer includes discrete symbol channels, continuous parameter channels, and multi-format converters;

[0051] Discrete symbol channel, used to output symbol sequences of pitch and ornament type, and achieve differentiable sampling through Gumbel-Softmax;

[0052] Continuous parameter channel, used for predicting parameters, including vibrato depth and portamento speed;

[0053] The continuous parameter channel also includes,using Tanh activation constraint range;

[0054] Multi-format converter, based on a rule engine, converts the internal representation into the target format.

[0055] According to two aspects of the present invention, there is provided an audio notation system for opera singing, the audio notation system for opera singing comprising:

[0056] Modules for multimodal feature extraction, adversarial music score training, layered transfer learning adaptation, streaming processing and multimodal verification, and symbol-parameter decoupling output.

[0057] A multimodal feature extraction module for attention-weighted fusion and dynamic adjustment of the contribution of time-frequency features;

[0058] A module for adversarial training of musical scores, used to verify the symbol sequences in musical scores using the opera music knowledge graph;

[0059] A module for layered transfer learning adaptation, used for adaptive training of different drama genres;

[0060] The streaming processing and multimodal verification module is used for multimodal alignment and dynamically adjusts the weights of model output and rule templates based on voting results.

[0061] The module for symbol-parameter decoupling output is used for symbol-parameter decoupling and outputting music score data in a target format.

[0062] According to three aspects of the present invention, there is provided an electronic device, characterized in that it includes: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0063] A computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method for notating audio of opera singing.

[0064] According to the four aspects of the present invention, a computer-readable storage medium is provided, characterized in that it stores a computer program that can be executed by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the method for notating audio of opera singing.

[0065] According to five aspects of the present invention, a music score generation platform is provided, characterized by comprising:

[0066] An electronic device for implementing the steps of the method for notating audio notation of opera singing;

[0067] A processor that runs a program, and when the program runs, the steps of the method for notating audio notation of opera singing are executed based on data output by the electronic device;

[0068] The storage medium is used to store a program, and when the program is running, it executes the steps of the method for notating the audio of opera singing for the data output from the electronic device.

[0069] Through the above solution, the following beneficial technical effects are achieved:

[0070] This application breaks through the performance bottleneck of traditional linear analysis by extracting heterogeneous multimodal features and processing time domain branches and frequency domain branches in parallel.

[0071] This application ensures style compliance by integrating opera theory rules into the generation adversarial process in the adversarial training music scores.

[0072] This application uses layered transfer learning adaptation to achieve rapid adaptation of multiple drama genres with extremely low data sample requirements.

[0073] This application introduces semantics such as body movements into music score generation through streaming processing and multimodal verification, forming a new closed loop.

[0074] This application uses decoupled musical notation representation: separating symbols and parameters to be compatible with the aesthetic expression of traditional musical notation.

[0075] In addition to identifying music score information, this application also adds recognition of action features through the voiceprint recognition method, enhances the ability to record and splice music score information, increases the reliability of music score information, and improves the accuracy and reliability of music score recording. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a flowchart of a method for notating audio of opera singing provided by one or more embodiments of the present invention.

[0077] Figure 2 It is a structural diagram of the audio notation system for opera singing provided by one or more embodiments of the present invention.

[0078] Figure 3A block diagram of the structure of an electronic device for a method for notating audio notation of opera singing provided by one or more embodiments of the present invention. DETAILED DESCRIPTION

[0079] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0080] Figure 1 This is a flowchart of a method for notating audio of opera singing provided by one or more embodiments of the present invention.

[0081] like Figure 1 The method for notating opera singing audio includes: multimodal feature extraction, adversarial training of music scores, layered transfer learning adaptation, streaming processing and multimodal verification, and symbol-parameter decoupling output.

[0082] The steps of multimodal feature extraction include attention weighted fusion and dynamic adjustment of the contribution of time-frequency features;

[0083] The steps of adversarial training of music scores include verifying the symbol sequence in the score through the opera music knowledge graph;

[0084] The steps of layered transfer learning adaptation include adaptive training for different drama genres;

[0085] The steps of stream processing and multimodal verification include multimodal alignment and dynamic adjustment of model output and rule template weights based on voting results;

[0086] The step of symbol-parameter decoupling and outputting includes symbol-parameter decoupling and outputting music score data in a target format.

[0087] In this embodiment, the multimodal feature extraction step includes a time domain branch and a frequency domain branch processed in parallel;

[0088] Extract transient feature information based on time domain branches;

[0089] The steps of multimodal feature extraction also include: extraction of motion features and extraction of audio features;

[0090] Transient features include,transient features of action features;

[0091] Extract steady-state characteristic information based on frequency domain branches;

[0092] Steady-state features include, steady-state features of audio features;

[0093] The steps of multimodal feature extraction also include,feature fusion;

[0094] Feature fusion includes using an attention-weighted fusion layer to dynamically adjust the contribution of time-frequency features.

[0095] In this embodiment, the attention weighted fusion layer includes: F fusion =a·F time +(1-a)·F freq , a=σ(W T [F time ; F freq ]);

[0096] F fusion It is the feature data after the fusion of time domain features and frequency domain features;

[0097] F time is the time domain feature, a·F time is the weighting of time domain features, and a is the weight coefficient;

[0098] F freq is the frequency domain feature, (1-a)·F freq is the weighting of frequency domain features, (1-a) is the weight coefficient;

[0099] Among them, σ is the Sigmoid function, W T is the learnable parameter matrix.

[0100] In this embodiment, the time domain branch and the frequency domain branch processed in parallel include:

[0101] For the input opera audio, the time domain branch and the frequency domain branch are processed in parallel;

[0102] The time domain branch uses a 1D-CNN network to extract the transient features of the articulation starting point and the breath position;

[0103] The frequency domain branch uses the STFT-CNN network to extract the steady-state characteristics of resonance peak distribution and harmonic energy.

[0104] In this embodiment, the steps of adversarial training of music scores include:

[0105] Input feature sequence after feature fusion;

[0106] Output the music score and parameter matrix according to the feature sequence;

[0107] Among them, the continuity of symbol sequences in musical scores is analyzed;

[0108] The continuity of the symbol sequence in the music score is analyzed and verified through the opera music knowledge graph.

[0109] In this embodiment, in addition to recognizing musical notation, voiceprint recognition also incorporates recognition of movement characteristics, such as the initial articulation of a word. For example, when detecting the movement of the "lying fish" in a video, the duration of the corresponding note in the musical notation is automatically extended. In addition to using voiceprints to identify "vocalization movements," this embodiment also incorporates "visual movements" and integrates them with voiceprints, making the notation process adaptive.

[0110] In this embodiment, based on the Transformer-Decoder structure, the feature sequence after feature fusion is input into the generator, and the generator outputs the musical score note symbols and parameter matrix.

[0111] In this embodiment, the discriminator includes a data-driven branch and a knowledge rule branch;

[0112] The driving branch uses Bi-LSTM to analyze the continuity of note sequences in the music score;

[0113] Knowledge rule branches are embedded in the opera music knowledge graph;

[0114] The discriminator performs verification, including verification of the compliance of the rhythm structure and the rationality of the ornaments.

[0115] In this embodiment, the step of adversarial training of the music score further includes:

[0116] Loss Function

[0117] In this embodiment, L CE (G) represents the cross entropy loss associated with the generator G;

[0118] 0.1E[D(G(x))] represents the expectation E of the discriminator D applied to the generated data G(x); the coefficient 0.1 represents the weight status of this item in the overall loss function;

[0119] Represents the weighted sum of the regularization term Ri; the coefficient 0.3 represents the weight state of the regularization in the loss function; R i is a rule violation penalty item, wherein the rule violation penalty item corresponds to the verification of the compliance of the banyan structure and the rationality of the ornamentation;

[0120] According to the loss function, the training of the generator is guided.

[0121] In this embodiment, the steps of layered transfer learning adaptation include: sharing coding layers;

[0122] The shared encoding layer is a Transformer network, pre-trained on a mixed dataset across genres;

[0123] Also included is a drama adapter;

[0124] The drama adapter uses a low-rank matrix fine-tuning module (LoRA) and inserts the FFN of each layer of Transformer;

[0125] It also includes training strategies: freezing the shared encoder and updating the genre adapter parameters;

[0126] Load the drama adapter in parallel, switch the drama genre by the drama ID and adapt it.

[0127] In this embodiment, the steps of stream processing and multimodal verification include: a causal convolutional layer;

[0128] Causal convolution layer, which constrains the convolution kernel to only access the current input and historical input data;

[0129] Also includes dynamic buffer window;

[0130] The dynamic buffer window expands dynamically with the beat intensity;

[0131] It also includes multimodal alignment: matching keyframes of motion features with keypoints of audio features through dynamic time warping;

[0132] It also includes the implementation of a correction mechanism: when a conflict between the score and the action is detected, a confidence-weighted vote is used;

[0133] Dynamically adjust the weights of model output and rule templates based on voting results.

[0134] In this embodiment, the confidence-weighted voting includes: Final_Note=β·Model_Output+(1-β)·Rule_Template, β=f(confidence)_score;

[0135] In this embodiment, Final_Note is the output result of the confidence-weighted voting;

[0136] β·Model_Output is the weighted part of the model output Model_Output, where β is the weight coefficient;

[0137] (1-β)·Rule_Template is the weighted part of the rule template Rule_Template, where (1-β) is the weight coefficient;

[0138] β is the result of a function f, which depends on confidence and is dynamically calculated by the model confidence score;

[0139] score is the evaluation indicator.

[0140] In this embodiment, the step of symbol-parameter decoupling output includes: a symbol-parameter decoupling output layer;

[0141] The symbol-parameter decoupling output layer includes discrete symbol channels, continuous parameter channels, and multi-format converters;

[0142] Discrete symbol channel, used to output symbol sequences of pitch and ornament type, and achieve differentiable sampling through Gumbel-Softmax;

[0143] Continuous parameter channel, used for predicting parameters, including vibrato depth and portamento speed;

[0144] The continuous parameter channel also includes,using Tanh activation constraint range;

[0145] Multi-format converter, based on a rule engine, converts the internal representation into the target format.

[0146] In this embodiment, the target format includes target formats corresponding to gongche notation and simplified notation;

[0147] The mapping rules of the rule engine include: 2 / 4 time notation, and the strong beat of each measure is marked with a ">" symbol;

[0148] The mapping rules also include 8 / 4 gongche notation, marked with the "huoqiang" symbol "△".

[0149] Figure 2 It is a structural diagram of the audio notation system for opera singing provided by one or more embodiments of the present invention.

[0150] like Figure 2 The audio notation system for opera singing shown includes:

[0151] Modules for multimodal feature extraction, adversarial music score training, layered transfer learning adaptation, streaming processing and multimodal verification, and symbol-parameter decoupling output.

[0152] A multimodal feature extraction module for attention-weighted fusion and dynamic adjustment of the contribution of time-frequency features;

[0153] A module for adversarial training of musical scores, used to verify the symbol sequences in musical scores using the opera music knowledge graph;

[0154] A module for layered transfer learning adaptation, used for adaptive training of different drama genres;

[0155] The streaming processing and multimodal verification module is used for multimodal alignment and dynamically adjusts the weights of model output and rule templates based on voting results.

[0156] The module for symbol-parameter decoupling output is used for symbol-parameter decoupling and outputting music score data in a target format.

[0157] It is worth noting that although the present system only discloses a module for multimodal feature extraction, a module for adversarial training of music scores, a module for layered transfer learning adaptation, a module for streaming processing and multimodal verification, and a module for symbol-parameter decoupling output, it does not mean that the present device is limited to the above-mentioned basic functional modules. On the contrary, what the present invention wants to express is that, based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with the existing technology to form an infinite number of embodiments or technical solutions. In other words, the present system is open rather than closed. Just because the present embodiment only discloses individual basic functional modules, it cannot be considered that the scope of protection of the claims of the present invention is limited to the above-mentioned basic functional modules.

[0158] Through the above solution, the following beneficial technical effects are achieved:

[0159] This application breaks through the performance bottleneck of traditional linear analysis by extracting heterogeneous multimodal features and processing time domain branches and frequency domain branches in parallel.

[0160] This application ensures style compliance by integrating opera theory rules into the generation adversarial process in the adversarial training music scores.

[0161] This application uses layered transfer learning adaptation to achieve rapid adaptation of multiple drama genres with extremely low data sample requirements.

[0162] This application introduces semantics such as body movements into music score generation through streaming processing and multimodal verification, forming a new closed loop.

[0163] This application uses decoupled musical notation representation: separating symbols and parameters to be compatible with the aesthetic expression of traditional musical notation.

[0164] In addition to identifying music score information, this application also adds recognition of action features through the voiceprint recognition method, enhances the ability to record and splice music score information, increases the reliability of music score information, and improves the accuracy and reliability of music score recording.

[0165] In the first specific embodiment, the disclosed audio notation strategy for opera singing is based on voiceprint recognition technology and includes: 1. Opera feature enhancement; (1) Establishing an opera-specific timbre library; (2) Designing an opera singing vibrato detection module. 2. Hybrid model architecture: A [original audio] --> B [opera type classification]; B --> | Yu Opera | C [special feature extractor]; B --> | Kunqu Opera | D [special feature extractor]; C / D --> E [adaptive encoder]; E --> F [multi-task output layer]. 3. Setting up an evaluation system; (1) Introducing subjective evaluation indicators from opera experts; (2) Developing an opera rhyme coincidence detection algorithm.

[0166] In this hybrid model architecture, the letters represent different modules or steps in the model processing flow. The specific meanings are as follows:

[0167] 1.A [raw audio] input layer: represents the unprocessed original opera audio data (such as .wav files).

[0168] 2.B{Opera Type Classification} Classifier: Uses a classification model to determine the opera type (such as Henan Opera, Kunqu Opera, etc.) of the input audio.

[0169] The output is the branch condition, which determines which dedicated feature extractor to use subsequently.

[0170] 3.C [Specialized Feature Extractor - Henan Opera]

[0171] Drama-specific module: a feature extractor designed for Henan Opera (may include feature extraction logic such as Henan Opera's unique timbre and rhythm).

[0172] 4.D [Specialized Feature Extractor - Kunqu Opera]

[0173] Drama-specific module: a feature extractor designed for Kunqu Opera (may focus on features such as the vocal cavity and pronunciation of Kunqu Opera).

[0174] 5.E [Adaptive Encoder]

[0175] Shared coding layer: Features extracted from different genres are uniformly encoded into high-level representations, which may include adaptive normalization or cross-genre feature fusion.

[0176] 6.F[Multi-task output layer]

[0177] Output layer: A multi-task learning head designed according to task requirements (such as timbre synthesis, vibrato generation, emotion classification, etc.).

[0178] In the second specific embodiment, the mapping between the core defects of the prior art and the technical solution of this embodiment is disclosed as shown in Table 1.

[0179] Table 1

[0180]

[0181] In this embodiment, the solution to the key technical defects is detailed:

[0182] 1. Improvements to portamento / tremolo processing defects

[0183] The root of the technical defects of the existing technology: When traditional pitch extraction algorithms (such as YIN or pYIN) process the continuously changing vibrato of opera, there are the following problems: (1) the pitch jump between frames leads to "staircase-like" MIDI output; (2) the error in detecting the start and end points of the glissando is >50ms.

[0184] The solution of this embodiment is: A [original audio] --> B [vibrato waveform analysis module]; B --> C [high-frequency vibrator detector]; B --> D [portamento slope calculator]; C --> E [time-frequency joint CNN]; D --> E; E --> F [continuous pitch trajectory generation]; F --> G [MIDI encoding with bend marks].

[0185] In this embodiment, the following are implemented: (1) high-frequency vibration detector: a 20-40 Hz bandpass filter group is designed to extract the throat vibration characteristics unique to opera; (2) portamento slope calculator: the portamento speed is calculated by the rate of change of the short-time autocorrelation function (unit: minute / second); (3) time-frequency joint CNN: the original waveform (time domain) and CQT spectrum (frequency domain) are input simultaneously, and dual-stream network fusion learning is adopted.

[0186] In the experimental data of this embodiment, on the test set of the Peking Opera "The Drunken Concubine", the F1-score of portamento detection was improved from 0.62 of the traditional solution to 0.89.

[0187] 2. Improvement of quantization problem of rubato rhythm

[0188] The root causes of technical defects in existing technologies:

[0189] Existing beat tracking algorithms (such as Madmom) rely on a fixed beat grid, resulting in: (1) free rhythms in rubato being incorrectly quantized to 1 / 8 or 1 / 16 notes; and (2) rhythm density distribution errors reaching ±35%.

[0190] Accordingly, the code design in this embodiment is: (python)

[0191]

[0192] Technical features in this embodiment: (1) - LSTM-CRF beat predictor: learning the elastic rhythm pattern of opera singing (such as the difference between "slow tempo" and "fast three-eye" in Henan opera); (2) non-uniform frame division strategy: dynamically adjusting the frame length according to the predicted instantaneous BPM (variable from 10 to 200ms); (3) rhythm symbolization rules: developing MusicXML extension tags for opera (such as <free-rhythm>).

[0193] The effect of this embodiment is verified: the test of Kunqu opera "The Peony Pavilion" shows that the rhythm conformity of the free-style paragraphs is improved from 58% to 82%.

[0194] 3. Improvements to special technique recognition issues

[0195] The root causes of the technical defects of the existing technology are: (1) the general music transcription model lacks the ability to perceive the acoustic characteristics of opera; (2) the soothing sound (rapid throat vibration) is misjudged as ordinary vibrato; (3) the coughing sound (breath explosion) is classified as noise.

[0196] Therefore, in this embodiment, the technical solution is:

[0197] The process of constructing the opera acoustic fingerprint library is as follows: original opera recording (200 hours) → expert annotation (marking the location of special techniques) → feature distillation (extracting technique MFCC-Pattern) → generative adversarial enhancement (WaveGAN generates variant samples) → storage in the feature retrieval library (Faiss index).

[0198] Improved recognition model: Added a technique detection branch (formula) to the standard Onsets and Frames model:

[0199] Loss = α\cdot Loss_{note} + β\cdot Loss_{ornament} + γ\cdot Loss_{articulation}; multi-scale CQT spectrum input (5 different resolutions) is used to capture technical details.

[0200] In this embodiment, the application case is to recognize the "hammering sound" technique of Hebei Bangzi Opera, with a recall rate of 91.2%.

[0201] 4. The comparative verification of the technical effects in this implementation is shown in Table 2. The performance improvement on the same test set (containing 50 segments each of Peking Opera, Kunqu Opera, and Henan Opera) is:

[0202] Table 2

[0203] index Traditional solutions The solution of this application Improvement Pitch continuity error (cents) 82.6 29.4 64.4％ Rhythm compliance (F1) 0.71 0.89 25.3％ Special technique recall rate 41.7％ 86.5％ 107.4％ Vocal separation purity 78.2dB 93.6dB 19.7％

[0204] 5. Summary of the solution in this embodiment: Breaking through the limitations of general music transcription technology through domain adaptability transformation:

[0205] 1. Injecting prior knowledge of opera acoustics: building a dedicated feature library and special symbol system;

[0206] 2. Hybrid architecture design: joint time-frequency network + dynamic beat tracking + multi-task learning;

[0207] 3. Innovation in the evaluation system: Introducing the opera rhythm similarity (XQSIM) evaluation index;

[0208] This embodiment has been applied in actual opera digital preservation projects, reducing the workload of manual proofreading by 73% compared with traditional technologies, and has significant technological advancement and industrial application value.

[0209] In the third specific embodiment, the technical goals set are: (1) to achieve end-to-end high-precision conversion of opera singing audio to musical scores (note F1 value ≥ 90%); (2) to support rapid migration of more than 21 opera genres (≤ 5 hours of data adaptation); (3) to achieve 200ms-level real-time processing capabilities; (4) to improve the completeness of musical score restoration of low-quality historical recordings to more than 75%.

[0210] The solution framework set in this embodiment is: adopting a modular deep learning framework: 1. Multimodal feature extraction → 2. Adversarial training music score generation → 3. Layered migration adaptation → 4. Streaming multimodal verification → 5. Decoupled music score output.

[0211] In this embodiment, a time-frequency dual-stream feature extraction network is provided: (1) input; (2) audio waveform (44.1kHz sampling, PCM encoding); and (3) optional body motion signal (50Hz frame rate, 6-dimensional vector encoding).

[0212] Parallel processing branches are used: (1) Time domain branch: 1D-CNN (64 layers, convolution kernel 64×1, stride 8); extract transient features such as articulation starting point and breath position (time resolution ≤ 10ms). (2) Frequency domain branch: STFT-CNN (window length 1024 points, Hanning window, 8 layers of convolution); extract steady-state features such as resonance peak distribution and harmonic energy (frequency resolution ≤ 5Hz).

[0213] Perform feature fusion: Dynamic Attention Weighted Fusion Layer (Formula): F fusion =a·F time +(1-a)·F freq , a=σ(W T [F time ; F freq ]);

[0214] (1)F fusion : It is the fused feature representation, which combines the time domain features and frequency domain features.

[0215] (2)F time : represents the time domain feature a·F time The weighted part of , where a is the weight coefficient.

[0216] (3)(1-a)·F freq : represents the frequency domain feature F freq The weighted part of , where (1-a) is the weight coefficient.

[0217] (4)a=σ(W T [F time ; F freq ]): a is the time domain feature F time and frequency domain features F freq Connect them and then pass a weight vector W T The transpose of is linearly transformed and finally obtained by an activation function σ (usually a sigmoid function). This activation function ensures that the value of a is between 0 and 1, thereby controlling the relative contribution of time domain and frequency domain features.

[0218] The above formula in this embodiment describes an adaptive fusion method that dynamically adjusts the weights of time domain and frequency domain features according to their relative importance to generate a more effective fused feature representation.

[0219] Where σ is the Sigmoid function and W is the learnable parameter matrix.

[0220] In this embodiment, a knowledge-guided adversarial training framework is included: (1) Generator (G). Based on the Transformer-Decoder structure (12 layers, 8 attention heads), it inputs a fused feature sequence and outputs preliminary musical notation symbols and parameter matrices.

[0221] (2) Discriminator (D). Data-driven branch: 3-layer Bi-LSTM analyzes the continuity of note sequences; knowledge rule branch: embeds the opera musicology knowledge graph (including the ban rule library and ornamentation syntax tree of 21 opera genres); performs the following verifications: banyan structure compliance (such as the strong beat position of each measure in the Peking Opera Xipi Liushuiban) and ornamentation context rationality (such as the non-consecutive occurrence of the sou sound more than 3 times).

[0222] (3) Loss function:

[0223] L CE (G): This term represents the cross entropy loss associated with the generator G. Cross entropy loss is often used in classification tasks to measure the difference between the predicted probability distribution and the true distribution.

[0224] 0.1E[D(G(x))]: This term represents the expectation E that the discriminator D should apply to the generated data G(x). The coefficient of 0.1 means that this term has a low weight in the overall loss function. This term is seen in generative adversarial networks (GANs), where the goal of the generator is to generate data that the discriminator cannot distinguish from real data.

[0225] This term represents the regularization term R i A weighted sum of , used to penalize violations of certain constraints. The coefficient 0.3 indicates the weight of this regularization in the loss function. The specific nature of the violation and the content of the regularization term depend on the context of the model.

[0226] In summary, this loss function combines the cross entropy loss, a term related to the discriminator output, and a regularization term to guide the training of the generator in GAN or similar models.

[0227] R i This is a penalty for rule violation (e.g., the strong beat position of the Peking Opera Xipi Liushuiban is wrong).

[0228] In this embodiment, a hierarchical transfer learning adaptation method is included: (1) the shared encoding layer includes a 12-layer Transformer (hidden layer 768 dimensions), pre-trained on a cross-genre mixed dataset (COAC); (2) the genre adapter includes a structure: a low-rank matrix fine-tuning module (LoRA, rank r = 8), inserted after the FFN of each layer of Transformer; the training strategy is to freeze the shared encoder and only update the adapter parameters (learning rate 5e-4); multiple genres support: parallel loading of adapters (such as Peking Opera Adapter_A and Kunqu Opera Adapter_B), and switching by genre ID.

[0229] In this embodiment, the stream processing engine design includes: (1) causal convolution layer: constraining the convolution kernel to access only the current and historical inputs to ensure real-time performance; (2) dynamic cache window: basic window length: 2 seconds of audio (about 4-8 phrases), dynamically expanded with the beat intensity (maximum 5 seconds); (3) multimodal alignment: matching body movement key frames and audio feature points through dynamic time warping (DTW); (4) real-time correction mechanism: when a conflict between the score and the action semantics is detected (e.g., the dragging of the [hair shake] is not extended), a confidence-weighted voting is adopted: Final_Note = β·Model_Output + (1-β)·

[0230] Rule_Template, β = f(confidence)_score; where Final_Note is the final output or result, combining the model output and the rule template. β·Model_Output: represents the weighted part of the model output Model_Output, where β is the weight coefficient. (1-β)·Rule_Template: represents the weighted part of the rule template Rule_Template, where (1-β) is the weight coefficient. β =

[0231] f(confidence)_score: Here β is the result of a function f() that depends on confidence.

[0232] Confidence can be the model's confidence in its output and is used to dynamically adjust the relative weights of the model output and the rule template. Score is a rating or evaluation metric.

[0233] The above formula in this embodiment describes an adaptive method that dynamically adjusts the weights of the model output and the predefined rule template according to the confidence of the model to generate the final output or result.

[0234] β is dynamically calculated from the model confidence score.

[0235] In this embodiment, the symbol-parameter decoupling output layer includes: (1) discrete symbol channel: outputs symbol sequences such as pitch and decoration type, and realizes differentiable sampling through Gumbel-Softmax; (2) continuous parameter channel: predicts parameters such as vibrato depth (0-1) and glissando speed (Hz / ms), and uses Tanh activation constraint range; (3) multi-format converter: converts the internal representation into the target format (such as gongche notation and simplified notation) based on the rule engine. The key mapping rules include: Peking Opera Erhuang original version → 2 / 4 beat simplified notation, with the strong beat of each measure marked with the symbol ">"; Kunqu Opera Zengban → 8 / 4 beat gongche notation, with the "huoqiang" symbol "△" marked.

[0236] The technical effectiveness was verified by comparing the performance on the COAC dataset (12 drama genres, 300 hours), as shown in Table 3:

[0237] Table 3

[0238]

[0239] In a fourth specific embodiment, the training process includes:

[0240] 1. Data preprocessing:

[0241] Audio normalization (-24dB LUFS), segmented windowing (1024 points, 512 points overlap); Among them, body motion encoding: convert video key frames into 6-dimensional vectors (hand coordinates, head posture, etc.).

[0242] 2. Pre-training stage:

[0243] The shared encoder is trained on the COAC dataset and the CTc loss function is used to optimize feature alignment.

[0244] 3. Adversarial training stage:

[0245] The encoder parameters are fixed, and the generator and discriminator are trained alternately, with an update ratio of G:D = 3:1 in each round.

[0246] 4. Migration and adaptation phase:

[0247] Load 5 hours of data for the target genre and fine-tune only the corresponding adapter (learning rate 5e-4, 50 epochs).

[0248] In a fifth specific embodiment, system deployment includes,

[0249] Edge device side:

[0250] Hardware: NVIDIA Jetson AGX Xavier, 32GB memory;

[0251] Deployment model: pruned two-stream network (parameter size compressed from 250M to 80M).

[0252] Cloud service: Provides an API interface: inputs an audio file and returns a musical score in JSON format (including symbols and parameter matrices).

[0253] The above implementation schemes are summarized as follows: an opera audio transcription method based on time-frequency dual-stream CNN and knowledge-guided adversarial training; a pluggable adapter and low-rank fine-tuning mechanism in layered transfer learning; and a multimodal real-time verification algorithm for the streaming processing engine.

[0254] In the sixth specific embodiment, an end-to-end deep learning framework is adopted, which includes the following core modules: 1. Multimodal feature extraction module; 2. Adversarial training music score generation module; 3. Hierarchical transfer learning adaptation module; 4. Streaming processing and multimodal verification module; 5. Symbol-parameter decoupling output module.

[0255] The system process includes: A[original audio]-->B[multi-channel sound source separation]; B-->C[dynamic frame preprocessing]; C-->D[time-frequency joint feature extraction]; D-->E[opera-specific AI model]; E-->F[pitch trajectory prediction]; E-->G[rhythm pattern analysis]; E-->H[technical feature matching]; F&G&H-->I[multimodal music score synthesis]; I-->J[opera symbol rendering engine]; J-->K[final music score output].

[0256] Among them, the core modules and steps include: B; C; D; E; J; end.

[0257] In this embodiment, the key technical solutions are described.

[0258] 1. The time-frequency dual-stream feature extraction network includes:

[0259] - Input layer: Receives audio waveforms (44.1kHz sampling rate) and optional body motion capture signals (50Hz frame rate).

[0260] - Parallel processing branches:

[0261] - Time-domain branch: 1D-CNN (convolution kernel size 64×1, stride 8, ReLU activation) to extract transient features such as the onset of articulation and breathing pauses.

[0262] - Frequency-domain branch: STFT-CNN (window length 1024 points, Hann window, 50% overlap), extracting steady-state features such as formant distribution and harmonic energy through 8 layers of convolution.

[0263] - Feature fusion: An attention-weighted fusion layer is used to dynamically adjust the contribution degrees of time-frequency features:

[0264] F fusion = a·F time +(1 - a)·F freq , a = σ(W T [F time ; F freq ); where σ is the Sigmoid function and W is a learnable parameter matrix.

[0265] 2. Knowledge-guided adversarial training framework:

[0266] - Generator (G): Based on the Transformer-Decoder structure (12 layers, 8-head attention), taking the fused feature sequence as input and outputting preliminary musical score symbols and parameter matrices.

[0267] - Discriminator (D):

[0268] - Data-driven branch: 3-layer Bi-LSTM to analyze the continuity of note sequences;

[0269] - Knowledge rule branch: Embedding the knowledge graph of traditional Chinese opera musicology (including the metrical pattern rule library and grace note grammar tree of 21 opera types), performing the following validations:

[0270] - Compliance of the metrical structure (such as the strong beat position in each measure of the Xipi Liushuiban in Beijing Opera)

[0271] - Rationality of grace note context (such as the tremolo not appearing continuously more than 3 times)

[0272] - Loss function:

[0273] R i is the penalty term for rule violation (such as the wrong strong beat position in the Xipi Liushuiban in Beijing Opera).

[0274] 3. Hierarchical transfer learning adaptation method:

[0275] - The shared encoding layer includes: 12-layer Transformer (hidden layer with 768 dimensions), pre-trained on a cross-opera type mixed dataset (COAC).

[0276] -Type adapters include:

[0277] (1) Structure: Low-rank matrix fine-tuning module (LoRA, rank r = 8), inserted after the FFN of each Transformer layer;

[0278] (2) Training strategy: freeze the shared encoder and only update the adapter parameters (learning rate 5e-4);

[0279] (3) Support for multiple genres: Load adapters in parallel (such as Peking Opera Adapter_A and Kunqu Opera Adapter_B) and switch by genre ID.

[0280] In the seventh specific embodiment, the prior art is compared with the present application.

[0281] 1. Comparison of core architectures of technical solutions

[0282] Existing technical architecture: audio input → [general preprocessing] → [standard Mel spectrum] →

[0283] [CNN+BiLSTM]→[MIDI encoder]→music score output.

[0284] Core flaws:

[0285] 1. Using universal music features (Mel spectrum) to ignore the acoustic characteristics of opera;

[0286] 2. The fixed frame strategy destroys the rhythmic structure of the rubato;

[0287] 3. Single-task output cannot capture special techniques;

[0288] The architecture in this embodiment is:

[0289] A[Opera Audio]-->B{Multimodal Preprocessing}; B-->C[Dynamic Framing]; B-->D[Sound Source Separation];

[0290] C-->E[Joint time-frequency features]; D-->E; E-->F[Xiju-specific AI model]; F-->G[Pitch / rhythm / technique multi-task output]; G-->H[Xiju symbol encoder]; H-->I[Music score with Xiju rhyme markings];

[0291] 2. Key Technical Differences and Improvements

[0292] 1. Comparison of feature extraction layers,

[0293]

[0294]

[0295] in,

[0296] - Opera Enhanced CQT: Designed with a 128-bin logarithmic frequency axis, focusing on covering the opera vocal frequency band (200Hz-2kHz);

[0297] -Dynamic framing algorithm: (python)

[0298] def adaptive_frame(audio,onset_detect):

[0299] frame_length = 1000 / (tempo_prediction * 2) # Unit: ms

[0300] return overlapping_frames(audio,frame_length,0.3)

[0301] - Voiceprint fingerprint matching: Use Faiss approximate nearest neighbor search to match pre-stored technical features.

[0302] 2. Model structure comparison,

[0303] Existing Model (Onsets and Frames):

[0304] mermaid

[0305] graph LR

[0306] A[Mel spectrum]-->B[CNN feature extraction]; B-->C[BiLSTM time series modeling]; C-->D[fully connected output layer].

[0307] The model of this embodiment (opera multi-task network):

[0308] mermaid

[0309] graph LR

[0310] A[Joint time-frequency input]-->B{Dual-stream feature extraction}; B-->C[Time domain CNN]; B-->D[Frequency domain CNN]; C-->E[Feature fusion gate]; D-->E; E-->F[LSTM-CRF rhythm prediction]; E-->G[Pitch trajectory regression]; E-->H[Technique classifier]; F&G&H-->I[Multi-task loss function].

[0311] Analysis of the technical features in this embodiment:

[0312] -Dual-stream input structure: Simultaneously processes time-domain waveforms (preserving phase information) and frequency-domain CQT (fine frequency resolution);

[0313] -LSTM-CRF Beat Prediction: Introduce conditional random field to constrain the rhythm conversion logic;

[0314] Multi-task Loss Function: (math)

[0315] \mathcal{L}=0.5\mathcal{L}_{pitch}+0.3\mathcal{L}_{rhythm}+

[0316] 0.2\mathcal{L}_{ornament}.

[0317] 3. Comparison in the post-processing stage,

[0318] Existing Technical Problems:

[0319] - Mechanical MIDI quantization loses the charm of traditional Chinese opera;

[0320] - Unable to represent special singing notations;

[0321] Solution of this Embodiment:

[0322] mermaid

[0323] graph TD

[0324] A[Original Output] --> B{Traditional Chinese Opera Symbolization Engine}; B --> C[Glissando Curvature Calculation]; B --> D[Free Rhythm Relaxation]; C --> E[Add MIDI Pitch Bend]; D --> E; E --> F[Generate MusicXML Extension Tags]; F --> G[Render Traditional Chinese Opera Special Music Score].

[0325] Among them, the key technologies are, - Glissando Quantization Algorithm: (python)

[0326]

[0327] - Traditional Chinese Opera XML Extension: Define custom tags such as <xiqu:ornament type="vibrato" / >`.

[0328] III. Verification of Objective Improvement Effects:

[0329] 1. Performance Comparison Experiment (Test Set: 300 Traditional Chinese Opera Audio Segments)

[0330]

[0331]

[0332] 2. Application Effect Cases

[0333] - Transcription of Beijing Opera "Empty City Stratagem":

[0334] - Traditional solution: 87% of the throat trills are lost, and the free rhythm is wrongly quantified as 4 / 4 beat;

[0335] - In this application: 13 "sounding" marks are completely retained to generate an elastic rhythm skeleton.

[0336] IV. Technical process of this embodiment

[0337] A [Original audio] -> B [Multi-channel sound source separation]; B -> C [Dynamic frame preprocessing]; C ->

[0338] D [Time-frequency joint feature extraction]; D -> E [AI model dedicated to opera]; E -> F [Pitch trajectory prediction]; E -> G [Rhythm pattern analysis]; E -> H [Technique feature matching]; F & G & H -> I [Multi-modal music score synthesis]; I -> J [Opera symbol rendering engine]; J -> K [Final music score output].

[0339] Core part: B; C; D; E; J; end.

[0340] V. Summary of the technical features of this application

[0341] 1. Adaptation transformation in the field

[0342] - Pioneering the acoustic feature fingerprint library and dedicated symbol system for opera;

[0343] - Developing a time-frequency joint modeling network structure.

[0344] 2. Innovation in algorithm architecture

[0345] - Co-design of dynamic frame strategy and LSTM-CRF rhythm prediction;

[0346] - Multi-task learning framework to achieve joint optimization of pitch / rhythm / technique.

[0347] 3. Breakthrough in application value

[0348] - The transcription accuracy of opera singing has exceeded the 90% mark for the first time; [[ID=4T]]

[0349] - The workload of manual proofreading is reduced by 76%, accelerating the digitalization process of intangible cultural heritage.

[0350] Technical effects

[0351] 1. Improvement in feature extraction dimension and accuracy (technical breakthrough)

[0352] Traditional audio spectrum conversion technology relies on Fourier transform (FFT) or short-time energy detection with a fixed frequency threshold, which has an insufficient capture rate of nonlinear acoustic features unique to opera, such as glissando and vibrato (experimental data shows only 62%). This solution adopts a composite neural network architecture:

[0353] -Time-frequency dual-stream convolution module: This module extracts time-domain transient features (such as articulation onset) and frequency-domain formant features (such as vocalization resonance) through parallel 1D-CNN and STFT-CNN, respectively.

[0354] -Attention-enhanced recurrent network: Using a Bi-LSTM + Attention mechanism to model long-term dependencies, it achieves 94% continuous tracking accuracy for pitch changes in drawn-out vocalizations (average duration of 1.2 seconds per syllable).

[0355] - Adversarial training mechanism: Introducing the Wasserstein GAN architecture, the discriminator network identifies the consistency of the opera style of the generated music scores, increasing the accuracy of Peking Opera Xipi Erhuang banshi recognition to 89%;

[0356] 2. Adaptability to multiple genres (technical generalization)

[0357] Traditional methods require separate modeling for different opera genres (for example, Kunqu opera gongche notation and Peking opera simplified notation require different rule libraries). This solution achieves this through:

[0358] -Layered transfer learning framework: A shared acoustic feature encoder (12-layer Transformer) at the bottom layer and pluggable genre-specific adapters at the top layer enable migration with just 5 hours of target genre data.

[0359] -Data augmentation strategy: Using physical acoustic simulation (room impulse response) and style transfer (converting between Mei style and Cheng style singing), the training data coverage was increased by 3.6 times;

[0360] -Symbolic representation learning: Decomposes musical elements into discrete symbols (pitch, duration, ornaments) and continuous parameters (vibrato depth, portamento speed), supporting output in multiple formats such as MIDI, gongche notation, staff notation, and simplified notation.

[0361] 3. Real-time interaction and error correction mechanism (user experience innovation)

[0362] Compared with the traditional offline processing mode (average delay > 30 seconds), the system:

[0363] -Streaming processing engine: uses causal convolution and dynamic caching mechanisms to achieve real-time music score generation at 200ms level;

[0364] - Multi-modal verification module: Construct a three-dimensional mapping space of singing, music score, and body movements. When conflicts are detected between the music score and stylized movements (such as the drawn-out notes accompanied by the [Water Sleeve] movement), it automatically triggers confidence-weighted correction.

[0365] - Human-machine collaborative annotation: Develop a progressive feedback interface that allows inheritors to annotate key singing nodes (such as the position of [Tremolo]), and the system completes model incremental updates within 24 hours.

[0366] 4. Digitalization efficiency of cultural heritage (application value)

[0367] Verified by the National Academy of Chinese Theatre Arts:

[0368] - In the project of restoring precious recordings of veteran artists, this system shortened the single-song filing time from the traditional 40 working hours to 2.3 hours.

[0369] - The completeness of music score restoration for monophonic historical recordings from the 1940s (signal-to-noise ratio ≤ 15 dB) reached 78%, far exceeding the 52% of traditional methods.

[0370] - Support parallel transfer learning for 21 local opera genres, reducing the digital protection cost of endangered opera genres (such as Haicheng Trumpet Opera) by 83%.

[0371] Verification of technical effects:

[0372] On the cross-opera test set (including 300 hours of audio from 12 opera genres), this solution achieved significant improvements compared to the baseline model:

[0373] - F1 value of note onset detection: 92.4% vs. 76.8% of the traditional HMM method

[0374] - Recognition rate of grace notes: 85.7% vs. 63.2% of the rule engine

[0375] - Restoration degree of the structure of musical patterns: 89.3% vs. 71.5% of the template matching method

[0376] - Cross-opera generalization error: 14.2% vs. 37.6% of the single-task learning model

[0377] The core innovation of this application lies in the deep integration of deep learning and the theory of Chinese opera musicology, realizing the quantification of the aesthetic features of Chinese opera singing that are difficult to capture by traditional acoustic feature engineering through a modular neural network architecture, providing an extensible technical paradigm for the digital inheritance of intangible cultural heritage.

[0378] In the eighth specific embodiment, an alternative solution is disclosed.

[0379] I. Alternative solution for feature extraction

[0380] 1. Alternative solution for time-frequency analysis;

[0381] - Wavelet transform replaces STFT;

[0382] Continuous wavelet transform (CWT) is used to construct the time-frequency domain feature extraction layer. The Morlet wavelet basis function is used to capture non-stationary signal characteristics (such as the instantaneous frequency changes of portamento), replacing the traditional short-time Fourier transform (STFT) to solve the problem of insufficient frequency resolution caused by fixed window length.

[0383] -Physical model driven feature extraction;

[0384] By embedding prior knowledge of acoustic physics (such as the source-filter model), separating the fundamental frequency and resonance peak through linear predictive coding (LPC), and combining it with neural networks for joint optimization, the characteristic analysis of special vocalization methods of opera (such as "back-of-the-head sound") is enhanced.

[0385] 2. Neural network architecture replacement;

[0386] -Graph neural network (GNN) modeling note relationships;

[0387] The temporal dependency between notes is constructed as a graph structure, where nodes represent note attributes (pitch, duration), and edges represent connection logic such as ornaments and glissando. GNN is used to learn the music score generation rules, replacing traditional RNN sequence modeling.

[0388] - Spiking neural networks (SNNs) process time series signals;

[0389] The spike timing-dependent plasticity (STDP) of SNN is used to simulate the perception mechanism of human auditory nerve to the rhythm of opera, which is particularly suitable for the bio-inspired recognition of banyan beats.

[0390] 2. Model Training and Optimization Alternatives

[0391] 1. Alternatives to adversarial training;

[0392] - Reinforcement learning replaces GAN;

[0393] A Markov decision process (MDP) for music score generation is constructed, and the reward function is designed based on the theoretical rules of opera music (such as the integrity of the plate structure). The generation strategy is optimized through the PPO algorithm to replace the discriminator mechanism in adversarial training.

[0394] -Knowledge distillation compression model;

[0395] A teacher-student network architecture is adopted to compress the composite neural network into a lightweight model (such as a MobileNet variant). Through attention transfer, the ability to capture high-order features such as ornaments is maintained, making it suitable for edge device deployment.

[0396] 2. Transfer learning alternatives;

[0397] -Meta-learning framework for rapid adaptation

[0398] Based on Model-Agnostic Meta-Learning (MAML), meta-feature representations shared by multiple drama genres are learned during the pre-training stage. Only a small number of samples (such as 10 singing segments) are needed to quickly adapt to new drama genres, replacing the fixed adapter design in hierarchical migration.

[0399] -Cross-modal pre-training enhances generalization;

[0400] By introducing opera video-audio-music score triplet data and constructing a cross-modal embedding space through contrastive learning, the model's robustness to missing information (such as historical recording noise) is improved.

[0401] 3. Music Score Generation and Post-Processing Alternatives

[0402] 1. Symbolically represent alternatives;

[0403] -Probabilistic grammar model constrains the generated results;

[0404] The music theory rules of opera are encoded into a probabilistic context-free grammar (PCFG), and the original music score output by the neural network is parsed and corrected to ensure that the generated result conforms to the ban style specifications of a specific opera genre (such as the "banyan" structure of Peking Opera).

[0405] - Differentiable music rendering engine;

[0406] Build an end-to-end differentiable synthesizer (such as DiffSynth) to jointly optimize music score generation and audio reconstruction, and correct pitch quantization errors through an auditory feedback loop.

[0407] 2. Interactive Correction Alternatives

[0408] -Active learning to optimize annotation efficiency;

[0409] An active learning strategy based on uncertainty sampling is designed to automatically identify error-prone musical score segments (such as densely packed ornamentation passages) and prioritize requests for manual annotation, replacing fixed-interval progressive feedback.

[0410] -Augmented reality (AR) assisted correction;

[0411] An AR visualization interface was developed to superimpose the generated music score with the actual lip shape and gestures of the inheritor during singing, and real-time correction was triggered through multimodal consistency detection.

[0412] 4. System Implementation Alternatives

[0413] 1. Hardware acceleration alternatives

[0414] -FPGA customized computing architecture;

[0415] In response to the parallel computing requirements of time-frequency dual-stream convolution, an FPGA-based pipeline processing unit is designed to implement heterogeneous computing acceleration of 1D-CNN and STFT-CNN through hardware description language (HDL), replacing general-purpose GPU operations.

[0416] -Neuromorphic chips process sequential signals;

[0417] By leveraging the asynchronous pulse computing characteristics of neuromorphic chips such as Loihi, the long-term dependency characteristics of opera singing (such as dragging notes lasting more than 10 seconds) can be efficiently processed, reducing power consumption by about 60%.

[0418] 2. Edge-cloud collaborative architecture;

[0419] -Lightweight model on the device + cloud-based fine-tuning;

[0420] Deploy the pruned feature extraction model (parameter size <1MB) on mobile devices, collect singing voices in real time and upload them to the cloud to complete music score generation, protect user data through differential privacy technology, and replace the pure end-to-end architecture.

[0421] 5. Complete technical path alternative

[0422] 1. Acoustic event detection driving scheme;

[0423] Abandon the end-to-end generation paradigm and adopt a two-stage processing flow:

[0424] -Phase 1: Detect acoustic events in singing (such as soothing and crying) based on CRNN and output symbolic labels with timestamps;

[0425] -Phase 2: Convert the symbol sequence into the target opera score template through the template filling method, and complete the ornaments by combining the expert rule library;

[0426] 2. Physical model reverse engineering solution;

[0427] Construct a physical modeling synthesizer for opera singing (such as vocal cord vibration simulation based on the mass-spring model), estimate the motion parameters of the vocal organs (such as the glottal opening curve) from the audio through parameter inversion technology, and then map these parameters into musical notation symbols.

[0428] 6. Data Acquisition and Enhancement Alternatives

[0429] 1. Synthetic data generation alternative;

[0430] -Neural vocoder simulates the sound of old records;

[0431] Use WaveGlow or HiFi-GAN to generate training data with specific noise characteristics (such as the scratching sound of a 78-rpm record) to enhance the model's adaptability to historical recordings.

[0432] -Style decoupling and reorganization enhancement;

[0433] Through β-VAE, the opera style and singing content characteristics in the singing are decoupled, and cross-genre mixed samples (such as Pingju lyrics + Yueju singing) are randomly recombined to improve the generalization of the model.

[0434] 2. Crowdsourcing data collection mechanism;

[0435] Build a distributed data platform empowered by blockchain to encourage opera enthusiasts to upload singing recordings and mark key nodes (such as breath points), and realize data ownership and quality verification through smart contracts.

[0436] The comparison of the technical effects of the alternative solutions is shown in Table 4.

[0437] Table 4

[0438]

[0439] Through the above alternatives, a multi-layered protection network can be constructed in the claims to prevent others from circumventing the originality of the present application through the following paths:

[0440] 1. Component-level replacement (e.g., replacing Bi-LSTM with GNN);

[0441] 2. Process reorganization (e.g. splitting end-to-end generation into two stages: detection and template filling);

[0442] 3. Technology crossover (e.g., combining neuromorphic chips with meta-learning frameworks);

[0443] 4. The key technical points of this embodiment are as follows:

[0444] 1. Time-frequency dual-stream feature extraction based on composite neural network architecture;

[0445] -Technical Approach: A dual-stream network consisting of 1D-CNN (time domain) and STFT-CNN (frequency domain) is deployed in parallel, combined with residual connections and feature fusion layers to collaboratively capture transient features (such as the starting point of articulation) and steady-state features (such as the distribution of resonance peaks) in opera singing.

[0446] In this embodiment, the limitations of the traditional single time-frequency analysis method are overcome. The time domain resolution (≤10ms) and frequency domain resolution (≤5Hz) are optimized separately through heterogeneous network branches, and the extraction accuracy of nonlinear features such as glissando and vibrato is improved to 94%.

[0447] In this embodiment, a parameter sharing mechanism of the dual-stream structure and a dynamic weight allocation algorithm of the feature fusion layer are used.

[0448] 2. Ensuring consistency of opera styles based on adversarial training;

[0449] -Technical Approach: A Wasserstein GAN (WGAN-GP) framework is introduced, in which the generator outputs a sequence of musical notation symbols, and the discriminator is embedded in the knowledge graph of opera musicology (such as banshi rules and ornamentation grammar). A gradient penalty mechanism is used to constrain the generated results to conform to the style of the opera.

[0450] In this embodiment, domain knowledge is encoded as a structured evaluation indicator of the discriminator (such as the beat density threshold of the Peking Opera "Xi Pi Liu Shui" style), rather than relying on pure data-driven, which improves the accuracy of style recognition from 71% to 89%.

[0451] In this embodiment, a rule-based knowledge graph embedding method in the discriminator and a joint optimization strategy of the generator and the discriminator are adopted.

[0452] 3. A multi-genre adaptive framework for hierarchical transfer learning;

[0453] -Technical means: Construct a hierarchical architecture of shared Transformer encoders + pluggable adapters. The bottom-level encoder learns common acoustic features across different genres, and the top-level adapter quickly adapts to the characteristics of the target genre through low-rank matrix fine-tuning (LoRA).

[0454] In this embodiment, only 5 hours of target drama data are needed to complete the migration (traditional methods require ≥50 hours), supporting parallel learning of 21 drama genres, and reducing the cross-genre generalization error to 14.2%.

[0455] In this embodiment, a modular design of the adapter and an adapter parameter reuse mechanism based on task similarity are adopted.

[0456] 4. Stream processing engine and multi-modal real-time verification;

[0457] -Technical means: Causal convolution + dynamic cache window is used to achieve streaming processing of vocal audio, and a three-dimensional mapping space of vocalization, music score and body movements is simultaneously constructed. When a conflict between the music score and the programmed movement is detected (such as the extended dragging of the voice accompanying the "hair-flipping" movement), a confidence-weighted correction is triggered.

[0458] In this embodiment, the multimodal relevance of opera performances is incorporated into the music score generation logic, the real-time processing delay is ≤200ms (traditional methods ≥30 seconds), and the action conflict false alarm rate is reduced to below 3%.

[0459] In this embodiment, a dynamic cache window sliding strategy and a multimodal confidence fusion algorithm are adopted.

[0460] 5. Musical score representation method with symbol-parameter decoupling;

[0461] - Technical means: Decompose musical score elements into discrete symbols (pitch, decoration type) and continuous parameters (vibrato depth, portamento speed), and generate symbol sequences and parameter matrices respectively through a dual-channel output layer, supporting multi-format output such as gongche notation, simplified notation, and MIDI staff notation.

[0462] In this embodiment, the problem of volume distortion of decorative sounds in traditional symbolic music generation is solved, and the degree of restoration of opera-specific techniques such as souyin and dùnyin reaches 85.7%.

[0463] In this embodiment, a joint training loss function of symbols and parameters and a unified intermediate representation for multi-format conversion are adopted.

[0464] Summary of this embodiment:

[0465] 1. Heterogeneous feature extraction: Breaking through the performance bottleneck of traditional linear analysis through a time-frequency dual-stream network.

[0466] 2. Knowledge-guided adversarial training: Integrate opera theory rules into the generative adversarial process to ensure style compliance.

[0467] 3. Modular migration architecture: enables rapid adaptation of multiple drama genres with extremely low data requirements.

[0468] 4. Multimodal real-time generation: For the first time, the semantics of body movements are introduced into the closed loop of music score generation.

[0469] 5. Decoupled musical notation representation: separating symbols and parameters to maintain compatibility with the aesthetic expression of traditional musical notation.

[0470] In a ninth embodiment, technical standards and data sets;

[0471] 1. Dataset: Chinese Opera Acoustic Corpus (COAC)*

[0472] -Issuing agency: Chinese Academy of Traditional Chinese Opera

[0473] -Content: Includes 12 opera genres, 300 hours of high-fidelity singing audio and gongche notation annotations.

[0474] - Application: Serves as a benchmark dataset for training and testing of the present invention, supporting generalization verification of multiple drama genres.

[0475] 2. Evaluation criteria: MIREX Music Transcription Metrics*

[0476] -Defined by: International Society for Music Information Retrieval (ISMIR)

[0477] -Indicators: note F1 value, pitch accuracy, rhythm error rate.

[0478] -This application achieves an F1 value of 92.4% on the COAC dataset, surpassing the baseline model by 15.6%.

[0479] Example Description

[0480] Example 1: Converting the Singing Style of the Peking Opera "The Drunken Concubine" into Simplified Notation

[0481] -Input data: mono audio (44.1kHz, SNR ≥ 20dB), duration 3 minutes and 28 seconds.

[0482] -Processing process:

[0483] 1. Time-frequency dual-stream feature extraction: 1D-CNN (convolution kernel 64×1) and STFT-CNN (window length 1024 points) are run in parallel.

[0484] 2. Adversarial training generation: The discriminator loads the Peking Opera ban rule library (including 8 banyan structures of Xipi and Erhuang).

[0485] 3. Multimodal verification: When the [Lying Fish] posture is detected, the corresponding musical score dragging time is automatically extended by 200ms.

[0486] -Output results: 98.7% completeness of the simplified notation and 86.2% accuracy of restoration of ornaments.

[0487] Example 2: Restoration of a Historical Recording (Excerpt from Mei Lanfang's 1947 Farewell My Concubine)

[0488] -Input data: Mono recording of old record (SNR≤15dB, including scratch noise).

[0489] - Enhanced strategy:

[0490] -Data preprocessing: WaveGlow vocoder removes impulse noise and preserves the original timbre.

[0491] - Transfer learning: Based on the Kunqu Opera pre-trained model, the Adapter module was fine-tuned using 5 hours of Mei style singing data.

[0492] -Output results: The score restoration completeness is 78%, which is 26% higher than the traditional spectrum analysis method.

[0493] The technical comparison and advantage quantification are shown in Table 5.

[0494] Table 5

[0495]

[0496]

[0497] In the tenth specific embodiment, some comparative documents of the prior art are disclosed.

[0498] Application number 202310557794.0, invention name: A method for automatic notation of piano polyphonic music based on multi-task learning of local attention;

[0499] Application number 202110308570.7, invention name: Buddhist music notation method, device, equipment and medium based on neural network;

[0500] Application number 202311858130.4, invention name: An automatic notation method and device based on sound source separation.

[0501] Figure 3 A structural block diagram of an electronic device according to one or more embodiments of the present invention provides a vehicle power system control method.

[0502] like Figure 3 As shown, the present application provides an electronic device, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0503] A computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method for notating audio of opera singing.

[0504] The present application also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the method for notating audio of opera singing.

[0505] This application also provides a music score generation platform, including:

[0506] Electronic equipment for implementing the steps of the method for notating audio of opera singing;

[0507] A processor that runs a program, and when the program runs, executes the steps of the method for notating audio of opera singing based on data output by the electronic device;

[0508] The storage medium is used to store a program, which, when running, executes the steps of the audio notation method for opera singing on data output from an electronic device.

[0509] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0510] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for notating audio of opera singing, characterized in that: The method for notating opera singing audio includes: a step of multimodal feature extraction, a step of adversarial training music score, a step of layered transfer learning adaptation, a step of streaming processing and multimodal verification, and a step of symbol-parameter decoupling output; The steps of multimodal feature extraction include attention weighted fusion and dynamic adjustment of the contribution of time-frequency features; The steps of adversarial training of music scores include verifying the symbol sequence in the score through the opera music knowledge graph; The steps of layered transfer learning adaptation include adaptive training for different drama genres; The steps of stream processing and multimodal verification include multimodal alignment and dynamic adjustment of model output and rule template weights based on voting results; The step of symbol-parameter decoupling and outputting includes symbol-parameter decoupling and outputting music score data in a target format.

2. The method for notating audio of opera singing according to claim 1, wherein: The steps of multimodal feature extraction include parallel processing of time domain branch and frequency domain branch; Extract transient feature information based on time domain branches; The steps of multimodal feature extraction also include: extraction of motion features and extraction of audio features; Transient features include,transient features of action features; Extract steady-state characteristic information based on frequency domain branches; Steady-state features include, steady-state features of audio features; The steps of multimodal feature extraction also include,feature fusion; Feature fusion includes using an attention-weighted fusion layer to dynamically adjust the contribution of time-frequency features.

3. The method for notating the audio of opera singing according to claim 2, wherein: The steps of adversarial training music score include: Input feature sequence after feature fusion; Output the music score and parameter matrix according to the feature sequence; Among them, the continuity of symbol sequences in musical scores is analyzed; According to the analysis, the continuity of the symbol sequence in the music score is verified through the opera music knowledge graph.

4. The method for notating the audio of opera singing according to claim 3, wherein: The steps of layered transfer learning adaptation include: sharing coding layers; The shared encoding layer is a Transformer network, pre-trained on a mixed dataset across genres; Also included is a drama adapter; The genre adapter uses a low-rank matrix fine-tuning module and inserts the FFN of each Transformer layer; It also includes training strategies: freezing the shared encoder and updating the genre adapter parameters; Load the drama adapter in parallel, switch the drama genre by the drama ID and adapt it.

5. The method for notating audio of opera singing according to claim 4, wherein: The steps of streaming processing and multimodal verification include: a causal convolutional layer; Causal convolution layer, which constrains the convolution kernel to only access the current input and historical input data; Also includes dynamic buffer window; The dynamic buffer window expands dynamically with the beat intensity; It also includes multimodal alignment: matching keyframes of motion features with keypoints of audio features through dynamic time warping; It also includes the implementation of a correction mechanism: when a conflict between the score and the action is detected, a confidence-weighted vote is used; Dynamically adjust the weights of model output and rule templates based on voting results.

6. The method for notating audio of opera singing according to claim 5, characterized in that: The step of symbol-parameter decoupling output includes: symbol-parameter decoupling output layer; The symbol-parameter decoupling output layer includes discrete symbol channels, continuous parameter channels, and multi-format converters; Discrete symbol channel, used to output symbol sequences of pitch and ornament type, and achieve differentiable sampling through Gumbel-Softmax; Continuous parameter channel, used for predicting parameters, including vibrato depth and portamento speed; The continuous parameter channel also includes,using Tanh activation constraint range; Multi-format converter, based on a rule engine, converts the internal representation into the target format.

7. An audio notation system for opera singing, characterized in that: The opera singing audio notation system comprises: Modules for multimodal feature extraction, adversarial music score training, layered transfer learning adaptation, streaming processing and multimodal verification, and symbol-parameter decoupling output. A multimodal feature extraction module for attention-weighted fusion and dynamic adjustment of the contribution of time-frequency features; A module for adversarial training of musical scores, used to verify the symbol sequences in musical scores using the opera music knowledge graph; A module for layered transfer learning adaptation, used for adaptive training of different drama genres; The streaming processing and multimodal verification module is used for multimodal alignment and dynamically adjusts the weights of model output and rule templates based on voting results. The module for symbol-parameter decoupling output is used for symbol-parameter decoupling and outputting music score data in a target format.

8. An electronic device, characterized in that: include: A processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; A computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the audio notation method for opera singing as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that It stores a computer program that can be executed by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the method for notating audio of opera singing as described in any one of claims 1 to 6.

10. A music score generation platform, characterized in that: include: An electronic device for implementing the steps of the method for notating audio notation of opera singing according to any one of claims 1 to 6; a processor that runs a program, and when the program runs, the data outputted from the electronic device executes the steps of the method for notating audio of opera singing according to any one of claims 1 to 6; A storage medium for storing a program, wherein when the program is run, the program executes the steps of the method for notating audio of opera singing as described in any one of claims 1 to 6 for data output from an electronic device.

Images