Method and system for audio notation of opera singing, electronic device and storage medium

Abstract

The application discloses a drama aria audio score recording method and system, electronic equipment and a storage medium, and relates to the field of voiceprint splicing. The method comprises the following steps: a multi-modal feature extraction step, an adversarial training score step, a hierarchical transfer learning adaptation step, a streaming processing and multi-modal verification step, and a symbol-parameter decoupling output step. The multi-modal feature extraction step comprises attention weighted fusion and dynamic adjustment of time-frequency feature contribution degree. The adversarial training score step comprises checking a symbol sequence in a score through a drama music knowledge graph. The hierarchical transfer learning adaptation step comprises adaptation training of different drama types. The streaming processing and multi-modal verification step comprises multi-modal alignment, and dynamic adjustment of the weight of model output and a rule template according to a voting result. The symbol-parameter decoupling output step comprises symbol-parameter decoupling, and output of score data in a target format.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of voiceprint splicing, in particular to a method and system for audio score notation of opera singing, an electronic device, a storage medium and a music score generation platform. BACKGROUND

[0002] Traditional operas have special performance characteristics. In addition to voiceprint recognition, they involve body coordination, nonlinear voiceprint expression such as glissando and vibrato, etc. Therefore, current music score generation mainly relies on manual listening and recording, which is inefficient. In addition, due to the experience and non-standard expression of inheritors, it is difficult to efficiently and large-scale carry out music score generation work.

[0003] The current technical solutions closest to "audio-generated music score" mainly fall into two categories:

[0004] 1. Traditional signal processing + machine learning hybrid solution:

[0005] (1) Extract features based on audio spectrum analysis (such as Mel spectrum, MPCC);

[0006] (2) Use Hidden Markov Model (HMM) or Random Forest for pitch / rhythm recognition;

[0007] (3) Post-processing to generate MIDI or XML format music score.

[0008] 2. End-to-end deep learning solution:

[0009] (1) Use CNN+BiLSTM network to process audio time-frequency graph;

[0010] (2) Achieve time alignment through attention mechanism;

[0011] (3) Output note sequence (such as piano roll representation);

[0012] (4) Typical representative: Google Magenta project's Onset and Frames model;

[0013] Using existing voiceprint recognition technology to solve the problem of traditional opera music score generation has limitations: 1. Pitch recognition error problem: glissando and vibrato in operas cause discretization error; 2. Rhythm quantization problem: it is difficult to match the standard beat system for scattered plate singing; 3. Special technique missing problem: unable to identify opera-specific "tossing" and "coughing" techniques; 4. Multi-voice part processing problem: traditional solutions are difficult to separate vocals and accompaniment parts.

[0014] Therefore, a voice recognition solution for opera singing audio score notation is needed, which is suitable for the performance characteristics of traditional operas and efficiently and accurately generates music scores for traditional operas. SUMMARY

[0015] The present application aims to provide a traditional opera singing audio score method, a traditional opera singing audio score system, an electronic device, a storage medium and a music score generation platform, at least solve the problem of automatic voiceprint recognition music score generation adapting to the performance characteristics of traditional operas, and solve one of the technical problems of efficiently and accurately generating music scores for traditional operas.

[0016] The present application provides the following solutions:

[0017] According to one aspect of the present application, a traditional opera singing audio score method is provided, which comprises the steps of multi-modal feature extraction, adversarial training of music scores, hierarchical transfer learning adaptation, streaming processing and multi-modal verification, and symbol-parameter decoupling output.

[0018] The multi-modal feature extraction step comprises attention weighted fusion and dynamic adjustment of time-frequency feature contribution degree.

[0019] The adversarial training of music scores step comprises verifying the symbol sequence in the music score through a traditional opera music knowledge graph.

[0020] The hierarchical transfer learning adaptation step comprises adaptation training of different types of operas.

[0021] The streaming processing and multi-modal verification step comprises multi-modal alignment and dynamic adjustment of the weights of model output and rule templates according to the voting results.

[0022] The symbol-parameter decoupling output step comprises symbol-parameter decoupling and output of music score data in a target format.

[0023] Further, the multi-modal feature extraction step comprises parallel processing of time domain branches and frequency domain branches.

[0024] The time domain branch extracts transient feature information.

[0025] The multi-modal feature extraction step further comprises extraction of action features and extraction of audio features.

[0026] The transient features include transient features of action features.

[0027] The frequency domain branch extracts steady-state feature information.

[0028] The steady-state features include steady-state features of audio features.

[0029] The multi-modal feature extraction step further comprises feature fusion.

[0030] The feature fusion comprises an attention weighted fusion layer and dynamic adjustment of time-frequency feature contribution degree.

[0031] Further, the step of adversarial training music score includes:

[0032] The feature sequence after input feature fusion;

[0033] Output music score and parameter matrix according to feature sequence;

[0034] Among them, analyze the continuity of the symbol sequence in the music score;

[0035] According to the analysis of the continuity of the symbol sequence in the music score, it is verified through the opera music knowledge graph.

[0036] Further, the step of hierarchical transfer learning adaptation includes sharing the encoding layer;

[0037] The shared encoding layer is a Transformer network, which is pre-trained on a cross-genre mixed dataset;

[0038] Also includes a genre adapter;

[0039] The genre adapter uses a low-rank matrix fine-tuning module (LoRA) inserted into the FFN of each layer of the Transformer;

[0040] Also includes training strategy: freeze the shared encoder, update the genre adapter parameters;

[0041] Parallel load genre adapter, switch genres by genre ID and perform adaptation.

[0042] Further, the step of streaming processing and multi-modal verification includes a causal convolution layer;

[0043] The causal convolution layer restricts the convolution kernel to only access the current input and historical input data;

[0044] Also includes a dynamic buffer window;

[0045] The dynamic buffer window dynamically expands with the beat intensity;

[0046] Also includes multi-modal alignment: match the key frames of action features and audio feature key points through dynamic time warping;

[0047] Also includes an implementation correction mechanism: when a conflict between music score and action is detected, use confidence weighted voting;

[0048] According to the voting result, dynamically adjust the weight of the model output and the rule template.

[0049] Further, the step of symbol-parameter decoupling output includes a symbol-parameter decoupling output layer;

[0050] The symbol-parameter decoupling output layer comprises a discrete symbol channel, a continuous parameter channel, and a multi-format converter.

[0051] The discrete symbol channel is used for outputting a symbol sequence of pitch and ornament type and realizing differentiable sampling through Gumbel-Softmax.

[0052] The continuous parameter channel is used for predicting parameters including tremolo depth and glissando speed.

[0053] The continuous parameter channel further comprises a Tanh activation for constraining a range.

[0054] The multi-format converter converts an internal representation into a target format based on a rule engine.

[0055] According to two aspects of the present application, an opera aria audio score system is provided, which comprises:

[0056] a multi-modal feature extraction module, an adversarial training score module, a hierarchical transfer learning adaptation module, a streaming processing and multi-modal verification module, and a symbol-parameter decoupling output module.

[0057] The multi-modal feature extraction module is used for attention weighted fusion and dynamic adjustment of time-frequency feature contribution.

[0058] The adversarial training score module is used for verifying a symbol sequence in a score through an opera music knowledge graph.

[0059] The hierarchical transfer learning adaptation module is used for adaptation training of different drama types.

[0060] The streaming processing and multi-modal verification module is used for multi-modal alignment and dynamic adjustment of weights of model output and rule templates according to a voting result.

[0061] The symbol-parameter decoupling output module is used for symbol-parameter decoupling and output of score data in a target format.

[0062] According to three aspects of the present application, an electronic device is provided, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus.

[0063] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes steps of the opera aria audio score method.

[0064] According to four aspects of the present application, a computer readable storage medium is provided, characterized in that it stores a computer program executable by an electronic device, which, when executed on the electronic device, causes the electronic device to perform the steps of the opera aria audio notation method.

[0065] According to five aspects of the present application, a music score generation platform is provided, characterized in that it comprises:

[0066] An electronic device for implementing the steps of the opera aria audio notation method.

[0067] A processor for running a program, which, when executed, performs the steps of the opera aria audio notation method on data output from the electronic device.

[0068] A storage medium for storing a program, which, when executed, performs the steps of the opera aria audio notation method on data output from the electronic device.

[0069] Through the above-mentioned scheme, the following beneficial technical effects are obtained:

[0070] The present application breaks through the performance bottleneck of traditional linear analysis by extracting heterogeneous multi-modal features and processing time domain branches and frequency domain branches in parallel.

[0071] The present application ensures style compliance by incorporating opera theoretical rules into the generative adversarial process in the adversarial training music score.

[0072] The present application realizes rapid adaptation of multiple genres with extremely low data sample requirements through hierarchical transfer learning adaptation.

[0073] The present application introduces semanticization of body movements and the like into music score generation through streaming processing and multi-modal verification, forming a new closed loop.

[0074] The present application decouples the score representation: separates symbols and parameters to be compatible with the aesthetic expression of traditional notation.

[0075] The present application, through the method of voiceprint recognition, not only identifies music score information, but also identifies movement characteristics, enhances music score information recording and splicing capabilities, increases music score information reliability, and improves the accuracy and reliability of music score recording. BRIEF DESCRIPTION OF DRAWINGS

[0076] Figure 1 is a flowchart of the opera aria audio notation method provided by one or more embodiments of the present application.

[0077] Figure 2 is a structural diagram of the opera aria audio notation system provided by one or more embodiments of the present application.

[0078] Figure 3An electronic device structure block diagram of a Peking opera aria audio score recording method is provided in one or more embodiments of the present application. DETAILED DESCRIPTION

[0079] The technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0080] Figure 1 A flowchart of a Peking opera aria audio score recording method is provided in one or more embodiments of the present application.

[0081] As shown in the Peking opera aria audio score recording method, the method comprises the following steps: Figure 1 a multi-modal feature extraction step, an adversarial training score step, a hierarchical transfer learning adaptation step, a streaming processing and multi-modal verification step, and a symbol-parameter decoupling output step.

[0082] The multi-modal feature extraction step comprises attention weighted fusion and dynamic adjustment of time-frequency feature contribution degree.

[0083] The adversarial training score step comprises verification of symbol sequences in the score through a Peking opera music knowledge graph.

[0084] The hierarchical transfer learning adaptation step comprises adaptation training of different types of operas.

[0085] The streaming processing and multi-modal verification step comprises multi-modal alignment and dynamic adjustment of the weights of model output and rule templates according to the voting results.

[0086] The symbol-parameter decoupling output step comprises symbol-parameter decoupling and output of score data in a target format.

[0087] In this embodiment, the multi-modal feature extraction step comprises parallel processing of a time domain branch and a frequency domain branch.

[0088] The time domain branch is used to extract transient feature information.

[0089] The multi-modal feature extraction step further comprises extraction of action features and extraction of audio features.

[0090] The transient features comprise transient features of the action features.

[0091] The frequency domain branch is used to extract steady-state feature information.

[0092] The steady-state features comprise steady-state features of the audio features.

[0093] The step of multi-modal feature extraction further comprises feature fusion.

[0094] The feature fusion comprises adopting an attention weighted fusion layer to dynamically adjust the contribution degree of the time-frequency features.

[0095] In the embodiment, the attention weighted fusion layer comprises F fusion = a·F time + (1-a)·F freq , a = σ(W T [F time ; F freq ]);

[0096] F fusion is feature data after fusion of the time domain features and the frequency domain features;

[0097] F time is the time domain feature, a·F time is the weighting of the time domain feature, and a is a weight coefficient;

[0098] F freq is the frequency domain feature, (1-a)·F freq is the weighting of the frequency domain feature, and (1-a) is a weight coefficient;

[0099] wherein σ is a Sigmoid function, W T is a learnable parameter matrix.

[0100] In the embodiment, the time domain branch and the frequency domain branch processed in parallel comprise:

[0101] The time domain branch and the frequency domain branch are processed in parallel corresponding to the input opera audio;

[0102] The time domain branch adopts a 1D-CNN network to extract transient features of the syllable starting point and the air outlet position;

[0103] The frequency domain branch adopts an STFT-CNN network to extract steady-state features of the formant distribution and the harmonic energy.

[0104] In the embodiment, the step of adversarial training of the score comprises:

[0105] The input feature sequence after feature fusion;

[0106] The score and the parameter matrix are output according to the feature sequence;

[0107] wherein the continuity of the symbol sequence in the score is analyzed;

[0108] The opera music knowledge graph is verified according to the continuity of the symbol sequence in the score.

[0109] In the embodiment, on the basis of voiceprint recognition, in addition to the recognition of the music score, the recognition of the action characteristics such as the action characteristics of the beginning of the pronunciation is added. For example, when detecting the body action of

lying fish

[0110] In the embodiment, based on the Transformer-Decoder structure, the feature sequence after feature fusion is input into the generator, and the generator outputs the music score symbol and the parameter matrix.

[0111] In the embodiment, the discriminator includes a data-driven branch and a knowledge rule branch.

[0112] The driving branch uses Bi-LSTM to analyze the continuity of the note sequence in the music score.

[0113] The knowledge rule branch embeds the opera music knowledge graph.

[0114] The discriminator performs verification, including the verification of the plate eye structure compliance and the rationality of the ornament.

[0115] In the embodiment, the step of adversarial training of the music score further includes:

[0116] Loss function

[0117] In the embodiment, L CE (G) represents the cross-entropy loss related to the generator G;

[0118] 0.1E[D(G(x))] represents the expectation E of the discriminator D applied to the generated data G(x); the coefficient 0.1 represents the weight state of the item in the overall loss function;

[0119] represents the weighted sum of the regularization terms Ri; the coefficient 0.3 represents the weight state of the regularization in the loss function; R i is a rule violation penalty term, wherein the rule violation penalty term corresponds to the verification of the plate eye structure compliance and the rationality of the ornament.

[0120] According to the loss function, the training of the generator is guided.

[0121] In the embodiment, the step of hierarchical transfer learning adaptation includes: sharing the encoding layer.

[0122] The shared encoding layer is a Transformer network, which is pre-trained on a cross-genre mixed data set.

[0123] It also includes a genre adapter.

[0124] The genre adapter adopts a low-rank matrix fine-tuning module (LoRA) to plug into the FFN of each layer of the Transformer;

[0125] It also includes a training strategy: freeze the shared encoder and update the genre adapter parameters;

[0126] The genre adapter is loaded in parallel, and the genre is switched and adapted through the genre ID.

[0127] In this embodiment, the steps of stream processing and multi-modal verification include a causal convolution layer;

[0128] The causal convolution layer restricts the convolution kernel to only access the current input and historical input data;

[0129] It also includes a dynamic buffer window;

[0130] The dynamic buffer window dynamically expands with the strength of the beat;

[0131] It also includes multi-modal alignment: matching the key frames of action features and the key points of audio features through dynamic time warping;

[0132] It also includes an implementation correction mechanism: when a conflict between the score and the action is detected, a confidence weighted voting is used;

[0133] According to the voting result, the weights of the model output and the rule template are dynamically adjusted.

[0134] In this embodiment, the confidence weighted voting includes: Final_Note=β·Model_Output+(1-β)·Rule_Template, β=f(confidence)_score;

[0135] In this embodiment, Final_Note is the output result of the confidence weighted voting;

[0136] β·Model_Output is the weighted part of the model output Model_Output, where β is the weight coefficient;

[0137] (1-β)·Rule_Template is the weighted part of the rule template Rule_Template, where (1-β) is the weight coefficient;

[0138] β is the result of a function f, which is dynamically calculated by the model confidence score depending on confidence;

[0139] score is an evaluation index.

[0140] In this embodiment, the step of symbol-parameter decoupling output includes: a symbol-parameter decoupling output layer;

[0141] The symbol-parameter decoupled output layer includes a discrete symbol channel, a continuous parameter channel, and a multi-format converter;

[0142] Discrete symbol channel is used to output symbol sequences of pitch and ornament types, and differentiable sampling is achieved through Gumbel-Softmax;

[0143] Continuous parameter channels are used to predict parameters, including vibrato depth and glissando speed.

[0144] The continuous parameter channel also includes using Tanh activation constraint range;

[0145] A multi-format converter, based on a rules engine, transforms internal representations into target formats.

[0146] In this embodiment, the target format includes the target formats corresponding to gongche notation and numbered musical notation;

[0147] The mapping rules of the rule engine include, in 2 / 4 time signature, marking each measure's strong beat with the ">" symbol;

[0148] The mapping rules also include marking the 8 / 4 beat gongche notation with the symbol "△".

[0149] Figure 2 This is a structural diagram of an audio notation system for opera singing provided in one or more embodiments of the present invention.

[0150] like Figure 2 The audio notation system for traditional Chinese opera singing shown includes:

[0151] The module includes modules for multimodal feature extraction, adversarial training score generation, hierarchical transfer learning adaptation, streaming processing and multimodal validation, and symbol-parameter decoupling output.

[0152] A multimodal feature extraction module is used for attention-weighted fusion to dynamically adjust the contribution of time-frequency features;

[0153] The adversarial training music score module is used to verify the symbol sequences in the music score through a knowledge graph of traditional Chinese opera music.

[0154] A hierarchical transfer learning adaptation module is used for adaptation training of different opera genres;

[0155] The module for streaming processing and multimodal validation is used for multimodal alignment, dynamically adjusting the weights of model output and rule templates based on voting results;

[0156] A module for symbol-parameter decoupling output, used for symbol-parameter decoupling, outputs music score data in a target format.

[0157] It is worth noting that, although the system only discloses the modules of multi-modal feature extraction, adversarial training music score, hierarchical transfer learning adaptation, streaming processing and multi-modal verification, and symbol-parameter decoupling output, it does not mean that the device is limited to the above basic functional modules. On the contrary, the meaning expressed by the present application is that on the basis of the above basic functional modules, a person skilled in the art can add one or more functional modules to form an infinite number of embodiments or technical solutions in combination with existing technology. That is to say, the system is open rather than closed, and the protection scope of the present application claimed in the present embodiment cannot be limited to the above disclosed basic functional modules.

[0158] Through the above scheme, the following beneficial technical effects are obtained:

[0159] The present application breaks through the performance bottleneck of traditional linear analysis by extracting heterogeneous multi-modal features and processing time domain branches and frequency domain branches in parallel.

[0160] The present application ensures style compliance by integrating opera theory rules into the generative adversarial process in the adversarial training music score.

[0161] The present application realizes rapid adaptation of multiple types of drama with extremely low data sample requirements through hierarchical transfer learning adaptation.

[0162] The present application introduces semanticization of body movements and other elements into music score generation through streaming processing and multi-modal verification, forming a new closed loop.

[0163] The present application decouples the symbol and parameter to be compatible with the aesthetic expression of traditional notation through decoupled music score representation.

[0164] In addition to identifying music score information, the present application also increases the recognition of action features through the method of voiceprint recognition, enhances the ability of music score information record splicing, increases the reliability of music score information, and improves the accuracy and reliability of music score recording.

[0165] In the first embodiment, the opera aria audio notation strategy is disclosed, which is based on voiceprint recognition technology, including: 1. Opera feature enhancement; (1) Establishing an opera-specific timbre library; (2) Designing an opera tremolo detection module. 2. Hybrid model architecture; A [raw audio]-->B {opera type classification}; B-->|Yuju|C [special feature extractor]; B-->|Kunqu|D [special feature extractor]; C / D-->E [adaptive encoder]; E-->F [multi-task output layer]. 3. Set up evaluation system; (1) Introduce subjective evaluation indicators of opera experts; (2) Develop opera rhyme compliance detection algorithm.

[0166] In this hybrid model architecture, letters represent different modules or steps in the model processing flow. The specific meanings are as follows:

[0167] 1. A [raw audio] input layer: represents the raw opera audio data (such as.wav files) that has not been processed.

[0168] 2. B {opera type classification} classifier: determines the opera genre (such as Yuju, Kunqu, etc.) of the input audio through a classification model.

[0169] The output is a branch condition that determines which special feature extractor to use next.

[0170] 3. C [special feature extractor - Yuju]

[0171] Genre-specific module: a feature extractor designed for Yuju (may include Yuju-specific timbre, rhythm, and other feature extraction logic).

[0172] 4. D [special feature extractor - Kunqu]

[0173] Genre-specific module: a feature extractor designed for Kunqu (may focus on Kunqu's voice, pronunciation, and other features).

[0174] 5. E [adaptive encoder]

[0175] Shared encoding layer: uniformly encodes the features extracted from different genres into high-level representations, which may include adaptive normalization or cross-genre feature fusion.

[0176] 6. F [multi-task output layer]

[0177] Output layer: multi-task learning head designed according to task requirements (such as timbre synthesis, tremolo generation, emotion classification, etc.).

[0178] In the second embodiment, the mapping of the core defects of the prior art as shown in Table 1 and the technical solution of the present embodiment is disclosed.

[0179] Table 1

[0180]

[0181] In this embodiment, the solution of key technical defects is described in detail:

[0182] 1. Glissando / vibrato processing defect improvement

[0183] The technical defect of the prior art is that: when the traditional pitch extraction algorithm (such as YIN or pYIN) processes the continuously changing opera vibrato, the following problems exist: (1) frame-to-frame pitch jump leads to "staircase" MIDI output; (2) glissando start / end point detection error > 50ms.

[0184] The solution of this embodiment is: A [original audio]-->B {vibrato waveform analysis module}; B-->C [high-frequency tremolo detector]; B-->D [glissando slope calculator]; C-->E [time-frequency joint CNN]; D-->E; E-->F [continuous pitch trajectory generation]; F-->G [MIDI encoding with bending marks].

[0185] In this embodiment, the following is implemented: (1) high-frequency tremolo detector: design a 20-40Hz band-pass filter set to extract the throat tremor features unique to opera; (2) glissando slope calculator: calculate the glissando speed (unit: cents / second) through the short-time autocorrelation function change rate; (3) time-frequency joint CNN: input the original waveform (time domain) and CQT spectrum (frequency domain) simultaneously, and use a dual-stream network to learn.

[0186] In this embodiment, the experimental data is that on the Beijing opera "The Drunken Beauty" test set, the glissando detection F1-score is improved from 0.62 of the traditional scheme to 0.89.

[0187] 2. Improvements in quantizing the rhythm of Sanban

[0188] The technical defect of the prior art is that:

[0189] The existing beat tracking algorithm (such as Madmom) relies on a fixed beat grid, which leads to: (1) the free rhythm of Sanban is incorrectly quantized as 1 / 8 or 1 / 16 notes; (2) the rhythm density distribution error is up to ±35%.

[0190] Accordingly, the code design in this embodiment is as follows (python):

[0191]

[0192] In this embodiment technical features: (1) - LSTM-CRF beat predictor: learn the flexible rhythm patterns of opera arias (such as the difference between "slow" and "fast three eyes" in Yu Opera); (2) non-uniform framing strategy: dynamically adjust the frame length (10-200 ms variable) according to the predicted instantaneous BPM (3) rhythm symbolization rules: develop MusicXML extension tags dedicated to opera (such as <free-rhythm>).

[0193] Effect verification in this embodiment: Kunqu "Peony Pavilion" test shows that the rhythm coincidence of the scattering board paragraph has been improved from 58% to 82%.

[0194] 3. Special technique identification missing improvement

[0195] Technical defects of prior art: (1) general music transcription model lacks opera acoustics feature perception ability; (2) posse sound (rapid throat vibration) is misjudged as ordinary vibrato; (3) cough sound (breath explosion sound) is classified as noise.

[0196] Accordingly, in this embodiment, the technical solutions are:

[0197] Opera acoustics fingerprint library construction process: original opera recording (200 hours) → expert annotation (marking special technique position) → feature distillation (extracting technique MFCC-Pattern) → generating adversarial enhancement (WaveGAN generating variant samples) → storing in feature retrieval library (Faiss index).

[0198] Improved recognition model: add technique detection branch (formula) in the standard Onsets and Frames model:

[0199] Loss=α\cdot Loss_{note}+β\cdot Loss_{ornament}+γ\cdot Loss_{articulation};Adopt multi-scale CQT spectrum input (5 different resolutions) to capture technique details.

[0200] Application case in this embodiment: identify Hebei Bangzi "ramming sound" technique, recall rate reaches 91.2%.

[0201] 4. The performance improvement on the same test set (containing 50 pieces of Peking opera, Kunqu and Henan opera) in this embodiment is shown in Table 2:

[0202] Table 2

[0203] Indicator Conventional scheme Scheme of the present application Lifting range Pitch continuity error (cents) 82.6 29.4 64.4％ Rhythm conformity (F1) 0.71 0.89 25.3％ Special technique recall rate 41.7％ 86.5％ 107.4％ Vocal separation purity 78.2 dB 93.6 dB 19.7％

[0204] 5. Summary of the scheme in this embodiment: breakthrough the limitations of general music transcription technology through domain adaptability improvement:

[0205] 1. Opera acoustics prior knowledge injection: build special feature library and special symbol system;

[0206] 2. Hybrid architecture design: time-frequency joint network + dynamic beat tracking + multi-task learning;

[0207] 3. Evaluation system innovation: introduce the opera rhythm similarity (XQSIM) evaluation index;

[0208] This embodiment has been applied in the actual opera digitization protection project, reducing the manual correction workload by 73% compared with traditional technology, with significant technological progress and industrial application value.

[0209] In the third embodiment, the technical goals are set as follows: (1) to realize end-to-end high-precision conversion of opera singing audio to music score (note F1 value ≥ 90%); (2) to support fast migration of more than 21 types of opera (≤ 5 hours of data adaptation); (3) to achieve 200ms level real-time processing capability; (4) to improve the completeness of music score restoration of low-quality historical recordings to more than 75%.

[0210] In this embodiment, the scheme framework is set as follows: adopt a modular deep learning framework: 1. Multi-modal feature extraction → 2. Adversarial training music score generation → 3. Hierarchical migration adaptation → 4. Streaming multi-modal verification → 5. Decoupled music score output.

[0211] In this embodiment, it includes a time-frequency dual-stream feature extraction network: (1) input; (2) audio waveform (44.1kHz sampling, PCM encoding); (3) optional body movement signal (50Hz frame rate, 6-dimensional vector encoding).

[0212] Parallel processing branches are adopted: (1) Time domain branch: 1D-CNN (64 layers, convolution kernel 64x1, step 8); extract transient features such as word starting point and air port position (time resolution ≤ 10ms). (2) Frequency domain branch: STFT-CNN (window length 1024 points, Hanning window, 8 layers of convolution); extract stable features such as formant distribution and harmonic energy (frequency resolution ≤ 5Hz).

[0213] Feature fusion is performed: dynamic attention weighted fusion layer (formula): F fusion = a·F time + (1-a)·F freq , a = σ(W T [F time ; F freq ]);

[0214] (1) F fusion : is the fused feature representation, which combines time domain features and frequency domain features.

[0215] (2) F time : represents the weighted part of the time domain feature a·F time , where a is the weight coefficient.

[0216] (3) (1-a)·F freq : represents the frequency domain feature F freq where (1-a) is the weight coefficient.

[0217] (4) a = σ(W T [F time ; F freq ]) : a is obtained by concatenating the time-domain feature F time and the frequency-domain feature F freq , then linearly transforming by the transpose of a weight vector W T , and finally passing through an activation function σ (usually sigmoid function). This activation function ensures that the value of a is between 0 and 1, thus controlling the relative contribution of time-domain and frequency-domain features.

[0218] The above formula in this embodiment describes an adaptive fusion method that dynamically adjusts the weights of time-domain and frequency-domain features according to their relative importance to generate a more effective fused feature representation.

[0219] where σ is the Sigmoid function and W is the learnable parameter matrix.

[0220] In this embodiment, the knowledge-guided adversarial training framework includes: (1) Generator (G). Based on the Transformer-Decoder structure (12 layers, 8 heads of attention), the input fused feature sequence is outputted as the preliminary music score and parameter matrix.

[0221] (2) Discriminator (D). Data-driven branch: 3-layer Bi-LSTM analyzes the continuity of the note sequence; knowledge rule branch: embeds the opera music knowledge graph (including the rule library of 21 types of opera board and the ornament syntax tree); performs the following checks: board eye structure compliance (such as the position of strong beat in the water board of Beijing Opera Xipi); the rationality of the context of ornaments (such as the discontinuous appearance of more than 3 times of ornaments).

[0222] (3) Loss function:

[0223] L CE (G) : This term represents the cross-entropy loss related to the generator G. Cross-entropy loss is commonly used in classification tasks to measure the difference between the predicted probability distribution and the true distribution.

[0224] 0.1E[D(G(x))] : This term represents the expectation E of the discriminator D applied to the generated data G(x). The coefficient 0.1 indicates that this term has a lighter weight in the overall loss function. This term is seen in Generative Adversarial Networks (GANs), where the goal of the generator is to generate data that the discriminator cannot distinguish from real data.

[0225] This term represents the regularization term R i The weighted sum is used to penalize violations of certain constraints. The coefficient 0.3 represents the weight of this regularization in the loss function. The specific nature of the violation and the content of the regularization term depend on the context of the model.

[0226] In summary, this loss function combines cross-entropy loss, a term related to the discriminator output, and a regularization term to guide the training of the generator in GANs or similar models.

[0227] R i Penalties for rule violations (e.g., incorrect placement of the forced clapping in Peking Opera Xipi Liushuiban).

[0228] In this embodiment, a hierarchical transfer learning adaptation method is included: (1) The shared encoding layer includes a 12-layer Transformer (768-dimensional hidden layer), pre-trained on a cross-genre hybrid dataset (COAC); (2) The genre adapter includes a structure: a low-rank matrix fine-tuning module (LoRA, rank r = 8), which is inserted into the FFN of each Transformer layer; training strategy: freeze the shared encoder and only update the adapter parameters (learning rate 5e-4); multi-genre support: load adapters in parallel (such as Peking Opera Adapter_A, Kunqu Opera Adapter_B), and switch by genre ID.

[0229] In this embodiment, the streaming engine design includes: (1) Causal convolutional layer: constraining the convolutional kernel to only access the current and historical inputs to ensure real-time performance; (2) Dynamic buffer window: basic window length: 2 seconds of audio (approximately 4-8 musical phrases), dynamically expanded with beat intensity (maximum 5 seconds); (3) Multimodal alignment: matching keyframes of body movements with audio feature points through dynamic time warping (DTW); (4) Real-time correction mechanism: when a semantic conflict between the musical score and the action is detected (e.g., the [hair flick] is accompanied by a prolonged vocalization without extension), a confidence-weighted voting method is used: Final_Note=β·Model_Output+(1-β)·

[0230] Rule_Template, β = f(confidence)_score; where Final_Note: is the final output or result, combining the model output and the rule template. β·Model_Output: represents the weighted part of the model output Model_Output, where β is the weight coefficient. (1-β)·Rule_Template: represents the weighted part of the rule template Rule_Template, where (1-β) is the weight coefficient. β =

[0231] f(confidence)_score: Here β is the result of a function f() that depends on confidence.

[0232] The confidence can be a degree of confidence of the model on its output, used to dynamically adjust the relative weights of the model output and the rule template. The score is a score or evaluation index.

[0233] The above formula in the embodiment describes an adaptive method of dynamically adjusting the weights of the model output and the pre-defined rule template according to the confidence of the model to generate the final output or result.

[0234] The β is dynamically calculated by the model confidence score.

[0235] In the embodiment, the symbol-parameter decoupling output layer includes: (1) a discrete symbol channel: outputting a symbol sequence such as pitch, ornament type, etc., and realizing differentiable sampling through Gumbel-Softmax; (2) a continuous parameter channel: predicting parameters such as tremolo depth (0-1), glissando speed (Hz / ms), etc., and adopting Tanh activation to constrain the range; and (3) a multi-format converter: converting the internal representation into a target format (such as staff notation, simplified notation) based on a rule engine, and key mapping rules include: Peking Opera Erhuang original plate→2 / 4 beat simplified notation, and marking "> " symbol for strong beat in each measure; Kunqu gift plate→8 / 4 beat staff notation, and marking "△" symbol for "Huaqiang".

[0236] The technical effect verification mode includes performance on the COAC data set (12 genres of 300 hours), as shown in Table 3:

[0237] Table 3

[0238]

[0239] In the fourth specific embodiment, the training process includes:

[0240] 1. Data preprocessing:

[0241] Audio normalization (-24 dB LUFS), segmented windowing (1024 points, 512 point overlap); wherein the segment action encoding: converting the video key frame into a 6-dimensional vector (hand coordinates, head posture, etc.).

[0242] 2. Pre-training stage:

[0243] Train the shared encoder on the COAC data set, and use the CTc loss function to optimize feature alignment.

[0244] 3. Adversarial training stage:

[0245] Fix the encoder parameters, and alternately train the generator and the discriminator, with an update ratio of G:D=3:1 per round.

[0246] 4. Migration adaptation stage:

[0247] Load target genre 5 hours data, only fine-tune corresponding adapter (learning rate 5e-4, 50 rounds).

[0248] In the fifth embodiment, the system deployment includes,

[0249] Edge device end:

[0250] Hardware: NVIDIA Jetson AGX Xavier, memory 32GB;

[0251] Deployed model: pruned dual-stream network (parameter quantity from 250M compressed to 80M).

[0252] Cloud service: provide API interface: input audio file, return JSON format score (including symbol and parameter matrix).

[0253] The above implementation scheme is summarized as a Peking Opera audio transposition method based on time-frequency dual-stream CNN and knowledge-guided adversarial training; a pluggable adapter and low-rank fine-tuning mechanism in hierarchical transfer learning; a multi-modal real-time verification algorithm of streaming processing engine.

[0254] In the sixth embodiment, an end-to-end deep learning framework is adopted, including the following core modules: 1. multi-modal feature extraction module; 2. adversarial training score generation module; 3. hierarchical transfer learning adaptation module; 4. streaming processing and multi-modal verification module; 5. symbol-parameter decoupling output module.

[0255] The system flow includes: A [original audio]-->B [multi-channel sound source separation]; B-->C [dynamic frame preprocessing]; C-->D [time-frequency joint feature extraction]; D-->E [Peking Opera AI model]; E-->F [pitch trajectory prediction]; E-->G [rhythm pattern analysis]; E-->H [technique feature matching]; F&G&H-->I [multi-modal score synthesis]; I-->J [Peking Opera symbol rendering engine]; J-->K [final score output].

[0256] Among them, the core modules and steps include: B; C; D; E; J; end.

[0257] In this embodiment, the key technical solutions are described.

[0258] 1. The time-frequency dual-stream feature extraction network includes:

[0259] - Input layer: receive audio waveform (44.1kHz sampling rate) and optional body motion capture signal (50Hz frame rate).

[0260] - Parallel processing branch:

[0261] - Time-domain branch: 1D-CNN (convolution kernel size 64x1, stride 8, ReLU activation), extracting transient features such as syllable onset, breath pauses, etc.

[0262] - Frequency-domain branch: STFT-CNN (window length 1024 points, Hanning window, 50% overlap), extracting steady-state features such as formant distribution, harmonic energy, etc. through 8 layers of convolution.

[0263] - Feature fusion: using attention weighted fusion layer to dynamically adjust the contribution of time-frequency features:

[0264] F fusion = a · F time + (1-a) · F freq , a = σ(W T [F time ; F freq ]); where σ is the Sigmoid function and W is a learnable parameter matrix.

[0265] 2. Knowledge-guided adversarial training framework:

[0266] - Generator (G): based on Transformer-Decoder structure (12 layers, 8 heads of attention), input fusion feature sequence, output preliminary music score symbols and parameter matrix.

[0267] - Discriminator (D):

[0268] - Data-driven branch: 3-layer Bi-LSTM analyzes the continuity of note sequences;

[0269] - Knowledge rule branch: embeds the opera music knowledge graph (including 21 types of board pattern rule base, ornamentation syntax tree), and performs the following checks:

[0270] - Board eye structure compliance (such as the strong beat position of Pihua board in Beijing Opera)

[0271] - Rationality of ornamentation context (such as the discontinuous appearance of more than 3 times)

[0272] - Loss function:

[0273] R i is the rule violation penalty term (such as the strong beat position error of Pihua board in Beijing Opera).

[0274] 3. Hierarchical transfer learning adaptation method:

[0275] - Shared encoding layer includes: 12-layer Transformer (hidden layer 768 dimensions), pre-trained on a mixed dataset across genres (COAC).

[0276] The genre adapter includes:

[0277] (1) Structure: Low Rank Matrix Adjustment Module (LoRA, rank r=8), inserted after the FFN of each layer of the Transformer;

[0278] (2) Training strategy: freeze the shared encoder, only update the adapter parameters (learning rate 5e-4);

[0279] (3) Multi-genre support: load adapters in parallel (such as Peking Opera Adapter_A, Kunqu Opera Adapter_B), switch through genre ID.

[0280] In the seventh embodiment, the prior art is compared with the present application.

[0281] I. Comparison of core architecture of technical solutions

[0282] Prior art architecture: audio input→[general preprocessing]→[standard Mel spectrum]→

[0283] [CNN+BiLSTM]→[MIDI encoder]→score output.

[0284] Core defects:

[0285] 1. Using general music features (Mel spectrum) ignores the acoustic characteristics of opera;

[0286] 2. Fixed framing strategy destroys the structure of the scattering plate rhythm;

[0287] 3. Single task output cannot capture special techniques;

[0288] Architecture in this embodiment:

[0289] A[opera audio]-->B{multimodal preprocessing};B-->C[dynamic framing];B-->D[source separation];

[0290] C-->E[time-frequency joint feature];D-->E;E-->F[opera-specific AI model];F-->G[pitch / rhythm / technique multi-task output];G-->H[opera symbol encoder];H-->I[score with opera rhyme marks];

[0291] II. Key technical differences and improvement implementation

[0292] 1. Comparison of feature extraction layers,

[0293]

[0294]

[0295] Among them,

[0296] - Opera Enhanced CQT: Design 128-bin log frequency axis, focusing on covering opera vocal frequency band (200Hz-2kHz);

[0297] - Dynamic Framing Algorithm: (python)

[0298] def adaptive_frame(audio, onset_detect):

[0299] frame_length = 1000 / (tempo_prediction*2) # unit: ms

[0300] return overlapping_frames(audio, frame_length, 0.3)

[0301] - Voiceprint Fingerprint Matching: Use Faiss approximate nearest neighbor search to match pre-stored technique features.

[0302] 2. Model structure comparison,

[0303] Existing model (Onsets and Frames):

[0304] mermaid

[0305] graph LR

[0306] A[Mel spectrum]-->B[CNN feature extraction];B-->C[BiLSTM time series modeling];C-->D[full connection output layer].

[0307] Model of this embodiment (opera multi-task network):

[0308] mermaid

[0309] graph LR

[0310] A[Joint time-frequency input]-->B{Dual-stream feature extraction};B-->C[Time-domain CNN];B-->D[Frequency-domain CNN];C-->E[Feature fusion gate];D-->E;E-->F[LSTM-CRF rhythm prediction];E-->G[Pitch trajectory regression];E-->H[Technique classifier];F&G&H-->I[Multi-task loss function].

[0311] Analysis of technical features in this embodiment:

[0312] -Dual-stream input structure: Simultaneously process time-domain waveform (preserve phase information) and frequency-domain CQT (fine frequency resolution);

[0313] -LSTM-CRF Beat Prediction: Introduce conditional random field to constrain the rhythm conversion logic;

[0314] Multi-task Loss Function: (math)

[0315] \(\mathcal{L}=0.5\mathcal{L}_{pitch}+0.3\mathcal{L}_{rhythm}+\)

[0316] \(0.2\mathcal{L}_{ornament}\).

[0317] 3. Comparison in the post-processing stage,

[0318] Problems of the existing technology:

[0319] - Mechanical MIDI quantization loses the charm of traditional Chinese opera;

[0320] - Unable to represent special singing notations;

[0321] Solution of this embodiment:

[0322] mermaid

[0323] graph TD

[0324] A[Original Output] --> B{Traditional Chinese Opera Symbolization Engine}; B --> C[Glissando Curvature Calculation]; B --> D[Free Rhythm Relaxation]; C --> E[Add MIDI Pitch Bend]; D --> E; E --> F[Generate MusicXML Extension Tags]; F --> G[Render Traditional Chinese Opera Special Score].

[0325] Among them, the key technologies are, - Glissando Quantization Algorithm: (python)

[0326]

[0327] - Traditional Chinese Opera XML Extension: Define custom tags such as `<xiqu:ornament type="擞音" / >`.

[0328] III. Verification of Objective Improvement Effects:

[0329] 1. Performance Comparison Experiment (Test Set: 300 Traditional Chinese Opera Audios)

[0330]

[0331]

[0332] 2. Application Effect Cases

[0333] - Transcription of Beijing Opera "Empty City Stratagem":

[0334] - Traditional solution: 87% of throat vibrations are lost, and the scattered notes are quantized as 4 / 4 beats by mistake;

[0335] - In this application: 13 "vibrato" marks are completely preserved, and a flexible rhythm skeleton is generated.

[0336] Four, technical process of the embodiment

[0337] A [original audio]-->B [multi-channel sound source separation]; B-->C [dynamic frame preprocessing]; C-->

[0338] D [time-frequency joint feature extraction]; D-->E [special AI model for opera]; E-->F [pitch trajectory prediction]; E-->G [rhythm pattern analysis]; E-->H [technique feature matching]; F&G&H-->I [multimodal score synthesis]; I-->J [opera symbol rendering engine]; J-->K [final score output].

[0339] Core part: B; C; D; E; J; end.

[0340] Five, summary of the technical features of the application

[0341] 1. Field adaptation

[0342] - First opera acoustic fingerprint library and special symbol system;

[0343] - Develop time-frequency joint modeling network structure.

[0344] 2. Algorithm architecture innovation

[0345] - Dynamic frame strategy and collaborative design of LSTM-CRF rhythm prediction;

[0346] - Multi-task learning framework realizes the joint optimization of pitch / rhythm / technique.

[0347] 3. Application value breakthrough

[0348] - The accuracy of opera singing transcription has broken through the 90% threshold for the first time;

[0349] - Artificial correction workload is reduced by 76%, accelerating the process of non-heritage digitization.

[0350] Technical effects

[0351] 1. Feature extraction dimension and precision improvement (technical breakthrough)

[0352] Traditional audio transposition techniques rely on Fourier transform (FFT) or short-time energy detection with fixed frequency thresholds, which fail to capture the nonlinear acoustic features unique to Chinese opera, such as glissando and vibrato (experimental data show only 62%). This solution uses a composite neural network architecture:

[0353] - Time-frequency dual-stream convolution module: Extracts time-domain transient features (e.g., word articulation onset) and frequency-domain formant features (e.g., vocal tract resonance) through parallel 1D-CNN and STFT-CNN;

[0354] - Attention-enhanced recurrent network: Models long-term dependencies using Bi-LSTM+Attention mechanism, achieving 94% continuous tracking accuracy for pitch gradual changes in vocal tracts (average duration per syllable: 1.2 seconds);

[0355] - Adversarial training mechanism: Introduces Wasserstein GAN architecture to judge the consistency of opera style in generated sheet music through a discriminator network, improving the identification accuracy of Beijing Opera Xipi Erhuang plate to 89%;

[0356] 2. Multi-genre adaptive capability (technical generalization)

[0357] Traditional methods require separate modeling for different genres (e.g., Kunqu Gongchi notation and Beijing Opera simplified notation require different rule libraries), this solution uses:

[0358] - Hierarchical transfer learning framework: Shared acoustic feature encoder (12-layer Transformer) at the bottom, and pluggable genre-specific adapter (Adapter) at the top, requiring only 5 hours of target genre data for transfer;

[0359] - Data augmentation strategy: Uses physical acoustic simulation (room impulse response) and style transfer (Mei and Chengpai singing styles mutual conversion), increasing the coverage of training data by 3.6 times;

[0360] - Symbolic representation learning: Decomposes sheet music elements into discrete symbols (pitch, duration, ornamentation) and continuous parameters (vibrato depth, glissando speed), supporting output in multiple formats such as MIDI, Gongchi notation, staff notation, and simplified notation.

[0361] 3. Real-time interaction and error correction mechanism (user experience innovation)

[0362] Compared to traditional offline processing mode (average delay > 30 seconds), the system:

[0363] - Streaming processing engine: Uses causal convolution and dynamic caching mechanisms to achieve 200ms-level real-time sheet music generation;

[0364] - Multi-modal verification module: Construct a three-dimensional mapping space of aria-melody-embodiment. When a conflict between melody and stylized action is detected (e.g., a

water sleeve

[0365] - Human-machine collaborative annotation: Develop a progressive feedback interface that allows inheritors to annotate key aria nodes (e.g.,

singing

[0366] 4. Digitalization efficiency of cultural heritage (application value)

[0367] According to the actual measurement verification by the National Opera Research Institute:

[0368] In the precious recording repair project of old artists, the system shortens the single song filing time from the traditional 40 man-hours to 2.3 hours;

[0369] The completeness of the melody restoration for single-channel historical recordings from the 1940s (SNR≤15dB) reaches 78%, far exceeding the traditional method of 52%;

[0370] Supporting parallel transfer learning for 21 local drama genres, the digital protection cost of endangered drama genres (such as Haicheng La Hua Opera) is reduced by 83%;

[0371] Technical effect verification:

[0372] On the cross-genre test set (including 12 genres of 300 hours of audio), the proposed scheme significantly outperforms the baseline model:

[0373] - Note onset detection F1 value: 92.4% vs. traditional HMM method 76.8%

[0374] - Ornament recognition rate: 85.7% vs. rule engine 63.2%

[0375] - Plate structure restoration degree: 89.3% vs. template matching method 71.5%

[0376] - Cross-genre generalization error: 14.2% vs. single-task learning model 37.6%

[0377] The core innovation of the present application lies in the deep integration of deep learning and opera music theory, which realizes the quantification of traditional acoustic feature engineering that cannot be captured by traditional acoustic feature engineering through a modular neural network architecture, providing an expandable technical paradigm for the digital inheritance of intangible cultural heritage.

[0378] In an eighth embodiment, an alternative solution is disclosed.

[0379] I. Feature extraction alternative

[0380] 1. Time-frequency analysis alternative

[0381] - Wavelet Transform instead of STFT;

[0382] Continuous Wavelet Transform (CWT) is used to construct the time-frequency domain feature extraction layer, which captures the non-stationary signal characteristics (such as the instantaneous frequency change of glissando) through Morlet wavelet basis function, replacing the traditional Short-Time Fourier Transform (STFT), solving the problem of insufficient frequency resolution caused by fixed window length.

[0383] - Physical model driven feature extraction;

[0384] Embedding acoustic physical prior knowledge (such as source-filter model), separating fundamental frequency and formant through Linear Predictive Coding (LPC), combining with neural network for joint optimization, enhancing the feature analysis of special singing methods of opera (such as "brain back sound").

[0385] 2. Neural network architecture replacement;

[0386] - Graph Neural Network (GNN) modeling note relationship;

[0387] The time-dependent relationship between notes is constructed as a graph structure, with nodes representing note attributes (pitch, time value) and edges representing connection logic such as ornamentation and glissando. Through GNN, the rules of music score generation are learned, replacing traditional RNN sequence modeling.

[0388] - Spiking Neural Network (SNN) processing time series signals;

[0389] Using the Spiking Temporal Dependence Plasticity (STDP) of SNN, the human auditory nerve's perception mechanism for opera rhythm is simulated, which is especially suitable for biologically inspired recognition of plate eye rhythm.

[0390] II. Model training and optimization replacement scheme

[0391] 1. Adversarial training replacement scheme;

[0392] - Reinforcement learning instead of GAN;

[0393] Constructing a Markov Decision Process (MDP) for music score generation, designing a reward function based on opera music theory rules (such as plate structure integrity), optimizing the generation strategy through PPO algorithm, replacing the discriminator mechanism in adversarial training.

[0394] - Knowledge distillation compresses the model;

[0395] Using teacher-student network architecture, compressing the composite neural network into a lightweight model (such as MobileNet variant), maintaining the ability to capture high-order features such as ornaments through Attention Transfer, and adapting to edge device deployment.

[0396] 2. Transfer learning alternatives

[0397] - Meta-learning framework enables fast adaptation

[0398] Based on Model-Agnostic Meta-Learning (MAML), learn a multi-drama shared meta-feature representation in the pre-training phase, and only need a small number of samples (such as 10 arias) to quickly adapt to new drama, replacing the fixed adapter design in hierarchical transfer.

[0399] - Cross-modal pre-training enhances generalization

[0400] Introduce opera video-audio-score triple data, build a cross-modal embedding space through contrastive learning, and improve the robustness of the model to missing information (such as historical recording noise).

[0401] III. Score generation and post-processing alternatives

[0402] 1. Symbolic representation alternatives

[0403] - Probabilistic grammar model constrains generation results

[0404] Encode the opera music theory rules as a probabilistic context-free grammar (PCFG), and perform syntax analysis and error correction on the raw score output by the neural network to ensure that the generated results conform to the specific drama's board specification (such as the "board eye" structure of Beijing Opera).

[0405] - Differentiable music rendering engine

[0406] Build an end-to-end differentiable synthesizer (such as DiffSynth) to jointly optimize score generation and audio reconstruction, and correct pitch quantization errors through an auditory feedback loop.

[0407] 2. Interactive correction alternatives

[0408] - Active learning optimizes annotation efficiency

[0409] Design an active learning strategy based on uncertainty sampling to automatically identify error-prone score segments (such as dense ornamentation passages) and prioritize manual annotation, replacing fixed-interval progressive feedback.

[0410] - Augmented reality (AR) assisted correction

[0411] Develop an AR visualization interface to superimpose the generated score with the actual singing of the inheritor's mouth shape and gesture movements, and trigger real-time correction through multi-modal consistency detection.

[0412] IV. System implementation alternatives

[0413] 1. Hardware acceleration alternatives

[0414] - FPGA customized computing architecture;

[0415] A pipeline processing unit based on FPGA is designed to meet the parallel computing requirements of time-frequency dual-flow convolution. The heterogeneous computing acceleration of 1D-CNN and STFT-CNN is realized through hardware description language (HDL), replacing general GPU operation.

[0416] - Neuromorphic chip processing timing signals;

[0417] The asynchronous pulse computing characteristics of Loihi and other neuromorphic chips are used to efficiently process the long-term dependence characteristics of opera arias (such as a 10-second-long aria), reducing power consumption by about 60%.

[0418] 2. Edge-cloud collaborative architecture;

[0419] - End-side lightweight model + cloud-side fine-tuning;

[0420] The pruned feature extraction model (parameter quantity <1MB) is deployed on mobile devices, real-time collection of arias is uploaded to the cloud to complete the music score generation, and differential privacy technology is used to protect user data, replacing the pure end-to-end architecture.

[0421] Five, complete technology path alternatives

[0422] 1. Acoustic event detection driven scheme;

[0423] Abandoning the end-to-end generation paradigm, a two-stage processing flow is adopted:

[0424] - Stage 1: Detect acoustic events (such as singing, crying) in arias based on CRNN, output timestamped symbolic labels;

[0425] - Stage 2: Convert symbolic sequences to target genre music score templates using template filling method, and complete the decoration sound combined with expert rule base;

[0426] 2. Physical model reverse engineering scheme;

[0427] A physical modeling synthesizer for opera singing is constructed (such as a vocal cord vibration simulation based on mass-spring model), the motion parameters of the sound-producing organs (such as glottal opening curve) are estimated from the audio through parameter inversion technology, and these parameters are mapped to music score symbols.

[0428] Six, data acquisition and enhancement alternatives

[0429] 1. Synthetic data generation alternative;

[0430] - Neural vocoder simulates the tone of old records;

[0431] Generate training data with specific noise characteristics (e.g., 78 rpm record scratches) using WaveGlow or HiFi-GAN to enhance the model's ability to adapt to historical recordings.

[0432] - Style decoupling and recombination enhancement;

[0433] Decouple the genre style and singing content features in the aria through β-VAE, and randomly recombine to generate cross-genre mixed samples (e.g., Peking Opera lyrics + Cantonese Opera singing), improving the model's generalization.

[0434] 2. Crowdsourcing data collection mechanism;

[0435] Build a distributed data platform empowered by blockchain, encourage opera enthusiasts to upload aria recordings and label key nodes (e.g., gas port position), and implement data rights and quality verification through smart contracts.

[0436] Alternative technical effects comparison, as shown in Table 4.

[0437] Table 4

[0438]

[0439] Through the above alternatives, a multi-level protection network can be constructed in the claims to prevent others from circumventing the originality of this application through the following paths:

[0440] 1. Component-level replacement (e.g., replace Bi-LSTM with GNN);

[0441] 2. Process reorganization (e.g., split the end-to-end generation into two stages of detection and template filling);

[0442] 3. Technology cross (e.g., combine neuromorphic chips and meta-learning frameworks);

[0443] Four, the key technical points of this embodiment are as follows:

[0444] 1. Time-frequency dual-flow feature extraction of composite neural network architecture;

[0445] - Technical means: Deploy a dual-flow network of 1D-CNN (time domain) and STFT-CNN (frequency domain) in parallel, combine residual connection and feature fusion layer, and realize the collaborative capture of transient features (e.g., word biting starting point) and steady-state features (e.g., resonance peak distribution) in opera arias.

[0446] In this embodiment, the limitations of traditional single time-frequency analysis methods are broken through, and the time domain resolution (≤10ms) and frequency domain resolution (≤5Hz) are optimized through heterogeneous network branches, and the extraction accuracy of nonlinear features such as glissando and vibrato is improved to 94%.

[0447] In this embodiment, the parameter sharing mechanism of the double-flow structure, and the dynamic weight distribution algorithm of the feature fusion layer.

[0448] 2. Play style consistency guarantee based on adversarial training;

[0449] -Technical means: Introduce the Wasserstein GAN (WGAN-GP) framework, where the generator outputs the score sequence, and the discriminator embeds the opera music knowledge graph (such as the board rule, the decoration sound grammar), and the gradient penalty mechanism is used to constrain the compliance of the genre style of the generated results.

[0450] In this embodiment, the domain knowledge is encoded as a structured evaluation indicator of the discriminator (such as the beat density threshold of the "Xipi Liushui" board of Peking Opera), rather than relying on pure data-driven, and the board identification accuracy is improved from 71% to 89%.

[0451] In this embodiment, the rule-based knowledge graph embedding method in the discriminator is adopted, and the generator and the discriminator are jointly optimized.

[0452] 3. Multi-genre adaptive framework of hierarchical transfer learning;

[0453] -Technical means: Build a hierarchical architecture with shared Transformer encoder + pluggable adapter (Adapter), the bottom encoder learns the common acoustic features across genres, and the top adapter quickly adapts to the characteristics of the target genre through low-rank matrix fine-tuning (LoRA).

[0454] In this embodiment, only 5 hours of target genre data is needed to complete the transfer (traditional method requires ≥ 50 hours), supporting parallel learning of 21 genres, and the cross-genre generalization error is reduced to 14.2%.

[0455] In this embodiment, the modular design of the Adapter is adopted, and the adapter parameter reuse mechanism based on task similarity is adopted.

[0456] 4. Stream processing engine and multi-modal real-time verification;

[0457] -Technical means: Use causal convolution + dynamic cache window to realize the stream processing of the aria audio, and simultaneously build a three-dimensional mapping space of aria-score-body action, and when the score and programmed action conflict (such as the extension of the "swing hair" action), trigger confidence weighted correction.

[0458] In this embodiment, the multi-modal correlation of opera performance is included in the score generation logic, and the real-time processing delay is ≤200ms (traditional method ≥30 seconds), and the action conflict false alarm rate is reduced to less than 3%.

[0459] In this embodiment, the sliding strategy of dynamic cache window and the multi-modal confidence fusion algorithm are adopted.

[0460] 5. Score representation method decoupled from symbols and parameters

[0461] - Technical means: decompose score elements into discrete symbols (pitch, ornament type) and continuous parameters (vibrato depth, slide speed), generate symbol sequences and parameter matrices through double-channel output layers, and support multi-format output such as staff score, staff score, and five-line score MIDI.

[0462] In this embodiment, the distortion problem of ornament quantization in traditional symbolic music generation is solved, and the restoration degree of opera characteristic techniques such as trill and staccato reaches 85.7%.

[0463] In this embodiment, the joint training loss function of symbols and parameters and the unified intermediate representation of multi-format conversion are adopted.

[0464] This embodiment scheme is summarized as follows:

[0465] 1. Heterogeneous feature extraction: Break through the performance bottleneck of traditional linear analysis through time-frequency dual-flow network.

[0466] 2. Knowledge-guided adversarial training: Integrate opera theory rules into the generation adversarial process to ensure style compliance.

[0467] 3. Modular migration architecture: Achieve rapid adaptation of multiple genres with extremely low data requirements.

[0468] 4. Multi-modal real-time generation: First introduce body movement semantics into the score generation closed loop.

[0469] 5. Decoupled score representation: Separate symbols and parameters to be compatible with the aesthetic expression of traditional notation.

[0470] In the ninth embodiment, technical standards and data sets

[0471] 1. Data set: Chinese Opera Acoustic Corpus (COAC)

[0472] - Issuing agency: China Drama College

[0473] - Content: Contains 12 genres, 300 hours of high-fidelity singing audio and staff score annotation.

[0474] - Application: As a benchmark data set for training and testing of the invention, it supports multi-genre generalization verification.

[0475] 2. Evaluation criteria: MIREX Music Transcription Metrics

[0476] -Definer: International Society for Music Information Retrieval (ISMIR)

[0477] -Indicators: Note F1 score, pitch accuracy, rhythm error rate.

[0478] -This application achieved an F1 score of 92.4% on the COAC dataset, surpassing the baseline model by 15.6%.

[0479] Example Description

[0480] Example 1: Peking Opera "The Drunken Beauty" singing melody conversion to simplified score

[0481] -Input data: Mono audio (44.1kHz, SNR≥20dB), duration 3 minutes 28 seconds.

[0482] -Processing flow:

[0483] 1. Time-frequency dual-stream feature extraction: 1D-CNN (convolution kernel 64x1) and STFT-CNN (window length 1024 points) parallel operation.

[0484] 2. Adversarial training generation: discriminator loads Peking Opera plate rule library (including Xipi, Erhuang 8 kinds of plate eye structure).

[0485] 3. Multi-modal verification: when detecting

lying fish

[0486] -Output results: Simplified score completeness 98.7%, ornament restoration accuracy 86.2%.

[0487] Example 2: Historical recording restoration (1947 Mei Lanfang "The Battle of the Two Beauties" segment)

[0488] -Input data: Mono old record recording (SNR≤15dB, containing scratch noise).

[0489] -Enhancement strategy:

[0490] -Data preprocessing: WaveGlow vocoder removes impulse noise and preserves original timbre.

[0491] -Migration learning: based on Kunqu pre-training model, fine-tune Adapter module through 5 hours of Mei singing melody data.

[0492] -Output results: Score restoration completeness 78%, improved by 26% compared with traditional spectral analysis method.

[0493] Technical comparison and advantage quantification as shown in Table 5.

[0494] Table 5

[0495]

[0496]

[0497] In a tenth specific embodiment, a portion of the prior art is disclosed.

[0498] Application No. 202310557794.0, entitled "A Multi-task Learning Piano Harmony Music Automatic Notation Method Based on Local Attention";

[0499] Application No. 202110308570.7, entitled "Buddhist Music Notation Method, Device and Equipment Based on Neural Network and Medium";

[0500] Application No. 202311858130.4, entitled "Automatic Notation Method and Device Based on Sound Source Separation";

[0501] Figure 3 The electronic device structure block diagram of the vehicle power supply system control method provided by one or more embodiments of the application.

[0502] As Figure 3 shown, the present application provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete the communication among each other through the communication bus.

[0503] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the opera aria audio notation method.

[0504] The present application also provides a computer readable storage medium storing a computer program executable by an electronic device, which makes the electronic device execute the steps of the opera aria audio notation method when the computer program runs on the electronic device.

[0505] The present application also provides a music score generation platform, comprising:

[0506] An electronic device for implementing the steps of the opera aria audio notation method.

[0507] A processor, which executes the steps of the opera aria audio notation method when the program runs on the data output from the electronic device.

[0508] A storage medium for storing the program, which executes the steps of the opera aria audio notation method when the program runs on the data output from the electronic device.

[0509] Those skilled in the art can clearly understand the application by the description of the above embodiments that the application can be implemented by means of software and the necessary universal hardware platform. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, and the like) execute the methods described in each of the embodiments or some parts of the embodiments of the application.

[0510] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, rather than limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A method for audio notation of a play aria, characterized by, The opera singing audio score recording method comprises the steps of multi-modal feature extraction, adversarial training score, hierarchical transfer learning adaptation, streaming processing and multi-modal verification, and symbol-parameter decoupling output; The multi-modal feature extraction step comprises synchronously collecting action video stream and audio stream of opera performance, extracting action transient feature through time domain branch 1D-CNN network, and extracting audio steady-state feature through frequency domain branch STFT-CNN network; An attention weighted fusion layer is used to dynamically adjust the contribution weight of action feature and audio feature; The adversarial training score step comprises verifying the symbol sequence in the score through the opera music knowledge graph; The hierarchical transfer learning adaptation step comprises adaptation training of different drama types; A shared encoding layer is constructed, and the shared encoding layer is a Transformer network pre-trained on a cross-drama mixed data set; A drama type adapter is inserted into the FFN layer of each layer of the Transformer, and the drama type adapter adopts a low-rank matrix fine-tuning module LoRA inserted into the FFN of each layer of the Transformer; During training, the shared encoder is frozen, and the parameters of the drama type adapter are updated; The drama type is switched and adapted through the drama type ID; The streaming processing and multi-modal verification step comprises realizing multi-modal alignment through dynamic time warping, and the dynamic time warping matches the key frames of the action feature and the key points of the audio feature; When a conflict between the score and the action is detected, a confidence weighted voting mechanism is used, which dynamically adjusts the weight of the model output and the rule template according to the voting result; The symbol-parameter decoupling output step comprises symbol-parameter decoupling and output of score data in a target format; The discrete symbol channel is used to output the symbol sequence of pitch and ornament type, and differentiable sampling is realized through Gumbel-Softmax; The continuous parameter channel is used to predict parameters, including the parameters of vibrato depth and glissando speed; The continuous parameter channel further comprises using Tanh activation to constrain the range; and a multi-format converter is used to output the target format score based on a rule engine to convert the internal representation into the target format.

2. The method of claim 1, wherein, The multi-modal feature extraction step comprises parallel processing of the time domain branch and the frequency domain branch; Transient feature information is extracted according to the time domain branch; The multi-modal feature extraction step further comprises extraction of action feature and extraction of audio feature; The transient feature comprises the transient feature of the action feature; Steady-state feature information is extracted according to the frequency domain branch; The steady-state feature comprises the steady-state feature of the audio feature; The multi-modal feature extraction step further comprises feature fusion; The feature fusion comprises using an attention weighted fusion layer to dynamically adjust the contribution degree of time-frequency feature; The 1D-CNN network of the time domain branch is used to extract the action transient feature, and the convolution kernel size of the 1D-CNN network is 64*1 and the step is 8; The action feature weight is predefined by a traditional opera action library, and an association rule of the action library includes automatically extending a corresponding music score drag time value when a lying fish body segment action in a video is detected; Wherein, the STFT-CNN of the frequency domain branch adopts a 1024-point Hanning window, a parameter setting of 50% overlap, and extracts 229-dimensional Mel spectrum features; Wherein, the attention weighted fusion layer introduces an action-singing tune alignment loss during training, and the alignment loss matches the key frames of the action features and the key points of the audio features based on dynamic time warping.

3. The method of claim 2, wherein, The step of adversarial training of the music score includes: The input feature sequence after feature fusion; Outputting the music score and the parameter matrix according to the feature sequence; Wherein, analyzing the continuity of the symbol sequence in the music score; According to the analysis of the continuity of the symbol sequence in the music score, the traditional opera music knowledge graph is verified; Wherein, the traditional opera music knowledge graph contains a board type rule library and a decoration sound library, the board type rule library covers the board type conversion logic of multiple drama types, and the decoration sound library records the annotation rules of the traditional opera specific decoration sound; Wherein, the discriminator contains a data-driven branch and a knowledge rule branch, the data-driven branch uses a 3-layer Bi-LSTM to analyze the continuity of the note sequence in the music score, and the knowledge rule branch embeds the traditional opera music knowledge graph.

4. The method of claim 3, wherein, Further comprising: In the board type rule library of the traditional opera music knowledge graph, multiple drama type board type conversion logic is included; in the decoration sound library, the annotation rules of the decoration sound, crying tune and glissando are included.

5. The method of claim 4, wherein, The step of hierarchical transfer learning adaptation includes sharing the encoding layer; The shared encoding layer is a Transformer network, which is pre-trained on a cross-drama mixed dataset; Further comprising a drama type adapter; The drama type adapter uses a low-rank matrix fine-tuning module inserted into the FFN of each layer of the Transformer; Further comprising a training strategy: freeze the shared encoder and update the drama type adapter parameters; Parallelly load the drama type adapter, switch the drama type through the drama type ID and perform adaptation; Wherein, the Transformer network of the shared encoding layer is a 12-layer structure, the hidden layer dimension is 768, and the pre-training dataset is a cross-drama mixed dataset; Wherein, the LoRA module of the drama type adapter is inserted into the FFN layer of each layer of the Transformer, the learning rate during training is set to 5e-4, and the iteration round is 50 rounds.

6. The method of claim 5, wherein, Further comprising: The cross-drama mixed dataset pre-trained by the shared encoding layer contains high-fidelity singing tune audio and staff notation of multiple drama types; When the drama type adapter is switched through the drama type ID, it supports fast response.

7. The method of claim 6, wherein, The step of stream processing and multi-modal verification includes a causal convolution layer; The causal convolution layer restricts the convolution kernel to only access the current input and historical input data; Further comprising a dynamic buffer window; The dynamic buffer window dynamically expands with the beat intensity; Further comprising multi-modal alignment: matching the key frames of the action features and the key points of the audio features through dynamic time warping; Further comprising an implementation correction mechanism: when a conflict between the music score and the action is detected, a confidence weighted voting is used; According to the voting result, the weights of the model output and the rule template are dynamically adjusted. Wherein, the causal convolution layer restricts the convolution kernel to only access the current and historical input data; Wherein, the dynamic buffer window dynamically expands with the beat intensity; Wherein, the weight calculation function of the confidence weighted voting is wherein is dynamically calculated by the model confidence score.

8. The method of claim 7, wherein, Further comprising: The length of the dynamic buffer window dynamically adjusts with the beat intensity; The alignment error of the dynamic time warping is controlled within a preset range.

9. The method of claim 8, wherein, The symbol-parameter decoupling output step comprises a symbol-parameter decoupling output layer; The symbol-parameter decoupling output layer comprises a discrete symbol channel, a continuous parameter channel, and a multi-format converter; The discrete symbol channel is used to output a symbol sequence of pitch and ornament type, and realizes differentiable sampling through Gumbel-Softmax; The continuous parameter channel is used to predict parameters, including parameters of vibrato depth and portamento speed; The continuous parameter channel further comprises a Tanh activation to constrain the range; The multi-format converter converts the internal representation into a target format based on a rule engine; The discrete symbol channel realizes differentiable sampling through Gumbel-Softmax; The continuous parameter channel adopts Tanh activation to constrain the range; The rule engine of the multi-format converter contains mapping rules of 2 / 4 time signature staff and 8 / 4 time signature staff, and each measure is marked with a strong beat symbol ">".

10. The method of claim 9, wherein, Further comprising: The multi-format converter supports outputting various general format staff data and staff data.

11. A system for audio notation of opera arias, characterized by The opera singing audio score system comprises: A multi-modal feature extraction module, an adversarial training score module, a hierarchical transfer learning adaptation module, a streaming processing and multi-modal verification module, and a symbol-parameter decoupling output module; The multi-modal feature extraction module is used to synchronously collect action video streams and audio streams of opera performances, extract action transient features through a time domain branch 1D-CNN network, and extract audio steady-state features through a frequency domain branch STFT-CNN network; An attention weighted fusion layer is used to dynamically adjust the contribution weights of the action features and the audio features; The adversarial training score module is used to verify the symbol sequence in the score through an opera music knowledge graph; The hierarchical transfer learning adaptation module is used for adaptation training of different drama types; A shared encoding layer is constructed, and the shared encoding layer is a Transformer network pre-trained on a cross-drama type mixed dataset; A drama type adapter is inserted into the FFN layer of each Transformer, and the drama type adapter adopts a low-rank matrix fine-tuning module LoRA inserted into the FFN of each Transformer; A strategy of updating the parameters of the drama type adapter while freezing the shared encoder is adopted during training; The drama type is switched and adapted through the drama type ID; The streaming processing and multi-modal verification module is used to realize multi-modal alignment through dynamic time warping, and the dynamic time warping matches the key frames of the action features and the key points of the audio features; When a conflict between the score and the action is detected, a confidence weighted voting mechanism is adopted, and the confidence weighted voting mechanism dynamically adjusts the weights of the model output and the rule template according to the voting result; A module for symbol-parameter decoupled output, used for symbol-parameter decoupling, and outputting music score data in a target format; Wherein, the symbol sequence is output through a discrete symbol channel, which is used for outputting the symbol sequence of pitch and ornament type, and the differentiable sampling is realized through Gumbel-Softmax; The parameters are predicted through a continuous parameter channel, which is used for predicting the parameters including the parameters of tremolo depth and glissando speed; The continuous parameter channel further comprises adopting Tanh activation to constrain the range; and outputting the music score in the target format through a multi-format converter, which converts the internal representation into the target format based on a rule engine.

12. An electronic device, comprising: Comprise: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete the communication among each other through the communication bus; The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the opera aria audio notation method in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, It stores a computer program executable by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the opera aria audio notation method in any one of claims 1 to 10.

14. A music score generation platform, characterized by, Comprise: An electronic device for implementing the steps of the opera aria audio notation method in any one of claims 1 to 10; A processor, the processor runs a program, and when the program runs, the data output from the electronic device executes the steps of the opera aria audio notation method in any one of claims 1 to 10; A storage medium for storing a program, and the program executes the steps of the opera aria audio notation method in any one of claims 1 to 10 when running on the data output from the electronic device.

Images