Harmonic texture and sequence reasoning fused audio-to-MIDI method and system

Abstract

The invention discloses a harmonic texture and sequence reasoning fused audio-to-MIDI method and system, and relates to the technical field of audio processing. Aiming at the problem of insufficient transcription accuracy of the existing polyphonic music, the method comprises the following steps: carrying out time-frequency transformation on the polyphonic music to generate a spectrogram; constructing a time-frequency target detection network, detecting notes based on harmonic comb texture features, and outputting a time-frequency bounding box; constructing a sequence inference network to extract time sequence context features; carrying out local alignment and weighted fusion on the time sequence context features by using a time-frequency bounding box by adopting an ROI-guided gating fusion mechanism; and inputting the fusion features into a decoder to generate an MIDI sequence, and correcting the note time value by using a time-frequency bounding box. According to the method, the visual texture features and the music sequence logic are effectively fused, and the time precision and robustness of polyphonic music transcription are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing, and specifically to an audio-to-MIDI method and system based on the fusion of harmonic texture detection and sequence reasoning. Background Technology

[0002] With the popularization of digital music creation and production, converting audio signals to MIDI (Musical Instrument Digital Interface) format has become a key technological requirement in the music industry. Audio-to-MIDI technology can automatically convert recorded musical performances into editable MIDI event sequences, providing convenience for music composition, arrangement, and education.

[0003] Currently, audio-to-MIDI technology mainly falls into two categories: methods based on spectral analysis and methods based on deep learning. Spectral analysis-based methods typically use Short-Time Fourier Transform (STFT) or Constant Q Transform (CQT) to convert the audio signal into a time-frequency representation, and then extract note information through peak detection and pitch estimation algorithms. These methods perform well with single instruments and simple musical passages, but their accuracy drops significantly when processing complex multi-part music.

[0004] With the development of deep learning technology, significant progress has been made in audio-to-MIDI conversion methods based on neural networks. CN112397044B discloses an automatic music transcription method based on deep learning. This method designs a stereo feature enhancement network and a temporal convolutional module network, which can effectively improve the automatic music transcription effect. This method overcomes the shortcomings of traditional solutions that simply take the average of stereo signals and convert them into mono signals for processing, thus improving the efficiency of utilizing the features of dual-channel signals.

[0005] Regarding multi-channel information fusion, CN116564253B proposes a piano music notation method based on cyclic spectrum. This method inputs the cyclic spectrum of the piano signal into detection networks with different targets, detecting the start point, end point, and duration of piano notes respectively, and then fuses these three channel features to obtain enhanced features. This method improves the accuracy and robustness of piano music notation through multi-channel information fusion.

[0006] Recent research has begun exploring the application of object detection technology in the field of audio transcription. CN118571262B discloses a note-level automatic singing transcription method based on object detection and linguistic features. This method performs object detection on slices of Mel spectrograms and phoneme posterior maps, and then performs post-processing and temporal adjustment on the left and right boundaries of the bounding boxes obtained from the object detection to obtain the start and end times of the notes. This method introduces two-dimensional image processing technology into the field of audio transcription, providing a new approach for the precise localization of note boundaries. However, singing voices are usually monophonic, and although this existing technology incorporates object detection, it does not optimize for the complex frequency overlap and harmonic interweaving problems in polyphonic music (where multiple instruments play simultaneously).

[0007] In the area of ​​time-frequency domain information fusion, CN114155879B proposes an abnormal sound detection method that utilizes time-frequency fusion to compensate for anomaly perception and stability. This method leverages the time-domain and frequency-domain information obtained from the original audio signal of the acoustic target to form perceptual complementarity, and designs a mechanism for fusing learnable features constructed from the time domain perspective with the Log-Mel spectrum. This win-win gain mechanism of time-frequency domain information complementarity effectively improves the stability of the system and the reliability of the detection results.

[0008] Furthermore, CN120564670A discloses a method for extracting semantics from classical music for large AIGC models. This method employs a convolutional neural network to extract local time-frequency patterns, generating a feature set containing note and chord correlations, and decouples sentiment and structural semantics through a variational autoencoder. This approach provides a new approach for advanced semantic understanding of music information.

[0009] However, existing technologies still have the following problems:

[0010] 1. Insufficient frequency domain feature extraction: Existing sequence models often compress the spectrogram into a one-dimensional vector sequence for processing, ignoring the rich texture features and geometric structures inherent in the spectrogram as a two-dimensional image, resulting in information loss. In particular, the special texture patterns presented in the spectrogram for notes with harmonic structures are not fully utilized.

[0011] 2. Ambiguous temporal boundary localization: Sequence models are less accurate than edge-detection-based visual algorithms in predicting the precise start and end times of notes. Existing methods struggle to accurately capture the physical duration of note sounds, resulting in MIDI files with poor rhythm and failing to accurately reflect the performance style of the original audio.

[0012] 3. Weak noise resistance and dealiasing capabilities: In complex multi-part musical scenes, the frequency components of different notes often overlap, and existing single-modal models struggle to effectively distinguish these overlapping frequency components, easily leading to misclassification. This is especially true when there is significant background noise or multiple instruments are playing simultaneously, resulting in a significant decrease in transcription accuracy.

[0013] 4. Lack of effective multimodal feature fusion mechanism: Although some methods attempt to combine time-domain and frequency-domain information, they lack adaptive fusion strategies for different audio qualities and complexities, and cannot dynamically adjust the weights of different features according to the characteristics of specific audio segments.

[0014] Therefore, there is an urgent need for an audio-to-MIDI method that can fully utilize the harmonic texture features in audio signals and effectively integrate time-frequency features with sequence reasoning, in order to improve the accuracy and robustness of polyphonic music transcription. Summary of the Invention

[0015] The technical problem to be solved by this invention is to provide an audio-to-MIDI method based on the fusion of harmonic texture detection and sequence reasoning, which addresses the shortcomings of existing technologies such as insufficient frequency domain feature extraction, ambiguous temporal boundary positioning, and weak noise resistance and dealiasing capabilities. This method can effectively improve the temporal accuracy of audio-to-MIDI conversion, enhance system robustness, improve interpretability, and effectively distinguish harmonic interference when multiple instruments are mixed.

[0016] The technical solution adopted by this invention to solve its technical problem is: to provide an audio-to-MIDI method based on the fusion of harmonic texture detection and sequence reasoning, including the following steps:

[0017] Step S1: Obtain the polyphonic music audio signal to be transcribed, and perform time-frequency transformation on it to generate a time-frequency spectrum containing energy distribution information;

[0018] Step S2: Construct a time-frequency target detection network. Take the time-frequency spectrum as input and detect and locate the note target based on the harmonic comb texture features of the note energy in the spectrum. The detection output is a time-frequency bounding box, where the horizontal span of the time-frequency bounding box represents the physical duration of the note, and the vertical position and height represent the fundamental pitch and harmonic bandwidth of the note.

[0019] Step S3: Construct a sequence reasoning network to extract the temporal context features of the polyphonic music audio signal, wherein the temporal context features include the tonality and rhythmic dependencies between notes;

[0020] Step S4: Using a region of interest (ROI)-guided gating fusion mechanism, the time-frequency bounding box generated in step S2 is used as a spatial index to perform local alignment and weighted fusion of the temporal context features extracted in step S3 to obtain multimodal hidden layer features.

[0021] Step S5: Input the multimodal hidden layer features into the decoder to generate a MIDI event sequence; when generating the note-off event, use the horizontal span information of the time-frequency bounding box in step S2 to quantize and correct the note's duration.

[0022] Preferably, in step S2, the harmonic comb texture feature refers to the combination of energy stripes arranged at equal intervals on the frequency axis by the fundamental frequency and multiple integer multiple overtones of the same note on the time-frequency spectrum; the time-frequency target detection network includes a harmonic sensing convolutional layer, which is used to extract the structural features of the energy stripe combination in order to distinguish different instrument sound sources with overlapping frequencies.

[0023] Preferably, the training loss function of the time-frequency target detection network in step S2 includes the time-frequency intersection-union ratio (TF-IoU) loss; the TF-IoU loss introduces a logarithmic scale weighting factor on the frequency axis when calculating the overlap between the predicted box and the ground truth box, so as to balance the difference in detection accuracy between low-frequency and high-frequency regions.

[0024] Preferably, the gating fusion mechanism based on region of interest guidance in step S4 specifically includes: calculating the local spectral entropy of the time-frequency spectrogram as a confidence index for measuring audio clarity; when the local spectral entropy is lower than a preset threshold, it indicates that the spectral texture is clear, and the gating mechanism increases the weight of the visual features corresponding to the time-frequency bounding box; when the local spectral entropy is higher than the preset threshold, it indicates that there is frequency aliasing or background noise, and the gating mechanism increases the weight of the temporal context features.

[0025] Preferably, in step S5, the time value of the note is quantized and corrected using the horizontal span information of the time-frequency bounding box, and the specific formula is as follows:

[0026]

[0027] in, This is the corrected timestamp for the end of the note. This is the end time predicted by the sequence reasoning network. The starting time coordinates of the time-frequency bounding box. The horizontal axis width of the time-frequency bounding box. This is the fusion coefficient.

[0028] The beneficial effects of this invention are as follows:

[0029] 1. Significantly Improved Temporal Accuracy: Compared to traditional single-modal sequence classification, this invention introduces object detection regression, utilizing the sensitivity of visual algorithms to edges to solve the pain point of "inaccurate rhythm" in MIDI transcription. By quantifying and correcting the note values ​​through the horizontal span information of the time-frequency bounding box, the generated MIDI files are more rhythmically closer to the subtle deviations in human performance, significantly improving the naturalness and accuracy of transcription.

[0030] 2. Enhanced Robustness: This invention employs a dual-modal complementary architecture. When audio is blurred, visual texture recognition is utilized; when visual occlusion (frequency overlap) occurs, sequence context reasoning is employed. Through a region-of-interest-guided gated fusion mechanism, the trust level of the visual and auditory streams is dynamically adjusted according to the local spectral entropy of the time-frequency spectrogram, effectively improving the system's stability in complex audio environments.

[0031] 3. High interpretability: The visual detection results (time-frequency bounding boxes) of the intermediate layer of this invention can be visualized, making it easy for users to understand which note segments the model has identified. The energy stripe combination structure features extracted by the harmonic perception convolutional layer enable the model to distinguish different instrument sound sources with overlapping frequencies, providing a more transparent interpretation mechanism for multi-part music transcription.

[0032] 4. Effectively distinguishes harmonic interference when multiple instruments are mixed: By extracting harmonic comb-like texture features and designing the time-frequency crossover ratio loss function, this invention can more accurately identify note boundaries in the spectrogram. Especially in complex multi-part scenarios, it effectively solves the misjudgment problem caused by severe frequency overlap and improves the accuracy of audio to MIDI conversion. Attached Figure Description

[0033] List of attached images:

[0034] Figure 1: Flowchart of the method

[0035] Figure 2: System Architecture and Data Flow Diagram

[0036] Figure 3: Schematic diagram of harmonic comb-like texture features and time-frequency bounding box

[0037] Figure 4: Schematic diagram of the principle of note value quantization correction

[0038] Detailed description of the attached diagram:

[0039] See the specific implementation details. Detailed Implementation

[0040] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0041] Example 1

[0042] Please seeFigure 1 , Figure 1 This is a schematic diagram of the method flow provided in an embodiment of the present invention. As shown in the figure, the method of the present invention mainly includes two parallel processing paths: the left side is a vision-based time-frequency target detection path, and the right side is an auditory-based sequence reasoning path, which ultimately converge in a fusion unit.

[0043] This invention provides an audio-to-MIDI conversion method and system that integrates harmonic textures and sequence reasoning. The embodiments of this invention are described in detail below.

[0044] Step S1: Obtain the polyphonic music audio signal to be transcribed, and perform time-frequency transformation on it to generate a time-frequency spectrogram containing energy distribution information.

[0045] In this step, the input polyphonic music audio signal is first received, and then a time-frequency transformation is performed on the audio signal using either a Log-Mel Spectrogram or a Constant Q Transform (CQT) to generate a time-frequency spectrogram reflecting the audio energy distribution. Preferably, if a Constant Q Transform (CQT) is used, the number of bins per octave is set to 36 to 48 to match the chromatic characteristics of the music; the lowest frequency (fmin) is set to 27.5 Hz (corresponding to the A0 key on a piano), and the highest frequency covers up to 16 kHz. Compared to the Short-Time Fourier Transform, CQT has higher frequency resolution in the low-frequency region, which is beneficial for distinguishing dense harmonic structures in the bass region. This spectrogram can be processed into a three-channel image tensor, and by calculating the first-order difference, dynamic feature channels can be added, thereby better characterizing the time-frequency energy distribution features of the audio.

[0046] Step S2: Construct a time-frequency target detection network, taking the time-frequency spectrogram as input, and detecting and locating note targets based on the harmonic comb texture features of the note energy in the spectrogram.

[0047] In this step, the harmonic comb texture feature refers to the energy stripes that appear on the time-frequency spectrum of the same note as a combination of the fundamental frequency and multiple integer multiples of overtones arranged at equal intervals on the frequency axis. Specifically, this is combined with... Figure 3 As shown, in a coordinate system with time on the horizontal axis and frequency on the vertical axis, a single musical note is represented by a set of parallel horizontal lines, which represent the fundamental frequency and overtones of the note, respectively. This invention uses a bounding box (shown by dashed lines) to frame this set of parallel lines, which exhibit a "harmonic comb texture" characteristic. Figure 3 As indicated by the annotation, the geometric properties of this time-frequency bounding box directly correspond to the physical properties of music: the horizontal width (Width) of the rectangle corresponds to the physical duration of the note's sound (...). The center height of the rectangle corresponds to the fundamental frequency pitch of the note. ).

[0048] The time-frequency target detection network includes harmonic-aware convolutional layers to extract structural features of energy stripe combinations to distinguish different instrument sound sources with overlapping frequencies.

[0049] Specifically, the harmonic sensing convolutional layer employs a dilated convolution structure. Preferably, the dilation rate of the convolution kernel is set to a combination of [1, 2, 4, 8] along the frequency axis. Since the harmonics of musical notes exhibit a fixed spacing pattern along the logarithmic frequency axis, the dilated convolution expanding along the frequency axis can increase the receptive field, enabling it to simultaneously cover the fundamental frequency and higher harmonics, thereby capturing the complete "comb texture." Furthermore, when training this detection network, the aspect ratio of the preset anchor boxes needs to be adjusted for musical notes, setting more elongated anchor boxes to accommodate notes with longer durations.

[0050] This network employs a deep convolutional neural network (such as CSPDarknet) as its backbone to construct a time-frequency target detection head. In this network, a note is defined as a quadruple on the time-frequency graph, including the note's center time point, center frequency (corresponding to pitch), duration, and bandwidth (covering the fundamental frequency and major harmonic range). The detection output is a time-frequency bounding box, where the horizontal axis span represents the physical duration of the note's sound, and the vertical axis position and height represent the fundamental frequency pitch and harmonic bandwidth of the note.

[0051] The training loss function of this network includes the Time-Frequency Intersection over Union (TF-IoU) Loss. This loss function introduces a logarithmic scaling factor on the frequency axis when calculating the overlap between the predicted and ground truth boxes to balance the difference in detection accuracy between low-frequency and high-frequency regions. Traditional IoU only calculates area overlap, while this method introduces pitch sensitivity, enabling the detection results to more accurately constrain the start and end points of notes.

[0052] Step S3: Construct a sequence reasoning network to extract the temporal context features of polyphonic music audio signals. The temporal context features include the tonality and rhythmic dependencies between notes.

[0053] In this step, the audio is segmented into a frame sequence, and frame-level feature vectors are extracted. These feature vectors are processed using a Transformer encoder, and long-distance dependencies are captured through a self-attention mechanism, thereby extracting temporal context features containing tonal and rhythmic dependencies between notes.

[0054] Step S4: Using a region of interest (ROI)-guided gating fusion mechanism, the time-frequency bounding box generated in step S2 is used as a spatial index to perform local alignment and weighted fusion of the temporal context features extracted in step S3 to obtain multimodal hidden layer features.

[0055] In this step, the gating fusion mechanism guided by the region of interest specifically includes:

[0056] The local spectral entropy of the time-frequency spectrum is calculated as a confidence metric for audio clarity. The local spectral entropy is calculated as follows: the time-frequency spectrum is divided into... Local blocks, energy distribution within each block Calculate Shannon entropy .when A lower value indicates that energy is concentrated at a few frequency points, meaning the harmonic structure is clear (visual characteristics are reliable); when A high value indicates a disordered energy distribution (such as percussion or noise), which means harmonic blurring. In this case, the weight of visual features should be reduced, and more reliance should be placed on temporal context features.

[0057] When the local spectral entropy is lower than the preset threshold, it indicates that the spectral texture is clear, and the gating mechanism increases the visual feature weights corresponding to the time-frequency bounding box.

[0058] When the local spectral entropy is higher than a preset threshold, it indicates the presence of frequency aliasing or background noise, and the gating mechanism increases the weight of the temporal context features.

[0059] In practice, the visual feature map is first transformed to include structural summary information in the frequency domain. Then, the fusion gating coefficient G is calculated:

[0060]

[0061] Where σ represents the activation function. and This is the weight matrix. The transformed visual features, Auditory characteristics This is the bias term. Multimodal hidden layer features are obtained by weighted fusion of visual and auditory features.

[0062] Step S5: Input the multimodal hidden layer features into the decoder to generate a MIDI event sequence; when generating the note-off event, use the horizontal span information of the time-frequency bounding box from step S2 to quantize and correct the note's duration. The correction principle is as follows: Figure 4 As shown in the figure, three parallel time axes are compared. The first axis shows the end time of the sequence inference network prediction. It may have rhythm deviations; the second axis shows the time-frequency bounding box obtained by visual detection, and its length It more accurately reflects the actual physical duration, with the end point being... The third axis shows the final corrected result. Through the fusion mechanism, the final note-off time is pulled toward the physical end point indicated by the visual bounding box.

[0063] In this step, the fused features are fed into a pointer network or autoregressive decoder to predict MIDI event sequences. Events include pitch probability, velocity probability, and state probability (note-on, note-off).

[0064] The specific formula for quantizing and correcting the note value using the horizontal span information of the time-frequency bounding box is as follows:

[0065]

[0066] in, This is the corrected timestamp for the end of the note. This is the end time predicted by the sequence reasoning network. The starting time coordinates of the time-frequency bounding box. The horizontal axis width of the time-frequency bounding box. The coefficients are used for fusion. This method enables frame-level precision MIDI generation, effectively combining the spatial precision of time-frequency target detection with the context-aware capabilities of sequence reasoning.

[0067] Through the above steps, this method can effectively convert polyphonic music audio into MIDI format, achieving high-precision note detection and transcription, and is particularly adaptable to complex multi-instrument mixed scenarios.

[0068] Example 2

[0069] like Figure 2 As shown, Figure 2This illustration shows the network architecture and data flow diagram provided by an embodiment of the present invention. The present invention provides an audio-to-MIDI system based on the fusion of harmonic texture detection and sequence reasoning. This system comprises four main functional parts: a time-frequency conversion module, a harmonic detection module, a sequence reasoning module, and a gated fusion decoding module.

[0070] The time-frequency conversion module receives the input audio signal and converts it into a time-frequency spectrum containing harmonic energy distribution. This module can calculate the log-Mel spectrum or the constant-Q transform spectrum and process it into a three-channel image tensor. By calculating the first-order difference, the time-frequency conversion module can also add dynamic feature channels, making the harmonic structure in the spectrum more clearly visible. This processing method enables the subsequent harmonic detection module to more accurately identify note information in the audio.

[0071] The harmonic detection module employs a deep convolutional neural network as its backbone to construct a time-frequency target detection head for identifying harmonic comb-like textures in the spectrogram. This module defines a note as a quadruple on the time-frequency graph, including the note's center time point, center frequency, duration, and bandwidth. By running a target detection algorithm, the module outputs a time-frequency bounding box representing pitch and duration. Employing a time-frequency intersection-over-union (IoU) loss function, this module not only calculates area overlap but also introduces pitch sensitivity to constrain the start and end times of notes with extremely high precision.

[0072] The sequence reasoning module is responsible for capturing the music theory-based logical dependencies in the audio signal. This module segments the audio into a frame sequence, extracts frame-level feature vectors, and processes them using a Transformer encoder. Through a self-attention mechanism, the sequence reasoning module can capture long-range dependencies, thereby understanding the logical structure and patterns in the music. The output feature sequence of this module needs to be downsampled or interpolated to align its time dimension with the feature time dimension output by the harmonic detection module.

[0073] The gated fusion decoding module is the core of the system, used to combine visual positioning accuracy with auditory logical probability to generate and correct MIDI files. To address the semantic misalignment between visual features and sequence features, this module employs an adaptive gated fusion unit. First, the gated fusion decoding module processes the visual feature map to include frequency domain structural summary information. Then, it calculates fusion gating coefficients and achieves feature fusion through Hadamard product. Finally, the fused features are input into a pointer-based network or autoregressive decoder to predict MIDI event sequences, including pitch probability, velocity probability, and state probability.

[0074] In a preferred embodiment, the gated fusion decoding module further utilizes the duration information output by the harmonic detection module to weight and correct the prediction results of the sequence model, thereby achieving frame-level precision MIDI generation. Specifically, when decoding the end time of a note, the system combines the visual positioning information provided by the harmonic detection module and the auditory logic prediction provided by the sequence reasoning module to determine the final note duration through a weighted fusion method. This fusion mechanism significantly improves the timing accuracy of the MIDI file.

[0075] During system operation, the time-frequency conversion module first converts the input audio into a spectrogram. Then, the harmonic detection module identifies the note positions and durations in the spectrogram. Simultaneously, the sequence reasoning module analyzes the musicological logic of the audio. Finally, the gated fusion decoding module fuses the two features and decodes them to generate a MIDI event sequence. The entire system, through audiovisual dual-modal fusion, ensures both precise positioning of note time boundaries and the coherence of musical logic, thereby generating high-quality MIDI files.

[0076] Example 3

[0077] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio-to-MIDI method based on the fusion of harmonic texture detection and sequence reasoning as described in Embodiment 1.

[0078] The computer-readable storage medium can be any medium capable of storing computer program code, such as read-only memory (ROM), random access memory (RAM), optical disc, magnetic disk, magnetic tape, flash memory, USB flash drive, etc.

[0079] When the program in the computer-readable storage medium is loaded and executed by the processor, the processor will execute the various steps of the audio-to-MIDI method based on harmonic texture detection and sequence reasoning fusion described in Embodiment 1, including acquiring the polyphonic music audio signal to be transcribed and generating a time-frequency spectrogram, constructing a time-frequency target detection network to detect note targets, constructing a sequence reasoning network to extract temporal context features, performing feature fusion using a region-of-interest guided gating fusion mechanism, and inputting multimodal hidden layer features into the decoder to generate a MIDI event sequence.

[0080] The computer program code can be written in programming languages ​​such as C++, Python, and MATLAB, and can run on general-purpose processors or dedicated audio processing chips. During program execution, the processor dynamically adjusts the weight ratio of harmonic texture detection and sequence inference according to the characteristics of the audio data to adapt to audio transcription tasks of varying complexity.

[0081] The program stored on the computer-readable storage medium may also include a user interface module that allows users to adjust transcription parameters, such as the sensitivity of time-frequency target detection, the context window size of sequence inference, and the threshold of gating fusion, to optimize the transcription effect of different music genres.

[0082] The above description is merely a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It is understood that other improvements and variations that are directly derived or conceived by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included within the protection scope of the present invention.

Claims

1. An audio-to-MIDI conversion method and system integrating harmonic texture and sequence reasoning, characterized in that, Includes the following steps: Step S1: Obtain the polyphonic music audio signal to be transcribed, and perform time-frequency transformation on it to generate a time-frequency spectrum containing energy distribution information; Step S2: Construct a time-frequency target detection network, taking the time-frequency spectrum as input, and detect and locate note targets based on the harmonic comb texture features of the note energy in the spectrum. The detection output is a time-frequency bounding box, wherein the horizontal span of the time-frequency bounding box represents the physical duration of the note, and the vertical position and height represent the fundamental pitch and harmonic bandwidth of the note. Step S3: Construct a sequence reasoning network to extract the temporal context features of the polyphonic music audio signal, wherein the temporal context features include the tonality and rhythmic dependencies between notes; Step S4: Using a region of interest (ROI)-guided gating fusion mechanism, the time-frequency bounding box generated in step S2 is used as a spatial index to perform local alignment and weighted fusion of the temporal context features extracted in step S3 to obtain multimodal hidden layer features. Step S5: Input the multimodal hidden layer features into the decoder to generate a MIDI event sequence; when generating the note-off event, use the horizontal span information of the time-frequency bounding box in step S2 to quantize and correct the note's duration.

2. The method according to claim 1, characterized in that, In step S2, the harmonic comb texture feature refers to the energy stripes that the same note represents on the time-frequency spectrum as a combination of the fundamental frequency and multiple integer multiple overtones arranged at equal intervals on the frequency axis; The time-frequency target detection network includes a harmonic-aware convolutional layer for extracting structural features of the energy stripe combinations to distinguish different musical instrument sound sources with overlapping frequencies.

3. The method according to claim 1, characterized in that, The training loss function of the time-frequency target detection network in step S2 includes the time-frequency intersection-union ratio (TF-IoU Loss). The time-frequency intersection-union loss introduces a logarithmic scale weighting factor on the frequency axis when calculating the overlap between the predicted bounding box and the ground truth bounding box, in order to balance the difference in detection accuracy between the low-frequency region and the high-frequency region.

4. The method according to claim 1, characterized in that, The gating fusion mechanism based on region of interest guidance described in step S4 specifically includes: Calculate the local spectral entropy of the time-frequency spectrum as a confidence metric for audio clarity; When the local spectral entropy is lower than a preset threshold, it indicates that the spectral texture is clear, and the gating mechanism increases the visual feature weight corresponding to the time-frequency bounding box. When the local spectral entropy is higher than a preset threshold, it indicates the presence of frequency aliasing or background noise, and the gating mechanism increases the weight of the temporal context features.

5. The method according to claim 1, characterized in that, The step S5 describes using the horizontal span information of the time-frequency bounding box to quantize and correct the note's duration. The specific formula is as follows: in, This is the corrected timestamp for the end of the note. This is the end time predicted by the sequence reasoning network. The starting time coordinates of the time-frequency bounding box. The horizontal axis width of the time-frequency bounding box. This is the fusion coefficient.

6. An audio-to-MIDI system based on the fusion of harmonic texture detection and sequence reasoning, characterized in that, include: The time-frequency conversion module is used to generate a time-frequency spectrum diagram containing the harmonic energy distribution; The harmonic detection module is used to run the target detection algorithm, identify the harmonic comb-like texture in the spectrogram, and output the time-frequency bounding box representing the pitch and duration. The sequence reasoning module is used to capture music theory logic dependencies in audio signals; The gated fusion decoding module is used to combine visual positioning accuracy and auditory logic probability to generate and correct MIDI files.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images