Audio spectrum searching method and system based on vector matching similarity

Abstract

The invention discloses an audio spectrum searching method and system based on vector matching similarity, and the method comprises the steps: (1) converting a music score into an MIDI format, and then converting the MIDI into a token sequence; (2) slicing the music score token sequence to generate a plurality of music score fragment token sequences; (3) training a similarity model for generating a feature vector of the token sequence of the music score fragment; (4) using the trained similarity model to generate feature vectors for all score fragment token sequences, and establishing a score fragment vector library; (5) transcribing the playing record of the user into an MIDI format in real time, and then converting the MIDI into a token sequence; (6) using the trained similarity model to generate feature vectors for the token sequence of the user playing recording; and (7) carrying out matching search in the music score fragment vector library to find out the music score fragment with the highest matching degree with the characteristic vector of the playing recording of the user. And the matching accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of audio information processing technology, and in particular to an audio spectrum search method and system based on vector matching similarity. Background Art

[0002] There is currently no technology that can search for music scores in real time based on the sound of playing, return results in seconds, and locate the position in the score. A similar existing technology is song recognition, which uses audio search and uses an audio fingerprinting algorithm to extract the fingerprint of each song and build a song fingerprint library. When a user requests a recording, song recognition will first extract the audio fingerprint of the recording, then compare and match the fingerprint to find the song with the highest match in the database. Song recognition technology requires a large amount of song audio to build the song fingerprint library, and cannot be used when there is only music score data but no music audio.

[0003] In order to solve the problem that users do not know the name of the music score when searching for music scores, but only know part of the melody of the music score, an audio music score search method and system based on vector matching similarity is urgently needed to be disclosed. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a technical solution: an audio spectrum search method and system based on vector matching similarity, comprising the following steps:

[0005] (1) Convert the music score into MIDI format, and then convert the MIDI into a token sequence;

[0006] (2) Slice the music score token sequence to generate multiple music score fragment token sequences;

[0007] (3) Train the similarity model to generate feature vectors of the token sequence of the music score;

[0008] (4) Use the trained similarity model to generate feature vectors for all music score fragment token sequences and establish a music score fragment vector library;

[0009] (5) Transcribe the user's performance recording into MIDI format in real time, and then convert the MIDI into a token sequence;

[0010] (6) Use the trained similarity model to generate a feature vector from the token sequence of the user's performance recording;

[0011] (7) Matching and searching in the music score segment vector library to find the music score segment with the highest matching degree with the feature vector of the user's performance recording.

[0012] Furthermore, the training step of the similarity model includes:

[0013] Generate training data, which is a triplet of three token sequences, including anchor samples, positive samples, and negative samples;

[0014] The positive samples are generated by performing a small random transformation on the anchor samples, and the negative samples are token sequences of music fragments that are unrelated to the anchor samples.

[0015] The similarity model adopts the structure of Embedding layer, LSTM layer, Attention layer and Dense layer, and the input is pitch token sequence and pitch time offset token sequence;

[0016] Using triplet loss as the loss function, the optimization goal is to make the similarity between the anchor sample and the positive sample greater than the similarity between the anchor sample and the negative sample. However, the larger the better, it is sufficient to exceed m. The main focus is on samples where the gap has not yet widened.

[0017] Furthermore, the calculation formula of the triplet loss is:

[0018] loss = max(0, cosine similarity(anchor sample, negative sample) - cosine similarity(anchor sample, positive sample) + m);

[0019] m is a constant greater than 0.

[0020] Furthermore, the music score segment vector library adopts a hierarchical index structure, including:

[0021] High-frequency music score fragment vector index;

[0022] Melody score fragment vector index;

[0023] Full score fragment vector index;

[0024] When searching, matches are performed in the high-frequency, melody, and full score fragment vector indexes in turn, with results in the high-frequency index being returned first.

[0025] The advantages of the invention compared with the prior art are:

[0026] 1. The implementation method of real-time score search for performance recordings uses the miditoken sequence feature vector matching search method to realize real-time audio score search and the user interaction process of real-time score search for performance recordings.

[0027] 2. High tolerance for performance errors. Deviations in rhythm, pitch, or speed are inevitable when playing, making traditional methods based on precise matching difficult to apply. This method uses a deep learning model to generate feature vectors and performs matching based on cosine similarity rather than strict alignment. This effectively tolerates subtle performance errors and improves matching accuracy.

[0028] 3. Efficient search, supporting real-time matching. Using hierarchical indexing (high-frequency library, melody library, and full library) to optimize the search process, prioritizing matching within the high-frequency music fragment library, reducing computational effort and improving response speed. Combined with the high-performance retrieval capabilities of the vector database, millisecond-level matching is achieved even with massive amounts of music data, meeting the needs of real-time interaction.

[0029] 4. Intelligent sorting improves user experience. It not only considers matching similarity, but also combines music popularity (such as popularity, user click-through rate) and user personalized weight (such as the songs that users often practice) to make the returned results more in line with user needs.

[0030] 5. Strong generalization capabilities, adaptable to a wide range of instruments and musical styles. Model training utilizes Triplet Loss, which clusters similar performance fragments in vector space and separates dissimilar fragments. This makes it suitable for a variety of instruments, including piano, guitar, and violin. Because the training data covers diverse styles such as classical, pop, and jazz, the system can adapt to diverse music score matching needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of an audio spectrum search method and system based on vector matching similarity of the present invention;

[0032] Figure 2 This is a structural diagram of the similarity model. DETAILED DESCRIPTION

[0033] The present invention will be described in further detail below with reference to the accompanying drawings.

[0034] The present invention will be described in detail with reference to the accompanying drawings.

[0035] The present invention provides an audio spectrum search method and system based on vector matching similarity in a specific implementation, comprising the following steps:

[0036] (1) Convert the music score into MIDI format, and then convert the MIDI into a token sequence;

[0037] (2) Slice the music score token sequence to generate multiple music score fragment token sequences;

[0038] (3) Train the similarity model to generate feature vectors of the token sequence of the music score;

[0039] (4) Use the trained similarity model to generate feature vectors for all music score fragment token sequences and establish a music score fragment vector library;

[0040] (5) Transcribe the user's performance recording into MIDI format in real time, and then convert the MIDI into a token sequence;

[0041] (6) Use the trained similarity model to generate a feature vector from the token sequence of the user's performance recording;

[0042] (7) Matching and searching in the music score segment vector library to find the music score segment with the highest matching degree with the feature vector of the user's performance recording.

[0043] Example 1: Real-time Piano Score Matching (Music Teaching Scenario)

[0044] background:

[0045] A user practiced "Für Alice" on the smart piano learning app, but due to unstable rhythm and some wrong notes, traditional methods made it difficult to accurately match the music score.

[0046] Implementation steps:

[0047] Music score preprocessing

[0048] Convert the score of "Für Alice" to MIDI and then parse it into a token sequence (pitch + time offset).

[0049] The token sequence is sliced ​​to generate multiple 16-bar segments, and the similarity model is trained to generate feature vectors, which are stored in the vector database.

[0050] User performance matching:

[0051] The user plays the first 8 measures (including some incorrect notes), and the system records them in real time and converts them into a MIDI token sequence.

[0052] The similarity model extracts the feature vector of the performance segment and searches first in the high-frequency music segment library.

[0053] The closest music score segment was matched (similarity 92%), and even if there was an error, the correct position could still be located.

[0054] Example 2: Searching for music scores by humming (music retrieval scenario)

[0055] background:

[0056] The user hummed a melody fragment from "Canon" in the hope of finding the corresponding music score, but the pitch of the humming was inaccurate and the rhythm was free.

[0057] Implementation steps:

[0058] Audio Processing:

[0059] The user's humming recording is converted into a MIDI approximate sequence through the Pitch Detection algorithm.

[0060] Due to the imprecise humming, the generated token sequence has time offset and pitch errors.

[0061] Fuzzy matching:

[0062] The system inputs the humming token sequence into the similarity model to generate a feature vector.

[0063] Searching in the melody fragment library, multiple "Canon" variation fragments were matched (similarity 85%-90%).

[0064] Sorting returns:

[0065] The top 3 results are returned based on the comprehensive matching degree and music popularity (the popularity of "Canon" is high).

[0066] The piano version and guitar version of the music score corresponding to the matching paragraph are displayed for the user to choose.

[0067] Effect:

[0068] Supports non-instrument input (such as human voice humming) to expand usage scenarios.

[0069] It can still recall the correct music score for inaccurate input, which is better than traditional melody retrieval technology.

[0070] As a further elaboration of the present invention, the training step of the similarity model includes:

[0071] Generate training data, which is a triplet of three token sequences, including anchor samples, positive samples, and negative samples;

[0072] The positive samples are generated by performing a small random transformation on the anchor samples, and the negative samples are token sequences of music fragments that are unrelated to the anchor samples.

[0073] The similarity model adopts the structure of Embedding layer, LSTM layer, Attention layer and Dense layer, and the input is pitch token sequence and pitch time offset token sequence;

[0074] Using triplet loss as the loss function, the optimization goal is to make the similarity between the anchor sample and the positive sample greater than the similarity between the anchor sample and the negative sample. However, the larger the better, it is sufficient to exceed m. The main focus is on samples where the gap has not yet widened.

[0075] As a further elaboration of the present invention, the calculation formula of the triplet loss is:

[0076] loss = max(0, cosine similarity(anchor sample, negative sample) - cosine similarity(anchor sample, positive sample) + m);

[0077] m is a constant greater than 0.

[0078] As a further elaboration of the present invention, the music score segment vector library adopts a hierarchical index structure, including:

[0079] High-frequency music score fragment vector index;

[0080] Melody score fragment vector index;

[0081] Full score fragment vector index;

[0082] When searching, matches are performed in the high-frequency, melody, and full score fragment vector indexes in turn, with results in the high-frequency index being returned first.

[0083] As a further elaboration of the present invention, the matching search ranking optimization step includes:

[0084] If the token sequence played by the user is short, it will be ranked comprehensively based on the fragment matching degree, music popularity and user usage weight;

[0085] If the token sequence played by the user is long and divided into multiple segments, it will be ranked comprehensively based on segment matching, continuity, score popularity and user usage weight.

[0086] As a further elaboration of the present invention, the score segment vectors are stored in a vector database, which supports massive distributed storage and high-performance vector retrieval. The vector database similarity metric is set to cosine similarity, which is consistent with the similarity model.

[0087] An audio spectrum search system, comprising:

[0088] Music score processing module, used to convert music scores into MIDI and generate token sequences and feature vectors;

[0089] Similarity model training module, used to train a deep learning model to generate token sequence feature vectors;

[0090] User performance processing module, used to convert user performance recordings into MIDI and generate token sequences and feature vectors;

[0091] A vector matching module is used to search for the music score segment with the highest matching degree in the music score segment vector library;

[0092] Hierarchical indexing module to optimize search efficiency and accuracy;

[0093] The sorting module is used to return results in a comprehensive sorting based on matching degree, continuity, popularity and weight.

[0094] As a further elaboration of the present invention, the system uses a vector database to store and manage feature vectors of music score segments.

[0095] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. An audio spectrum search method based on vector matching similarity, characterized in that: The following steps are involved: (1) Convert the music score into MIDI format, and then convert the MIDI into a token sequence; (2) Slice the music score token sequence to generate multiple music score fragment token sequences; (3) Train the similarity model to generate feature vectors of the token sequence of the music score; (4) Use the trained similarity model to generate feature vectors for all music score fragment token sequences and establish a music score fragment vector library; (5) Transcribe the user's performance recording into MIDI format in real time, and then convert the MIDI into a token sequence; (6) Use the trained similarity model to generate a feature vector from the token sequence of the user's performance recording; (7) Matching and searching in the music score segment vector library to find the music score segment with the highest matching degree with the feature vector of the user's performance recording.

2. The audio spectrum search method based on vector matching similarity according to claim 1, characterized in that: The training steps of the similarity model include: Generate training data, which is a triplet of three token sequences, including anchor samples, positive samples, and negative samples; The positive samples are generated by performing a small random transformation on the anchor samples, and the negative samples are token sequences of music fragments that are unrelated to the anchor samples. The similarity model adopts the structure of Embedding layer, LSTM layer, Attention layer and Dense layer, and the input is pitch token sequence and pitch time offset token sequence; Using triplet loss as the loss function, the optimization goal is to make the similarity between the anchor sample and the positive sample greater than the similarity between the anchor sample and the negative sample. However, the larger the better, it is sufficient to exceed m. The main focus is on samples where the gap has not yet widened.

3. The audio spectrum search method based on vector matching similarity according to claim 2, characterized in that: The calculation formula of the triplet loss is: loss = max(0, cosine similarity(anchor sample, negative sample) - cosine similarity(anchor sample, positive sample) + m); m is a constant greater than 0.

4. The audio spectrum search method based on vector matching similarity according to claim 1, characterized in that: The music score segment vector library adopts a hierarchical index structure, including: High-frequency music score fragment vector index; Melody score fragment vector index; Full score fragment vector index; When searching, matches are performed in the high-frequency, melody, and full score fragment vector indexes in turn, with results in the high-frequency index being returned first.

5. The audio spectrum search method based on vector matching similarity according to claim 1, characterized in that: The matching search sorting optimization step includes: If the token sequence played by the user is short, it will be ranked comprehensively based on the fragment matching degree, music popularity and user usage weight; If the token sequence played by the user is long and divided into multiple segments, it will be ranked comprehensively based on segment matching, continuity, score popularity and user usage weight.

6. The audio spectrum search method based on vector matching similarity according to claim 1, characterized in that: The score segment vectors are stored in a vector database, which supports massive distributed storage and high-performance vector retrieval. The vector database similarity metric is set to cosine similarity, which is consistent with the similarity model.

7. An audio spectrum search system, characterized in that: include: Music score processing module, used to convert music scores into MIDI and generate token sequences and feature vectors; Similarity model training module, used to train a deep learning model to generate token sequence feature vectors; User performance processing module, used to convert user performance recordings into MIDI and generate token sequences and feature vectors; A vector matching module is used to search for the music score segment with the highest matching degree in the music score segment vector library; Hierarchical indexing module to optimize search efficiency and accuracy; The sorting module is used to return results in a comprehensive sorting based on matching degree, continuity, popularity and weight.

8. The audio spectrum search system according to claim 7, characterized in that: The system uses a vector database to store and manage music score segment feature vectors.

Images