Song fragment search methods, devices, equipment, media, and products

Abstract

This application discloses a song fragment search method, apparatus, device, medium, and product thereof. The method includes: obtaining the encoding information corresponding to the song fragment to be searched submitted by the client; using a feature extraction model to extract a high-dimensional index vector representing deep semantic information of the song fragment at multiple scales based on the encoding information; determining the high-dimensional index vector of the song fragment to be searched that is not similar to the high-dimensional index vector in a first song fragment feature library as the matching vector; the high-dimensional index vector in the first song fragment feature library is used as the feature representation of the song fragment without a main melody extracted by the feature extraction model; and matching the target song fragment that is similar to the matching vector from the high-dimensional index vectors corresponding to multiple song fragments stored in a second song fragment feature library. Through the above process, a song search service can be quickly, efficiently, and accurately realized, allowing users to find target song fragments similar to the song fragment to be searched.

Description

Technical Field

[0001] This application relates to the field of music information retrieval technology, and in particular to a song fragment search method and corresponding apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rise of short videos, live streaming, and radio, the number of cover songs is increasing, and the scenarios requiring music recognition are becoming more complex. Compared to the original version, cover versions may differ from, or even be completely different from, other musical components such as timbre, fundamental frequency, rhythm, tempo, harmony, lyrics, singing style, and overall structure. Therefore, cover song recognition is a very challenging research task.

[0003] There are various existing technologies related to cover song recognition, and each of these technologies has certain shortcomings. For example: (1) Traditional landmark-based song recognition technology can only recognize songs with the same source version and cannot recognize the cover versions with certain differences mentioned above; (2) Traditional humming recognition technology based on melody matching can only recognize clean a cappella / humming and cannot recognize the cover versions with background accompaniment mentioned above; (3) Traditional cover song recognition technology mainly extracts audio features such as pitch class profile (PCP) and then uses algorithms such as dynamic programming to calculate the similarity distance between songs. Due to the diversity of cover versions, the above solutions are only applicable to cover versions with smaller modifications, have low accuracy, and are slow to recognize, making them unsuitable for searching massive amounts of music.

[0004] Song cover song recognition, especially short clips, presents a significant challenge. These clips typically last only a few seconds to a dozen seconds, offering limited information, yet the goal is to quickly identify the corresponding original song and return the result to the user. Due to this characteristic, short clip cover song recognition has always been a difficult problem in the industry. Firstly, there is a lack of effective training sets for short clip covers; secondly, the recognition rate is low (due to strong interference, distortion, and channel attenuation in the user's audio environment); and thirdly, short clips are prone to misjudgment (the clips are too short, lacking sufficient effective content; clips without melody, such as accompaniment, speech, background noise, or repetitive DJ background sounds, are all prone to misjudgment, with the absence of melody being the most significant source of misjudgment).

[0005] In summary, the technical solutions for song fragment recognition lack universal applicability, have low recognition accuracy, and are inefficient. Therefore, the applicant is trying to explore more effective technical solutions. Summary of the Invention

[0006] The primary objective of this application is to solve at least one of the aforementioned problems by providing a song fragment search method and corresponding apparatus, computer equipment, computer-readable storage medium, and computer program product.

[0007] To achieve the various objectives of this application, the following technical solution is adopted:

[0008] A song fragment search method provided for one of the purposes of this application includes the following steps:

[0009] Obtain the encoding information corresponding to the song fragment submitted by the client for searching;

[0010] A high-dimensional index vector representing deep semantic information at multiple scales of the song segment to be searched is extracted based on the encoded information using a feature extraction model that has been trained to convergence.

[0011] The high-dimensional index vector of the song segment to be searched that is not similar to the high-dimensional index vector in the first song segment feature library is determined as the matching vector; the high-dimensional index vector in the first song segment feature library is the feature representation of the song segment without main melody extracted by the feature extraction model;

[0012] The target song segment that is similar to the vector to be matched is selected from the high-dimensional index vectors corresponding to multiple song segments stored in the second song segment feature library; the high-dimensional index vectors in the second song segment feature library are the feature representations of song segments with a main melody extracted by the feature extraction model.

[0013] In a more detailed embodiment, obtaining the encoding information corresponding to the song segment submitted by the client for searching includes the following steps:

[0014] Receive a song segment search request submitted by the client and obtain the audio data of the song segment to be searched specified in the request;

[0015] The system checks whether the audio data contains human vocal information; if not, it terminates the subsequent execution.

[0016] The audio data is formatted to be between the minimum preset duration and the maximum preset duration.

[0017] The audio data is encoded to obtain the corresponding encoding information.

[0018] In a more detailed embodiment, when the feature extraction model is invoked, the following steps are performed:

[0019] Multiple convolutional blocks in the shared network of a feature extraction model that has been trained to convergence are used to sequentially perform multi-level feature extraction on the encoded information to obtain intermediate feature information that has extracted the deep semantic information of the encoded information.

[0020] After extracting features from the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks in the feature extraction model, the features are converted into output feature vectors of corresponding scales. The deep semantic information contained in the output feature vectors of each branch network is different.

[0021] The feature extraction model outputs the feature vectors of each branch network as the high-dimensional index vector.

[0022] In a further embodiment, the step of extracting features from the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks of the feature extraction model, and then converting it into an output feature vector of the corresponding scale, includes any two or more of the following steps:

[0023] The intermediate feature information is extracted by using multiple convolutional blocks in the first branch network to obtain global feature information, and the global feature information is pooled into a global scale output feature vector.

[0024] After extracting features from the intermediate feature information using multiple convolutional blocks in the second branch network, the network is divided into multiple parts by channel and pooled to obtain the corresponding channel-scale output feature vector.

[0025] After extracting features from the intermediate feature information using multiple convolutional blocks in the third branch network, the network is divided into multiple parts according to frequency band and pooled to obtain the corresponding output feature vector at the frequency band scale.

[0026] In a preferred embodiment, when the first branch network performs the pooling operation, it employs mean pooling and / or max pooling to obtain one or two output feature vectors at the global scale; when the second branch network performs the pooling operation, it employs mean pooling for one or more channels to obtain one or more output feature vectors at the channel scale; and when the third branch network performs the pooling operation, it employs mean pooling for one or more frequency bands to obtain one or more output feature vectors at the frequency band scale.

[0027] In a further embodiment, when the convolutional block is invoked, the following steps are performed:

[0028] The information in the input is subjected to convolutional transformation to obtain transformation feature information;

[0029] The transformed feature information is combined into spliced ​​feature information after being processed by instance normalization and batch normalization respectively, and the spliced ​​feature information is activated and output.

[0030] The concatenated feature information of the activation output is processed through multiple convolution operations and batch normalization to obtain residual information;

[0031] The residual information is superimposed onto the input information to activate the output.

[0032] In a preferred embodiment, in the shared network, at least one of the convolutional blocks applies an attention module to extract key information from the audio data of the song segment, wherein the attention module is a spatial attention module or a channel attention module.

[0033] In a preferred embodiment, the source of the encoded information is any one of the time-spectrum information, Mel spectrum information, CQT filter information, pitch contour information, and Chroma feature information of the corresponding audio data.

[0034] In a more detailed embodiment, determining the high-dimensional index vector of the searched song segment that is not similar to the high-dimensional index vector in the first song segment feature library as the matching vector includes the following steps:

[0035] The first song fragment feature library is invoked. The first song fragment feature library stores multiple preset high-dimensional index vectors corresponding to song fragments without a main melody. The high-dimensional index vectors are obtained by the feature extraction model by extracting the corresponding song fragments.

[0036] Calculate the similarity between the high-dimensional index vector of the song segment to be searched and each high-dimensional index vector in the first song segment feature library, and obtain the corresponding similarity value;

[0037] Determine whether there is a song segment without a main melody whose similarity value exceeds a preset threshold. If it exists, determine that the song segment to be searched does not contain a song melody and terminate the process. If it does not exist, determine that the song segment to be searched contains a song melody and determine its high-dimensional index vector as the matching vector.

[0038] In a more detailed embodiment, the step of matching the target song segment that is similar to the vector to be matched from the high-dimensional index vectors corresponding to multiple song segments stored in the second song segment feature library includes the following steps:

[0039] The second song fragment feature library is invoked to obtain the high-dimensional index vector corresponding to each song fragment. The high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of a song fragment at different scales as a whole.

[0040] Calculate the similarity between the vector to be matched and each high-dimensional index vector in the second song segment feature library to obtain the corresponding similarity sequence;

[0041] The song segment corresponding to the largest similarity value in the similarity sequence that exceeds a preset threshold is identified as the target song segment that is similar to the song segment to be searched.

[0042] The link to the song containing the target song fragment is retrieved from the music library and pushed to the client device.

[0043] In an extended embodiment, the training process of the feature extraction model includes the following iterative training steps:

[0044] A training sample is retrieved from the training set, and the encoding information of the training sample is determined. The training sample is a song segment containing the melody of a song that is pre-collected according to a predetermined duration.

[0045] The encoded information is input into the feature extraction model and trained to obtain the corresponding output feature vectors.

[0046] Each output feature vector is classified and predicted to map to the corresponding classification label.

[0047] The loss value of the feature extraction model is calculated using the supervision label corresponding to the training sample and the classification label, and the feature extraction model is updated according to the gradient based on the loss value;

[0048] Determine whether the loss value has reached a preset threshold. If it has not reached the preset threshold, call the next training sample in the training set to continue iterative training of the feature extraction model until the loss value reaches the preset threshold.

[0049] In an extended embodiment, the song fragment search method includes the following steps for constructing a second song fragment feature library:

[0050] The audio data of candidate songs is obtained from the music library, and the audio data is divided into multiple song segments according to the preset duration to obtain the audio data corresponding to the multiple song segments.

[0051] The audio data of each song segment is encoded to obtain its encoding information;

[0052] For each song segment, the feature extraction model is invoked to extract deep semantic information of its encoded information at multiple scales, thereby obtaining its high-dimensional index vector.

[0053] For each song segment, calculate the similarity between its high-dimensional index vector and each high-dimensional index vector in the first song segment feature library, and obtain the corresponding similarity value;

[0054] For each song segment, determine whether there is a song segment without a main melody whose similarity value exceeds a preset threshold. If it exists, discard the song segment. If it does not exist, store the mapping relationship data between the song segment and its high-dimensional index vector in the second song segment feature library.

[0055] A song fragment search device provided for one of the purposes of this application includes: a fragment encoding module, a semantic extraction module, a fragment filtering module, and a fragment matching module. The fragment encoding module is used to acquire encoding information corresponding to a song fragment submitted by a client for searching. The semantic extraction module is used to extract high-dimensional index vectors representing deep semantic information at multiple scales of the song fragment to be searched based on the encoding information using a feature extraction model trained to convergence. The fragment filtering module is used to determine high-dimensional index vectors of song fragments to be searched that are not similar to high-dimensional index vectors in a first song fragment feature library as matching vectors. The high-dimensional index vectors in the first song fragment feature library are feature representations of song fragments without a main melody extracted by the feature extraction model. The fragment matching module is used to match target song fragments similar to the matching vectors from multiple high-dimensional index vectors corresponding to song fragments stored in a second song fragment feature library. The high-dimensional index vectors in the second song fragment feature library are feature representations of song fragments with a main melody extracted by the feature extraction model.

[0056] A computer device provided for one of the purposes of this application includes a central processing unit and a memory, the central processing unit being configured to invoke and run a computer program stored in the memory to perform the steps of the song fragment search method described in this application.

[0057] A computer-readable storage medium is provided for another purpose of this application, which stores, in the form of computer-readable instructions, a computer program implemented according to the described song fragment search method, which, when invoked by a computer, performs the steps included in the method.

[0058] A computer program product provided for another purpose of this application includes a computer program / instructions that, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0059] Compared with existing technologies, the advantages of this application are as follows:

[0060] First, this application uses the encoded information obtained from the encoding of the song segment to be searched, and employs a pre-trained feature extraction model to convergence to obtain a high-dimensional index vector representing the style-invariant features of the deep semantic information. Based on a first song segment feature library used to store high-dimensional index vectors of song segments without a main melody, it filters and determines whether the song segment to be searched belongs to the category of song segments without a main melody. The high-dimensional index vectors of the song segments to be searched that are determined to have a main melody are determined as the matching vectors. Then, similar target song segments are matched in a second song segment feature library. Since the high-dimensional index vectors of the song segments in the second song segment feature library are also extracted using the same feature extraction model, they all realize the deep semantic representation of the corresponding audio data of the song segments at different semantic scales. Therefore, semantic matching can be performed with the song segment to be searched, thereby accurately matching similar song segments based on semantics. This end-to-end model architecture enables client devices to quickly query similar songs.

[0061] Secondly, this application addresses the characteristics of short song fragments and insufficient information by employing multi-scale features to enhance the capture of semantic information and by using a feature library of first song fragments without a main melody for filtering and preprocessing the song fragments to be searched. The former can provide rich verification information for the search matching process by enhancing the semantic information representation capability, while the latter can achieve the goal of searching and matching only song fragments with melody by filtering out redundant information without melody. The combination of the two comprehensively improves the accuracy of the search matching results.

[0062] Furthermore, since this application implements multi-scale feature extraction of deep semantic information of the audio data of song segments in its feature extraction model, the obtained high-dimensional index vector is more representative, such as representing global feature information, salient feature information, channel feature information, frequency band feature information, etc. of the audio data of the song segment. This enables a more effective indexing of the audio data of the corresponding song segment. Based on this, the song segment search task can be performed to obtain a more accurate and efficient matching effect, which can be used to serve various business needs such as cover song recognition, song recognition, humming recognition, and infringement comparison.

[0063] Furthermore, this application, based on end-to-end representation learning capabilities and supplemented by a retrieval matching mechanism, can achieve significant scalability. It can be deployed in the backend of online music service platforms to implement standardized interfaces, thereby serving the needs of various application scenarios, providing comprehensive and multi-purpose open services, and enhancing the economic advantages of the platform's music information retrieval. Attached Figure Description

[0064] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0065] Figure 1 This is a flowchart illustrating a typical embodiment of the song fragment search method of this application;

[0066] Figure 2 This is a flowchart illustrating the process of obtaining encoded information in one embodiment of this application;

[0067] Figure 3 This is a flowchart illustrating the operation of a feature extraction model in one embodiment of this application.

[0068] Figure 4 This is a schematic diagram of the network architecture of a feature extraction model in one embodiment of this application;

[0069] Figure 5 This is a schematic diagram of the network architecture of the feature extraction model in another embodiment of this application;

[0070] Figure 6 This is a flowchart illustrating the working process of the residual convolutional block used in the feature extraction model of this application.

[0071] Figure 7 This is a flowchart illustrating the specific process of filtering the song fragments to be searched using the first song fragment feature library in an embodiment of this application.

[0072] Figure 8 This is a flowchart illustrating the specific process of matching the song fragments to be searched using the second song fragment feature library in an embodiment of this application.

[0073] Figure 9 This is a flowchart illustrating the process of training the feature extraction model of this application.

[0074] Figure 10 This is a schematic diagram of the classification model that the feature extraction model of this application is connected to during the training phase.

[0075] Figure 11 This is a flowchart illustrating the process of constructing the second song fragment feature library in an embodiment of this application;

[0076] Figure 12 This is a schematic block diagram of the song fragment search device of this application;

[0077] Figure 13 This is a schematic diagram of the structure of a computer device used in this application. Detailed Implementation

[0078] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0079] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0080] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0081] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant) that may include a radio frequency receiver, pager, internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0082] The hardware referred to by the names "server," "client," and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0083] It should be noted that the concept of "server" used in this application can also be extended to the case of server clusters. Based on the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method in this application.

[0084] One or more of the technical features of this application, unless explicitly specified herein, can be deployed on a server and accessed by a client remotely calling the online service interface provided by the server, or can be directly deployed and run on a client for access.

[0085] Unless otherwise specified, the neural network models referenced or potentially referenced in this application may be deployed on a remote server and invoked remotely on the client, or deployed on a client with the capability to invoke directly. In some embodiments, when running on the client, the corresponding intelligence may be acquired through transfer learning in order to reduce the requirements on the client's hardware resources and avoid excessive consumption of the client's hardware resources.

[0086] Unless otherwise specified, all data involved in this application may be stored remotely on a server or on a local terminal device, as long as it is suitable for use by the technical solution of this application.

[0087] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0088] Unless otherwise expressly stated, the various embodiments disclosed in this application can be combined in a cross-cutting manner to flexibly construct new embodiments, as long as such combination does not depart from the inventive spirit of this application and can meet the needs of the prior art or solve a certain deficiency in the prior art. Those skilled in the art should be aware of such modifications.

[0089] The song fragment search method of this application can be programmed into a computer program product and deployed on a server to run. In this way, the client can access the interface opened by the computer program product after it runs in the form of a web program or application program, and realize human-computer interaction with the process of the computer program product through a graphical user interface.

[0090] Please see Figure 1 The song fragment search method of this application, in its typical embodiment, includes the following steps:

[0091] Step S1100: Obtain the encoding information corresponding to the song fragment to be searched submitted by the client:

[0092] The technical solution of this application can be deployed on the server of an online music platform, providing a variety of song search services to a massive number of platform users, including but not limited to song recognition, humming recognition, and cover song recognition.

[0093] In the song recognition service, users can record a segment of audio data, usually from an external source, on their client and submit it to the server as a song fragment to be searched. The server then uses this data to find the original song or a corresponding cover song.

[0094] In the humming-based music recognition service, users can record a melody sung by a human voice on their client to obtain the corresponding audio data. Similarly, this data is submitted to the server as a song fragment to be searched. The server then finds songs with the same melody based on the similarity of the melody.

[0095] In the cover song recognition service, users can specify or submit the audio data of a song to the server on their client as a song segment to be searched. The server then uses this data to find the same song or other different versions of the song, thereby determining whether there is a cover song or a similar song relationship between the two.

[0096] The song fragments referred to in this application can usually be predefined as song fragments whose duration falls within a specific duration range. In practice, song fragments can be defined based on the timestamps of the lyrics combined with a specific duration range, thereby making the song fragments more independent.

[0097] After the song fragment submitted by the user arrives at the server, the server will encode it accordingly in order to obtain its corresponding encoding information. During the encoding process, the audio data of the song fragment to be searched can be adaptively processed according to the specific situation.

[0098] The audio data of the song segment to be searched submitted to the server can be in any format such as MP3, WMA, M4A, WAV, etc., or it can be audio data obtained by separating the audio from various video files. The audio data of the song segment to be searched is usually composed of multiple speech data packets in the time domain. Based on this, the speech data packets are transformed according to the specific encoding information type to obtain the corresponding encoding information.

[0099] The encoded information primarily describes style-invariant features in the audio data of a song segment. It can be of various types, including but not limited to time-spectrum information, Mel-spectrum information, CQT filter information, pitch contour information, and chroma feature information extracted from the speech data packets of the audio data. This type of information can be encoded using appropriate algorithms to obtain the corresponding type of encoded information. In this application, any of the above types of encoded information can be used to achieve feature extraction. In practice, it is recommended to use the experimentally optimal CQT filter information to obtain the encoded information.

[0100] Those skilled in the art will understand that all the above-mentioned encoded information can be encoded using corresponding algorithms. During the encoding process, the audio data of the song segment needs to undergo conventional processing such as pre-emphasis, framing, and windowing before time-domain or frequency-domain analysis is performed, thus achieving speech signal analysis. Pre-emphasis aims to enhance the high-frequency components of the speech signal, smoothing the spectrum; pre-emphasis is typically achieved using a first-order high-pass filter. Before analyzing the speech signal, it needs to be framed, usually with each frame length set to 20ms. Considering frame shift, there can be a 10ms overlap between adjacent frames. Framing can be achieved by windowing the speech signal. Different window selections will affect the results of speech signal analysis; a common approach is to use the window function corresponding to the Hamm window to perform the windowing operation.

[0101] Based on the preprocessing required for speech signal analysis of the audio data of the song segment, further time-domain and frequency-domain analysis can be performed to achieve encoding and obtain the corresponding encoded information:

[0102] Regarding the aforementioned time-spectrum information, the speech data of each speech data packet in the time domain is pre-emphasized, framed, windowed, and transformed into the frequency domain using a short-time Fourier transform (STFT), thereby obtaining the corresponding data of the spectrogram and thus constituting the aforementioned time-spectrum information.

[0103] The Mel spectrum information can be obtained by filtering the time-spectrum information using a Mel-scale filter bank. Similarly, the corresponding Mel cepstral information can be obtained by taking the logarithm of the Mel spectrum information and performing a DCT transform. It can be understood that Mel spectrum information and its cepstral information can better describe the stylistically invariant features of a song, such as pitch, intonation, and timbre.

[0104] Regarding the CQT filtering information, since all notes in music are composed of several octaves of equal temperament (12-tone equal temperament), corresponding to the twelve semitones of an octave on a piano, the frequency ratio between adjacent semitones is 2. 1 / 12Clearly, for two octaves of the same pitch, the higher octave has twice the frequency of the lower octave. Therefore, in music, sound is distributed exponentially, but the audio spectrum obtained by Fourier transform is linearly distributed. The frequency points of the two cannot be directly correlated, which can lead to errors in the estimation of certain pitch frequencies. Therefore, the CQT (Constant Q Transform) time-frequency transform algorithm can be used to replace the Fourier transform for speech analysis. CQT refers to a filter bank where the center frequency is distributed exponentially, with different filtering bandwidths but a constant center frequency to bandwidth ratio of Q. Unlike the Fourier transform, the horizontal axis frequency of its spectrum is not linear but based on log2, and the filter window length can be adjusted according to different spectral line frequencies to obtain better performance. Since CQT has the same frequency distribution as the musical scale, the amplitude values ​​of the musical signal at each note frequency can be directly obtained by calculating the CQT spectrum of the music signal, making it more suitable for music signal processing. Therefore, this embodiment recommends using this information to perform corresponding encoding to obtain the corresponding encoded information, which is then used as the input to the neural network model of this application.

[0105] The pitch profile information mentioned can include either PCP (Pitch Class Profile) or HPCP (Harmonic Pitch Class Profile), aiming to extract the corresponding pitch sequence from the audio data of a song segment. After normalization, merging, and segmentation, it is transformed into a melody profile sequence. Then, the standard pitch difference generated using standard pitch is converted into the corresponding feature representation. The encoded information constructed based on pitch profile information has good robustness to environmental noise.

[0106] The chroma feature information mentioned above is a collective term for chroma vectors and chromagrams. A chroma vector is a vector containing 12 elements, each representing the energy of one of the 12 pitch levels within a time period (e.g., one frame). The energy of the same pitch level across different octaves is accumulated. The chromagram is a sequence of chroma vectors. Specifically, after performing a short-time Fourier transform on the audio data packets of a song segment from the time domain to the frequency domain, some noise reduction processing is performed, followed by tuning. The absolute time is converted into frames according to the length of the selected window, and the energy of each pitch within each frame is recorded, becoming a pitch map. Based on the pitch map, the energy (measured by loudness) of notes at the same time, pitch level, and different octaves is superimposed onto the elements of that pitch level in the chroma vector, forming the chroma map. The data corresponding to this chroma map is the aforementioned chroma feature information.

[0107] Any of the specific encoding information described above can be used as input to the feature extraction model of this application. To facilitate processing by the feature extraction model, the encoding information can be organized according to a certain preset format. For example, the encoding information corresponding to each speech packet can be organized into a row vector. For the entire encoded audio data, the row vectors of each speech packet can be organized into rows in time sequence to obtain a two-dimensional matrix as its complete encoding information. Such methods can be preset to suit the feature extraction model and can be flexibly implemented by those skilled in the art.

[0108] It should be noted that the encoding principle mentioned here is adopted to adapt to the input requirements of the feature extraction model, and therefore applies to all audio data that the feature extraction model needs to process for representation learning. For example, in this application, it applies not only to the audio data of the song segment to be searched, but also to the processing of the audio data of the training samples by the feature extraction model during the training phase. Similarly, it also applies to the processing of the corresponding audio data of the song segments included in the first and second song segment feature libraries of this application. Those skilled in the art will understand this.

[0109] Step S1200: Using a feature extraction model trained to convergence, extract a high-dimensional index vector representing the deep semantic information of the song segment to be searched at multiple scales based on the encoded information.

[0110] This application presents a feature extraction model based on a convolutional neural network model for extracting deep semantic information from song fragments. The model is pre-trained to convergence and, after training, acquires the ability to extract deep semantic information from the audio data of song fragments at multiple scales based on the encoded information. This enables the model to learn the representation of style-invariant features of the audio data of the corresponding song fragments, which can then be used for querying, retrieving, and matching between songs.

[0111] The feature extraction model of this application is implemented to extract deep semantic information at multiple scales from the same audio data. This deep semantic information is represented as one or more high-dimensional index vectors to achieve feature representation of the audio data from multiple different aspects and / or angles. The high-dimensional index vector is essentially a high-dimensional vector, which serves as an index representation of the encoded information of the corresponding audio data at the semantic level. These different scales include global scales based on encoded information, or scales based on frequency bands, channels, etc., obtained through feature extraction. For a song segment, selecting any number of deep semantic information at two or more scales corresponding to its encoded information and representing them as high-dimensional index vectors enables feature representation of the multi-scale deep semantic information of the corresponding song segment.

[0112] Once the feature extraction model implemented based on the above principles is trained to convergence, it can open a service interface for the technical solution of this embodiment to call. The encoded information of the song segment to be searched is fed into it, and the feature extraction model performs feature extraction based on the encoded information to obtain the high-dimensional index vector corresponding to the song segment to be searched.

[0113] It should be understood that, since the feature extraction model can extract deep semantic information from song segments at multiple scales, different organizational forms can be used when converting this deep semantic information at different scales into the high-dimensional index vector. For example, the high-dimensional index vector can be represented as a single high-dimensional vector, which typically represents the deep semantic information of a song segment as a whole; or, the high-dimensional index vector can be represented as multiple discrete high-dimensional vectors according to the scale correspondence, with each high-dimensional vector corresponding to a scale. In any case, those skilled in the art can flexibly organize these high-dimensional vectors according to the needs of the actual scale semantic information, so as to facilitate the retrieval of the representation data of the overall deep semantic information of the song.

[0114] In this step, the feature extraction model extracts features from the encoded information of the song segment to be searched, and finally obtains the high-dimensional index vector corresponding to the song segment to be searched, which can be used for subsequent similarity matching.

[0115] Step S1300: Determine the high-dimensional index vector of the song segment to be searched that is not similar to the high-dimensional index vector in the first song segment feature library as the matching vector; the high-dimensional index vector in the first song segment feature library is the feature representation of the song segment without a main melody extracted by the feature extraction model:

[0116] This application pre-constructs a first song fragment feature library, which stores a massive amount of song fragments without a main melody and the mapping relationship data between them and their corresponding high-dimensional index vectors. The high-dimensional index vectors are obtained by extracting the corresponding song fragments without a main melody using the feature extraction model of this application. These song fragments without a main melody can be pre-collected from songs without a main melody. These songs can be purely instrumental accompaniment songs, or they can be intros, transitions, or endings of songs with a main melody, etc. They can be detected and extracted using public recognition techniques or obtained through manual processing.

[0117] To reduce the misjudgment rate of matching similar song segments to the searched song segment, the high-dimensional index vector of the searched song segment is compared with the high-dimensional index vectors of each melody-less song segment in the first song segment feature library. Then, based on the similarity value, it is determined whether there is a melody-less song segment similar to the searched song. If there is, it means that the searched song segment does not contain the main melody of the song, so there is no need to continue with the subsequent steps, and the corresponding client device can be directly notified. If there is no melody, it means that the searched song segment contains the main melody of the song, and its high-dimensional index vector can be determined as the matching vector. Then, the matching vector is used to search for similar target song segments for the searched song segment.

[0118] When performing similarity calculations, any algorithm suitable for calculating the similarity distance between data can be used, such as the cosine similarity algorithm, the Euclidean distance algorithm, the Pearson coefficient algorithm, the Jaccard similarity algorithm, or the nearest neighbor search algorithm. Those skilled in the art can implement this flexibly.

[0119] Step S1400: Match the target song segment that is similar to the vector to be matched from the high-dimensional index vectors corresponding to multiple song segments stored in the second song segment feature library; the high-dimensional index vectors in the second song segment feature library are the feature representations of song segments with a main melody extracted by the feature extraction model.

[0120] This application pre-constructs a second song fragment feature library. The construction process of the second song fragment feature library is detailed in the corresponding embodiments below. In summary, the second song fragment feature library stores massive amounts of mapping data between song fragments with a main melody and their corresponding high-dimensional index vectors. The high-dimensional index vectors are obtained by extracting the corresponding song fragments with a main melody using the feature extraction model of this application. These song fragments with a main melody can be pre-obtained from songs in a music library, for example, by extracting fragments corresponding to the lyrics of a song using timestamps in the lyrics and combining them with a specific duration range. Alternatively, they can be obtained through manual identification and processing.

[0121] In an exemplary online music service platform, a music library is prepared, which stores a massive amount of audio data corresponding to songs, providing song fragment materials for the second song fragment feature library. Therefore, for the song fragment to be searched, the song fragments pointed to by the second song fragment feature library are candidate song fragments in the search process, and these candidate song fragments can correspond to the target song in the music library.

[0122] The song fragment feature library, based on the high-dimensional index vector of the song fragment to be searched (i.e., the matching vector) and the corresponding high-dimensional index vectors of each song fragment in the second song fragment feature library, can apply a preset similarity calculation formula to calculate the similarity value between the song fragment to be searched and each song fragment. The similarity calculation formula can be implemented using any algorithm suitable for calculating the similarity distance between data, such as cosine similarity algorithm, Euclidean distance algorithm, Pearson coefficient algorithm, Jaccard similarity algorithm, nearest neighbor search algorithm, etc., and those skilled in the art can implement it flexibly. After the similarity calculation, a similarity sequence is obtained between the matching vector of the song fragment to be searched and the corresponding high-dimensional index vectors of each song fragment in the second song fragment feature library. This similarity sequence stores the similarity values ​​between the song fragment to be searched and each song fragment in the second song fragment feature library.

[0123] After determining the similarity sequence corresponding to the song segment to be searched, a preset threshold, which can be an empirical or experimental threshold, can be used to filter the similarity values ​​in the similarity sequence, filtering out all elements whose similarity values ​​exceed the preset threshold. If the number of elements exceeding the preset threshold is 0, it indicates that there are no similar song segments in the second song segment feature library that are similar to the song segment to be searched. If multiple similarity values ​​are obtained after filtering, only the song segment corresponding to the largest similarity value can be selected as the similar song segment corresponding to the song segment to be searched, that is, the target song segment obtained by searching for a song.

[0124] At this point, based on the song fragment submitted or specified by the client device, this application identifies a target song fragment that is semantically similar to it. Furthermore, it can retrieve the corresponding audio data of the song to which the target song fragment belongs from the music library, obtain the access link to the song's audio data, encapsulate it into a playable page or playable data containing the link, and push it to the corresponding client device for further user access. Therefore, whether it's song recognition by listening, song recognition by humming, or cover song recognition, users only need to submit or specify the corresponding song fragment to be searched to obtain similar songs.

[0125] In other embodiments disclosed subsequently in this application, the process of searching for songs by song has various variations, which will not be discussed here. It is only based on the description of this typical embodiment that it is understood that the implementation of this application has numerous advantages, including but not limited to the following aspects:

[0126] First, this application uses the encoded information obtained from the encoding of the song segment to be searched, and employs a pre-trained feature extraction model to convergence to obtain a high-dimensional index vector representing the style-invariant features of the deep semantic information. Based on a first song segment feature library used to store high-dimensional index vectors of song segments without a main melody, it filters and determines whether the song segment to be searched belongs to the category of song segments without a main melody. The high-dimensional index vectors of the song segments to be searched that are determined to have a main melody are determined as the matching vectors. Then, similar target song segments are matched in a second song segment feature library. Since the high-dimensional index vectors of the song segments in the second song segment feature library are also extracted using the same feature extraction model, they all realize the deep semantic representation of the corresponding audio data of the song segments at different semantic scales. Therefore, semantic matching can be performed with the song segment to be searched, thereby accurately matching similar song segments based on semantics. This end-to-end model architecture enables client devices to quickly query similar songs.

[0127] Secondly, this application addresses the characteristics of short song fragments and insufficient information by employing multi-scale features to enhance the capture of semantic information, and by using a feature library of first song fragments without a main melody for filtering and preprocessing the song fragments to be searched. The former can provide rich verification information for the search matching process by enhancing the semantic information representation capability, while the latter can achieve the goal of searching and matching only song fragments with a main melody by filtering out redundant information without a main melody. The combination of the two comprehensively improves the accuracy of the search matching results.

[0128] Furthermore, since this application implements multi-scale feature extraction of deep semantic information of the audio data of song segments in its feature extraction model, the obtained high-dimensional index vector is more representative, such as representing global feature information, salient feature information, channel feature information, frequency band feature information, etc. of the audio data of the song segment. This enables a more effective indexing of the audio data of the corresponding song segment. Based on this, the song segment search task can be performed to obtain a more accurate and efficient matching effect, which can be used to serve various business needs such as cover song recognition, song recognition, humming recognition, and infringement comparison.

[0129] Furthermore, this application, based on end-to-end representation learning capabilities and supplemented by a retrieval matching mechanism, can achieve significant scalability. It can be deployed in the backend of online music service platforms to implement standardized interfaces, thereby serving the needs of various application scenarios, providing comprehensive and multi-purpose open services, and enhancing the economic advantages of the platform's music information retrieval.

[0130] Please see Figure 2In a more detailed embodiment, step S1100, obtaining the encoding information corresponding to the song segment to be searched submitted by the client, includes the following steps:

[0131] Step S1110: Receive the song fragment search request submitted by the client, and obtain the audio data of the song fragment to be searched specified in the request:

[0132] In this embodiment, the user records or specifies an audio segment on the song segment search page displayed on their client device. If necessary, this audio segment can be pre-defined by the application for a preset duration. This triggers a song segment search request, using the audio data as the audio data required for the searched song segment. The request is submitted to the server providing the song search service. The server parses the request to obtain the corresponding audio data for the searched song segment. The audio data submitted by the client can be either audio data with accompaniment and a vocal melody, or audio data consisting solely of a human humming the song's melody. Therefore, it can be data extracted from a song file or audio data sampled from the client's recording device. The song file can be either locally stored on the client or a remote file provided by the client via an access link.

[0133] Step S1120: Detect whether the audio data contains human vocal information; if not, terminate subsequent execution.

[0134] To improve matching accuracy, the server can preprocess the received audio data. For example, it can use a VAD logic module to detect whether the audio data contains the main melody of a human vocal performance. If the audio data does not contain the main melody, the subsequent execution of this application can be terminated, and a corresponding notification can be directly returned to the client device. When it is confirmed that audio data corresponding to the main melody exists, subsequent steps can continue. The VAD logic module can also be executed using various existing technologies well known to those skilled in the art; it is recommended to use an end-to-end pre-trained neural network model to achieve convergence.

[0135] Step S1130: Format the audio data into audio data between the minimum preset duration and the maximum preset duration:

[0136] Although the duration of user-submitted audio data can be standardized on the client side to conform to a song segment within the expected length range, further standardization can be achieved on the server side through this step by performing further preprocessing on the user-submitted audio data to expand compatibility with user-submitted audio data.

[0137] Specifically, the server can process the audio data of the song segment to be searched into a preset specific duration range, such as 10 to 15 seconds. The corresponding audio data within this range can be determined using the detection methods from the previous step to include the main melody of the song, i.e., the vocal part. If the total duration of the song segment submitted by the user does not reach the minimum preset duration of this specific duration range, the matching process for that song segment can be terminated. If the total duration of the song segment submitted by the user exceeds the maximum preset duration of the specific duration range, the audio data containing the main melody of the song can be extracted from the corresponding specific duration range as the audio data for encoding.

[0138] Step S1140: Encode the audio data to obtain corresponding encoding information:

[0139] For audio data that has already undergone voice detection and formatting, the encoding principle described above in this application can be used to encode it and obtain the corresponding encoding information. As mentioned earlier, it is recommended to use CQT filtering information to construct the corresponding encoding information for the audio data of the song segment to be searched.

[0140] This embodiment preprocesses the audio data of the song segment to be searched, thereby filtering out some audio data without voice and standardizing the audio data. This avoids the corresponding online service frequently responding to invalid song segments to be searched, which can improve the matching accuracy of song search and save the system overhead of backend services.

[0141] Please see Figure 3 In a more detailed embodiment, when the feature extraction model is invoked, the following steps are performed:

[0142] Step S2100: Using multiple convolutional blocks in the shared network of the feature extraction model that has been trained to convergence, multi-level feature extraction is performed on the encoded information to obtain intermediate feature information that has extracted the deep semantic information of the encoded information.

[0143] The feature extraction model described herein is constructed based on the multi-branch concept of this application and can be flexibly modified to adapt to the requirements of different embodiments of this application. In a typical embodiment, the feature extraction model is as follows: Figure 4As shown in the principle block diagram, the feature extraction model consists of a shared network and multiple branch networks. The shared network includes multiple convolutional blocks for progressively extracting deep semantic information from encoded information to obtain intermediate feature information. The multiple branch networks each perform different types of deep semantic information extraction based on the intermediate feature information to obtain corresponding output feature information. Each branch network contains a portion of the same structure, which includes multiple convolutional blocks for progressively extracting deep semantic information. After the last convolutional block is output, it can be processed differently according to the different functions of each branch network.

[0144] The convolutional blocks can be implemented using convolutional layers based on CNNs or RNNs, preferably convolutional blocks based on residual convolution principles. To achieve contextual analysis and extract key information from the audio data of song segments, an attention mechanism can be applied to any of the convolutional blocks, adding a corresponding attention module, specifically a Spatial Attention Module (SAM) or a Channel Attention Module (CAM). In an enhanced embodiment, instance normalization (IN) and batch normalization (BN) operations are applied to the convolutional blocks to divide the input information into two parts. One part undergoes instance normalization to learn style-invariant features, while the other part undergoes batch normalization for normalization. Therefore, the commonly known IBN architecture is applied. Using this architecture, highly diverse musical attribute invariant features of the song segment's audio data, such as notes, rhythm, and timbre, can be learned, while preserving version information.

[0145] Therefore, it is easy to understand that the feature extraction model adapts to different application scenarios, enables different branch networks, and is trained to convergence using a pre-selected training set to obtain the corresponding feature extraction capabilities. This makes it suitable for performing tasks corresponding to the application scenario, extracting the output feature information corresponding to the audio data of the input song segment from the encoded information of the audio data of that song segment. The training process of the feature extraction model will be given in the exemplary embodiments of this application, and will not be described here.

[0146] In this step, as Figure 4 In the architecture shown, the encoded information is processed by multiple convolutional blocks of the shared network for feature extraction, and in particular, the key information is extracted by the last convolutional block. This yields intermediate feature information containing the key information of the encoded information. This intermediate feature information is then divided into multiple outputs to the multiple branch networks so that different deep semantic information can be extracted from different perspectives in each branch network.

[0147] Step S2200: After extracting features from the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks of the feature extraction model, the extracted features are converted into output feature vectors of corresponding scales. The deep semantic information contained in the output feature vectors of each branch network is different.

[0148] As mentioned above, Figure 4 In the architecture shown, the various branch networks can be flexibly selected and combined. Therefore, the specific number of branch networks can be determined based on the resulting architecture. The intermediate feature information output by the shared network is then input into each of the branch networks for further feature extraction processing.

[0149] according to Figure 4 The architecture shown has two convolutional blocks in each branch network that belong to the same structural part. The two convolutional blocks extract features from the output in sequence. The extracted output can then be processed differently according to the specific structure of different branch networks.

[0150] Specifically, different branch networks, adapting to the different deep semantic information they extract, can perform different processing on parts with different structures. For example, one branch network can be subjected to max pooling or mean pooling output; one branch network can be connected to a Dropout layer to randomly discard redundant features before max pooling output; in another branch network, the intermediate feature information output from the last convolutional block can be segmented into equal channels and then mean pooled before output; in yet another branch network, the intermediate feature information output from the last convolutional block can be segmented into equal frequency bands and then mean pooled before output. Through various processing of the feature information output from the last convolutional block, output feature information containing different deep semantic information can be obtained. These output feature information describe the deep semantic information of the song segment's audio data from different scales, including global information and various local information of the song segment's audio data, such as global information that abstracts the salient features of the encoded information of the song segment's audio data, and local information that abstracts the channel or frequency band features of the encoded information of the song segment's audio data, and so on. Based on this, multiple output feature information with different representations can be obtained. These output feature information can be called independently or combined as needed.

[0151] In this application, the output feature information output by each branch network is normalized into an output feature vector representation. Therefore, multiple branch networks can obtain multiple output feature vectors accordingly. Each output feature vector represents the deep semantic information of the audio data of the song segment at different aspects or scales. The deep semantic information contained in each output feature vector is different from that in the others.

[0152] In practice, two or more branch networks are typically employed to obtain two or more output feature vectors. This allows for the use of two or more deep semantic information layers to represent the audio data of a song segment. For example, the output feature vector representing the global information of the audio data of the song segment can be combined with the output feature vector representing the channel information of the audio data of the song segment. Alternatively, the output feature vector representing the global information of the audio data of the song segment can be combined with the output feature vector representing the frequency band information of the audio data of the song segment. Or, the output feature vector representing the channel information of the audio data of the song segment can be combined with the output feature vector representing the frequency band information of the audio data of the song segment. Alternatively, all output feature vectors can be combined. Such methods can be utilized as needed by those skilled in the art.

[0153] Step S2300: The feature extraction model outputs the output feature vectors of each branch network as the high-dimensional index vector.

[0154] The output feature vectors obtained by each branch network can be ultimately converted into high-dimensional index vectors for storage or direct use. These high-dimensional index vectors serve as indexes for the audio data of corresponding song segments. Since each branch network has normalized its output feature information into output feature vectors, the high-dimensional index vectors can be processed flexibly depending on the specific application of the feature extraction model. For example, for applications requiring only storage for later use and separate retrieval, each output feature vector can be stored as multiple corresponding high-dimensional index vectors in the first and second song segment feature libraries mentioned in this application, allowing for on-demand retrieval, querying, and matching of the high-dimensional index vectors output by different branch networks. Furthermore, for specific tasks such as song recognition, cover song recognition, and humming recognition, all output feature vectors from all the structured branch networks can be sequentially concatenated to obtain a single high-dimensional index vector. This high-dimensional index vector can be stored or used for matching immediately, for example, stored in the first song segment feature library (without a main melody) and the second song segment feature library (with a main melody). Thus, the representation learning of the audio data of the song segment is achieved through the aforementioned high-dimensional index vector. In this application, for convenience, the high-dimensional index vectors of the song segments stored in the first song segment feature library and the second song segment feature library, as well as the high-dimensional index vector of the song segment to be searched, can all be implemented in the form of a single high-dimensional vector concatenated from multiple output feature vectors.

[0155] Based on the principles disclosed above in this exemplary embodiment, the first and second song fragment feature libraries can be prepared for some or all of the songs in the online music service platform's music library according to the process of this embodiment. By applying the steps of this embodiment to the audio data of the song fragments of each corresponding song in the music library, high-dimensional index vectors corresponding to the audio data of each song fragment can be obtained. These high-dimensional index vectors are associated and stored with the corresponding song fragments to construct the song fragment feature library. If a song fragment has no melody information, its high-dimensional index vector can be added to the first song fragment feature library; if a song fragment has melody information, its high-dimensional index vector can be added to the second song fragment feature library. Subsequently, the high-dimensional index vectors in the song fragment feature library can be directly called for retrieval, query, matching, and other operations.

[0156] In addition to the various applications disclosed in this application, the mining and utilization based on the high-dimensional index vector obtained in this application may have many other uses, which can be flexibly applied by those skilled in the art based on the principles disclosed herein, without affecting the inventiveness of this application.

[0157] Based on the above description of the execution process and network architecture of the feature extraction model, it can be understood that this embodiment contains a wealth of beneficial effects, including but not limited to the following aspects:

[0158] First, the feature extraction model encodes the audio data of the song segment into corresponding encoded information to obtain style-invariant features of the audio data of the song segment. Then, an intermediate feature information is extracted from the encoded information through a shared network. Based on the intermediate feature information, deep semantic information of the audio data of the song segment is extracted from different perspectives through multiple branch networks to obtain corresponding output feature information. Finally, these output feature information are used as the high-dimensional index vector corresponding to the audio data of the song segment, completing the end-to-end representation learning of the audio data of the song segment.

[0159] Secondly, because the feature extraction model employs a combination of shared networks and multiple branch networks, it achieves multi-angle feature extraction of deep semantic information from the audio data of song segments. This makes the obtained high-dimensional index vector more representative, such as representing global feature information, salient feature information, channel feature information, and frequency band feature information of the audio data of the song segments. This enables a more effective indexing of the audio data of the corresponding song segments. Based on this, downstream processing such as retrieval, querying, and matching of the audio data of song segments can achieve more accurate and efficient matching results. It can be universally used in various application scenarios such as cover song recognition, song recognition by listening, humming recognition, and song copyright determination.

[0160] Furthermore, the output feature vectors obtained from the multiple branch networks of the feature extraction model can be combined into a single high-dimensional index vector or used independently as different high-dimensional index vectors, which can be flexibly determined according to the required deep semantic information. It has a wide range of uses and flexible applications. When processing the representation learning of massive amounts of audio data of song segments, it can achieve significant scalability. It can be deployed in the backend of online music service platforms to realize standardized interfaces, thereby serving the needs of various application scenarios, providing comprehensive and multi-purpose open services, and enhancing the economic advantages of the platform's music information retrieval.

[0161] In a further embodiment, step S2200, which involves extracting features from the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks of the feature extraction model, and then converting the extracted features into output feature vectors of the corresponding scale, includes any two or more of the following steps:

[0162] Step S2210: Use multiple convolutional blocks in the first branch network to extract features from the intermediate feature information to obtain global feature information, and pool the global feature information into a global-scale output feature vector:

[0163] Figure 4 In the exemplary first branch network, after extracting the intermediate feature information through two convolutional blocks with the same structure as other branch networks, the output of the last convolutional block is divided into two paths. One path directly performs mean pooling to obtain its overall feature information, while the other path randomly discards some time-frequency region information through a Dropout layer and then extracts the salient feature information globally through max pooling. This results in two corresponding global output feature vectors. Based on this architecture, during model training, it improves the model's generalization ability to audio data with local time-frequency domain changes such as missing or inserted segments in song clips, and also helps prevent overfitting to some extent. Furthermore, the two global output feature vectors—one capturing overall features and the other capturing salient features—enhance the model's recognition ability.

[0164] Step S2220: After extracting features from the intermediate feature information using multiple convolutional blocks in the second branch network, the network is divided into multiple parts by channel and pooled to obtain the corresponding channel-scale output feature vector.

[0165] Since the feature information output by each convolutional block is usually represented as "number of channels * number of bandwidths * number of frames", it can be segmented according to the number of channels. Figure 4 In the exemplary second branch network, after extracting the intermediate feature information step-by-step through two convolutional blocks with the same structure as the other branch networks, the output of the last convolutional block is divided into multiple channels, such as two outputs. These outputs are then passed through 1*1 convolutional layers and subjected to mean pooling to obtain the corresponding channel output feature information for each of the two channels. In this process, the two channel branches focus on capturing local audio features. For audio with significant variations in adaptation and where much information is obscured by strong noise or other interference, feature representations can be established from a few salient, common local features.

[0166] Step S2230: After extracting features from the intermediate feature information using multiple convolutional blocks in the third branch network, the network is divided into multiple parts according to frequency band and pooled to obtain the corresponding output feature vector at the frequency band scale.

[0167] Figure 4In the exemplary third branch network, after extracting the intermediate feature information step-by-step through two convolutional blocks with the same structure as other branch networks, the output of the last convolutional block is average-pooled and then divided into multiple outputs, such as two outputs. After average-pooling, the corresponding frequency band output feature information is obtained. In this process, each frequency band branch focuses on extracting the feature information of its corresponding frequency band. This has a significant effect on resisting frequency band selective attenuation in harsh sound pickup environments, balancing the contribution of high and low frequency information in feature composition, and resisting the addition or deletion of content in a fixed range of frequency bands (such as adding or removing a drum sound) or strong interference in a fixed frequency range.

[0168] It is understandable that multiple output feature vectors obtained from the same branch network can be further processed into the same output feature vector by concatenation or mean pooling. Those skilled in the art can implement this flexibly.

[0169] In this embodiment, a rich branch network is used to extract multi-faceted and multi-scale feature information from the audio data of the song segment. This enables the obtained output feature vector to obtain rich deep semantic information representation, which not only represents the global and salient information of the audio data of the song segment, but also represents the relevant local information of the audio data of the song segment by channel and frequency band. Considering that the intermediate feature information has already captured the key information of the audio data of the song segment under the action of the shared network, this embodiment realizes the indexing value of the audio data of the song segment from multiple aspects. The high-dimensional index vector obtained subsequently can improve the accuracy of various aspects when used for querying, retrieval and matching.

[0170] Since this embodiment can capture the deep semantic information of the audio data of song segments from multiple perspectives, the feature extraction applicable to the audio data of song segments can effectively mine the deep semantic information of such short-duration audio data, making the information represented by the corresponding high-dimensional index vector richer and more accurate, and achieving a more precise matching effect.

[0171] Please see Figure 5 Based on the previous embodiment, the network structure of the feature extraction model in this application is improved. It can be seen that... Figure 5 Network architecture and Figure 4 The difference in network architecture is that, Figure 5In the first branch network, the output of the last convolutional block is directly subjected to max pooling to obtain the global output feature vector, capturing the salient feature information of the encoded audio data of the song segment. In the second branch network, the output of the last convolutional block is equally divided into four parts corresponding to the feature information of each channel. The feature information corresponding to each channel is then subjected to mean pooling and reassembled into the corresponding output feature vector. It is easy to understand that through the segmentation and construction of local branches, the obtained output feature vector can learn better local feature information.

[0172] This embodiment exemplarily provides a method based on... Figure 4 The modified network architecture shown is relatively lightweight. Based on this example, it is easy to understand that the inventive spirit of this application lies in the flexible combination and use of multiple described branch networks. Those skilled in the art, based on the principles disclosed in this application and the characteristics of the multi-scale deep semantic information possessed by the output feature vectors obtained from each branch network, can adapt to different specific applications and select feature extraction models constructed from different combinations of branch networks to create various other embodiments of this application to meet needs such as humming recognition, song recognition, cover song recognition, and infringement comparison.

[0173] Please see Figure 6 In a further embodiment, when the convolutional block is invoked, the following steps are performed:

[0174] Step S3100: Perform convolution transformation on the input information to obtain transformed feature information:

[0175] In the feature extraction model of this application, each convolutional block, for the input information, whether it is the encoded information or the intermediate feature information output by the previous convolutional block, first undergoes a convolution operation through a 1*1 convolutional kernel to obtain the corresponding transformed feature information.

[0176] Step S3200: After performing instance normalization and batch normalization on the transformed feature information, combine them into concatenated feature information, and activate the output of the concatenated feature information:

[0177] After the first convolution, an instance batch normalization (IN) layer is applied to process the transformed feature information. This transformed feature information is divided into two paths. A batch normalization block (BN) is used to perform batch normalization on half of the channels, while instance normalization is applied to the remaining channels. The latter allows the corresponding convolutional block to capture the style-invariant features of the song segment's audio data. Therefore, for song representations with diverse styles within a single dataset, better utilization of the song representation is achieved. The two channels, after different normalization processes, can be concatenated into the same concatenated feature information for activation output.

[0178] Step S3300: After performing multiple convolution operations and batch normalization on the spliced ​​feature information of the activation output, the residual information is obtained.

[0179] The concatenated feature information from the activated output is further processed through multiple convolutional layers to extract more features. Each such convolutional layer is followed by a batch normalization layer for normalization before output. The last convolutional layer uses a 1x1 kernel to avoid the representation learning ability of the entire feature extraction model being attenuated due to normalization of multiple instances in multiple convolutional blocks. Therefore, the final output feature information is the residual information from the residual convolution process.

[0180] Step S3400: Superimpose the residual information onto the input information and activate the output:

[0181] Finally, based on the principle of residual convolution, referring to the transformation feature information obtained from the first convolution, the residual information is superimposed on it and then the output is activated to obtain the intermediate feature information output after the current convolution block undergoes residual convolution operation.

[0182] In this embodiment, the convolutional blocks required for the feature extraction model of this application are constructed by combining residual convolution with instance batch normalization. The residual convolutional network is based on the basic model of the ResNet series and improved, and the IBN architecture is superimposed. The feature extraction model built in this way is easier to train and can achieve more accurate feature extraction results, which is particularly suitable for feature extraction of audio data of song segments.

[0183] Please see Figure 7 In a more detailed embodiment, step S1300, determining the high-dimensional index vector of the searched song segment that is not similar to the high-dimensional index vector in the first song segment feature library as the matching vector, includes the following steps:

[0184] Step S1310: Call the first song fragment feature library. The first song fragment feature library stores multiple preset high-dimensional index vectors corresponding to song fragments without a main melody. The high-dimensional index vectors are obtained by the feature extraction model extracting the corresponding song fragments.

[0185] As disclosed in the preceding embodiments, this application pre-constructs a first song fragment feature library, which stores high-dimensional index vectors corresponding to multiple song fragments without a main melody. These song fragments without a main melody are usually the accompaniment parts at the beginning, middle, and end of a song, or instrumental music. Here, for the need to filter the song fragments to be searched, the first song fragment feature library can be called.

[0186] Step S1320: Calculate the similarity between the high-dimensional index vector of the song segment to be searched and each high-dimensional index vector in the first song segment feature library, and obtain the corresponding similarity values:

[0187] In this embodiment, both the high-dimensional index vector of the song segment to be searched and the high-dimensional index vector of the song segments in the first song segment feature library are single high-dimensional vectors that represent the multi-scale deep semantic information of the corresponding song segments as a whole. To determine whether the two match, various similarity algorithms can be applied by calculating the data distance between them. As mentioned above, specific algorithms such as cosine similarity algorithm, Euclidean distance algorithm, Pearson coefficient algorithm, Jaccard similarity algorithm, nearest neighbor search algorithm, etc., can be used to implement this. Those skilled in the art can implement it flexibly.

[0188] After similarity calculation, a similarity sequence is obtained between the high-dimensional index vector of the song segment to be searched and the high-dimensional index vectors of each song segment in the first song segment feature library. This similarity sequence stores the similarity values ​​between the song segment to be searched and each song segment without a main melody in the first song segment feature library.

[0189] Step S1330: Determine whether there is a song fragment without a main melody whose similarity value exceeds a preset threshold. If it exists, determine that the song fragment to be searched does not contain the main melody of the song and terminate the execution. If it does not exist, determine that the song fragment to be searched contains the main melody of the song and determine its high-dimensional index vector as the vector to be matched.

[0190] To determine whether a song segment to be searched is audio data without a melody, a preset threshold can be set and compared with the similarity values ​​of song segments in a first song segment feature library. It's easy to understand that if the similarity value between a song segment in the first song segment feature library and the song segment to be searched is greater than the preset threshold, then the song segment to be searched can be considered a song segment without a melody; conversely, if the similarity value between a song segment in the first song segment feature library and the song segment to be searched is less than the preset threshold, then the song segment to be searched can be considered a song segment with a melody. Based on this principle, simply compare the maximum similarity value in the similarity sequence with the preset threshold. If the former is greater than the latter, the song segment to be searched is determined to be a song segment without a melody, and the execution of subsequent steps can be terminated, simply notifying the user; if the former is less than the latter, the song segment to be searched is determined to be a song segment with a melody, and the high-dimensional index vector of the song segment to be searched can be determined as the matching vector required for subsequent matching.

[0191] This embodiment employs a similarity algorithm. By calculating the data distance between the high-dimensional index vectors of the song fragment to be searched and the song fragments without a melody in the first song fragment feature library, it determines and filters out the song fragments without a melody, and only determines the high-dimensional index vectors of the song fragments with a melody as the matching vectors. This enables the backend service to perform the frontend filtering operation, thereby ensuring higher accuracy in the backend service when searching and comparing song fragments, saving system overhead for the backend service, and avoiding frequent responses to redundant and invalid information.

[0192] Please see Figure 8 In a more detailed embodiment, step S1400, matching the target song segment that is similar to the vector to be matched from the high-dimensional index vectors corresponding to multiple song segments stored in the second song segment feature library, includes the following steps:

[0193] Step S1410: Call the second song fragment feature library to obtain the high-dimensional index vectors corresponding to each song fragment. The high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of a song fragment at different scales.

[0194] As disclosed in the preceding embodiments, this application pre-constructs a second song fragment feature library, which stores multiple high-dimensional index vectors corresponding to song fragments with a main melody. These song fragments with a main melody are usually the parts corresponding to the lyrics of the song. Here, for the need to match the song fragments to be searched, the second song fragment feature library can be called.

[0195] Step S1420: Calculate the similarity between the vector to be matched and each high-dimensional index vector in the second song segment feature library to obtain the corresponding similarity sequence:

[0196] In this embodiment, both the high-dimensional index vector of the song segment to be searched (i.e., the matching vector) and the high-dimensional index vector of the song segments in the second song segment feature library are single high-dimensional vectors that collectively represent the multi-scale deep semantic information of the corresponding song segments. To determine whether the two match, various similarity algorithms can be applied by calculating the data distance between them. As mentioned earlier, specific algorithms such as cosine similarity algorithm, Euclidean distance algorithm, Pearson coefficient algorithm, Jaccard similarity algorithm, nearest neighbor search algorithm, etc., can be used. Any algorithm suitable for calculating the similarity distance between data can be implemented flexibly by those skilled in the art.

[0197] After similarity calculation, a similarity sequence is obtained between the high-dimensional index vector of the song segment to be searched and the high-dimensional index vectors of each song segment in the second song segment feature library. This similarity sequence stores the similarity values ​​between the song segment to be searched and each song segment with a main melody in the second song segment feature library.

[0198] Step S1430: Determine the song segment corresponding to the largest similarity value in the similarity sequence that exceeds a preset threshold as the target song segment similar to the song segment to be searched.

[0199] To determine whether a target song segment similar to the searched song segment exists in the second song segment feature library, a preset threshold can be used. This preset threshold can be an empirical threshold or an experimental threshold, determined flexibly by those skilled in the art. Accordingly, all elements in the similarity sequence are compared one by one with the preset threshold, identifying all elements with similarity values ​​exceeding the preset threshold. When one or more elements have similarity values ​​exceeding the preset threshold, generally, the song segment corresponding to the element with the highest similarity value is identified as the target song segment similar to the searched song segment.

[0200] Step S1440: Obtain the access link of the song to which the target song fragment belongs from the music library and push it to the client device:

[0201] As mentioned earlier, the song fragments in the second song fragment feature library belong to a song in the music library of the online music service platform, and a corresponding relationship has been established in advance. Therefore, in this step, the song containing the target song fragment can be obtained from the music library based on this corresponding relationship, the access link of the song can be encapsulated into the corresponding result notification information, and then the result notification information can be pushed to the user who submitted the song fragment to be searched, and delivered to their corresponding client device.

[0202] In this embodiment, the matching vector of the song segment to be searched after filtering for the absence of a main melody is matched with the data distance in a second song segment feature library that stores high-dimensional index vectors of main melody song segments. This determines the target song segment that is similar to the song segment to be searched. Then, the target song in the music library is determined based on the target song segment. Finally, the corresponding access link of the target song is pushed to the client. This completes the business loop, allowing the user who submits the song segment to be searched to obtain the expected similar song, realizing the tracing of the source of the song segment to be searched or the matching of similar songs.

[0203] Please see Figure 9 In an extended embodiment, the training process of the feature extraction model includes the following iterative training steps:

[0204] Step S4100: Retrieve a training sample from the training set and determine the encoding information of the training sample. The training sample is a song segment containing the main melody of the song, pre-collected according to a predetermined duration.

[0205] Those skilled in the art will understand that, to adapt to different downstream tasks, different training sets can be constructed for training the feature extraction model, each training set containing a sufficient number of training samples, and each training sample having a corresponding supervision label.

[0206] The training samples can be pre-collected by those skilled in the art. Each training sample is the audio data of a song fragment. These song fragments can be MIDI song fragments with a main melody, song fragments with accompaniment and a main melody, song fragments with unaccompanied vocal parts, song fragments without a main melody, song fragments with a main melody, etc. Different versions of the same song fragment sung by different audio sources can be merged into the same category, that is, corresponding to the same supervision label, to enhance the generalization ability of the model classification.

[0207] In one embodiment, each song in the music library of an online music service platform is taken, and each song is divided into multiple song segments according to the timestamp of the lyrics of each song and according to a specific duration range. Some of the song segments are song segments with a main melody, and the other part is song segments without a main melody. Subsequently, after the feature extraction model of this application is trained to a convergent state, the feature extraction model can be used to extract features from the two parts of the song segments to obtain their corresponding high-dimensional index vectors. The song segments without a main melody and their high-dimensional index vectors are stored in the first song segment feature library of this application, and the song segments with a main melody and their high-dimensional index vectors are stored in the second song segment feature library of this application, thus completing the construction of the first and second song segment feature libraries respectively.

[0208] In another improved embodiment, song segments that are less than the specified duration range can be deleted to improve the accuracy of the samples required for model training. In yet another embodiment, which will be disclosed later, the feature extraction model of this application can be used to extract high-dimensional index vectors of certain candidate song segments, calculate their similarity with the high-dimensional index vectors of each song segment in the pre-built first song segment feature library, determine whether these candidate song segments belong to song segments with a main melody based on the similarity, and then add the song segments with a main melody and their high-dimensional index vectors to the second song segment feature library.

[0209] For the training samples in the training set, for the convenience of model training, the encoding information corresponding to the audio data of the song segments can be prepared in advance, or the corresponding encoding information can be obtained in real time when the audio data of each song segment is used to train the feature extraction model. As for the specific encoding principle, please refer to the corresponding process disclosed above in this application.

[0210] Step S4200: Input the encoded information into the feature extraction model and train it to obtain the corresponding output feature vectors:

[0211] During the training process of a training sample, the corresponding encoded information of the training sample is output to the feature extraction model for feature extraction. The principle of feature extraction is explained in the preceding embodiments and will not be repeated here. In this process, the feature extraction model learns the representation of the training sample to obtain the corresponding output feature vectors.

[0212] Step S4300: Perform classification prediction on each of the output feature vectors to map them to the corresponding classification labels:

[0213] In this application, the training task of the feature extraction model is understood as a classification task. Therefore, by connecting the output feature vectors of the feature extraction model to the corresponding pre-prepared classification models, examining the classification results of each classification model, and supervising them with corresponding supervision labels, the model can be trained. Based on this principle, during the training phase, when training the feature extraction model implemented in any embodiment of this application, a classification model is connected to the output end of each output feature vector of each branch network.

[0214] The classification model described above uses, for example... Figure 10 The structure shown employs a batch normalization layer to perform batch normalization on the output feature vector, and then a fully connected layer to map the output feature vector to the classification space. The classification function calculates the classification probability of each classification label, thereby determining the classification label corresponding to the training sample as the one with the highest classification probability.

[0215] The classifier in the classification model can be constructed using a multi-class classifier implemented with the Softmax function, or it can be constructed using a multi-class classifier implemented with the AM-Softmax function, which can enhance intra-class compactness and expand inter-class sparsity. The latter obviously has better classification advantages.

[0216] Step S4400: Calculate the loss value of the feature extraction model using the supervision labels corresponding to the training samples and the classification labels, and perform gradient updates on the feature extraction model based on the loss value.

[0217] The classification model employs the batch normalization layer to achieve a balance between the triplet loss and the cross-entropy classification loss. Subsequently, the triplet loss can be calculated on the batch normalization layer, and the cross-entropy classification loss can be calculated on the fully connected layer. By combining these two losses, the output feature vector can be optimized.

[0218] Therefore, after the training samples are predicted with corresponding classification labels, the loss value between the supervision label and the classification label can be calculated based on the corresponding supervision label. Then, the feature extraction model is updated with gradients based on the loss value, and the weight parameters of each part of the model are corrected to promote model convergence.

[0219] Since there are multiple branch networks, each branch network may have multiple output feature vectors, and there are corresponding multiple classification models. Therefore, when calculating the loss value, a weighted approach can be used. That is, the triplet loss and classification loss in each classification model are first weighted and summed to obtain the loss value corresponding to each output feature vector. Then, the loss values ​​corresponding to each output feature vector are weighted and summed again to obtain the final loss value. The gradient update of the entire feature extraction model can be performed using this loss value.

[0220] Step S4500: Determine whether the loss value has reached a preset threshold. If it has not reached the preset threshold, call the next training sample in the training set to continue iterative training of the feature extraction model until the loss value reaches the preset threshold.

[0221] For each training sample, the loss value is determined by whether it infinitely approaches 0 or whether it reaches a preset threshold. When these conditions are met, the feature extraction model is considered to have converged, and training can be terminated, allowing the model to be deployed to production. If convergence has not been achieved, the next training sample in the training set can be used to iteratively train the feature extraction model until it converges.

[0222] This embodiment reveals the training principle and process of the feature extraction model of this application. As can be seen from this embodiment, by training the feature extraction model with a pre-prepared training set, the feature extraction model can learn the ability to extract the corresponding output feature vector from the encoded information of the audio data of a song segment, thereby achieving effective representation learning of the deep semantic information of the audio data of the song segment. Furthermore, the output feature vectors of the audio data of the same song segment at multiple scales can be jointly trained, resulting in higher training efficiency and richer model functions. When put into the production stage, it can quickly obtain the deep semantic information corresponding to multiple scales of the audio data of the same song segment.

[0223] The classification model in this embodiment employs a multi-classifier with a batch normalization layer and an AM-Softmax function. This balances the triplet loss and classification loss during gradient updates, enabling faster training and convergence. The trained model can then more effectively learn representations of the deep semantic information in the song segment's audio data. When the output feature vectors are combined as needed, they can more effectively represent the feature information of the song segment's audio data, resulting in more efficient matching.

[0224] This embodiment also demonstrates the scalability and compatibility of the feature extraction model in this application. Specifically, this embodiment allows the feature extraction model to be trained using training samples corresponding to different downstream tasks to meet the needs of serving different downstream tasks. This enables the feature extraction model to serve different downstream tasks. Therefore, it is a relatively basic improvement with better economic efficiency.

[0225] Please see Figure 11 In an extended embodiment, the song fragment search method includes the following steps for constructing a second song fragment feature library:

[0226] Step S5100: Obtain the audio data of candidate songs from the music library, and divide the audio data into multiple song segments according to the preset duration to obtain the audio data corresponding to the multiple song segments:

[0227] Taking an online music service platform as an example, it has a music library containing a massive number of songs, which can be used as candidate songs for constructing the second song fragment feature library in this embodiment.

[0228] To scientifically segment each candidate song to obtain multiple corresponding song segments, a preset duration is set. This preset duration can be a fixed value or a range. For the latter, the preset duration is usually a specific range, falling between the minimum and maximum preset durations, such as any value between 10 and 15 seconds.

[0229] To accommodate situations where the preset duration is represented as a specific duration range, multiple target lyrics falling within the specific duration range can be calculated based on the timestamps in the lyrics of each candidate song and the timestamps of consecutive lyrics. Audio data segments corresponding to the song segments singing these target lyrics are then extracted from the candidate songs based on their timestamps. This allows for the acquisition of multiple song segments with a main melody from each candidate song. Furthermore, considering the similarity in the main melody and background music of songs with the same lyrics, duplicate song segments from each candidate song can be removed based on the lyrics; that is, only one of the song segments with identical content in each candidate song needs to be retained. Thus, each candidate song has multiple corresponding song segments, and the mapping relationship data between each song segment and its corresponding candidate song can be flexibly stored for later retrieval.

[0230] Step S5200: Encode the audio data of each song segment to obtain its encoding information:

[0231] Since it is necessary to extract the deep semantic information of each song segment in each candidate song, this embodiment still uses the feature extraction model of this application, which has been pre-trained to a convergent state, to process each song segment. For this purpose, the audio data of the corresponding song segment needs to be encoded to obtain the corresponding encoding information. The principle and process of encoding have been disclosed in the foregoing embodiments of this application, and will not be repeated here.

[0232] Step S5300: For each song segment, the feature extraction model is invoked to extract deep semantic information of its encoded information at multiple scales, thereby obtaining its high-dimensional index vector.

[0233] Once the encoding information of a song segment is determined, the feature extraction model can be invoked to extract features from the encoding information. Under the action of this feature extraction model, multi-scale deep semantic information of the corresponding song segment is extracted based on the encoding information, and finally organized into a high-dimensional index vector. As mentioned above, this application recommends concatenating the deep semantic information at multiple scales into a single high-dimensional vector as the high-dimensional index vector.

[0234] Step S5400: For each song segment, calculate the similarity between its high-dimensional index vector and each high-dimensional index vector in the first song segment feature library, and obtain the corresponding similarity value:

[0235] To improve the accuracy of data entry, the pre-built first song fragment feature library is further utilized. The similarity between the high-dimensional index vector of each song fragment without a main melody stored therein and the high-dimensional index vector of the song fragment to be entered into the library is calculated to obtain the corresponding similarity sequence. This similarity sequence stores the similarity values ​​corresponding to each song fragment without a main melody.

[0236] Step S5500: For each song segment, determine whether there is a song segment without a main melody whose similarity value exceeds a preset threshold. If so, discard the song segment. If not, store the mapping relationship data between the song segment and its high-dimensional index vector in the second song segment feature library.

[0237] Finally, following the principles revealed in the preceding embodiments, the system determines whether there are song fragments without a main melody whose similarity value exceeds a preset threshold based on whether the maximum similarity value in the similarity sequence is higher than a preset threshold. If such a fragment exists, it indicates that the song fragment to be added to the database is highly likely to be a song fragment without a main melody, and it can be discarded and not added to the database. Conversely, if the maximum similarity value in the similarity sequence is higher than a song fragment with a main melody, the song fragment and its high-dimensional index vector mapping data can be added to the second song fragment feature database. This process is repeated to continuously enrich the second song fragment feature database, completing its construction. This database can then be used in this application to match similar target song fragments for the song fragment to be searched.

[0238] This embodiment further details the construction process of the second song fragment feature library of this application. As can be seen from this embodiment, the song fragments in the second song fragment feature library can be selected from the music library of an online music service platform, and a mapping relationship is established between the song fragments and the corresponding candidate songs in the music library. Therefore, when it is necessary to search for similar songs (fragments) for a song fragment to be searched, the index feature vectors of each song fragment in the second song fragment feature library can be used for similarity matching, and the similar songs in the music library are determined based on the matched similar song fragments. In this process, since the song fragments in the second song fragment feature library have already been filtered and selected using the first song fragment feature library, they are generally song fragments with a main melody. Therefore, when performing search matching, the accuracy of the similarity values ​​obtained by the algorithm is higher, and false positives are less likely to occur.

[0239] Please see Figure 12 This application provides a song fragment search device, functionally deployed to adapt to the song fragment search method of this application, including: a fragment encoding module 1100, a semantic extraction module 1200, a fragment filtering module 1300, and a fragment matching module 1400. The fragment encoding module 1100 is used to obtain the encoding information corresponding to the song fragment to be searched submitted by the client; the semantic extraction module 1200 is used to extract a high-dimensional index vector representing the deep semantic information of the song fragment to be searched at multiple scales based on the encoding information using a feature extraction model trained to convergence; the fragment filtering module 1300... Module 1300 is used to determine the high-dimensional index vector of the searched song segment that is not similar to the high-dimensional index vector in the first song segment feature library as the matching vector; the high-dimensional index vector in the first song segment feature library is the feature representation of the song segment without main melody extracted by the feature extraction model; the segment matching module 1400 is used to match the target song segment that is similar to the matching vector from the high-dimensional index vectors corresponding to multiple song segments stored in the second song segment feature library; the high-dimensional index vector in the second song segment feature library is the feature representation of the song segment with main melody extracted by the feature extraction model.

[0240] In a further embodiment, the segment encoding module 1100 includes: a request analysis submodule, used to receive a song segment search request submitted by a client and obtain the audio data of the song segment to be searched specified in the request; a human voice detection submodule, used to detect whether the audio data contains human voice singing information, and if not, to terminate subsequent execution; a formatting processing submodule, used to format the audio data into audio data between a minimum preset duration and a maximum preset duration; and an encoding execution submodule, used to encode the audio data to obtain corresponding encoding information.

[0241] In a more detailed embodiment, the feature extraction model includes the following components for implementing its function: a shared extraction module, used to sequentially perform multi-level feature extraction on the encoded information using multiple convolutional blocks in the shared network of the feature extraction model that has been trained to convergence, to obtain intermediate feature information that extracts the deep semantic information of the encoded information; a branch extraction module, used to perform feature extraction on the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks of the feature extraction model, and then convert it into output feature vectors of corresponding scales, wherein the deep semantic information contained in the output feature vectors of each branch network is different; and a vector output module, used by the feature extraction model to output the output feature vectors of each branch network as the high-dimensional index vector.

[0242] In a further embodiment, the branch extraction module is configured to include two or more sub-modules: a first extraction sub-module, used to extract features from the intermediate feature information using multiple convolutional blocks in the first branch network to obtain global feature information, and pooling the global feature information into a global-scale output feature vector; a second extraction sub-module, used to extract features from the intermediate feature information using multiple convolutional blocks in the second branch network, and then divide it into multiple parts by channel for pooling, thereby obtaining a channel-scale output feature vector; and a third extraction sub-module, used to extract features from the intermediate feature information using multiple convolutional blocks in the third branch network, and then divide it into multiple parts by frequency band for pooling, thereby obtaining a frequency band-scale output feature vector.

[0243] In a preferred embodiment, when the first branch network performs the pooling operation, it employs mean pooling and / or max pooling to obtain one or two output feature vectors at the global scale; when the second branch network performs the pooling operation, it employs mean pooling for one or more channels to obtain one or more output feature vectors at the channel scale; and when the third branch network performs the pooling operation, it employs mean pooling for one or more frequency bands to obtain one or more output feature vectors at the frequency band scale.

[0244] In a further embodiment, the convolutional block is configured to include the following structures for implementing its function: a convolutional transformation unit for performing convolutional transformation on the input information to obtain transformed feature information; a normalization processing unit for performing instance normalization and batch normalization processing on the transformed feature information respectively and combining them into concatenated feature information, and activating the output of the concatenated feature information; a residual calculation unit for performing multiple convolutional operations and batch normalization processing on the activated output concatenated feature information to obtain residual information; and an activation output unit for superimposing the residual information onto the input information to activate the output.

[0245] In a preferred embodiment, in the shared network, at least one of the convolutional blocks applies an attention module to extract key information from the audio data of the song segment, wherein the attention module is a spatial attention module or a channel attention module.

[0246] In a preferred embodiment, the source of the encoded information is any one of the time-spectrum information, Mel spectrum information, CQT filter information, pitch contour information, and Chroma feature information of the corresponding audio data.

[0247] In a more detailed embodiment, the segment filtering module 1300 includes: a melody-less calling submodule, used to call a first song segment feature library, which stores multiple preset high-dimensional index vectors corresponding to song segments without a melody, the high-dimensional index vectors being obtained by the feature extraction model extracting the corresponding song segments; a melody-less calculation submodule, used to calculate the similarity between the high-dimensional index vector of the song segment to be searched and each high-dimensional index vector in the first song segment feature library, and obtain the corresponding similarity value; and a melody-less judgment submodule, used to determine whether there are song segments without a melody whose similarity value exceeds a preset threshold. If such a segment exists, the searched song segment is determined not to contain the main melody of the song and execution is terminated. If not, the searched song segment is determined to contain the main melody of the song and its high-dimensional index vector is determined as the matching vector.

[0248] In a more detailed embodiment, the segment matching module 1400 includes: a melody calling submodule, used to call a second song segment feature library to obtain high-dimensional index vectors corresponding to each song segment therein, wherein the high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of a song segment at different scales; a melody calculation submodule, used to calculate the similarity between the vector to be matched and each high-dimensional index vector in the second song segment feature library to obtain a corresponding similarity sequence; a melody matching submodule, used to determine that the song segment corresponding to the largest similarity value exceeding a preset threshold in the similarity sequence is a target song segment similar to the song segment to be searched; and a link push submodule, used to obtain the access link of the song to which the target song segment belongs from the music library and push it to the client device.

[0249] In an extended embodiment, the song fragment search device includes the following structure for training the feature extraction model: a sample retrieval module for retrieving a training sample from the training set and determining the encoding information of the training sample, wherein the training sample is a song fragment containing the main melody of a song pre-collected for a predetermined duration; a representation learning module for inputting the encoding information into the feature extraction model to train it and obtain corresponding output feature vectors; a classification prediction module for performing classification prediction on each of the output feature vectors to map them to corresponding classification labels; a gradient update module for calculating the loss value of the feature extraction model using the supervision label corresponding to the training sample and the classification label, and performing gradient update on the feature extraction model based on the loss value; and an iterative decision module for determining whether the loss value reaches a preset threshold. If the preset threshold is not reached, the next training sample in the training set is retrieved to continue iterative training of the feature extraction model until the loss value reaches the preset threshold.

[0250] In an extended embodiment, the song fragment search device includes a structure for constructing a second song fragment feature library. This structure includes: a song segmentation module, used to obtain audio data of candidate songs from a music library, and segment the audio data into multiple song fragments according to a preset duration to obtain audio data corresponding to multiple song fragments; a segmentation encoding module, used to encode the audio data of each song fragment to obtain its encoding information; a fragment extraction module, used to call the feature extraction model to extract deep semantic information of the encoded information at multiple scales for each song fragment, and obtain its high-dimensional index vector accordingly; a similarity calculation module, used to calculate the similarity between the high-dimensional index vector of each song fragment and each high-dimensional index vector in the first song fragment feature library to obtain the corresponding similarity value; and a fragment storage module, used to determine whether there is a song fragment without a main melody whose similarity value exceeds a preset threshold for each song fragment. If so, the song fragment is discarded; if not, the mapping relationship data between the song fragment and its high-dimensional index vector is stored in the second song fragment feature library.

[0251] To address the aforementioned technical problems, embodiments of this application also provide computer equipment. For example... Figure 13The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store control information sequences. When the computer-readable instructions are executed by the processor, the processor can implement a song fragment search method. The processor of the computer device provides computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the song fragment search method of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0252] In this embodiment, the processor is used to execute... Figure 12 The system contains the specific functions of each module and its sub-modules. The memory stores the program code and various data required to execute these modules or sub-modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules / sub-modules in the song fragment search device of this application. The server can call the server's program code and data to execute the functions of all sub-modules.

[0253] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the song fragment search method of any embodiment of this application.

[0254] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0255] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0256] In summary, this application utilizes a feature extraction model to learn the representation of deep semantic information of song fragments at multiple scales, obtaining their high-dimensional index vectors. Based on these high-dimensional index vectors, similar song fragments are matched, resulting in more accurate and efficient performance in song querying, retrieval, and matching. This can serve various downstream tasks such as song recognition by listening, humming recognition, and cover song recognition, thereby enhancing the comprehensive service capabilities of online music platforms.

[0257] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in the prior art that are similar to those disclosed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0258] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for searching song fragments, characterized in that, Includes the following steps: Obtain the encoding information corresponding to the song fragment submitted by the client for searching; A high-dimensional index vector representing deep semantic information at multiple scales of the song segment to be searched is extracted based on the encoded information using a feature extraction model that has been trained to convergence. The high-dimensional index vector of the song segment to be searched that is not similar to the high-dimensional index vector in the first song segment feature library is determined as the matching vector; the high-dimensional index vector in the first song segment feature library is the feature representation of the song segment without main melody extracted by the feature extraction model; The target song segment that is similar to the vector to be matched is matched from the high-dimensional index vectors corresponding to multiple song segments stored in the second song segment feature library; the high-dimensional index vectors in the second song segment feature library are the feature representations of song segments with a main melody extracted by the feature extraction model. When the feature extraction model is invoked, the following steps are performed: Multiple convolutional blocks in the shared network of a feature extraction model that has been trained to convergence are used to sequentially perform multi-level feature extraction on the encoded information to obtain intermediate feature information that has extracted the deep semantic information of the encoded information. After extracting features from the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks in the feature extraction model, the features are converted into output feature vectors of corresponding scales. The deep semantic information contained in the output feature vectors of each branch network is different. The feature extraction model outputs the feature vectors of each branch network as the high-dimensional index vector.

2. The song fragment search method according to claim 1, characterized in that, The process of obtaining the encoding information corresponding to the song segment submitted by the client includes the following steps: Receive a song segment search request submitted by the client and obtain the audio data of the song segment to be searched specified in the request; The system checks whether the audio data contains human vocal information; if not, it terminates the subsequent execution. The audio data is formatted to be between the minimum preset duration and the maximum preset duration. The audio data is encoded to obtain the corresponding encoding information.

3. The song fragment search method according to claim 1, characterized in that, The step of extracting intermediate feature information at different scales by using multiple convolutional blocks from two or more branches of the feature extraction model and then converting it into an output feature vector of the corresponding scale includes any two or more of the following steps: The intermediate feature information is extracted by using multiple convolutional blocks in the first branch network to obtain global feature information, and the global feature information is pooled into a global scale output feature vector. After extracting features from the intermediate feature information using multiple convolutional blocks in the second branch network, the network is divided into multiple parts by channel and pooled to obtain the corresponding channel-scale output feature vector. After extracting features from the intermediate feature information using multiple convolutional blocks in the third branch network, the network is divided into multiple parts according to frequency band and pooled to obtain the corresponding output feature vector at the frequency band scale.

4. The song fragment search method according to claim 3, characterized in that: When the first branch network performs the pooling operation, it uses mean pooling and / or max pooling to obtain one or two output feature vectors at the global scale. When the second branch network performs the pooling operation, it uses mean pooling for one or more channels to obtain one or more output feature vectors at the channel scale. When the third branch network performs the pooling operation, it uses mean pooling for one or more frequency bands to obtain one or more output feature vectors at the frequency band scale.

5. The song fragment search method according to claim 1, characterized in that, When the convolutional block is invoked, the following steps are performed: The information in the input is subjected to convolutional transformation to obtain transformation feature information; The transformed feature information is combined into spliced ​​feature information after being processed by instance normalization and batch normalization respectively, and the spliced ​​feature information is activated and output. The concatenated feature information of the activation output is processed through multiple convolution operations and batch normalization to obtain residual information; The residual information is superimposed onto the input information to activate the output.

6. The song fragment search method according to claim 1, characterized in that, In the shared network, at least one of the convolutional blocks applies an attention module to extract key information from the audio data of a song segment. The attention module is either a spatial attention module or a channel attention module.

7. The song fragment search method according to claim 1, characterized in that, The source of the encoded information is any one of the following: time-spectrum information, Mel spectrum information, CQT filter information, pitch contour information, and Chroma feature information of the corresponding audio data.

8. The song fragment search method according to claim 1, characterized in that, The step of determining the high-dimensional index vector of the searched song segment that is not similar to the high-dimensional index vector in the first song segment feature library as the matching vector includes the following steps: The first song fragment feature library is invoked. The first song fragment feature library stores multiple preset high-dimensional index vectors corresponding to song fragments without a main melody. The high-dimensional index vectors are obtained by the feature extraction model by extracting the corresponding song fragments. Calculate the similarity between the high-dimensional index vector of the song segment to be searched and each high-dimensional index vector in the first song segment feature library, and obtain the corresponding similarity value; Determine whether there is a song segment without a main melody whose similarity value exceeds a preset threshold. If it exists, determine that the song segment to be searched does not contain the main melody of the song and terminate the execution. If it does not exist, determine that the song segment to be searched contains the main melody of the song and determine its high-dimensional index vector as the vector to be matched.

9. The song fragment search method according to claim 1, characterized in that, The step of matching the target song segment that is similar to the vector to be matched from the high-dimensional index vectors corresponding to multiple song segments stored in the second song segment feature library includes the following steps: The second song fragment feature library is invoked to obtain the high-dimensional index vector corresponding to each song fragment. The high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of a song fragment at different scales as a whole. Calculate the similarity between the vector to be matched and each high-dimensional index vector in the second song segment feature library to obtain the corresponding similarity sequence; The song segment corresponding to the largest similarity value in the similarity sequence that exceeds a preset threshold is identified as the target song segment that is similar to the song segment to be searched. The link to the song containing the target song fragment is retrieved from the music library and pushed to the client device.

10. The song fragment search method according to claim 1, characterized in that, The training process of the feature extraction model includes the following iterative training steps: A training sample is retrieved from the training set, and the encoding information of the training sample is determined. The training sample is a song segment containing the main melody of the song that is pre-collected according to a predetermined duration. The encoded information is input into the feature extraction model and trained to obtain the corresponding output feature vectors. Each output feature vector is classified and predicted to map to the corresponding classification label. The loss value of the feature extraction model is calculated using the supervision label corresponding to the training sample and the classification label, and the feature extraction model is updated according to the gradient based on the loss value; Determine whether the loss value has reached a preset threshold. If it has not reached the preset threshold, call the next training sample in the training set to continue iterative training of the feature extraction model until the loss value reaches the preset threshold.

11. The song fragment search method according to any one of claims 1 to 10, characterized in that, This method includes the following steps for constructing a second song fragment feature library: The audio data of candidate songs is obtained from the music library, and the audio data is divided into multiple song segments according to the preset duration to obtain the audio data corresponding to the multiple song segments. The audio data of each song segment is encoded to obtain its encoding information; For each song segment, the feature extraction model is invoked to extract deep semantic information of its encoded information at multiple scales, thereby obtaining its high-dimensional index vector. For each song segment, calculate the similarity between its high-dimensional index vector and each high-dimensional index vector in the first song segment feature library, and obtain the corresponding similarity value; For each song segment, determine whether there is a song segment without a main melody whose similarity value exceeds a preset threshold. If it exists, discard the song segment. If it does not exist, store the mapping relationship data between the song segment and its high-dimensional index vector in the second song segment feature library.

12. A song fragment search device, characterized in that, include: The segment encoding module is used to obtain the encoding information corresponding to the song segments submitted by the client for searching; The semantic extraction module is used to extract a high-dimensional index vector representing deep semantic information at multiple scales of the song segment to be searched based on the encoded information using a feature extraction model that has been trained to convergence. The segment filtering module is used to determine the high-dimensional index vectors of the searched song segments that are not similar to the high-dimensional index vectors in the first song segment feature library as the matching vectors; the high-dimensional index vectors in the first song segment feature library are the feature representations of song segments without a main melody extracted by the feature extraction model; The segment matching module is used to match the target song segment that is similar to the vector to be matched from the high-dimensional index vectors corresponding to multiple song segments stored in the second song segment feature library; the high-dimensional index vectors in the second song segment feature library are the feature representations of song segments with a main melody extracted by the feature extraction model. The feature extraction model includes the following structure for implementing its function: a shared extraction module, which uses multiple convolutional blocks in the shared network of the feature extraction model that has been trained to convergence to sequentially perform multi-level feature extraction on the encoded information to obtain intermediate feature information that has extracted the deep semantic information of the encoded information; The branch extraction module is used to extract features from the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks of the feature extraction model, and then convert them into output feature vectors of the corresponding scales. The deep semantic information contained in the output feature vectors of each branch network is different. The vector output module is used to output the output feature vectors of each branch network as the high-dimensional index vector by the feature extraction model.

13. A computer device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 11, which, when invoked by a computer, executes the steps included in the corresponding method.

15. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 11.

Images