Song search methods, devices, equipment, media, and products

Abstract

This application discloses a song search method, apparatus, device, medium, and product. The method includes: obtaining the encoding information corresponding to a song submitted by a client; using a feature extraction model trained to convergence to extract a high-dimensional index vector representing deep semantic information at multiple scales of the song to be searched based on the encoding information; calculating the similarity between the high-dimensional index vector and high-dimensional index vectors representing deep semantic information at multiple scales of each candidate song extracted by the feature extraction model in a preset song feature library, obtaining a similarity sequence; filtering and determining target songs in the similarity sequence whose similarity values ​​exceed a preset threshold and are the most similar, and constructing a corresponding access link for the target song and pushing it to the client device. Through the above process, a song search service can be quickly, efficiently, and accurately realized, allowing users to find target songs similar to the song to be searched.

Description

Technical Field

[0001] This application relates to the field of music information retrieval technology, and in particular to a song search method and corresponding apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rise of short videos, live streaming, and radio, the number of cover songs is increasing, and the scenarios requiring music recognition are becoming more complex. Compared to the original version, cover versions may differ from, or even be completely different from, other musical components such as timbre, fundamental frequency, rhythm, tempo, harmony, lyrics, singing style, and overall structure. Therefore, cover song recognition is a very challenging research task.

[0003] There are various existing technologies related to cover song recognition, and each of these technologies has certain shortcomings. For example: (1) Traditional landmark-based song recognition technology can only recognize songs with the same source version and cannot recognize the cover versions with certain differences mentioned above; (2) Traditional humming recognition technology based on melody matching can only recognize clean a cappella / humming and cannot recognize the cover versions with background accompaniment mentioned above; (3) Traditional cover song recognition technology mainly extracts audio features such as pitch class profile (PCP) and then uses algorithms such as dynamic programming to calculate the similarity distance between songs. Due to the diversity of cover versions, the above solutions are only applicable to cover versions with smaller modifications, have low accuracy, and are slow to recognize, making them unsuitable for searching massive amounts of music.

[0004] Common applications of cover song recognition include not only matching and recognizing the content sung by different singers, but also matching the content sung by the same singer; it also includes matching between longer and more complete songs, as well as matching between partial song fragments and complete song fragments. There are many scenarios, and in order to meet these requirements, the proposed technical solution should be able to handle as many scenarios as possible.

[0005] Given that existing technologies for song recognition lack universal applicability, have low accuracy, and are inefficient, the applicant is exploring more effective technical solutions. Summary of the Invention

[0006] The primary objective of this application is to solve at least one of the aforementioned problems by providing a song search method and corresponding apparatus, computer equipment, computer-readable storage medium, and computer program product.

[0007] To achieve the various objectives of this application, the following technical solution is adopted:

[0008] A song search method provided for one of the purposes of this application includes the following steps:

[0009] Obtain the encoding information corresponding to the song submitted by the client for searching;

[0010] A high-dimensional index vector representing deep semantic information at multiple scales of the song to be searched is extracted from the encoded information using a feature extraction model that has been trained to convergence.

[0011] Calculate the similarity between the high-dimensional index vector and the high-dimensional index vector in the preset song feature library that represents the deep semantic information of each candidate song at multiple scales extracted by the feature extraction model, and obtain a similarity sequence;

[0012] Target songs with similarity values ​​exceeding a preset threshold and being the most similar in the similarity sequence are selected and their access links are constructed and pushed to the client device.

[0013] In a more detailed embodiment, obtaining the encoding information corresponding to the song to be searched submitted by the client includes the following steps:

[0014] Receive a song search request submitted by the client and obtain the audio data of the song to be searched specified in the request;

[0015] The system checks whether the audio data contains human vocal information; if not, it terminates the subsequent execution.

[0016] The audio data of the song to be searched is encoded to obtain the corresponding encoding information.

[0017] In one further embodiment, calculating the similarity between the high-dimensional index vector and the high-dimensional index vectors in the preset song feature library that represent the deep semantic information of each candidate song at multiple scales extracted by the feature extraction model, to obtain a similarity sequence, includes the following steps:

[0018] A preset song feature library is invoked to obtain the high-dimensional index vector corresponding to each candidate song. The high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of a candidate song at different scales.

[0019] The similarity between the high-dimensional index vector of the song to be searched and each high-dimensional index vector in the song feature library is calculated to obtain the corresponding similarity sequence; the similarity sequence stores the similarity value corresponding to each candidate song in the song feature library;

[0020] The similarity sequence is sorted in reverse order based on the similarity value to obtain the sorted similarity sequence output.

[0021] In another embodiment of the refinement, the step of calculating the similarity between the high-dimensional index vector and the high-dimensional index vector in the preset song feature library that represents the deep semantic information of each candidate song at multiple scales extracted by the feature extraction model, and obtaining a similarity sequence, includes the following steps:

[0022] A preset song feature library is invoked to obtain the high-dimensional index vector corresponding to each candidate song. The high-dimensional index vector is a set of high-dimensional vectors that represent the deep semantic information of a candidate song at various scales.

[0023] The similarity between the high-dimensional vector of the song to be searched and the corresponding high-dimensional vector of each candidate song in the preset song feature library is calculated according to the high-dimensional vector corresponding to different semantic scales, so as to obtain the similarity sequence corresponding to each semantic scale.

[0024] Based on the correspondence between high-dimensional vectors at the semantic scale, the similarity values ​​in the similarity sequences corresponding to various semantic scales are summarized and merged to obtain a summary similarity sequence.

[0025] The summarized similarity sequence is sorted in reverse order based on the similarity values ​​to obtain the sorted similarity sequence output.

[0026] In a further embodiment, obtaining the encoding information corresponding to the song to be searched submitted by the client includes the following steps:

[0027] Receive a song search request submitted by the client and obtain the audio data of the song to be searched specified in the request;

[0028] If the audio data of the song to be searched exceeds a preset duration, it is divided into multiple audio data corresponding to multiple song segments according to the preset duration; otherwise, it is retained as the whole audio data.

[0029] Each audio data segment is encoded separately to obtain the corresponding encoding information for each audio data segment, so that the feature extraction model can extract the corresponding high-dimensional index vectors for each song segment.

[0030] According to another further embodiment, the step of calculating the similarity between the high-dimensional index vector and the high-dimensional index vector in the preset song feature library that represents the deep semantic information of each candidate song at multiple scales extracted by the feature extraction model, and obtaining a similarity sequence, includes the following steps:

[0031] A preset song feature library is invoked to obtain the high-dimensional index vectors corresponding to the song segments of each candidate song. The high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of a song segment at different scales as a whole.

[0032] For each audio data segment, the similarity between its high-dimensional index vector and the corresponding high-dimensional index vector of each candidate song in the preset song feature library is calculated to obtain the similarity sequence corresponding to each audio data segment.

[0033] The similarity values ​​in all similarity sequences are aggregated and merged according to the same candidate songs to obtain the aggregated similarity sequence;

[0034] The summarized similarity sequence is sorted in reverse order based on the similarity values ​​to obtain the sorted similarity sequence output.

[0035] In a preferred embodiment, when the feature extraction model is invoked, the following steps are performed:

[0036] Multiple convolutional blocks in the shared network of a feature extraction model that has been trained to convergence are used to sequentially perform multi-level feature extraction on the encoded information to obtain intermediate feature information that has extracted the deep semantic information of the encoded information.

[0037] After extracting features from the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks in the feature extraction model, the features are converted into output feature vectors of corresponding scales. The deep semantic information contained in the output feature vectors of each branch network is different.

[0038] The feature extraction model outputs the feature vectors of each branch network as the high-dimensional index vector.

[0039] In a further embodiment, the step of extracting features from the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks of the feature extraction model, and then converting it into an output feature vector of the corresponding scale, includes any two or more of the following steps:

[0040] The intermediate feature information is extracted by using multiple convolutional blocks in the first branch network to obtain global feature information, and the global feature information is pooled into a global scale output feature vector.

[0041] After extracting features from the intermediate feature information using multiple convolutional blocks in the second branch network, the network is divided into multiple parts by channel and pooled to obtain the corresponding channel-scale output feature vector.

[0042] After extracting features from the intermediate feature information using multiple convolutional blocks in the third branch network, the network is divided into multiple parts according to frequency band and pooled to obtain the corresponding output feature vector at the frequency band scale.

[0043] In a preferred embodiment, when the first branch network performs the pooling operation, it employs mean pooling and / or max pooling to obtain one or two output feature vectors at the global scale; when the second branch network performs the pooling operation, it employs mean pooling for one or more channels to obtain one or more output feature vectors at the channel scale; and when the third branch network performs the pooling operation, it employs mean pooling for one or more frequency bands to obtain one or more output feature vectors at the frequency band scale.

[0044] In an optional embodiment, the source of the encoded information is any one of the time-spectrum information, Mel spectrum information, CQT filter information, pitch contour information, and Chroma feature information of the corresponding audio data.

[0045] In a preferred embodiment, in the shared network, at least one of the convolutional blocks applies an attention module to extract key information from the song's audio data, and the attention module is a spatial attention module or a channel attention module.

[0046] In the optimized embodiment, when the convolutional block is invoked, the following steps are performed:

[0047] The information in the input is subjected to convolutional transformation to obtain transformation feature information;

[0048] The transformed feature information is combined into spliced ​​feature information after being processed by instance normalization and batch normalization respectively, and the spliced ​​feature information is activated and output.

[0049] The concatenated feature information of the activation output is processed through multiple convolution operations and batch normalization to obtain residual information;

[0050] The residual information is superimposed onto the input information to activate the output.

[0051] In an extended embodiment, the training process of the feature extraction model includes the following iterative training steps:

[0052] A training sample is retrieved from the training set, and the encoding information of the training sample is determined. The training sample is pre-collected song audio data, which is a complete song or a segment thereof.

[0053] The encoded information is input into the feature extraction model and trained to obtain the corresponding output feature vectors.

[0054] Each output feature vector is classified and predicted to map to the corresponding classification label.

[0055] The loss value of the feature extraction model is calculated using the supervision label corresponding to the training sample and the classification label, and the feature extraction model is updated according to the gradient based on the loss value;

[0056] Determine whether the loss value has reached a preset threshold. If it has not reached the preset threshold, call the next training sample in the training set to continue iterative training of the feature extraction model until the loss value reaches the preset threshold.

[0057] A song search device provided for one of the purposes of this application includes: a song encoding module, a semantic extraction module, a similarity matching module, and a filtering and push module. The song encoding module is used to obtain encoding information corresponding to a song submitted by a client for searching. The semantic extraction module is used to extract a high-dimensional index vector representing deep semantic information at multiple scales of the song to be searched based on the encoding information using a feature extraction model trained to convergence. The similarity matching module is used to calculate the similarity between the high-dimensional index vector and high-dimensional index vectors representing deep semantic information at multiple scales of each candidate song extracted by the feature extraction model in a preset song feature library, thereby obtaining a similarity sequence. The filtering and push module is used to filter and determine target songs in the similarity sequence whose similarity values ​​exceed a preset threshold and are the most similar, and constructs a corresponding access link for the target song to push to the client device.

[0058] A computer device provided for one of the purposes of this application includes a central processing unit and a memory, the central processing unit being configured to invoke and run a computer program stored in the memory to perform the steps of the song search method described in this application.

[0059] A computer-readable storage medium is provided for another purpose of this application, which stores, in the form of computer-readable instructions, a computer program implemented according to the described song search method, which, when invoked by a computer, performs the steps included in the method.

[0060] A computer program product provided for another purpose of this application includes a computer program / instructions that, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0061] Compared with existing technologies, the advantages of this application are as follows:

[0062] First, in response to the request to search for songs by song, this application obtains a high-dimensional index vector representing the style-invariant features of a deep semantic information based on the encoded information of the song to be searched, using a pre-trained feature extraction model that has reached a convergence state. This high-dimensional index vector is then compared with the corresponding high-dimensional index vectors of candidate songs in a pre-defined song feature library for similarity calculation. Based on the similarity calculation results, target songs similar to the song to be searched are determined, and the link to the target song is then pushed to the client device. Since the high-dimensional index vectors of the candidate songs in the song feature library are also extracted using the same feature extraction model, they all achieve deep semantic representation of the corresponding audio data of the songs at different semantic scales. Therefore, semantic matching can be performed with the song to be searched, thereby accurately matching similar songs based on semantics. This end-to-end model architecture enables the client device to quickly query similar songs.

[0063] Secondly, because this application implements multi-scale feature extraction of deep semantic information of song audio data in its feature extraction model, the obtained high-dimensional index vector is more representative, such as representing global feature information, salient feature information, channel feature information, frequency band feature information, etc. of the song audio data, thereby achieving a more effective index of the corresponding song audio data. Based on this, downstream processing such as retrieval, querying, and matching of song audio data can achieve more accurate and efficient matching results, and can be used to serve a variety of application scenarios such as cover song recognition, song recognition, and humming recognition.

[0064] Furthermore, this application, based on end-to-end representation learning capabilities and supplemented by a retrieval matching mechanism, can achieve significant scalability. It can be deployed in the backend of online music service platforms to implement standardized interfaces, thereby serving the needs of various application scenarios, providing comprehensive and multi-purpose open services, and enhancing the economic advantages of the platform's music information retrieval. Attached Figure Description

[0065] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0066] Figure 1 This is a flowchart illustrating a typical embodiment of the song search method of this application;

[0067] Figure 2 This is a flowchart illustrating the process of obtaining encoded information in one embodiment of this application;

[0068] Figure 3 This is a flowchart illustrating the process of calculating and obtaining a similarity sequence in one embodiment of this application;

[0069] Figure 4This is a flowchart illustrating the process of calculating and obtaining a similarity sequence in another embodiment of this application;

[0070] Figure 5 This is a flowchart illustrating the process of obtaining encoded information in another embodiment of this application;

[0071] Figure 6 This is a flowchart illustrating the process of calculating and obtaining a similarity sequence in another embodiment of this application;

[0072] Figure 7 This is a flowchart illustrating the operation of a feature extraction model in one embodiment of this application.

[0073] Figure 8 This is a schematic diagram of the network architecture of a feature extraction model in one embodiment of this application;

[0074] Figure 9 This is a schematic diagram of the network architecture of the feature extraction model in another embodiment of this application;

[0075] Figure 10 This is a flowchart illustrating the working process of the residual convolutional block used in the feature extraction model of this application.

[0076] Figure 11 This is a flowchart illustrating the process of training the feature extraction model of this application.

[0077] Figure 12 This is a schematic diagram of the classification model that the feature extraction model of this application is connected to during the training phase.

[0078] Figure 13 This is a schematic block diagram of the song search device of this application;

[0079] Figure 14 This is a schematic diagram of the structure of a computer device used in this application. Detailed Implementation

[0080] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0081] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0082] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0083] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant) that may include a radio frequency receiver, pager, internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0084] The hardware referred to by the names "server," "client," and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0085] It should be noted that the concept of "server" used in this application can also be extended to the case of server clusters. Based on the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method in this application.

[0086] One or more of the technical features of this application, unless explicitly specified herein, can be deployed on a server and accessed by a client remotely calling the online service interface provided by the server, or can be directly deployed and run on a client for access.

[0087] Unless otherwise specified, the neural network models referenced or potentially referenced in this application may be deployed on a remote server and invoked remotely on the client, or deployed on a client with the capability to invoke directly. In some embodiments, when running on the client, the corresponding intelligence may be acquired through transfer learning in order to reduce the requirements on the client's hardware resources and avoid excessive consumption of the client's hardware resources.

[0088] Unless otherwise specified, all data involved in this application may be stored remotely on a server or on a local terminal device, as long as it is suitable for use by the technical solution of this application.

[0089] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0090] Unless otherwise expressly stated, the various embodiments disclosed in this application can be combined in a cross-cutting manner to flexibly construct new embodiments, as long as such combination does not depart from the inventive spirit of this application and can meet the needs of the prior art or solve a certain deficiency in the prior art. Those skilled in the art should be aware of such modifications.

[0091] The song search method of this application can be programmed into a computer program product and deployed on a server to run. In this way, the client can access the interface opened by the computer program product after it runs in the form of a web program or application, and realize human-computer interaction with the process of the computer program product through a graphical user interface.

[0092] Please see Figure 1 The song search method of this application, in its typical embodiment, includes the following steps:

[0093] Step S1100: Obtain the encoding information corresponding to the song to be searched submitted by the client:

[0094] The technical solution of this application can be deployed on the server of an online music platform, providing a variety of song search services to a massive number of platform users, including but not limited to song recognition, humming recognition, and cover song recognition.

[0095] In the music recognition service, users can record a piece of audio data, usually from an external source, on their client and submit it to the server as the song to be searched. The server then uses this data to find the original song or a corresponding cover song.

[0096] In the humming-based music recognition service, users can record a melody sung by a human voice on their client to obtain the corresponding audio data. Similarly, this data can be submitted to the server as a song to be searched, and the server will find songs with the same melody based on the similarity of the melodies.

[0097] In the cover song recognition service, users can specify or submit the audio data of a song to the server on their client as the song to be searched. The server then uses this data to find the same song or other different versions of the song, thereby determining whether there is a cover song or a similar song relationship between the two.

[0098] After the song submitted by the user arrives at the server, the server will first encode it accordingly in order to obtain its corresponding encoding information. During the encoding process, the audio data of the song to be searched can be adaptively processed according to the specific situation.

[0099] The audio data of the song to be searched submitted to the server can be in any format such as MP3, WMA, M4A, WAV, etc., or it can be audio data obtained by separating the audio from various video files. The audio data of the song to be searched is usually composed of multiple speech data packets in the time domain. Based on this, the speech data packets are transformed according to the specific encoding information type to obtain the corresponding encoding information.

[0100] The encoded information primarily describes style-invariant features in the song's audio data and can be of various types, including but not limited to time-spectrum information, Mel-spectrum information, CQT filter information, pitch contour information, and chroma feature information extracted from the audio data's speech packets. This type of information can be encoded using appropriate algorithms to obtain the corresponding type of encoded information. In this application, any of the above types of encoded information can be used for feature extraction. In practice, it is recommended to use the experimentally optimal CQT filter information to obtain the encoded information.

[0101] Those skilled in the art will understand that all the above-mentioned encoded information can be encoded using corresponding algorithms. During the encoding process, the song audio data needs to undergo conventional processing such as pre-emphasis, framing, and windowing before time-domain or frequency-domain analysis, i.e., speech signal analysis. Pre-emphasis aims to enhance the high-frequency components of the speech signal, smoothing the spectrum; pre-emphasis is typically achieved using a first-order high-pass filter. Before analyzing the speech signal, it needs to be framed, usually with each frame length set to 20ms. Considering frame shift, there can be a 10ms overlap between adjacent frames. Framing can be achieved by windowing the speech signal. Different window selections will affect the results of speech signal analysis; commonly, the Hamm window and its corresponding window function are used for windowing.

[0102] After completing the preprocessing required for speech signal analysis of the song audio data, further time-domain and frequency-domain analysis can be performed to achieve encoding and obtain the corresponding encoded information.

[0103] Regarding the aforementioned time-spectrum information, the speech data of each speech data packet in the time domain is pre-emphasized, framed, windowed, and transformed into the frequency domain using a short-time Fourier transform (STFT), thereby obtaining the corresponding data of the spectrogram and thus constituting the aforementioned time-spectrum information.

[0104] The Mel spectrum information can be obtained by filtering the time-spectrum information using a Mel-scale filter bank. Similarly, the corresponding Mel cepstral information can be obtained by taking the logarithm of the Mel spectrum information and performing a DCT transform. It can be understood that Mel spectrum information and its cepstral information can better describe the stylistically invariant features of a song, such as pitch, intonation, and timbre.

[0105] Regarding the CQT filtering information, since all notes in music are composed of several octaves of equal temperament (12-tone equal temperament), corresponding to the twelve semitones of an octave on a piano, the frequency ratio between adjacent semitones is 2. 1 / 12Clearly, for two octaves of the same pitch, the higher octave has twice the frequency of the lower octave. Therefore, in music, sound is distributed exponentially, but the audio spectrum obtained by Fourier transform is linearly distributed. The frequency points of the two cannot be directly correlated, which can lead to errors in the estimation of certain pitch frequencies. Therefore, the CQT (Constant Q Transform) time-frequency transform algorithm can be used to replace the Fourier transform for speech analysis. CQT refers to a filter bank where the center frequency is distributed exponentially, with different filtering bandwidths but a constant center frequency to bandwidth ratio of Q. Unlike the Fourier transform, the horizontal axis frequency of its spectrum is not linear but based on log2, and the filter window length can be adjusted according to different spectral line frequencies to obtain better performance. Since CQT has the same frequency distribution as the musical scale, the amplitude values ​​of the musical signal at each note frequency can be directly obtained by calculating the CQT spectrum of the music signal, making it more suitable for music signal processing. Therefore, this embodiment recommends using this information to perform corresponding encoding to obtain the corresponding encoded information, which is then used as the input to the neural network model of this application.

[0106] The pitch profile information mentioned above, including both PCP (Pitch Class Profile) and HPCP (Harmonic Pitch Class Profile), aims to extract the corresponding pitch sequence from the song's audio data. After normalization, merging, and segmentation, it is transformed into a melody profile sequence. Then, the standard pitch difference generated using standard pitch is converted into the corresponding feature representation. The encoded information constructed based on pitch profile information has good robustness to environmental noise.

[0107] The chroma feature information mentioned above is a collective term for chroma vectors and chromagrams. A chroma vector is a vector containing 12 elements, each representing the energy of one of the 12 pitch levels within a time period (e.g., one frame). The energy of the same pitch level across different octaves is accumulated. The chromagram is a sequence of chroma vectors. Specifically, after performing a short-time Fourier transform on the speech data packets of the song audio data to convert them from the time domain to the frequency domain, some noise reduction processing is performed, followed by tuning. The absolute time is converted into frames according to the length of the selected window, and the energy of each pitch within each frame is recorded, becoming a pitch map. Based on the pitch map, the energy (measured by loudness) of notes at the same time, pitch level, and different octaves is superimposed onto the elements of that pitch level in the chroma vector, becoming the chroma map. The data corresponding to this chroma map is the aforementioned chroma feature information.

[0108] Any of the specific encoding information described above can be used as input to the feature extraction model of this application. To facilitate processing by the feature extraction model, the encoding information can be organized according to a certain preset format. For example, the encoding information corresponding to each speech packet can be organized into a row vector. For the entire audio data, the row vectors of each speech packet can be organized into rows in time sequence to obtain a two-dimensional matrix as its complete encoding information. Such methods can be preset to suit the feature extraction model and can be flexibly implemented by those skilled in the art.

[0109] It should be noted that the encoding principle mentioned here applies not only to the song to be searched, but also to the processing of training samples by the feature extraction model during the training phase, which should be understood by those skilled in the art.

[0110] Step S1200: Using a feature extraction model trained to convergence, extract a high-dimensional index vector representing the deep semantic information of the song to be searched at multiple scales based on the encoded information.

[0111] This application presents a feature extraction model based on a convolutional neural network model for extracting deep semantic information from songs. The model is pre-trained to convergence and, after training, acquires the ability to extract deep semantic information from the audio data of a song at multiple scales based on the encoded information. This enables the model to learn the representation of style-invariant features of the corresponding song's audio data, and can then be used for querying, retrieving, and matching between songs.

[0112] The feature extraction model of this application is implemented to extract deep semantic information from multiple scales of the same audio data. This deep semantic information is represented as one or more high-dimensional index vectors to achieve feature representation of the audio data from multiple different aspects and / or angles. The high-dimensional index vector is essentially a high-dimensional vector, which serves as an index representation of the encoded information of the corresponding audio data at the semantic level. These different scales include global scales based on encoded information, or scales based on frequency bands, channels, etc., obtained through feature extraction. For a song, selecting any number of deep semantic information at two or more scales corresponding to its encoded information and representing them as high-dimensional index vectors allows for the feature representation of the multi-scale deep semantic information of the corresponding song.

[0113] Once the feature extraction model implemented based on the above principles is trained to convergence, it can open a service interface for the technical solution of this embodiment to call. The encoded information of the song to be searched is fed into the model, and the feature extraction model performs feature extraction based on the encoded information to obtain the high-dimensional index vector corresponding to the song to be searched.

[0114] It should be understood that, since the feature extraction model can extract deep semantic information of a song from multiple scales, different organizational forms can be used when converting this deep semantic information at different scales into the high-dimensional index vector. For example, the high-dimensional index vector can be represented as a single high-dimensional vector, which typically represents the deep semantic information of a song as a whole; or, the high-dimensional index vector can be represented as multiple discrete high-dimensional vectors according to the scale correspondence, with each high-dimensional vector corresponding to a scale. In any case, those skilled in the art can flexibly organize these high-dimensional vectors according to the needs of the actual scale semantic information, so as to facilitate the retrieval of the representation data of the overall deep semantic information of the song.

[0115] In this step, the feature extraction model extracts features from the encoded information of the song to be searched, and finally obtains the high-dimensional index vector corresponding to the song to be searched, which can be used for subsequent similarity matching.

[0116] Step S1300: Calculate the similarity between the high-dimensional index vector and the high-dimensional index vector in the preset song feature library that represents the deep semantic information of each candidate song at multiple scales extracted by the feature extraction model, and obtain a similarity sequence:

[0117] In an exemplary online music service platform, a music library is prepared, storing a massive amount of audio data corresponding to songs. To achieve the goal of searching for songs by song, the feature extraction model is used beforehand to extract the deep semantic information of each song in the music library, representing it as a corresponding high-dimensional index vector. Then, the mapping relationship data between these high-dimensional index vectors and their corresponding songs is stored in a preset song feature library. This song feature library stores the high-dimensional index vectors corresponding to each song. Subsequently, the high-dimensional index vectors corresponding to each song in the song feature library can be called to perform similarity calculations, so as to search for similar songs corresponding to the song to be searched. Therefore, for the song to be searched, the songs pointed to by the song feature library are the candidate songs in the search process.

[0118] The function of the feature extraction model in extracting high-dimensional index vectors from the audio data of songs has been described above, and the same applies to this step. Specifically, after the feature extraction model is trained to convergence, it can be used in the production stage. Online music service platforms can use this feature extraction model to extract the corresponding high-dimensional index vectors for each song in their music library, and then organize them into mapping relationship data stored in the song feature library, thus serving the needs of this application. Similarly, since the feature extraction model is suitable for extracting multi-scale features, the high-dimensional index vectors extracted from the encoded information of the audio data of each song in the music library also realize the multi-scale representation of the deep semantic information of the song, and there is usually an organizational correspondence between them and the high-dimensional index vectors of the song to be searched. Of course, those skilled in the art can sometimes adjust this correspondence as needed.

[0119] Based on the high-dimensional index vector of the song to be searched and the corresponding high-dimensional index vectors of each candidate song in the song feature library, a preset similarity calculation formula can be applied to calculate the similarity value between the song to be searched and each candidate song. The similarity calculation formula can be implemented using any algorithm suitable for calculating the similarity distance between data, such as cosine similarity algorithm, Euclidean distance algorithm, Pearson coefficient algorithm, Jaccard similarity algorithm, nearest neighbor search algorithm, etc. Those skilled in the art can implement it flexibly. After the similarity calculation, a similarity sequence is obtained between the high-dimensional index vector of the song to be searched and the corresponding high-dimensional index vectors of each candidate song in the song feature library. This similarity sequence stores the similarity values ​​between the song to be searched and each candidate song in the song feature library.

[0120] Step S1400: Filter and determine target songs in the similarity sequence whose similarity values ​​exceed a preset threshold and are the most similar, and construct the corresponding access link for the target song and push it to the client device:

[0121] After determining the similarity sequence corresponding to the song to be searched, a preset threshold, which can be an empirical or experimental threshold, can be used to filter the similarity values ​​in the similarity sequence, filtering out all elements whose similarity values ​​exceed the preset threshold. If the number of elements exceeding the preset threshold is 0, it indicates that there are no similar songs in the song feature library that are similar to the song to be searched. If multiple similarity values ​​are obtained after filtering, only the candidate song corresponding to the highest similarity value can be selected as the similar song corresponding to the song to be searched, that is, the target song obtained by searching for a song using a song.

[0122] Based on the song submitted or specified by the client device, this application identifies target songs that are semantically similar. It can then retrieve the corresponding audio data from the music library, obtain an access link to the audio data, encapsulate it into a playable page or playable data containing the link, and push it to the corresponding client device for further user access. Therefore, whether it's song recognition by listening, humming, or cover song identification, users only need to submit or specify the corresponding song to search for to obtain the appropriate result.

[0123] In other embodiments disclosed subsequently in this application, the process of searching for songs by song has various variations, which will not be discussed here. It is only based on the description of this typical embodiment that it is understood that the implementation of this application has numerous advantages, including but not limited to the following aspects:

[0124] First, in response to the request to search for songs by song, this application obtains a high-dimensional index vector representing the style-invariant features of a deep semantic information based on the encoded information of the song to be searched, using a pre-trained feature extraction model that has reached a convergence state. This high-dimensional index vector is then compared with the corresponding high-dimensional index vectors of candidate songs in a pre-defined song feature library for similarity calculation. Based on the similarity calculation results, target songs similar to the song to be searched are determined, and the link to the target song is then pushed to the client device. Since the high-dimensional index vectors of the candidate songs in the song feature library are also extracted using the same feature extraction model, they all achieve deep semantic representation of the corresponding audio data of the songs at different semantic scales. Therefore, semantic matching can be performed with the song to be searched, thereby accurately matching similar songs based on semantics. This end-to-end model architecture enables the client device to quickly query similar songs.

[0125] Secondly, because this application implements multi-scale feature extraction of deep semantic information of song audio data in its feature extraction model, the obtained high-dimensional index vector is more representative, such as representing global feature information, salient feature information, channel feature information, frequency band feature information, etc. of the song audio data, thereby achieving a more effective index of the corresponding song audio data. Based on this, downstream processing such as retrieval, querying, and matching of song audio data can achieve more accurate and efficient matching results, and can be used to serve a variety of application scenarios such as cover song recognition, song recognition, and humming recognition.

[0126] Furthermore, this application, based on end-to-end representation learning capabilities and supplemented by a retrieval matching mechanism, can achieve significant scalability. It can be deployed in the backend of online music service platforms to implement standardized interfaces, thereby serving the needs of various application scenarios, providing comprehensive and multi-purpose open services, and enhancing the economic advantages of the platform's music information retrieval.

[0127] Please see Figure 2 In a more detailed embodiment, step S1100, obtaining the encoding information corresponding to the song to be searched submitted by the client, includes the following steps:

[0128] Step S1111: Receive the song search request submitted by the client and obtain the audio data of the song to be searched specified in the request.

[0129] In this embodiment, a user records audio data on the song search page displayed on their client device, triggering a song search request that uses the audio data as the audio data required for the song to be searched. The request is then submitted to the server that provides the song search service in this application. The server parses the request and obtains the corresponding audio data.

[0130] Step S1112: Detect whether the audio data contains human vocal information; if not, terminate subsequent execution.

[0131] To improve matching accuracy, the server can preprocess the received audio data. For example, it can use a VAD logic module to detect whether the audio data contains a vocal melody. If the audio data does not contain a vocal melody, the subsequent execution of this application can be terminated, and a corresponding notification can be returned directly to the client device. When it is confirmed that there is audio data corresponding to a vocal melody, subsequent steps can continue. The VAD logic module can also be executed using various existing technologies well known to those skilled in the art; it is recommended to use an end-to-end pre-trained neural network model to achieve convergence.

[0132] Step S1113: Encode the audio data of the song to be searched to obtain the corresponding encoding information:

[0133] For audio data that has passed voice detection, the encoding principle described above in this application can be used to encode it to obtain the corresponding encoding information. As mentioned earlier, it is recommended to use CQT filtering information to construct the corresponding encoding information for the audio data of the song to be searched.

[0134] This embodiment preprocesses the audio data to be searched to filter out some audio data without voice, thereby avoiding frequent responses from online services to invalid songs and improving the matching accuracy of song search.

[0135] Please see Figure 3In one further embodiment, step S1300, calculating the similarity between the high-dimensional index vector and the high-dimensional index vector in the preset song feature library that represents the deep semantic information of each candidate song at multiple scales extracted by the feature extraction model, to obtain a similarity sequence, includes the following steps:

[0136] Step S1311: Call the preset song feature library to obtain the high-dimensional index vector corresponding to each candidate song. The high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of a candidate song at different scales.

[0137] In this embodiment, when the feature extraction model extracts and represents the encoded information of each candidate song in the music library, it sequentially concatenates the deep semantic information at various scales extracted from each candidate song to form a single high-dimensional vector, which serves as a separate high-dimensional index vector. This high-dimensional index vector represents the deep semantic information at different scales of its corresponding candidate song as a whole. Therefore, correspondingly, when the feature extraction model extracts and represents the features of the song to be searched, it also obtains a single high-dimensional vector with the same organization as the corresponding high-dimensional index vector.

[0138] Step S1312: Calculate the similarity between the high-dimensional index vector of the song to be searched and each high-dimensional index vector in the song feature library to obtain the corresponding similarity sequence; this similarity sequence stores the similarity values ​​corresponding to each candidate song in the song feature library:

[0139] Since each song corresponds to only one high-dimensional index vector, the computational complexity of similarity matching between the song to be searched and the candidate songs can be simplified. Here, the similarity calculation formula is directly applied based on their respective high-dimensional index vectors to calculate the similarity between the song to be searched and each candidate song, obtain the corresponding similarity value, and construct a separate similarity sequence. This similarity sequence stores the similarity values ​​corresponding to each candidate song in the song feature library.

[0140] Step S1313: Reverse sort the similarity sequence according to the similarity value to obtain the sorted similarity sequence output:

[0141] To facilitate subsequent screening and filtering, the elements in the similarity sequence can be sorted in reverse order, i.e., from largest to smallest, and then the sorted similarity sequence can be output.

[0142] In this embodiment, by limiting the high-dimensional index vector output by the feature extraction model to a single high-dimensional vector, the similarity calculation between the song to be searched and the candidate songs can be performed based on a simple formula, thereby simplifying system resource consumption, improving computational efficiency, and increasing the matching speed when searching for songs.

[0143] Please see Figure 4 In another embodiment of the deepening approach, step S1300, calculating the similarity between the high-dimensional index vector and the high-dimensional index vector in the preset song feature library that represents the deep semantic information of each candidate song at multiple scales extracted by the feature extraction model, to obtain a similarity sequence, includes the following steps:

[0144] Step S1321: Call the preset song feature library to obtain the high-dimensional index vector corresponding to each candidate song. The high-dimensional index vector is a set of high-dimensional vectors that represent the deep semantic information of a candidate song at various scales.

[0145] Unlike the previous embodiment, in this embodiment, when the feature extraction model extracts and represents features from the audio data of a song, it obtains a separate high-dimensional vector for each scale. Instead of concatenating these high-dimensional vectors, it directly associates them with their corresponding candidate songs and stores them in the song feature library, thus forming a high-dimensional index vector. That is, this high-dimensional index vector consists of multiple dispersed high-dimensional vectors, each representing the deep semantic information of a scale within the same song. Therefore, it can be understood that, adapting to this characteristic of the feature extraction model in this embodiment, when applying this model to extract and represent features from the search song, the corresponding high-dimensional index vector also includes multiple dispersed high-dimensional vectors, each representing the deep semantic information of a scale. Accordingly, it can be seen that the multiple high-dimensional vectors of the search song and the multiple high-dimensional vectors of the candidate songs correspond one-to-one with each other regarding the semantic scale obtained from the feature extraction by the feature extraction model.

[0146] Step S1322: Calculate the similarity between the high-dimensional vector of the song to be searched and the corresponding high-dimensional vector of each candidate song in the preset song feature library according to the high-dimensional vector corresponding to different semantic scales, and obtain the similarity sequence corresponding to each semantic scale:

[0147] Given that the high-dimensional index vector is composed of high-dimensional vectors at multiple scales, this embodiment calculates the similarity between the song to be searched and each candidate song according to their semantic scale correspondence, obtaining a similarity sequence for each semantic scale. Therefore, it is easy to understand that a corresponding similarity sequence can be obtained for each semantic scale.

[0148] Step S1323: Based on the correspondence between high-dimensional vectors at the semantic scale, summarize and merge the similarity values ​​in the similarity sequences corresponding to various semantic scales to obtain the summarized similarity sequence:

[0149] Since there are multiple similarity sequences corresponding to different semantic scales, it is necessary to integrate the results of each similarity sequence. In one embodiment, for each similarity sequence, the similarity values ​​corresponding to the same candidate song at different scales can be directly summed, or weighted summed, or averaged, so that the similarity values ​​corresponding to each candidate song at different scales are unified into the same similarity value, thereby forming a final aggregated similarity sequence.

[0150] Step S1324: Reverse sort the summarized similarity sequence according to the similarity values ​​to obtain the sorted similarity sequence output:

[0151] Similarly, the summarized similarity sequence can also be sorted in reverse, and the sorted similarity sequence can be output.

[0152] In this embodiment, the high-dimensional index vector output by the adaptive feature extraction model includes multiple dispersed high-dimensional vectors that correspond one-to-one with different semantic scales. This provides flexibility in accessing the deep semantic information extracted by the feature extraction model, allowing those skilled in the art to flexibly organize and use the high-dimensional vectors corresponding to different semantic scales according to their chosen scale. This can serve different business needs. For example, in the business scenario of song recognition, more attention can be paid to high-dimensional vectors representing the global scale and channel scale of audio data; in the business scenario of cover song recognition, more attention can be paid to high-dimensional vectors representing its global scale and frequency band scale. Such implementations are flexible and adaptable.

[0153] Please see Figure 5 In another further embodiment, step S1100, obtaining the encoding information corresponding to the song to be searched submitted by the client, includes the following steps:

[0154] Step S1121: Receive the song search request submitted by the client and obtain the audio data of the song to be searched specified in the request.

[0155] In this embodiment, a user records audio data on the song search page displayed on their client device, triggering a song search request that uses the audio data as the audio data required for the song to be searched. The request is then submitted to the server that provides the song search service in this application. The server parses the request and obtains the corresponding audio data.

[0156] Step S1122: Detect whether the audio data of the song to be searched exceeds a preset duration. If it does, divide it into multiple audio data segments corresponding to multiple song fragments according to the preset duration; otherwise, retain it as a whole audio data segment.

[0157] When the song to be searched is long, directly encoding and extracting features may sometimes affect accuracy. Therefore, in this case, the server can perform duration detection on the audio data of the song to be searched to see if it exceeds a preset duration. If it does not exceed the preset duration, the audio data of the song to be searched is treated as a single song segment; if it exceeds the preset duration, the song to be searched can be divided into multiple song segments according to the preset duration, obtaining the audio data corresponding to multiple song segments.

[0158] Step S1123: Encode each audio data segment to obtain the corresponding encoding information for each audio data segment, so that the feature extraction model can extract the corresponding high-dimensional index vectors for each song segment:

[0159] Regardless of whether the result of the previous processing steps is a single audio segment or multiple audio segments, each audio segment is independently encoded to obtain the corresponding encoding information. Subsequently, a feature extraction model is used to independently extract the multi-scale deep semantic information of each audio segment's encoding information to obtain multiple high-dimensional index vectors corresponding to the multiple audio segments.

[0160] This embodiment performs time-division processing on the song to be searched, allowing the feature extraction model to extract and represent features for each segment of the audio data of the song to be searched separately. This is expected to obtain the deep semantic information of the song to be searched in a more refined manner. Subsequently, when matching based on the corresponding high-dimensional index vectors of each segment of audio data, more accurate search matching results are expected to be obtained.

[0161] Please see Figure 6 According to the previous embodiment, step S1300, calculating the similarity between the high-dimensional index vector and the high-dimensional index vector in the preset song feature library that represents the deep semantic information of each candidate song at multiple scales extracted by the feature extraction model, and obtaining a similarity sequence, includes the following steps:

[0162] Step S1331: Call the preset song feature library to obtain the high-dimensional index vectors corresponding to the song segments of each candidate song. The high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of a song segment at different scales.

[0163] Adapted to the previous embodiment, when constructing the song feature library, each song in the library can be segmented according to a certain preset duration, and the same song can be processed into audio data corresponding to multiple song segments. Then, the feature extraction model is used to extract features for each song segment to obtain high-dimensional index vectors that represent the multi-scale semantic information of each song segment. Thus, a song has multiple high-dimensional index vectors corresponding to multiple song segments. Each high-dimensional index vector is associated with the song and stored in the song feature library. Each high-dimensional index vector can be used to represent the deep semantic information of a song segment at multiple scales.

[0164] Step S1332: Calculate the similarity between the high-dimensional index vector of each audio data segment and the corresponding high-dimensional index vector of each candidate song in the preset song feature library, to obtain the similarity sequence corresponding to each audio data segment:

[0165] Given that the high-dimensional index vector of each candidate song is composed of high-dimensional index vectors of multiple song segments, and that the song to be searched may also be segmented into multiple song segments to obtain multiple high-dimensional index vectors, this embodiment calculates the similarity between the song to be searched and the candidate songs by comparing the high-dimensional index vector corresponding to each song segment of the song to be searched with all high-dimensional index vectors in the song feature library. Therefore, it is easy to understand that a corresponding similarity sequence can be obtained for each song segment of the song to be searched. During this process, the high-dimensional index vector corresponding to each song segment of each candidate song will be compared with the high-dimensional index vector corresponding to each song segment of the song to be searched, forming a many-to-many computational relationship and achieving more in-depth interactive computation.

[0166] Step S1333: Summarize and merge the similarity values ​​in all similarity sequences according to the same candidate songs to obtain the summarized similarity sequence:

[0167] Since there are multiple similarity sequences corresponding to the song segments to be searched, and each similarity sequence may contain similarity values ​​between the song segment and multiple song segments of the same candidate song, it is necessary to integrate the results of each similarity sequence.

[0168] In the recommended embodiment, the similarity values ​​of each similarity sequence can be filtered first. Specifically, since a similarity sequence contains similarity values ​​between a specific song segment of the song to be searched and various song segments of a candidate song in the song feature library, the element with the largest similarity value is retained based on the relationship between the multiple similarity values ​​belonging to the same candidate song in the similarity sequence. This indicates that the specific song segment of the song to be searched is most similar to the song segment of the candidate song corresponding to the retained element, and is relatively dissimilar to other song segments of the candidate song. After this processing, in each similarity sequence, for each candidate song, one song segment whose similarity value is relatively highly similar to a song segment of the song to be searched is retained. For each song segment of the song to be searched, there is a corresponding similarity sequence.

[0169] Furthermore, the filtered similarity values ​​are aggregated according to their corresponding candidate songs. That is, the similarity values ​​of the same candidate songs in each similarity sequence are summed, weighted, or averaged to form a final aggregated similarity sequence. Each element in this similarity sequence can comprehensively represent the similarity value between the song to be searched and each candidate song.

[0170] In the transformation embodiment, based on the correspondence between song segments of candidate songs, the element with the largest similarity value can be selected from each similarity sequence. This element represents which song segment of the candidate song is most similar to the song segment of the search song, thereby obtaining an intermediate similarity sequence. This intermediate similarity sequence contains the similarity values ​​between each song segment of each candidate song and the song segment of the search song that is most similar to it. On this basis, according to the relationship of belonging to the same candidate song, the similarity values ​​of all song segments of the same candidate song are summed, averaged, etc., to obtain the final summarized similarity sequence.

[0171] Step S1334: Reverse sort the summarized similarity sequence according to the similarity values ​​to obtain the sorted similarity sequence output:

[0172] Similarly, the summarized similarity sequence can also be sorted in reverse, and the sorted similarity sequence can be output.

[0173] This embodiment utilizes song segmentation to perform similarity matching between the song to be searched and candidate songs in the music library, achieving compatible handling of various complex situations. On the one hand, when the song to be searched is too long, segmentation can improve the accuracy of matching; on the other hand, even if the song to be searched is a short segment, since the candidate songs are divided into multiple song segments for deep semantic information representation, it also helps to achieve more precise matching of specific song segments between the song to be searched and the candidate songs, facilitating location. In addition, based on the principle of segmentation, when calculating similarity, the similarity calculation between the song to be searched and each candidate song constitutes a many-to-many calculation relationship, realizing a deeper interactive calculation, making the correspondence between the song segments more accurate, and the calculated similarity value is theoretically more accurate, thus improving the overall accuracy of matching.

[0174] Please see Figure 7 In a preferred embodiment, when the feature extraction model is invoked, the following steps are performed:

[0175] Step S2100: Using multiple convolutional blocks in the shared network of the feature extraction model that has been trained to convergence, multi-level feature extraction is performed on the encoded information to obtain intermediate feature information that has extracted the deep semantic information of the encoded information.

[0176] The feature extraction model described herein is constructed based on the multi-branch concept and can be flexibly modified to adapt to the requirements of different embodiments of this application. A typical embodiment of the feature extraction model is as follows: Figure 8 As shown in the principle block diagram, the feature extraction model consists of a shared network and multiple branch networks. The shared network includes multiple convolutional blocks for progressively extracting deep semantic information from encoded information to obtain intermediate feature information. The multiple branch networks each perform different types of deep semantic information extraction based on the intermediate feature information to obtain corresponding output feature information. Each branch network contains a portion of the same structure, which includes multiple convolutional blocks for progressively extracting deep semantic information. After the last convolutional block is output, it can be processed differently according to the different functions of each branch network.

[0177] The convolutional blocks can be implemented using convolutional layers based on CNNs or RNNs, preferably convolutional blocks based on the residual convolution principle. To achieve contextual analysis and extract key information from the song audio data, an attention mechanism can be applied to any of the convolutional blocks, adding a corresponding attention module, specifically a Spatial Attention Module (SAM) or a Channel Attention Module (CAM). In an enhanced embodiment, instance normalization (IN) and batch normalization (BN) operations are applied to the convolutional blocks to divide the input information into two parts. One part undergoes instance normalization to learn style-invariant features, while the other part undergoes batch normalization for normalization. Therefore, the commonly known IBN architecture is applied. Using this architecture, highly diverse musical attribute invariant features of the song audio data, such as notes, rhythm, and timbre, can be learned while preserving version information.

[0178] Therefore, it is easy to understand that the feature extraction model adapts to different application scenarios, enables different branch networks, and is trained to convergence using a pre-selected training set to obtain the corresponding feature extraction capabilities. This makes it suitable for performing tasks corresponding to the application scenario, extracting the output feature information corresponding to the song audio data from the encoded information of the input song audio data. The training process of the feature extraction model will be given in the exemplary embodiments of this application, and will not be described here.

[0179] In this step, as Figure 8 In the architecture shown, the encoded information is processed by multiple convolutional blocks of the shared network for feature extraction, and in particular, the key information is extracted by the last convolutional block. This yields intermediate feature information containing the key information of the encoded information. This intermediate feature information is then divided into multiple outputs to the multiple branch networks so that different deep semantic information can be extracted from different perspectives in each branch network.

[0180] Step S2200: After extracting features from the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks of the feature extraction model, the extracted features are converted into output feature vectors of corresponding scales. The deep semantic information contained in the output feature vectors of each branch network is different.

[0181] As mentioned above, Figure 8In the architecture shown, the various branch networks can be flexibly selected and combined. Therefore, the specific number of branch networks can be determined based on the resulting architecture. The intermediate feature information output by the shared network is then input into each of the branch networks for further feature extraction processing.

[0182] according to Figure 8 The architecture shown has two convolutional blocks in each branch network that belong to the same structural part. The two convolutional blocks extract features from the output in sequence. The extracted output can then be processed differently according to the specific structure of different branch networks.

[0183] Specifically, different branch networks, adapting to the different deep semantic information they extract, can perform different processing on parts with different structures. For example, one branch network can be subjected to max pooling or mean pooling output; one branch network can be connected to a Dropout layer to randomly discard redundant features before max pooling output; in another branch network, the intermediate feature information output from the last convolutional block can be segmented into equal channels and then mean pooled before output; in yet another branch network, the intermediate feature information output from the last convolutional block can be segmented into equal frequency bands and then mean pooled before output. Through various processing of the feature information output from the last convolutional block, output feature information containing different deep semantic information can be obtained. These output feature information describe the deep semantic information of the song audio data from different scales, including global information and various local information, such as global information that abstracts the salient features of the encoded information of the song audio data, and local information that abstracts the channel or frequency band features of the encoded information of the song audio data, etc. Based on this, multiple output feature information with different representations can be obtained. These output feature information can be called independently or combined as needed.

[0184] In this application, the output feature information output by each branch network is normalized into an output feature vector representation. Therefore, multiple branch networks can obtain multiple output feature vectors accordingly. Each output feature vector represents the deep semantic information of the song audio data in different aspects or at different scales. The deep semantic information contained in each output feature vector is different from that in the others.

[0185] In practice, two or more branch networks are typically used to obtain two or more output feature vectors, thereby utilizing two or more deep semantic information to represent the song audio data. For example, the output feature vector representing the global information of the song audio data can be combined with the output feature vector representing the channel information of the song audio data; alternatively, the output feature vector representing the global information of the song audio data can be combined with the output feature vector representing the frequency band information of the song audio data; or the output feature vector representing the channel information of the song audio data can be combined with the output feature vector representing the frequency band information of the song audio data; or all output feature vectors can be combined. Such methods can be utilized as needed by those skilled in the art.

[0186] Step S2300: The feature extraction model outputs the output feature vectors of each branch network as the high-dimensional index vector.

[0187] The output feature vectors obtained by each branch network can be ultimately converted into high-dimensional index vectors for storage or direct use. These high-dimensional index vectors serve as indexes for the corresponding song audio data. Since each branch network has normalized its output feature information into output feature vectors, the high-dimensional index vectors can be processed flexibly depending on the specific application of the feature extraction model. For example, for applications requiring only storage for later use and separate retrieval, each output feature vector can be stored as multiple corresponding high-dimensional index vectors in the song feature library mentioned in this application, allowing for on-demand retrieval, querying, and matching of the high-dimensional index vectors output by different branch networks. Furthermore, for specific tasks such as song recognition, cover song recognition, and humming recognition, all output feature vectors from all the structured branch networks can be sequentially concatenated to obtain a single high-dimensional index vector. This high-dimensional index vector can be stored or used for matching immediately, for example, stored in the song feature library. Thus, the representation learning of the song audio data is achieved through the high-dimensional index vectors.

[0188] Based on the principles disclosed above in this typical embodiment, the song feature library can be prepared for some or all of the songs in the music library of an online music service platform according to the process of this embodiment. By applying the steps of this embodiment to the audio data of each corresponding song or its song fragment in the music library, a high-dimensional index vector corresponding to the audio data of each song can be obtained. By associating and storing these high-dimensional index vectors with the corresponding songs, a song feature library can be constructed. Subsequently, the high-dimensional index vector corresponding to any song can be directly called from the song feature library for retrieval, query, matching and other operations.

[0189] In addition to the various applications disclosed in this application, the mining and utilization based on the high-dimensional index vector obtained in this application may have many other uses, which can be flexibly applied by those skilled in the art based on the principles disclosed herein, without affecting the inventiveness of this application.

[0190] Based on the above description of the execution process and network architecture of the feature extraction model, it can be understood that this embodiment contains a wealth of beneficial effects, including but not limited to the following aspects:

[0191] First, the feature extraction model encodes corresponding coded information using the audio information of the song audio data to obtain style-invariant features of the song audio data. Then, an intermediate feature information is extracted from the coded information through a shared network. Based on the intermediate feature information, deep semantic information of the song audio data is extracted from different angles through multiple branch networks to obtain corresponding output feature information. Finally, these output feature information are used as the high-dimensional index vector corresponding to the song audio data to complete the end-to-end representation learning of the song audio data.

[0192] Secondly, since the feature extraction model adopts a combination of shared networks and multiple branch networks to achieve multi-angle feature extraction of deep semantic information of song audio data, the obtained high-dimensional index vector is more representative, such as representing global feature information, salient feature information, channel feature information, frequency band feature information, etc. of the song audio data. This enables a more effective indexing of the corresponding song audio data. Based on this, downstream processing such as retrieval, querying, and matching of song audio data can achieve more accurate and efficient matching results. It can be used in various application scenarios such as cover song recognition, song recognition by listening, humming recognition, and song copyright determination.

[0193] Furthermore, the output feature vectors obtained from the multiple branch networks of the feature extraction model can be combined into a single high-dimensional index vector or used independently as different high-dimensional index vectors, which can be flexibly determined according to the required deep semantic information. It has a wide range of uses and flexible applications. When processing representation learning of massive amounts of song audio data, it can achieve significant scalability. It can be deployed in the backend of online music service platforms to realize standardized interfaces, thereby serving the needs of various application scenarios, providing comprehensive and multi-purpose open services, and enhancing the economic advantages of the platform's music information retrieval.

[0194] In a further embodiment, step S2200, which involves extracting features from the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks of the feature extraction model, and then converting the extracted features into output feature vectors of the corresponding scale, includes any two or more of the following steps:

[0195] Step S2210: Use multiple convolutional blocks in the first branch network to extract features from the intermediate feature information to obtain global feature information, and pool the global feature information into a global-scale output feature vector:

[0196] Figure 8 In the exemplary first branch network, after extracting the intermediate feature information through two convolutional blocks with the same structure as other branch networks, the output of the last convolutional block is divided into two paths. One path directly performs mean pooling to obtain its overall feature information, while the other path randomly discards some time-frequency region information through a Dropout layer and then extracts the salient global feature information through max pooling. This results in two corresponding global output feature vectors. Based on this architecture, during model training, it improves the model's generalization ability to audio data with local time-frequency domain changes such as missing or inserted segments, and also helps prevent overfitting to some extent. Furthermore, the two global output feature vectors—one capturing overall features and the other capturing salient features—enhance the model's recognition ability.

[0197] Step S2220: After extracting features from the intermediate feature information using multiple convolutional blocks in the second branch network, the network is divided into multiple parts by channel and pooled to obtain the corresponding channel-scale output feature vector.

[0198] Since the feature information output by each convolutional block is usually represented as "number of channels * number of bandwidths * number of frames", it can be segmented according to the number of channels. Figure 8 In the exemplary second branch network, after extracting the intermediate feature information step-by-step through two convolutional blocks with the same structure as the other branch networks, the output of the last convolutional block is divided into multiple channels, such as two outputs. These outputs are then passed through 1*1 convolutional layers and subjected to mean pooling to obtain the corresponding channel output feature information for each of the two channels. In this process, the two channel branches focus on capturing local audio features. For audio with significant variations in adaptation and where much information is obscured by strong noise or other interference, feature representations can be established from a few salient, common local features.

[0199] Step S2230: After extracting features from the intermediate feature information using multiple convolutional blocks in the third branch network, the network is divided into multiple parts according to frequency band and pooled to obtain the corresponding output feature vector at the frequency band scale.

[0200] Figure 8In the exemplary third branch network, after extracting the intermediate feature information step-by-step through two convolutional blocks with the same structure as other branch networks, the output of the last convolutional block is average-pooled and then divided into multiple outputs, such as two outputs. After average-pooling, the corresponding frequency band output feature information is obtained. In this process, each frequency band branch focuses on extracting the feature information of its corresponding frequency band. This has a significant effect on resisting frequency band selective attenuation in harsh sound pickup environments, balancing the contribution of high and low frequency information in feature composition, and resisting the addition or deletion of content in a fixed range of frequency bands (such as adding or removing a drum sound) or strong interference in a fixed frequency range.

[0201] It is understandable that multiple output feature vectors obtained from the same branch network can be further processed into the same output feature vector by concatenation or mean pooling. Those skilled in the art can implement this flexibly.

[0202] In this embodiment, a rich branch network is used to extract multi-faceted and multi-scale feature information from the song audio data, enabling the obtained output feature vector to obtain rich deep semantic information representation. This represents not only the global and salient information of the song audio data, but also the relevant local information of the song audio data by channel and frequency band. Considering that the intermediate feature information has already captured the key information of the song audio data under the action of the shared network, this embodiment realizes the indexing value of the song audio data from multiple aspects. The high-dimensional index vector obtained subsequently can improve the accuracy of various aspects when used for querying, retrieval, and matching.

[0203] Because this embodiment can capture the deep semantic information of song audio data from multiple perspectives, it is particularly suitable for feature extraction of song audio data with a relatively large amount of data, and is especially suitable for application scenarios of long song processing. For such application scenarios, it can achieve a more accurate matching effect.

[0204] Please see Figure 9 Based on the previous embodiment, the network structure of the feature extraction model in this application is improved. It can be seen that... Figure 9 Network architecture and Figure 8 The difference in network architecture is that Figure 9 In the first branch network, the output of the last convolutional block is directly subjected to max pooling to obtain the global output feature vector, capturing the salient feature information of the encoded information of the song audio data. In the second branch network, the output of the last convolutional block is equally divided into four parts corresponding to the feature information of each channel. The feature information corresponding to each channel is subjected to mean pooling and then reassembled into the corresponding output feature vector. It is easy to understand that through the segmentation and construction of local branches, the obtained output feature vector can learn better local feature information.

[0205] Similarly, as a response to Figure 9 In the equivalent replacement embodiment of segmentation by channel, the output of the last convolutional block can also be segmented by frequency band instead of by channel to obtain the feature information corresponding to the four frequency bands. Then, the mean pooling process is performed as described above, and the segments are reassembled into the corresponding output feature vector.

[0206] This embodiment exemplarily provides a method based on... Figure 8 The modified network architecture shown is relatively lightweight. Based on this example, it is easy to understand that the inventive spirit of this application lies in the flexible combination and use of multiple described branch networks. Those skilled in the art, based on the principles disclosed in this application and the characteristics of the multi-scale deep semantic information possessed by the output feature vectors obtained from each branch network, can adapt to different specific applications and select feature extraction models constructed from different combinations of branch networks to create various other embodiments of this application to meet needs such as humming recognition, song recognition, and cover song recognition.

[0207] Please see Figure 10 In the optimized embodiment, when the convolutional block in the feature extraction model is invoked, the following steps are performed:

[0208] Step S3100: Perform convolution transformation on the input information to obtain transformed feature information:

[0209] In the feature extraction model of this application, each convolutional block, for the input information, whether it is the encoded information or the intermediate feature information output by the previous convolutional block, first undergoes a convolution operation through a 1*1 convolutional kernel to obtain the corresponding transformed feature information.

[0210] Step S3200: After performing instance normalization and batch normalization on the transformed feature information, combine them into concatenated feature information, and activate the output of the concatenated feature information:

[0211] After the first convolution, an instance batch normalization (IN) layer is applied to process the transformed feature information. This transformed feature information is divided into two paths. A batch normalization block (BN) is used to perform batch normalization on half of the channels, while instance normalization is applied to the remaining channels. The latter allows the corresponding convolutional block to capture the style-invariant features of the song's audio data. Therefore, for song representations with diverse styles within a single dataset, better utilization of the song representation is achieved. The two channels, after different normalization processes, are then concatenated into the same concatenated feature information for activation output.

[0212] Step S3300: After performing multiple convolution operations and batch normalization on the spliced ​​feature information of the activation output, the residual information is obtained.

[0213] The concatenated feature information from the activated output is further processed through multiple convolutional layers to extract more features. Each such convolutional layer is followed by a batch normalization layer for normalization before output. The last convolutional layer uses a 1x1 kernel to avoid the representation learning ability of the entire feature extraction model being attenuated due to normalization of multiple instances in multiple convolutional blocks. Therefore, the final output feature information is the residual information from the residual convolution process.

[0214] Step S3400: Superimpose the residual information onto the input information and activate the output.

[0215] Finally, based on the principle of residual convolution, referring to the transformation feature information obtained from the first convolution, the residual information is superimposed on it and then the output is activated to obtain the intermediate feature information output after the current convolution block undergoes residual convolution operation.

[0216] In this embodiment, the convolutional blocks required for the feature extraction model of this application are constructed by combining residual convolution with instance batch normalization. The residual convolutional network is an improvement on the basic model of the ResNet series, and is superimposed with the IBN architecture. The feature extraction model built in this way is easier to train and can achieve more accurate feature extraction results, which is particularly suitable for feature extraction of song audio data.

[0217] Please see Figure 11 In an extended embodiment, the training process of the feature extraction model includes the following iterative training steps:

[0218] Step S4100: Retrieve a training sample from the training set and determine the encoding information of the training sample. The training sample is pre-collected song audio data, which is a complete song or a segment thereof.

[0219] Those skilled in the art will understand that, to adapt to different downstream tasks, different training sets can be constructed for training the feature extraction model, each training set containing a sufficient number of training samples, and each training sample having a corresponding supervision label.

[0220] The training samples can be pre-collected by those skilled in the art. Each training sample is a song audio data set, adaptable to different downstream tasks. These song audio data sets can be complete songs, MIDI melody fragments, songs with accompaniment, songs with unaccompanied vocal parts, song fragments without melody, song fragments with melody, etc. Different versions of the same song can be merged into the same category, i.e., corresponding to the same supervision label, to enhance the model's generalization ability. When the duration of the song audio data in the training samples is too long, it can be further divided into multiple song fragments according to a certain preset duration, serving as multiple training samples associated with the same supervision label for training. When segmenting a song into fragments, the timestamps of the song lyrics can be referenced, allowing the song fragments to be segmented based on one or more complete lines of lyrics.

[0221] For the training samples in the training set, for the convenience of model training, the encoding information corresponding to their song audio data can be prepared in advance, or the corresponding encoding information can be obtained in real time when each song audio data is used to train the feature extraction model. As for the specific encoding principle, please refer to the corresponding process disclosed above in this application.

[0222] Step S4200: Input the encoded information into the feature extraction model and train it to obtain the corresponding output feature vectors:

[0223] During the training process of a training sample, the corresponding encoded information of the training sample is output to the feature extraction model for feature extraction. The principle of feature extraction is explained in the preceding embodiments and will not be repeated here. In this process, the feature extraction model learns the representation of the training sample to obtain the corresponding output feature vectors.

[0224] Step S4300: Perform classification prediction on each of the output feature vectors to map them to the corresponding classification labels:

[0225] In this application, the training task of the feature extraction model is understood as a classification task. Therefore, by connecting the output feature vectors of the feature extraction model to the corresponding pre-prepared classification models, examining the classification results of each classification model, and supervising them with corresponding supervision labels, the model can be trained. Based on this principle, during the training phase, when training the feature extraction model implemented in any embodiment of this application, a classification model is connected to the output end of each output feature vector of each branch network.

[0226] The classification model described above uses, for example... Figure 12The structure shown employs a batch normalization layer to perform batch normalization on the output feature vector, and then a fully connected layer to map the output feature vector to the classification space. The classification function calculates the classification probability of each classification label, thereby determining the classification label corresponding to the training sample as the one with the highest classification probability.

[0227] The classifier in the classification model can be constructed using a multi-class classifier implemented with the Softmax function, or it can be constructed using a multi-class classifier implemented with the AM-Softmax function, which can enhance intra-class compactness and expand inter-class sparsity. The latter obviously has better classification advantages.

[0228] Step S4400: Calculate the loss value of the feature extraction model using the supervision labels corresponding to the training samples and the classification labels, and perform gradient updates on the feature extraction model based on the loss value.

[0229] The classification model employs the batch normalization layer to achieve a balance between the triplet loss and the cross-entropy classification loss. Subsequently, the triplet loss can be calculated on the batch normalization layer, and the cross-entropy classification loss can be calculated on the fully connected layer. By combining these two losses, the output feature vector can be optimized.

[0230] Therefore, after the training samples are predicted with corresponding classification labels, the loss value between the supervision label and the classification label can be calculated based on the corresponding supervision label. Then, the feature extraction model is updated with gradients based on the loss value, and the weight parameters of each part of the model are corrected to promote model convergence.

[0231] Since there are multiple branch networks, each branch network may have multiple output feature vectors, and there are corresponding multiple classification models. Therefore, when calculating the loss value, a weighted approach can be used. That is, the triplet loss and classification loss in each classification model are first weighted and summed to obtain the loss value corresponding to each output feature vector. Then, the loss values ​​corresponding to each output feature vector are weighted and summed again to obtain the final loss value. The gradient update of the entire feature extraction model can be performed using this loss value.

[0232] Step S4500: Determine whether the loss value has reached a preset threshold. If it has not reached the preset threshold, call the next training sample in the training set to continue iterative training of the feature extraction model until the loss value reaches the preset threshold.

[0233] For each training sample, the loss value is determined by whether it infinitely approaches 0 or whether it reaches a preset threshold. When these conditions are met, the feature extraction model is considered to have converged, and training can be terminated, allowing the model to be deployed in production, such as for feature extraction from songs in a music library or for other downstream tasks. If convergence has not been achieved, the next training sample in the training set can be used to iteratively train the feature extraction model until it converges.

[0234] This embodiment reveals the training principle and process of the feature extraction model of this application. As can be seen from this embodiment, by training the feature extraction model with a pre-prepared training set, the feature extraction model can learn the ability to extract the corresponding output feature vector from the encoded information of the song audio data, thereby achieving effective representation learning of the deep semantic information of the song audio data. Furthermore, the output feature vectors of multiple scales of the same song audio data can be jointly trained, resulting in higher training efficiency and richer model functions. When put into the production stage, it can quickly obtain the deep semantic information corresponding to multiple scales of the same song audio data.

[0235] The classification model in this embodiment employs a multi-classifier with batch normalization and the AM-Softmax function. This allows for a balance between the triplet loss and the classification loss during gradient updates, enabling the model to converge more quickly. Furthermore, the trained model can more effectively learn representations of the deep semantic information in the song audio data. When the output feature vectors are subsequently combined as needed, they can more effectively represent the feature information of the song audio data, resulting in more efficient matching.

[0236] This embodiment also demonstrates the scalability and compatibility of the feature extraction model in this application. Specifically, this embodiment allows the feature extraction model to be trained using training samples corresponding to different downstream tasks to meet the needs of serving different downstream tasks. This enables the feature extraction model to serve different downstream tasks. Therefore, it is a relatively basic improvement with better economic efficiency.

[0237] Please see Figure 13This application provides a song search device, functionally deployed to adapt to the song search method of this application, including: a song encoding module 1100, a semantic extraction module 1200, a similarity matching module 1300, and a filtering and push module 1400. The song encoding module 1100 is used to obtain the encoding information corresponding to the song to be searched submitted by the client; the semantic extraction module 1200 is used to extract a high-dimensional index vector representing deep semantic information at multiple scales of the song to be searched based on the encoding information using a feature extraction model trained to convergence; the similarity matching module 1300 is used to calculate the similarity between the high-dimensional index vector and the high-dimensional index vectors representing deep semantic information at multiple scales of each candidate song extracted by the feature extraction model in a preset song feature library, obtaining a similarity sequence; the filtering and push module 1400 is used to filter and determine target songs in the similarity sequence whose similarity values ​​exceed a preset threshold and are the most similar, constructing a corresponding access link for the target song and pushing it to the client device.

[0238] In a further embodiment, the song encoding module 1100 includes: a request analysis submodule, used to receive a song search request submitted by a client and obtain the audio data of the song to be searched specified in the request; a voice detection submodule, used to detect whether the audio data contains human voice singing information, and if not, to terminate subsequent execution; and an encoding processing submodule, used to encode the audio data of the song to be searched to obtain corresponding encoding information.

[0239] In one further embodiment, the similarity matching module 1300 includes: a candidate invocation submodule, used to invoke a preset song feature library to obtain high-dimensional index vectors corresponding to each candidate song therein, wherein the high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of a candidate song at different scales; a unified calculation submodule, used to calculate the similarity between the high-dimensional index vector of the song to be searched and each high-dimensional index vector in the song feature library, to obtain a corresponding similarity sequence; the similarity sequence stores the similarity values ​​corresponding to each candidate song in the song feature library; and a sorting processing submodule, used to reverse sort the similarity sequence according to the similarity values ​​to obtain a sorted similarity sequence output.

[0240] In another embodiment, the similarity matching module 1300 includes: a candidate invocation submodule, used to invoke a preset song feature library to obtain high-dimensional index vectors corresponding to each candidate song, wherein the high-dimensional index vectors are multiple high-dimensional vectors that disperse and represent deep semantic information of a candidate song at various scales; a scale calculation submodule, used to calculate the similarity between the high-dimensional vector of the song to be searched and the corresponding semantic scale high-dimensional vector of each candidate song in the preset song feature library according to the high-dimensional vectors corresponding to different semantic scales, and obtain a similarity sequence corresponding to each semantic scale; a scale aggregation submodule, used to aggregate and merge the similarity values ​​in the similarity sequences corresponding to various semantic scales according to the correspondence of high-dimensional vectors at semantic scales, and obtain an aggregated similarity sequence; and a sorting processing submodule, used to reverse sort the aggregated similarity sequence according to the similarity values, and obtain a sorted similarity sequence output.

[0241] In another further embodiment, the song encoding module 1100 includes: a request analysis submodule, used to receive a song search request submitted by a client and obtain the audio data of the song to be searched specified in the request; a detection and segmentation submodule, used to detect whether the audio data of the song to be searched exceeds a preset duration; if it does, it is segmented into multiple audio data corresponding to multiple song segments according to the preset duration; otherwise, it is retained as a whole audio data segment; and a segment encoding submodule, used to encode each audio data segment separately to obtain the encoding information corresponding to each audio data segment, so that the feature extraction model can extract the corresponding high-dimensional index vectors of each song segment respectively.

[0242] According to another embodiment of the refinement, the similarity matching module 1300 includes: a candidate invocation submodule, used to invoke a preset song feature library to obtain high-dimensional index vectors corresponding to song segments of each candidate song, wherein the high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of a song segment at different scales; a segment calculation submodule, used to calculate the similarity between the high-dimensional index vector of each audio data segment and the corresponding high-dimensional index vector of each candidate song in the preset song feature library, to obtain a similarity sequence corresponding to each audio data segment; a segment aggregation submodule, used to aggregate and merge the similarity values ​​in all similarity sequences according to the same candidate songs, to obtain an aggregated similarity sequence; and a sorting processing submodule, used to reverse sort the aggregated similarity sequence according to the similarity values, to obtain a sorted similarity sequence output.

[0243] In a preferred embodiment, the feature extraction model is implemented with the following structure: a shared extraction module configured to sequentially perform multi-level feature extraction on the encoded information using multiple convolutional blocks in a shared network of the feature extraction model that has been trained to convergence, thereby obtaining intermediate feature information that extracts deep semantic information from the encoded information; a branch extraction module configured to perform feature extraction on the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks of the feature extraction model, and then convert it into output feature vectors of corresponding scales, wherein the deep semantic information contained in the output feature vectors of each branch network is different; and an output processing module configured to output the output feature vectors of each branch network as the high-dimensional index vector.

[0244] In a further embodiment, the branch extraction module includes any two or more of the following modules: a first extraction submodule, configured to use multiple convolutional blocks in the first branch network to extract features from the intermediate feature information to obtain global feature information, and pool the global feature information into a global-scale output feature vector; a second extraction submodule, configured to use multiple convolutional blocks in the second branch network to extract features from the intermediate feature information, and then divide it into multiple parts by channel for pooling, thereby obtaining a channel-scale output feature vector; and a third extraction submodule, configured to use multiple convolutional blocks in the third branch network to extract features from the intermediate feature information, and then divide it into multiple parts by frequency band for pooling, thereby obtaining a frequency band-scale output feature vector.

[0245] In a preferred embodiment, when the first branch network performs the pooling operation, it employs mean pooling and / or max pooling to obtain one or two output feature vectors at the global scale; when the second branch network performs the pooling operation, it employs mean pooling for one or more channels to obtain one or more output feature vectors at the channel scale; and when the third branch network performs the pooling operation, it employs mean pooling for one or more frequency bands to obtain one or more output feature vectors at the frequency band scale.

[0246] In an optional embodiment, the source of the encoded information is any one of the time-spectrum information, Mel spectrum information, CQT filter information, pitch contour information, and Chroma feature information of the corresponding audio data.

[0247] In a preferred embodiment, in the shared network, at least one of the convolutional blocks applies an attention module to extract key information from the song's audio data, and the attention module is a spatial attention module or a channel attention module.

[0248] In an optimized embodiment, the convolutional block is implemented with the following structure: an initial convolutional unit for performing convolutional transformation on the input information to obtain transformed feature information; a normalization processing unit for performing instance normalization and batch normalization on the transformed feature information and combining them into concatenated feature information, and activating the output of the concatenated feature information; a residual calculation unit for performing multiple convolutional operations and batch normalization on the activated output concatenated feature information to obtain residual information; and an activation output unit for superimposing the residual information onto the input information to activate the output.

[0249] In an extended embodiment, the feature extraction model is iteratively trained within a training task implemented by the following structure: a sample retrieval module, used to retrieve a training sample from the training set and determine the encoding information of the training sample, wherein the training sample is pre-collected song audio data, and the song audio data is a complete song or a segment thereof; a representation learning module, used to input the encoding information into the feature extraction model to train it and obtain corresponding output feature vectors; a classification prediction module, used to perform classification prediction on each of the output feature vectors respectively, so that they map to the corresponding classification labels; a loss calculation module, used to calculate the loss value of the feature extraction model using the supervision label corresponding to the training sample and the classification label, and to perform gradient updates on the feature extraction model based on the loss value; and an iterative decision module, used to determine whether the loss value reaches a preset threshold. If the preset threshold is not reached, the next training sample in the training set is retrieved to continue iterative training of the feature extraction model until the loss value reaches the preset threshold.

[0250] To address the aforementioned technical problems, embodiments of this application also provide computer equipment. For example... Figure 14 The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store a sequence of control information. When the computer-readable instructions are executed by the processor, the processor can implement a song search method. The processor of the computer device provides computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions, which, when executed by the processor, can cause the processor to execute the song search method of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 14The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0251] In this embodiment, the processor is used to execute... Figure 13 The system defines the specific functions of each module and its sub-modules. The memory stores the program code and various data required to execute these modules or sub-modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules / sub-modules in the song search device of this application. The server can call the server's program code and data to execute the functions of all sub-modules.

[0252] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the song search method of any embodiment of this application.

[0253] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0254] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0255] In summary, this application utilizes a feature extraction model to learn the representation of deep semantic information of song audio data at multiple scales to obtain high-dimensional index vectors. Based on these high-dimensional index vectors, similar songs are matched, resulting in more accurate and efficient performance in song querying, retrieval, and matching. This approach can serve various downstream tasks such as song recognition by listening, humming recognition, and cover song recognition, thereby enhancing the comprehensive service capabilities of online music platforms.

[0256] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in the prior art that are similar to those disclosed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0257] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A song search method, characterized in that, Includes the following steps: Obtain the encoding information corresponding to the song submitted by the client for searching; A high-dimensional index vector representing deep semantic information at multiple scales of the song to be searched is extracted from the encoded information using a feature extraction model that has been trained to convergence. Calculate the similarity between the high-dimensional index vector and the high-dimensional index vector in the preset song feature library that represents the deep semantic information of each candidate song at multiple scales extracted by the feature extraction model, and obtain a similarity sequence; Select target songs in the similarity sequence whose similarity values ​​exceed a preset threshold and are the most similar; construct the corresponding access link for the target song and push it to the client device; the target song is a cover song. When the feature extraction model is invoked, the following steps are performed: Multiple convolutional blocks in the shared network of a feature extraction model that has been trained to convergence are used to sequentially perform multi-level feature extraction on the encoded information to obtain intermediate feature information that has extracted the deep semantic information of the encoded information. After extracting features from the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks in the feature extraction model, the features are converted into output feature vectors of corresponding scales. The deep semantic information contained in the output feature vectors of each branch network is different. The feature extraction model outputs the feature vectors of each branch network as the high-dimensional index vector. The step of extracting intermediate feature information at different scales by using multiple convolutional blocks from two or more branches of the feature extraction model and then converting it into an output feature vector of the corresponding scale includes any two or more of the following steps: The intermediate feature information is extracted by using multiple convolutional blocks in the first branch network to obtain global feature information, and the global feature information is pooled into a global scale output feature vector. After extracting features from the intermediate feature information using multiple convolutional blocks in the second branch network, the network is divided into multiple parts by channel and pooled to obtain the corresponding channel-scale output feature vector. After extracting features from the intermediate feature information using multiple convolutional blocks in the third branch network, the network is divided into multiple parts according to frequency band and pooled to obtain the corresponding output feature vector at the frequency band scale.

2. The song search method according to claim 1, characterized in that, The process of obtaining the encoding information corresponding to the song to be searched submitted by the client includes the following steps: Receive a song search request submitted by the client and obtain the audio data of the song to be searched specified in the request; The system checks whether the audio data contains human vocal information; if not, it terminates the subsequent execution. The audio data of the song to be searched is encoded to obtain the corresponding encoding information.

3. The song search method according to claim 1, characterized in that, The step of calculating the similarity between the high-dimensional index vector and the high-dimensional index vector representing the deep semantic information of each candidate song at multiple scales extracted by the feature extraction model in the preset song feature library, and obtaining a similarity sequence, includes the following steps: A preset song feature library is invoked to obtain the high-dimensional index vector corresponding to each candidate song. The high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of a candidate song at different scales. The similarity between the high-dimensional index vector of the song to be searched and each high-dimensional index vector in the song feature library is calculated to obtain the corresponding similarity sequence; the similarity sequence stores the similarity value corresponding to each candidate song in the song feature library; The similarity sequence is sorted in reverse order based on the similarity value to obtain the sorted similarity sequence output.

4. The song search method according to claim 1, characterized in that, The step of calculating the similarity between the high-dimensional index vector and the high-dimensional index vector representing the deep semantic information of each candidate song at multiple scales extracted by the feature extraction model in the preset song feature library, and obtaining a similarity sequence, includes the following steps: A preset song feature library is invoked to obtain the high-dimensional index vector corresponding to each candidate song. The high-dimensional index vector is a set of high-dimensional vectors that represent the deep semantic information of a candidate song at various scales. The similarity between the high-dimensional vector of the song to be searched and the corresponding high-dimensional vector of each candidate song in the preset song feature library is calculated according to the high-dimensional vector corresponding to different semantic scales, so as to obtain the similarity sequence corresponding to each semantic scale. Based on the correspondence between high-dimensional vectors at the semantic scale, the similarity values ​​in the similarity sequences corresponding to various semantic scales are summarized and merged to obtain a summary similarity sequence. The summarized similarity sequence is sorted in reverse order based on the similarity values ​​to obtain the sorted similarity sequence output.

5. The song search method according to claim 2, characterized in that, The process of obtaining the encoding information corresponding to the song to be searched submitted by the client includes the following steps: Receive a song search request submitted by the client and obtain the audio data of the song to be searched specified in the request; If the audio data of the song to be searched exceeds a preset duration, it is divided into multiple audio data corresponding to multiple song segments according to the preset duration; otherwise, it is retained as the whole audio data. Each audio data segment is encoded separately to obtain the corresponding encoding information for each audio data segment, so that the feature extraction model can extract the corresponding high-dimensional index vectors for each song segment.

6. The song search method according to claim 5, characterized in that, The step of calculating the similarity between the high-dimensional index vector and the high-dimensional index vector representing the deep semantic information of each candidate song at multiple scales extracted by the feature extraction model in the preset song feature library, and obtaining a similarity sequence, includes the following steps: A preset song feature library is invoked to obtain the high-dimensional index vectors corresponding to the song segments of each candidate song. The high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of a song segment at different scales as a whole. For each audio data segment, the similarity between its high-dimensional index vector and the corresponding high-dimensional index vector of each candidate song in the preset song feature library is calculated to obtain the similarity sequence corresponding to each audio data segment. The similarity values ​​in all similarity sequences are aggregated and merged according to the same candidate songs to obtain the aggregated similarity sequence; The summarized similarity sequence is sorted in reverse order based on the similarity values ​​to obtain the sorted similarity sequence output.

7. The song search method according to claim 1, characterized in that, The source of the encoded information is any one of the following: time-spectrum information, Mel spectrum information, CQT filter information, pitch contour information, and Chroma feature information of the corresponding audio data.

8. The song search method according to claim 1, characterized in that, In the shared network, at least one of the convolutional blocks applies an attention module to extract key information from the song's audio data. The attention module is either a spatial attention module or a channel attention module.

9. The song search method according to claim 1, characterized in that, When the convolutional block is invoked, the following steps are performed: The information in the input is subjected to convolutional transformation to obtain transformation feature information; The transformed feature information is combined into spliced ​​feature information after being processed by instance normalization and batch normalization respectively, and the spliced ​​feature information is activated and output. The concatenated feature information of the activation output is processed through multiple convolution operations and batch normalization to obtain residual information; The residual information is superimposed onto the input information to activate the output.

10. The song search method according to claim 1, characterized in that, The training process of the feature extraction model includes the following iterative training steps: A training sample is retrieved from the training set, and the encoding information of the training sample is determined. The training sample is pre-collected song audio data, which is a complete song or a segment thereof. The encoded information is input into the feature extraction model and trained to obtain the corresponding output feature vectors. Each output feature vector is classified and predicted to map to the corresponding classification label. The loss value of the feature extraction model is calculated using the supervision label corresponding to the training sample and the classification label, and the feature extraction model is updated according to the gradient based on the loss value; Determine whether the loss value has reached a preset threshold. If it has not reached the preset threshold, call the next training sample in the training set to continue iterative training of the feature extraction model until the loss value reaches the preset threshold.

11. The song search method according to any one of claims 1 to 10, characterized in that: When the first branch network performs the pooling operation, it uses mean pooling and / or max pooling to obtain one or two output feature vectors at the global scale. When the second branch network performs the pooling operation, it uses mean pooling for one or more channels to obtain one or more output feature vectors at the channel scale. When the third branch network performs the pooling operation, it uses mean pooling for one or more frequency bands to obtain one or more output feature vectors at the frequency band scale.

12. A song search device, characterized in that, include: The song encoding module is used to obtain the encoding information corresponding to the song submitted by the client for searching; The semantic extraction module is used to extract a high-dimensional index vector representing deep semantic information at multiple scales of the song to be searched based on the encoded information using a feature extraction model that has been trained to convergence. The similarity matching module is used to calculate the similarity between the high-dimensional index vector and the high-dimensional index vector in the preset song feature library that represents the deep semantic information of each candidate song at multiple scales extracted by the feature extraction model, and to obtain a similarity sequence. The filtering and push module is used to filter and determine target songs in the similarity sequence whose similarity values ​​exceed a preset threshold and are the most similar, and to construct the corresponding access link of the target song and push it to the client device. The target song is a cover song. The feature extraction model is implemented as follows, which includes a shared extraction module configured to sequentially perform multi-level feature extraction on the encoded information using multiple convolutional blocks in the shared network of the feature extraction model that has been trained to convergence, thereby obtaining intermediate feature information that has extracted the deep semantic information of the encoded information. The branch extraction module is configured to extract features from the intermediate feature information at different scales using multiple convolutional blocks in two or more branch networks of the feature extraction model, and then convert them into output feature vectors of the corresponding scales. The deep semantic information contained in the output feature vectors of each branch network is different. The output processing module is configured to output the output feature vectors of each branch network as the high-dimensional index vector. The branch extraction module includes any two or more of the following modules: a first extraction submodule, configured to use multiple convolutional blocks in the first branch network to extract features from the intermediate feature information to obtain global feature information, and pool the global feature information into a global-scale output feature vector; a second extraction submodule, configured to use multiple convolutional blocks in the second branch network to extract features from the intermediate feature information, and then divide it into multiple parts by channel for pooling, thereby obtaining a channel-scale output feature vector; and a third extraction submodule, configured to use multiple convolutional blocks in the third branch network to extract features from the intermediate feature information, and then divide it into multiple parts by frequency band for pooling, thereby obtaining a frequency band-scale output feature vector.

13. A computer device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 11, which, when invoked by a computer, executes the steps included in the corresponding method.

15. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 11.

Images