Song indexing methods, devices, equipment, media, and products

Abstract

This application discloses a song indexing method, apparatus, device, medium, and product. The method includes: encoding audio information in song audio data to obtain corresponding encoded information; inputting the encoded information into a common network of a feature extraction model trained to convergence, and sequentially performing multi-level feature extraction on the encoded information through multiple convolutional blocks in the common network to obtain intermediate feature information; inputting the intermediate feature information into two or more branch networks of the feature extraction model, and after feature extraction by multiple convolutional blocks in each branch network, converting the intermediate feature information into a corresponding output feature vector, wherein the deep semantic information contained in the output feature vectors of each branch network is different; and using the output feature vector as a high-dimensional index vector representing the deep semantic information of the song audio data. This application improves the representation and learning capabilities of deep semantic information in song audio data.

Description

Technical Field

[0001] This application relates to the field of music information retrieval technology, and in particular to a song indexing method and corresponding apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rise of short videos, live streaming, and radio, the number of cover songs is increasing, and the scenarios requiring music recognition are becoming more complex. Compared to the original version, cover versions may differ from, or even be completely different from, other musical components such as timbre, fundamental frequency, rhythm, tempo, harmony, lyrics, singing style, and overall structure. Therefore, cover song recognition is a very challenging research task.

[0003] There are various existing technologies related to cover song recognition, and each of these technologies has certain shortcomings. For example: (1) Traditional landmark-based song recognition technology can only recognize songs with the same source version and cannot recognize the cover versions with certain differences mentioned above; (2) Traditional humming recognition technology based on melody matching can only recognize clean a cappella / humming and cannot recognize the cover versions with background accompaniment mentioned above; (3) Traditional cover song recognition technology mainly extracts audio features such as pitch class profile (PCP) and then uses algorithms such as dynamic programming to calculate the similarity distance between songs. Due to the diversity of cover versions, the above solutions are only applicable to cover versions with smaller modifications, have low accuracy, and are slow to recognize, making them unsuitable for searching massive amounts of music.

[0004] Therefore, existing technologies for song recognition lack universal applicability, have low accuracy, and are inefficient. It is necessary to explore more effective technical solutions. Summary of the Invention

[0005] The primary objective of this application is to solve at least one of the aforementioned problems by providing a song indexing method and corresponding apparatus, computer equipment, computer-readable storage medium, and computer program product.

[0006] To achieve the various objectives of this application, the following technical solution is adopted:

[0007] A song indexing method provided for one of the purposes of this application includes the following steps:

[0008] Encode the audio information in the song's audio data to obtain the corresponding encoded information;

[0009] The encoded information is input into a shared network of a feature extraction model that has been trained to convergence. Multiple convolutional blocks in the shared network sequentially perform multi-level feature extraction on the encoded information to obtain intermediate feature information that has extracted the deep semantic information of the song audio data.

[0010] The intermediate feature information is input into two or more branch networks of the feature extraction model. After feature extraction by multiple convolutional blocks in each branch network, the intermediate feature information is converted into a corresponding output feature vector. The deep semantic information contained in the output feature vectors of each branch network is different.

[0011] The output feature vectors of each branch network of the feature extraction model are used as high-dimensional index vectors representing the deep semantic information of the song audio data.

[0012] In a more detailed embodiment, the step of inputting the intermediate feature information into two or more branch networks of the feature extraction model includes any two or more of the following steps:

[0013] The intermediate feature information is input into the first branch network for extracting global information. In the first branch network, multiple convolutional blocks are used to extract features from the intermediate feature information to obtain global feature information. The global feature information is then pooled into a global output feature vector.

[0014] The intermediate feature information is input into the second branch network for extracting local information. In the second branch network, multiple convolutional blocks are used to extract the intermediate feature information and then pool it by channel to obtain the channel output feature vector of the song audio data.

[0015] The intermediate feature information is input into a third branch network for extracting local information. In the third branch network, multiple convolutional blocks are used to extract features from the intermediate feature information and then pooling is performed according to frequency bands to obtain the frequency band output feature vector of the song audio data.

[0016] In a preferred embodiment, when the first branch network performs the pooling operation, it employs mean pooling and / or max pooling to obtain one or two global output feature vectors; when the second branch network performs the pooling operation, it employs mean pooling for one or more channels to obtain one or more channel output feature vectors; when the third branch network performs the pooling operation, it employs mean pooling for one or more frequency bands to obtain one or more frequency band output feature vectors.

[0017] In an optional embodiment, in the step of encoding the audio information in the song audio data, the audio information is any one of the time-spectrum information, Mel spectrum information, CQT filter information, pitch contour information, and Chroma feature information of the song audio data.

[0018] In a specific embodiment, in the shared network, at least one of the convolutional blocks applies an attention module to extract key information from the song audio data, and the attention module is a spatial attention module or a channel attention module.

[0019] In a preferred embodiment, the convolutional block is a convolutional block that applies residual convolution.

[0020] In a further embodiment, the convolutional block is used to perform the following steps:

[0021] The information in the input is subjected to convolutional transformation to obtain transformation feature information;

[0022] The transformed feature information is combined into spliced ​​feature information after being processed by instance normalization and batch normalization respectively, and the spliced ​​feature information is activated and output.

[0023] The concatenated feature information of the activation output is processed through multiple convolution operations and batch normalization to obtain residual information;

[0024] The residual information is superimposed onto the input information to activate the output.

[0025] In an extended embodiment, the training process of the feature extraction model includes the following iterative training steps:

[0026] A training sample is retrieved from the training set, and the encoding information of the audio information of the training sample is determined. The training sample is pre-collected song audio data, which is a complete song or a segment thereof.

[0027] The encoded information is input into the feature extraction model and trained to obtain the corresponding output feature vectors.

[0028] Each output feature vector is classified and predicted to map to the corresponding classification label.

[0029] The loss value of the feature extraction model is calculated using the supervision label corresponding to the training sample and the classification label, and the feature extraction model is updated according to the gradient based on the loss value;

[0030] Determine whether the loss value has reached a convergence state. If it has not reached the preset threshold, call the next training sample in the training set to continue iterative training of the feature extraction model until the feature extraction model is trained to a convergence state.

[0031] In an extended embodiment, after the step of using the output feature vectors of each branch network of the feature extraction model as high-dimensional index vectors representing the deep semantic information of the song audio data, the following steps are included:

[0032] In response to a query request for audio data, the feature extraction model is invoked to extract the corresponding high-dimensional index vector as the query vector.

[0033] The similarity between the query vector and multiple high-dimensional index vectors in the preset song feature library is calculated to obtain a similarity data sequence. The high-dimensional index vectors stored in the song feature library are obtained by the feature extraction model by extracting the corresponding song audio data.

[0034] The song audio data corresponding to the maximum similarity exceeding a preset threshold in the similarity data sequence is identified as the similar songs of the query audio data.

[0035] In a specific embodiment, before the step of calculating the similarity between the query vector and multiple high-dimensional index vectors in a preset song feature library, the following steps are included:

[0036] Calculate the similarity between the query vector and the high-dimensional index vector of each song audio data in the no-melody feature library to obtain the corresponding similarity sequence. The high-dimensional index vector in the no-melody feature library is extracted by the feature extraction model from each song audio data without melody information.

[0037] The similarity of each song audio data in the similarity sequence is compared to whether it is lower than a preset threshold. If the similarity of all song audio data is lower than the preset threshold, the subsequent steps continue; otherwise, the subsequent steps are terminated.

[0038] In an extended embodiment, after the step of using the output feature vectors of each branch network of the feature extraction model as high-dimensional index vectors representing the deep semantic information of the song audio data, the following steps are included:

[0039] Obtain the audio data to be compared;

[0040] The feature extraction model is invoked to determine the high-dimensional index vector corresponding to the queried audio data;

[0041] Calculate the similarity between the high-dimensional index vector of the queried audio data and the high-dimensional index vector of the song audio data;

[0042] Determine whether the similarity exceeds a preset threshold. If it exceeds the preset threshold, determine that the queried audio data and the song audio data constitute a similar song.

[0043] In an optional embodiment, in the step of using the output feature vectors of each branch network of the feature extraction model as high-dimensional index vectors representing the deep semantic information of the song audio data, the output feature vectors of each branch network are concatenated into the same high-dimensional index vector or distributed into multiple high-dimensional index vectors and stored in the song feature library.

[0044] A song indexing device provided for one of the purposes of this application includes: an encoding processing module, a common extraction module, a branch extraction module, and an index processing module. The encoding processing module encodes audio information in song audio data to obtain corresponding encoded information. The common extraction module inputs the encoded information into a common network of a feature extraction model trained to convergence, and performs multi-level feature extraction on the encoded information through multiple convolutional blocks in the common network to obtain intermediate feature information that extracts deep semantic information from the song audio data. The branch extraction module inputs the intermediate feature information into two or more branch networks of the feature extraction model, and performs feature extraction on the intermediate feature information through multiple convolutional blocks in each branch network, converting it into corresponding output feature vectors. The deep semantic information contained in the output feature vectors of each branch network is different. The index processing module uses the output feature vectors of each branch network of the feature extraction model as high-dimensional index vectors representing the deep semantic information of the song audio data.

[0045] A computer device provided for one of the purposes of this application includes a central processing unit and a memory, the central processing unit being configured to invoke and run a computer program stored in the memory to perform the steps of the song indexing method described in this application.

[0046] A computer-readable storage medium is provided for another purpose of this application, which stores, in the form of computer-readable instructions, a computer program implemented according to the described song indexing method, which, when invoked by a computer, performs the steps included in the method.

[0047] A computer program product provided for another purpose of this application includes a computer program / instructions that, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0048] Compared with existing technologies, the advantages of this application are as follows:

[0049] First, this application uses the audio information of the song audio data to encode corresponding encoding information to obtain style-invariant features of the song audio data. Then, an intermediate feature information is extracted from the encoded information through a shared network. Based on the intermediate feature information, deep semantic information of the song audio data is extracted from different angles through multiple branch networks to obtain corresponding output feature information. Finally, these output feature information are used as the high-dimensional index vector corresponding to the song audio data to complete the end-to-end representation learning of the song audio data.

[0050] Secondly, because this application employs a combination of shared networks and multiple branch networks in its feature extraction model, it achieves multi-angle feature extraction of deep semantic information of song audio data. Therefore, the obtained high-dimensional index vector is more representative, such as representing global feature information, salient feature information, channel feature information, and frequency band feature information of the song audio data. This enables a more effective indexing of the corresponding song audio data. Based on this, downstream processing such as retrieval, querying, and matching of song audio data can achieve more accurate and efficient matching results. It can be universally used in various application scenarios such as cover song recognition, song recognition by listening, humming recognition, and song copyright determination.

[0051] Furthermore, the output feature vectors obtained from the multiple branch networks in this application can be combined into a single high-dimensional index vector or used independently as different high-dimensional index vectors, which can be flexibly determined according to the required deep semantic information. It has a wide range of uses and flexible applications. When processing representation learning of massive amounts of song audio data, it can achieve significant scalability. It can be deployed in the backend of online music service platforms to realize standardized interfaces, thereby serving the needs of various application scenarios, providing comprehensive and multi-purpose open services, and enhancing the economic advantages of the platform's music information retrieval. Attached Figure Description

[0052] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0053] Figure 1 This is a flowchart illustrating a typical embodiment of the song indexing method of this application;

[0054] Figure 2 This is a block diagram illustrating the network architecture principle of a feature extraction model in one embodiment of this application;

[0055] Figure 3 This is a flowchart illustrating the process of obtaining output feature vectors using multiple branch networks in one embodiment of this application.

[0056] Figure 4This is a block diagram illustrating the network architecture of a feature extraction model in another embodiment of this application.

[0057] Figure 5 This is a flowchart illustrating the working process of the residual convolutional block used in the feature extraction model of this application.

[0058] Figure 6 This is a flowchart illustrating the process of training the feature extraction model of this application.

[0059] Figure 7 This is a schematic diagram of the classification model that the feature extraction model of this application is connected to during the training phase.

[0060] Figure 8 This is a flowchart illustrating one implementation of the feature extraction model of this application for similar song matching.

[0061] Figure 9 This is a flowchart illustrating another implementation of the feature extraction model of this application for similar song matching, which uses a melody-free feature library for filtering.

[0062] Figure 10 This is a flowchart illustrating another implementation of the feature extraction model of this application for similar song matching, which serves the purpose of song infringement comparison.

[0063] Figure 11 This is a schematic block diagram of the song indexing device of this application;

[0064] Figure 12 This is a schematic diagram of the structure of a computer device used in this application. Detailed Implementation

[0065] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0066] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0067] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0068] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant) that may include a radio frequency receiver, pager, internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0069] The hardware referred to by the names "server," "client," and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0070] It should be noted that the concept of "server" used in this application can also be extended to the case of server clusters. Based on the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method in this application.

[0071] One or more of the technical features of this application, unless explicitly specified herein, can be deployed on a server and accessed by a client remotely calling the online service interface provided by the server, or can be directly deployed and run on a client to access the service.

[0072] Unless otherwise specified, the neural network models referenced or potentially referenced in this application may be deployed on a remote server and invoked remotely on the client, or deployed on a client with the capability to invoke directly. In some embodiments, when running on the client, the corresponding intelligence may be acquired through transfer learning in order to reduce the requirements on the client's hardware resources and avoid excessive consumption of the client's hardware resources.

[0073] Unless otherwise specified, all data involved in this application may be stored remotely on a server or on a local terminal device, as long as it is suitable for use by the technical solution of this application.

[0074] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0075] Unless otherwise expressly stated, the various embodiments disclosed in this application can be combined in a cross-cutting manner to flexibly construct new embodiments, as long as such combination does not depart from the inventive spirit of this application and can meet the needs of the prior art or solve a certain deficiency in the prior art. Those skilled in the art should be aware of such modifications.

[0076] The song indexing method of this application can be programmed into a computer program product and deployed on a server for execution. In this way, the client can access the interface opened by the computer program product after it runs in the form of a web page program or application program, and realize human-computer interaction with the process of the computer program product through a graphical user interface.

[0077] Please see Figure 1 The song indexing method of this application, in its typical embodiment, includes the following steps:

[0078] Step S1100: Encode the audio information in the song audio data to obtain the corresponding encoding information:

[0079] The aforementioned song audio data can be in any format such as MP3, WMA, M4A, WAV, etc., or it can be audio data obtained by separating audio from various video files. Song audio data typically consists of multiple time-domain speech data packets. The song audio data can originate from various songs pre-stored in the music library of an online music service platform, or it can be songs or a cappella audio submitted by users in real-time. This application can flexibly respond and process according to specific tasks to serve different types of needs. For example, when constructing a song feature library for song audio data in the music library to serve needs such as song recognition, humming recognition, and cover song recognition, feature extraction needs to be performed on each song audio data in the library. Similarly, when retrieving, querying, and matching songs to serve song recognition, humming recognition, cover song recognition, and song copyright determination, it is necessary to obtain the song audio data submitted by the client for feature extraction. Based on this, various transformations are typically performed on the speech data packets to encode the audio information in the song audio data as required by this application, thereby obtaining the corresponding encoded information.

[0080] The audio information primarily describes style-invariant features in the song's audio data and can be of various types, including but not limited to time-spectrum information, Mel-spectrum information, CQT filtering information, pitch contour information, and chroma feature information extracted from the speech data packets of the song's audio data. This type of information can be encoded using appropriate algorithms to obtain corresponding types of encoded information. In this application, any of the above types of encoded information can be used for feature extraction. In practice, it is recommended to use the experimentally optimal CQT filtering information to obtain the encoded information.

[0081] Those skilled in the art will understand that all the above-mentioned audio information can be encoded using corresponding algorithms. During encoding, the song audio data needs to undergo conventional processing such as pre-emphasis, framing, and windowing before time-domain or frequency-domain analysis, i.e., speech signal analysis. Pre-emphasis aims to enhance the high-frequency components of the speech signal, smoothing the spectrum; pre-emphasis is typically achieved using a first-order high-pass filter. Before analyzing the speech signal, it needs to be framed, usually with each frame length set to 20ms, and considering frame shift, there is a 10ms overlap between adjacent frames. Framing can be achieved by windowing the speech signal. Different window selections will affect the results of speech signal analysis; a common approach is to use the window function corresponding to the Hamm window to perform the windowing operation.

[0082] After completing the preprocessing required for speech signal analysis of the song audio data, further time-domain and frequency-domain analysis can be performed to achieve encoding and obtain the corresponding encoded information.

[0083] Regarding the aforementioned time-spectrum information, the speech data of each speech data packet in the time domain is pre-emphasized, framed, windowed, and transformed into the frequency domain using a short-time Fourier transform (STFT), thereby obtaining the corresponding data of the spectrogram and thus constituting the aforementioned time-spectrum information.

[0084] The Mel spectrum information can be obtained by filtering the time-spectrum information using a Mel-scale filter bank. Similarly, the corresponding Mel cepstral information can be obtained by taking the logarithm of the Mel spectrum information and performing a DCT transform. It can be understood that Mel spectrum information and its cepstral information can better describe the stylistically invariant features of a song, such as pitch, intonation, and timbre.

[0085] Regarding the CQT filtering information, since all notes in music are composed of several octaves of equal temperament (12-tone equal temperament), corresponding to the twelve semitones of an octave on a piano, the frequency ratio between adjacent semitones is 2. 1 / 12Clearly, for two octaves of the same pitch, the higher octave has twice the frequency of the lower octave. Therefore, in music, sound is distributed exponentially, but the audio spectrum obtained by Fourier transform is linearly distributed. The frequency points of the two cannot be directly correlated, which can lead to errors in the estimation of certain pitch frequencies. Therefore, the CQT (Constant Q Transform) time-frequency transform algorithm can be used to replace the Fourier transform for speech analysis. CQT refers to a filter bank where the center frequency is distributed exponentially, with different filtering bandwidths but a constant center frequency to bandwidth ratio of Q. Unlike the Fourier transform, the horizontal axis frequency of its spectrum is not linear but based on log2, and the filter window length can be adjusted according to different spectral line frequencies to obtain better performance. Since CQT has the same frequency distribution as the musical scale, the amplitude values ​​of the musical signal at each note frequency can be directly obtained by calculating the CQT spectrum of the music signal, making it more suitable for music signal processing. Therefore, this embodiment recommends using this information to perform corresponding encoding to obtain the corresponding encoded information, which is then used as the input to the neural network model of this application.

[0086] The pitch profile information mentioned above, including both PCP (Pitch Class Profile) and HPCP (Harmonic Pitch Class Profile), aims to extract the corresponding pitch sequence from the song's audio data. After normalization, merging, and segmentation, it is transformed into a melody profile sequence. Then, the standard pitch difference generated using standard pitch is converted into the corresponding feature representation. The encoded information constructed based on pitch profile information has good robustness to environmental noise.

[0087] The chroma feature information mentioned above is a collective term for chroma vectors and chromagrams. A chroma vector is a vector containing 12 elements, each representing the energy of one of the 12 pitch levels within a time period (e.g., one frame). The energy of the same pitch level across different octaves is accumulated. The chromagram is a sequence of chroma vectors. Specifically, after performing a short-time Fourier transform on the speech data packets of the song audio data to convert them from the time domain to the frequency domain, some noise reduction processing is performed, followed by tuning. The absolute time is converted into frames according to the length of the selected window, and the energy of each pitch within each frame is recorded, becoming a pitch map. Based on the pitch map, the energy (measured by loudness) of notes at the same time, pitch level, and different octaves is superimposed onto the elements of that pitch level in the chroma vector, becoming the chroma map. The data corresponding to this chroma map is the aforementioned chroma feature information.

[0088] Any of the specific audio information described above can be used as input to the feature extraction model of this application. To facilitate processing by the feature extraction model, the audio information can be converted into corresponding encoded information according to a certain preset format. For example, the audio information corresponding to each voice packet can be organized into a row vector, and the row vectors of each voice data packet in the song audio data can be organized into a row to obtain a two-dimensional matrix as the encoded information. Such methods can be preset to suit the feature extraction model and can be flexibly implemented by those skilled in the art.

[0089] Step S1200: Input the encoded information into a shared network of a feature extraction model that has been trained to convergence. Multiple convolutional blocks in this shared network sequentially perform multi-level feature extraction on the encoded information to obtain intermediate feature information that extracts the deep semantic information of the song's audio data.

[0090] In order to extract features from the corresponding song audio data based on the encoded information, this application proposes a novel feature extraction model based on a neural network model architecture. By pre-training the model to a convergent state, it learns the ability to extract deep semantic information from the song audio data based on the encoded information to obtain the corresponding output feature vector, thereby completing the representation learning of the style-invariant features of the song audio data for querying, retrieval, and matching.

[0091] The feature extraction model described herein is constructed based on the multi-branch concept and can be implemented in various more specific embodiments. A typical embodiment of the feature extraction model is as follows: Figure 2 As shown in the principle block diagram, the feature extraction model consists of a common network and multiple branch networks. The common network includes multiple convolutional blocks for progressively extracting deep semantic information from encoded information to obtain intermediate feature information. Each branch network extracts different types of deep semantic information based on the intermediate feature information to obtain corresponding output feature information. Each branch network contains a portion of the same structure, which includes multiple convolutional blocks for progressively extracting deep semantic information. After the last convolutional block is output, it can be processed differently according to the different functions of each branch network.

[0092] The convolutional blocks can be implemented using convolutional layers based on CNNs or RNNs, preferably convolutional blocks based on the residual convolution principle. To achieve contextual analysis and extract key information from the song audio data, an attention mechanism can be applied to any of the convolutional blocks, adding a corresponding attention module, specifically a Spatial Attention Module (SAM) or a Channel Attention Module (CAM). In an enhanced embodiment, instance normalization (IN) and batch normalization (BN) operations are applied to the convolutional blocks to divide the input information into two parts. One part undergoes instance normalization to learn style-invariant features, while the other part undergoes batch normalization for normalization. Therefore, the commonly known IBN architecture is applied. Using this architecture, highly diverse musical attribute invariant features of the song audio data, such as notes, rhythm, and timbre, can be learned while preserving version information.

[0093] Therefore, it is easy to understand that the feature extraction model is suitable for different application scenarios, enabling different branch networks. By first training it to a convergent state using a pre-selected training set, the corresponding feature extraction capability can be obtained, thus making it suitable for performing tasks corresponding to the application scenario, extracting the output feature information corresponding to the song audio data from the encoded information of the input song audio data. The training process of the feature extraction model will be given in the exemplary embodiments of this application, and will not be described here.

[0094] In this step, as Figure 2 In the architecture shown, the encoded information is processed by multiple convolutional blocks of a shared network for feature extraction, and in particular, the key information is extracted by the last convolutional block. This yields intermediate feature information containing the key information of the encoded information. This intermediate feature information is then divided into multiple outputs to the multiple branch networks so that different deep semantic information can be extracted from each branch network.

[0095] Step S1300: Input the intermediate feature information into two or more branch networks of the feature extraction model. After feature extraction by multiple convolutional blocks in each branch network, the intermediate feature information is converted into a corresponding output feature vector. The deep semantic information contained in the output feature vectors of each branch network is different.

[0096] As mentioned above, Figure 2In the architecture shown, the various branch networks can be flexibly selected and combined. Therefore, the specific number of branch networks can be determined based on the combined architecture. The intermediate feature information output by the common network is fed into each of the branch networks for further feature extraction processing.

[0097] according to Figure 2 The architecture shown has two convolutional blocks in each branch network that belong to the same structural part. The two convolutional blocks extract features from the output in sequence. The extracted output can then be processed differently according to the specific structure of different branch networks.

[0098] Specifically, different branch networks, adapting to the different deep semantic information they extract, can perform different processing on parts with different structures. For example, one branch network can be subjected to max pooling or mean pooling output; one branch network can be connected to a Dropout layer to randomly discard redundant features before max pooling output; in another branch network, the intermediate feature information output from the last convolutional block can be segmented into equal channels and then mean pooled before output; in yet another branch network, the intermediate feature information output from the last convolutional block can be segmented into equal frequency bands and then mean pooled before output. Through various processing of the feature information output from the last convolutional block, output feature information containing different deep semantic information can be obtained. These output feature information describe the deep semantic information of the song audio data from different perspectives, including global information and various local information, such as global information that abstracts the salient features of the encoded information of the song audio data, and local information that abstracts the channel or frequency band features of the encoded information of the song audio data, etc. Based on this, multiple corresponding output feature information with different representations can be obtained. These output feature information can be called independently or combined as needed.

[0099] In this application, the output feature information output by each branch network is normalized into an output feature vector representation. Therefore, multiple branch networks can obtain multiple output feature vectors accordingly. Each output feature vector represents the deep semantic information of the song audio data in different aspects, and the deep semantic information contained in each output feature vector is different from each other.

[0100] In practice, two or more branch networks are typically used to obtain two or more output feature vectors, thereby utilizing two or more deep semantic information to represent the song audio data. For example, the output feature vector representing the global information of the song audio data can be combined with the output feature vector representing the channel information of the song audio data; alternatively, the output feature vector representing the global information of the song audio data can be combined with the output feature vector representing the frequency band information of the song audio data; or the output feature vector representing the channel information of the song audio data can be combined with the output feature vector representing the frequency band information of the song audio data; or all output feature vectors can be combined. Such methods can be utilized as needed by those skilled in the art.

[0101] Step S1400: Use the output feature vectors of each branch network of the feature extraction model as high-dimensional index vectors representing the deep semantic information of the song audio data.

[0102] The output feature vectors obtained by each branch network can be ultimately converted into high-dimensional index vectors for storage or direct use. These high-dimensional index vectors serve as indexes for the corresponding song audio data. Since each branch network has normalized its output feature information into output feature vectors, the high-dimensional index vectors can be flexibly processed depending on the specific application of the feature extraction model. For example, for applications requiring only storage and separate retrieval, each output feature vector can be stored as multiple corresponding high-dimensional index vectors in a song feature library, allowing for on-demand retrieval, querying, and matching using the high-dimensional index vectors output by different branch networks. Furthermore, for specific tasks such as song recognition, cover song identification, humming recognition, and copyright comparison, all output feature vectors from all the structured branch networks can be sequentially concatenated to obtain a single high-dimensional index vector, which can be stored or used for matching immediately. Thus, the representation learning of the song audio data is achieved through the high-dimensional index vector.

[0103] Based on the principles disclosed above in this typical embodiment, a song feature library can be prepared for some or all of the songs in the music library of an online music service platform by following the process of this embodiment. By applying the steps of this embodiment to the audio data of each corresponding song or its segment in the music library, a high-dimensional index vector corresponding to the audio data of each song can be obtained. By associating and storing these high-dimensional index vectors with the corresponding songs, a song feature library can be constructed. Subsequently, the high-dimensional index vector corresponding to any song can be directly called from the song feature library for retrieval, query, matching and other operations.

[0104] Similarly, another application can be derived: applying the above steps, extracting the corresponding high-dimensional index vectors from the audio data of two songs or their segments, then comparing the similarity of the two high-dimensional index vectors, examining the data distance between them, and using a preset threshold to determine whether the two are similar. If they are similar, they can be considered identical; otherwise, they are different. This can be used for determining song copyright infringement or for simple matching between two songs.

[0105] In addition to the above applications, the mining and utilization based on the high-dimensional index vector obtained in this application may have many other uses, which can be flexibly applied by those skilled in the art based on the principles disclosed herein, without affecting the inventiveness of this application.

[0106] Based on the description of this exemplary embodiment, it can be understood that the implementation of this application has numerous advantages, including but not limited to the following aspects:

[0107] First, this application uses the audio information of the song audio data to encode corresponding encoding information to obtain style-invariant features of the song audio data. Then, an intermediate feature information is extracted from the encoded information through a shared network. Based on the intermediate feature information, deep semantic information of the song audio data is extracted from different angles through multiple branch networks to obtain corresponding output feature information. Finally, these output feature information are used as the high-dimensional index vector corresponding to the song audio data to complete the end-to-end representation learning of the song audio data.

[0108] Secondly, because this application employs a combination of shared networks and multiple branch networks in its feature extraction model, it achieves multi-angle feature extraction of deep semantic information of song audio data. Therefore, the obtained high-dimensional index vector is more representative, such as representing global feature information, salient feature information, channel feature information, and frequency band feature information of the song audio data. This enables a more effective indexing of the corresponding song audio data. Based on this, downstream processing such as retrieval, querying, and matching of song audio data can achieve more accurate and efficient matching results. It can be universally used in various application scenarios such as cover song recognition, song recognition by listening, humming recognition, and song copyright determination.

[0109] Furthermore, the output feature vectors obtained from the multiple branch networks in this application can be combined into a single high-dimensional index vector or used independently as different high-dimensional index vectors, which can be flexibly determined according to the required deep semantic information. It has a wide range of uses and flexible applications. When processing representation learning of massive amounts of song audio data, it can achieve significant scalability. It can be deployed in the backend of online music service platforms to realize standardized interfaces, thereby serving the needs of various application scenarios, providing comprehensive and multi-purpose open services, and enhancing the economic advantages of the platform's music information retrieval.

[0110] Please combine Figure 2 network architecture and Figure 3 The flowchart shown is based on Figure 2 In a specific implementation, and in a more detailed embodiment, step S1200, the step of inputting the intermediate feature information into two or more branch networks of the feature extraction model, includes any two or more of the following steps:

[0111] Step S1210: Input the intermediate feature information into the first branch network for extracting global information. In the first branch network, multiple convolutional blocks are used to extract features from the intermediate feature information to obtain global feature information. The global feature information is then pooled into a global output feature vector.

[0112] Figure 2 In the exemplary first branch network, after extracting the intermediate feature information through two convolutional blocks with the same structure as other branch networks, the output of the last convolutional block is divided into two paths. One path directly performs mean pooling to obtain its overall feature information, while the other path randomly discards some time-frequency region information through a Dropout layer and then extracts the salient global feature information through max pooling. This results in two corresponding global output feature vectors. Based on this architecture, during model training, it improves the model's generalization ability to audio data with local time-frequency domain changes such as missing or inserted segments, and also helps prevent overfitting to some extent. Furthermore, the two global output feature vectors—one capturing overall features and the other capturing salient features—enhance the model's recognition ability.

[0113] Step S1220: Input the intermediate feature information into the second branch network for extracting local information. In the second branch network, multiple convolutional blocks are used to extract features from the intermediate feature information, and then pooling is performed by channel to obtain the channel output feature vector of the song audio data.

[0114] Since the feature information output by each convolutional block is usually represented as "number of channels * number of bandwidths * number of frames", it can be segmented according to the number of channels. Figure 2In the exemplary second branch network, after extracting the intermediate feature information step-by-step through two convolutional blocks with the same structure as the other branch networks, the output of the last convolutional block is divided into multiple channels, such as two outputs. These outputs are then passed through 1*1 convolutional layers and subjected to mean pooling to obtain the corresponding channel output feature information for each of the two channels. In this process, the two channel branches focus on capturing local audio features. For audio with significant variations in adaptation and where much information is obscured by strong noise or other interference, feature representations can be established from a few salient, common local features.

[0115] Step S1230: Input the intermediate feature information into the third branch network for extracting local information. In the third branch network, multiple convolutional blocks are used to extract features from the intermediate feature information and then pooling is performed according to frequency bands to obtain the frequency band output feature vector of the song audio data.

[0116] Figure 2 In the exemplary third branch network, after extracting the intermediate feature information step-by-step through two convolutional blocks with the same structure as other branch networks, the output of the last convolutional block is average-pooled and then divided into multiple outputs, such as two outputs. After average-pooling, the corresponding frequency band output feature information is obtained. In this process, each frequency band branch focuses on extracting the feature information of its corresponding frequency band. This has a significant effect on resisting frequency band selective attenuation in harsh sound pickup environments, balancing the contribution of high and low frequency information in feature composition, and resisting the addition or deletion of content in a fixed range of frequency bands (such as adding or removing a drum sound) or strong interference in a fixed frequency range.

[0117] It is understandable that multiple output feature vectors obtained from the same branch network can be further processed into the same output feature vector by concatenation or mean pooling. Those skilled in the art can implement this flexibly.

[0118] In this embodiment, a rich branch network is used to extract multifaceted feature information from the song audio data, enabling the obtained output feature vector to obtain rich deep semantic information representation. This represents not only the global and salient information of the song audio data, but also the relevant local information of the song audio data by channel and frequency band. Considering that the intermediate feature information has already captured the key information of the song audio data under the action of the shared network, this embodiment realizes the indexing value of the song audio data from multiple aspects. The high-dimensional index vector obtained subsequently can improve the accuracy of various aspects when used for querying, retrieval, and matching.

[0119] Because this embodiment can capture the deep semantic information of song audio data from multiple perspectives, it is particularly suitable for feature extraction of song audio data with a relatively large amount of data. It is especially suitable for application scenarios involving whole song processing. For application scenarios such as whole song matching, cover song recognition, and song copyright comparison, it can achieve a more accurate matching effect.

[0120] Please see Figure 4 Based on the previous embodiment, the network structure of the feature extraction model in this application is improved. It can be seen that... Figure 4 Network architecture and Figure 2 The difference in network architecture is that Figure 4 In the first branch network, the output of the last convolutional block is directly subjected to max pooling to obtain the global output feature vector, capturing the salient feature information of the encoded information of the song audio data. In the second branch network, the output of the last convolutional block is equally divided into four parts corresponding to the feature information of each channel. The feature information corresponding to each channel is subjected to mean pooling and then reassembled into the corresponding output feature vector. It is easy to understand that through the segmentation and construction of local branches, the obtained output feature vector can learn better local feature information.

[0121] This embodiment exemplarily provides a method based on... Figure 2 The modified network architecture shown is relatively lightweight. Based on this principle, it is easy to understand that the focus of this application's inventive spirit lies in the flexible combination and use of multiple branch networks. Those skilled in the art, based on the principles disclosed in this application and the characteristics of the deep semantic information possessed by the output feature vectors obtained from each branch network, can adapt to different specific applications and select feature extraction models constructed from different combinations of branch networks to create various other embodiments of this application to meet needs such as humming recognition, song recognition, cover song recognition, and song copyright comparison.

[0122] Please see Figure 5 In a further embodiment, the convolutional block is implemented based on residual convolution and is used to perform the following steps:

[0123] Step S2100: Perform convolution transformation on the input information to obtain transformed feature information:

[0124] In the feature extraction model of this application, each convolutional block, for the input information, whether it is the encoded information or the intermediate feature information output by the previous convolutional block, first undergoes a convolution operation through a 1*1 convolutional kernel to obtain the corresponding transformed feature information.

[0125] Step S2200: After performing instance normalization and batch normalization on the transformed feature information, combine them into concatenated feature information, and activate the output of the concatenated feature information:

[0126] After the first convolution, an instance batch normalization (IN) layer is applied to process the transformed feature information. This transformed feature information is divided into two paths. A batch normalization block (BN) is used to perform batch normalization on half of the channels, while instance normalization is applied to the remaining channels. The latter allows the corresponding convolutional block to capture the style-invariant features of the song's audio data. Therefore, for song representations with diverse styles within a single dataset, better utilization of the song representation is achieved. The two channels, after different normalization processes, are then concatenated into the same concatenated feature information for activation output.

[0127] Step S2300: After performing multiple convolution operations and batch normalization on the spliced ​​feature information of the activation output, the residual information is obtained.

[0128] The concatenated feature information from the activated output is further processed through multiple convolutional layers to extract more features. Each such convolutional layer is followed by a batch normalization layer for normalization before output. The last convolutional layer uses a 1x1 kernel to avoid the representation learning ability of the entire feature extraction model being attenuated due to normalization of multiple instances in multiple convolutional blocks. Therefore, the final output feature information is the residual information from the residual convolution process.

[0129] Step S2400: Superimpose the residual information onto the input information and activate the output:

[0130] Finally, based on the principle of residual convolution, referring to the transformation feature information obtained from the first convolution, the residual information is superimposed on it and then the output is activated to obtain the intermediate feature information output after the current convolution block undergoes residual convolution operation.

[0131] In this embodiment, the convolutional blocks required for the feature extraction model of this application are constructed by combining residual convolution with instance batch normalization. The residual convolutional network is an improvement on the basic model of the ResNet series, and is superimposed with the IBN architecture. The feature extraction model built in this way is easier to train and can achieve more accurate feature extraction results, which is particularly suitable for feature extraction of song audio data.

[0132] Please see Figure 6 In an extended embodiment, the training process of the feature extraction model includes the following iterative training steps:

[0133] Step S3100: A training sample is retrieved from the training set, and the encoding information of the audio information of the training sample is determined. The training sample is pre-collected song audio data, which is a complete song or a segment thereof.

[0134] Those skilled in the art will understand that, to adapt to different downstream tasks, different training sets can be constructed for training the feature extraction model, each training set containing a sufficient number of training samples, and each training sample having a corresponding supervision label.

[0135] The training samples can be pre-collected by those skilled in the art. Each training sample is a song audio data set, adaptable to different downstream tasks. These song audio data sets can be complete songs, MIDI melody fragments, songs with accompaniment, songs with unaccompanied vocal parts, song fragments without melody, song fragments with melody, etc. Different versions of the same song can be merged into the same category, i.e., corresponding to the same supervision label, to enhance the model's generalization ability. When the duration of the song audio data in the training samples is too long, it can be further divided into multiple song fragments according to a certain duration, serving as multiple training samples associated with the same supervision label for training. When segmenting a song into fragments, the timestamps of the song lyrics can be referenced, allowing the song fragments to be segmented based on one or more complete lines of lyrics.

[0136] For the training samples in the training set, for the convenience of model training, the encoding information corresponding to their song audio data can be prepared in advance, or the corresponding encoding information can be obtained in real time when each song audio data is used to train the feature extraction model. As for the specific encoding principle, please refer to the corresponding process disclosed above in this application.

[0137] Step S3200: Input the encoded information into the feature extraction model and train it to obtain the corresponding output feature vectors:

[0138] During the training process of a training sample, the corresponding encoded information of the training sample is output to the feature extraction model for feature extraction. The principle of feature extraction is explained in the preceding embodiments and will not be repeated here. In this process, the feature extraction model learns the representation of the training sample to obtain the corresponding output feature vectors.

[0139] Step S3300: Perform classification prediction on each of the output feature vectors to map them to the corresponding classification labels:

[0140] In this application, the training task of the feature extraction model is understood as a classification task. Therefore, by connecting the output feature vectors of the feature extraction model to the corresponding pre-prepared classification models, examining the classification results of each classification model, and supervising them with corresponding supervision labels, the model can be trained. Based on this principle, during the training phase, when training the feature extraction model implemented in any embodiment of this application, a classification model is connected to the output end of each output feature vector of each branch network.

[0141] The classification model described above uses, for example... Figure 7 The structure shown employs a batch normalization layer to perform batch normalization on the output feature vector, and then a fully connected layer to map the output feature vector to the classification space. The classification function calculates the classification probability of each classification label, thereby determining the classification label corresponding to the training sample as the one with the highest classification probability.

[0142] The classifier in the classification model can be constructed using a multi-class classifier implemented with the Softmax function, or it can be constructed using a multi-class classifier implemented with the AM-Softmax function, which can enhance intra-class compactness and expand inter-class sparsity. The latter obviously has better classification advantages.

[0143] Step S3400: Calculate the loss value of the feature extraction model using the supervision labels corresponding to the training samples and the classification labels, and perform gradient updates on the feature extraction model based on the loss value.

[0144] The classification model employs the batch normalization layer to achieve a balance between the triplet loss and the cross-entropy classification loss. Subsequently, the triplet loss can be calculated on the batch normalization layer, and the cross-entropy classification loss can be calculated on the fully connected layer. By combining these two losses, the output feature vector can be optimized.

[0145] Therefore, after the training samples are predicted with corresponding classification labels, the loss value between the supervision label and the classification label can be calculated based on the corresponding supervision label. Then, the feature extraction model is updated with gradients based on the loss value, and the weight parameters of each part of the model are corrected to promote model convergence.

[0146] Since there are multiple branch networks, each branch network may have multiple output feature vectors, and there are corresponding multiple classification models. Therefore, when calculating the loss value, a weighted approach can be used. That is, the triplet loss and classification loss in each classification model are first weighted and summed to obtain the loss value corresponding to each output feature vector. Then, the loss values ​​corresponding to each output feature vector are weighted and summed again to obtain the final loss value. The gradient update of the entire feature extraction model can be performed using this loss value.

[0147] Step S3500: Determine whether the loss value has reached a convergence state. If it has not reached a preset threshold, call the next training sample in the training set to continue iterative training of the feature extraction model until the feature extraction model is trained to a convergence state.

[0148] For each training sample, the loss value is determined by whether it infinitely approaches 0 or whether it reaches a preset threshold. When these conditions are met, the feature extraction model is considered to have converged, and training can be terminated, allowing the model to be deployed in production, such as for feature extraction from songs in a music library or for other downstream tasks. If convergence has not been achieved, the next training sample in the training set can be used to iteratively train the feature extraction model until it converges.

[0149] This embodiment reveals the training principle and process of the feature extraction model of this application. As can be seen from this embodiment, by training the feature extraction model with a pre-prepared training set, the feature extraction model can learn the ability to extract the corresponding output feature vector from the encoded information of the song audio data, thereby achieving effective representation learning of the deep semantic information of the song audio data. Furthermore, multiple output feature vectors of the same song audio data can be jointly trained, resulting in higher training efficiency and richer model functions. When put into production, it can quickly obtain multiple aspects of the deep semantic information of the same song audio data.

[0150] The classification model in this embodiment employs a multi-classifier with a batch normalization layer and an AM-Softmax function. This balances the triplet loss and classification loss during gradient updates, enabling faster training and convergence. Furthermore, the trained model can more effectively learn representations of the deep semantic information in the song audio data. When the output feature vectors are combined as needed, they can more effectively represent the features of the song audio data, resulting in more efficient matching.

[0151] This embodiment also demonstrates the scalability and compatibility of the feature extraction model in this application. Specifically, this embodiment allows the feature extraction model to be trained using training samples corresponding to different downstream tasks to meet the needs of serving different downstream tasks. This enables the feature extraction model to serve different downstream tasks. Therefore, it is a relatively basic improvement with better economic efficiency.

[0152] Please see Figure 8To adapt to the need for matching songs in a music library based on a specified song or its fragments, in the extended embodiment, after step S1400 and the step of using the output feature vectors of each branch network of the feature extraction model as high-dimensional index vectors representing the deep semantic information of the song audio data, the following steps are included:

[0153] Step S4100: In response to the query request for audio data, the feature extraction model is invoked to extract the corresponding high-dimensional index vector as the query vector.

[0154] In this embodiment, the feature extraction model described in this application can be used to receive query requests submitted by users for the purposes of song recognition, cover song recognition, and humming song recognition. Based on the user's query request, the model determines the song provided or specified by the user as the query audio data, and then identifies whether the corresponding query audio data is similar to the songs in the song feature library. Based on the degree of similarity, the model identifies the same song (song recognition, humming song recognition) or determines whether it belongs to a certain original song (cover song recognition).

[0155] It is understood that in the song feature library, the feature extraction model of this application has been used to pre-extract the corresponding high-dimensional index vectors for each song audio data in a song library. The extraction process can be implemented according to steps S1100 to S1400. The high-dimensional index vector can be a high-dimensional vector obtained by concatenating multiple output feature vectors, or it can be obtained by combining multiple output feature vectors stored separately as needed. It is easy to understand that for song audio data that has been cut into multiple song segments in the feature extraction stage, multiple sets of high-dimensional index vectors corresponding to multiple song segments are generated. In this case, each song segment can be regarded as a song. If the duration of the query audio data is too long, it can also be divided into multiple song segments according to a fixed duration for processing. The query results of each song segment can then be comprehensively processed.

[0156] The song feature library typically stores mapping data between specific songs and their corresponding high-dimensional index vectors. Based on this, the summary information of the specific song audio data corresponding to the high-dimensional index vector can be quickly determined so as to output the results.

[0157] Regardless of whether the query audio data is a complete object or divided into multiple song segments, each segment is individually input into the feature extraction model of this application for feature extraction, obtaining its corresponding high-dimensional index vector. This high-dimensional index vector is generated according to the organization of the high-dimensional index vectors of the extracted song audio data in the song feature library. That is, if the high-dimensional index vector of each song audio data in the song feature library is independent, then the high-dimensional index vector of the query audio data is also correspondingly independent; if the former is multiple scattered vectors obtained from different branch networks and different outputs, then the latter is also multiple corresponding vectors. In other words, the final output structure of the feature extraction model of this application is consistent when performing feature extraction on the song library and when performing feature extraction on the query audio data. Accordingly, the high-dimensional index vector obtained from the query audio data constitutes the query vector corresponding to the query audio data.

[0158] Step S4400: Calculate the similarity between the query vector and multiple high-dimensional index vectors in the preset song feature library to obtain a similarity data sequence. The high-dimensional index vectors stored in the song feature library are obtained by the feature extraction model from the corresponding song audio data.

[0159] After obtaining the query vector, a preset similarity calculation formula can be used to calculate the similarity between the query vector and the corresponding high-dimensional index vectors of each song in the song feature library. The similarity calculation formula can be implemented using any algorithm suitable for calculating the similarity distance between data, such as cosine similarity algorithm, Euclidean distance algorithm, Pearson coefficient algorithm, Jaccard similarity algorithm, nearest neighbor search algorithm, etc. Those skilled in the art can implement it flexibly. After the similarity calculation, a similarity data sequence of the high-dimensional index vectors of each song in the song feature library corresponding to the query vector is obtained. This similarity data sequence stores the similarity values ​​between the query vector and each song in the song feature library.

[0160] It should be noted that if both the query vector and the high-dimensional index vector in the song feature library exist in a scattered form, multiple similarity data sequences can be obtained after calculating them separately. For the convenience of subsequent calculations, multiple similarity data sequences can be merged into a single similarity data sequence by averaging, weighting, or simply summing the similarity values ​​corresponding to the same songs.

[0161] Similarly, if the query audio data is segmented into multiple song segments and feature extraction is performed to obtain multiple query vectors, then after each query vector is calculated to obtain the single similarity data sequence, the final similarity data sequence of the query audio data can be obtained by summarizing the single similarity data sequences corresponding to multiple query vectors. The summarization method used is the same as described above, and can be any form such as averaging, simple summation, or weighted averaging.

[0162] Step S4500: Determine the song audio data corresponding to the maximum similarity score exceeding a preset threshold in the similarity data sequence as the similar songs of the query audio data.

[0163] After determining the final similarity data sequence corresponding to the query audio data, a preset threshold, which can be an empirical threshold, can be used to filter the similarity data sequence, filtering out all elements whose similarity values ​​exceed the preset threshold. If the number of elements exceeding the preset threshold is 0, it indicates that there are no similar songs in the music library that are similar to the query audio data. If multiple similarity values ​​are obtained after filtering, only the song corresponding to the highest similarity value can be selected as the similar song corresponding to the query audio data. Thus, the query audio data is matched with its corresponding similar songs.

[0164] This embodiment utilizes the feature extraction model of this application to extract a high-dimensional index vector from the query audio data as a query vector. It then calculates the similarity between this query vector and the high-dimensional index vectors of each song in a pre-constructed song feature library. Based on the specific similarity value, it matches the query audio data with corresponding similar songs, enabling downstream tasks such as song recognition by listening, song recognition by humming, and cover song recognition. This allows for the provision of corresponding services to a large number of users, finding similar songs based on user-submitted specified songs or song fragments. Because this application uses the high-dimensional index vectors of songs obtained through the feature extraction model to index the songs, and the similarity calculation between high-dimensional index vectors is efficient and fast, song matching is extremely rapid, efficient, and accurate.

[0165] Please see Figure 9 Based on the previous embodiment, in a specific embodiment, before step S4400, which calculates the similarity between the query vector and multiple high-dimensional index vectors in a preset song feature library, the following steps are included:

[0166] Step S4200: Calculate the similarity between the query vector and the high-dimensional index vectors of each song audio data in the melody-free feature library to obtain the corresponding similarity sequence. The high-dimensional index vectors in the melody-free feature library are extracted by the feature extraction model from each song audio data without melody information.

[0167] In the previous embodiment, when performing vector similarity calculation, the case of matching short song segments with a duration less than a predetermined length was not considered, as the segmented song segments might not have vocal melodies. However, song matching is usually based primarily on vocal melodies. To address this, this application can achieve filtering by pre-constructing a melody-free feature library. This melody-free feature library stores high-dimensional index vectors corresponding to the melody-free parts of the song segments segmented from the song audio data in the music library. These high-dimensional index vectors are pre-extracted using the feature extraction model of this application. Thus, the melody-free feature library can be used to filter the query audio data.

[0168] Specifically, after obtaining the query vector corresponding to the query audio data in step S4100, the similarity between the query vector and the high-dimensional index vector of all song audio data in the melody-free feature library can be calculated in the same way as the aforementioned similarity calculation to obtain the corresponding similarity sequence.

[0169] Step S4300: Compare the similarity of each song audio data in the similarity sequence to a preset threshold. If the similarity of all song audio data is lower than the preset threshold, continue with the subsequent steps; otherwise, terminate the subsequent steps.

[0170] To determine whether the query vector is similar to any high-dimensional index vector in the melody-free feature library, those skilled in the art can set a preset threshold for similarity based on prior knowledge or empirical data. Then, each similarity value of the similarity sequence obtained in the previous step is compared with the preset threshold. If the similarity values ​​of all elements in the similarity sequence are lower than the preset threshold, it indicates that there is no melody-free song segment in the melody-free feature library that is similar to the query audio data. Accordingly, steps S4400 and S4500 can be executed to further determine similar songs. Otherwise, if there is at least one element whose similarity value is higher than the preset threshold, it indicates that the query audio data is similar to at least one song segment in the melody-free feature library, and the song segment is obviously a melody-free segment. In this case, the subsequent steps can be terminated to save system resource consumption.

[0171] This embodiment calculates the similarity between the query vector corresponding to the query audio data and the high-dimensional index vector in the no-melody feature library, and then determines whether the query audio data belongs to the no-melody song fragment in the no-melody feature library according to a preset threshold. This achieves filtering of query requests, which can save server resource consumption and improve the efficiency of song query, retrieval and matching. For short fragment recognition, since short fragments are more likely to contain content without vocal melody, the effectiveness of the recognition process can be significantly improved and the processing efficiency of the corresponding downstream tasks can be improved.

[0172] Please see Figure 10 To meet the need for similarity assessment of two songs or song segments to determine whether there is any infringement, in the extended embodiment, after step S1400, which involves using the output feature vectors of each branch network of the feature extraction model as high-dimensional index vectors representing the deep semantic information of the song audio data, the following steps are included:

[0173] Step S5100: Obtain the query audio data to be compared:

[0174] In this embodiment, the song audio data extracted in the preceding embodiments can be used as the original audio data to be compared, or the copyrighted original song. According to the preceding embodiments, the high-dimensional index vector corresponding to the original song has been extracted by the feature extraction model of this application.

[0175] Furthermore, another song to be compared can be obtained, along with its corresponding query audio data. The song corresponding to the query audio data obtained here can be a song submitted or published by a user creator on the online music platform, or it can be an individual target song obtained by the online music platform itself from the target music library where infringement needs to be determined.

[0176] Step S5200: Call the feature extraction model to determine the high-dimensional index vector corresponding to the queried audio data:

[0177] Since infringement determination is required for the queried audio data, similarly, the feature extraction model described in this application is used to extract the corresponding high-dimensional index vector from the queried audio data in the same manner as described in the foregoing embodiments. Thus, the high-dimensional index vector corresponding to the original song and the high-dimensional index vector corresponding to the queried audio data are obtained. As can be seen from the preceding description of the correspondence in the organization of the two high-dimensional index vectors used to calculate similarity, the organization of the two high-dimensional index vectors here is also the same.

[0178] Step S5300: Calculate the similarity between the high-dimensional index vector of the queried audio data and the high-dimensional index vector of the song audio data.

[0179] Referring to the previous embodiment, the similarity calculation formula corresponding to any similarity algorithm such as cosine similarity algorithm, Euclidean distance algorithm, Pearson coefficient algorithm, Jaccard similarity algorithm, nearest neighbor search algorithm, etc., is used to calculate the similarity between the high-dimensional index vector (also called query vector) of the query audio data and the high-dimensional index vector of the original song, and the corresponding similarity value can be obtained.

[0180] Step S5400: Determine whether the similarity exceeds a preset threshold. If it exceeds the preset threshold, determine that the queried audio data and the song audio data constitute a similar song.

[0181] After obtaining the similarity score between two songs, a preset threshold, determined by a person skilled in the art based on empirical data or prior knowledge, is used to determine whether the two songs constitute similar songs. Specifically, the similarity score is compared to see if it exceeds the preset threshold. If it exceeds the preset threshold, it indicates that the queried audio data and the original song's audio data constitute similar songs, thus determining that the queried audio data constitutes an infringing work. Conversely, if it does not exceed the preset threshold, it indicates that the two songs do not constitute similar songs, requiring no special processing, or simply informing the relevant users.

[0182] This embodiment exemplifies the process by which the feature extraction model of this application is used to compare two songs to determine whether they constitute suspected infringement. It can be seen that, since the feature extraction model of this application has the ability to more accurately represent the feature information of related songs, it can more accurately make judgments and obtain corresponding infringement judgment results when comparing song infringements. This helps online service platforms to quickly investigate various song audio data, stop infringement in a timely manner, and prevent platform infringement risks.

[0183] Please see Figure 11This application provides a song indexing device, functionally deployed to adapt to the song indexing method of this application, including: an encoding processing module 1100, a common extraction module 1200, a branch extraction module 1300, and an indexing processing module 1400. The encoding processing module 1100 is used to encode audio information in song audio data to obtain corresponding encoded information. The common extraction module 1200 is used to input the encoded information into a common network of a feature extraction model trained to convergence, and perform multi-level feature extraction on the encoded information through multiple convolutional blocks in the common network. The intermediate feature information from which the deep semantic information of the song audio data is extracted is obtained; the branch extraction module 1300 is used to input the intermediate feature information into two or more branch networks of the feature extraction model, and after multiple convolutional blocks in each branch network extract the intermediate feature information, it is converted into a corresponding output feature vector. The deep semantic information contained in the output feature vectors of each branch network is different; the index processing module 1400 is used to use the output feature vectors of each branch network of the feature extraction model as high-dimensional index vectors representing the deep semantic information of the song audio data.

[0184] In a further embodiment, the branch extraction module 1300 includes any two or more of the following sub-modules: a first branch sub-module, used to input the intermediate feature information into a first branch network for extracting global information, where multiple convolutional blocks are used in the first branch network to extract features from the intermediate feature information to obtain global feature information, and pooling the global feature information into a global output feature vector; a second branch sub-module, used to input the intermediate feature information into a second branch network for extracting local information, where multiple convolutional blocks are used in the second branch network to extract features from the intermediate feature information and then pooling by channel to obtain the channel output feature vector of the song audio data; and a third branch sub-module, used to input the intermediate feature information into a third branch network for extracting local information, where multiple convolutional blocks are used in the third branch network to extract features from the intermediate feature information and then pooling by frequency band to obtain the frequency band output feature vector of the song audio data.

[0185] In a preferred embodiment, when the first branch network performs the pooling operation, it employs mean pooling and / or max pooling to obtain one or two global output feature vectors; when the second branch network performs the pooling operation, it employs mean pooling for one or more channels to obtain one or more channel output feature vectors; when the third branch network performs the pooling operation, it employs mean pooling for one or more frequency bands to obtain one or more frequency band output feature vectors.

[0186] In an optional embodiment, in the encoding processing module 1100, the audio information is any one of the time-spectrum information, Mel spectrum information, CQT filter information, pitch contour information, and Chroma feature information of the song audio data.

[0187] In a specific embodiment, in the shared network, at least one of the convolutional blocks applies an attention module to extract key information from the song audio data, and the attention module is a spatial attention module or a channel attention module.

[0188] In a preferred embodiment, the convolutional block is a convolutional block that applies residual convolution.

[0189] In a further embodiment, the convolutional block is constructed to include the following units: a convolutional transformation unit, used to perform convolutional transformation on the input information to obtain transformation feature information; a normalization processing unit, used to combine the transformation feature information into concatenated feature information after performing instance normalization and batch normalization processing respectively, and to activate the output of the concatenated feature information; an intermediate convolutional unit, used to obtain residual information by performing multiple convolutional operations and batch normalization processing on the activated output concatenated feature information; and a residual processing unit, used to superimpose the residual information onto the input information to activate the output.

[0190] In an extended embodiment, the feature extraction model is trained using a training task architecture with the following structure: a sample retrieval module, used to retrieve a training sample from the training set and determine the encoding information of the audio information of the training sample, wherein the training sample is pre-collected song audio data, and the song audio data is a complete song or a segment thereof; a training execution module, used to input the encoding information into the feature extraction model to train it and obtain corresponding output feature vectors; a classification prediction module, used to perform classification prediction on each of the output feature vectors respectively, so that they map to the corresponding classification labels; a gradient update module, used to calculate the loss value of the feature extraction model using the supervision label corresponding to the training sample and the classification label, and to perform gradient update on the feature extraction model according to the loss value; and a loop iteration module, used to determine whether the loss value has reached a convergence state. When it has not reached a preset threshold, the next training sample in the training set is retrieved to continue iterative training of the feature extraction model until the feature extraction model is trained to a convergence state.

[0191] In an extended embodiment, the song indexing device further includes: a request response module, used to respond to a query request for querying audio data and call the feature extraction model to extract its corresponding high-dimensional index vector as a query vector; a similarity calculation module, used to calculate the similarity between the query vector and multiple high-dimensional index vectors in a preset song feature library to obtain a similarity data sequence, wherein the high-dimensional index vectors stored in the song feature library are obtained by the feature extraction model from each corresponding song audio data; and a similarity determination module, used to determine the song audio data corresponding to the maximum similarity exceeding a preset threshold in the similarity data sequence as similar songs to the query audio data.

[0192] In a specific embodiment, the song indexing device further includes: a melody-free similarity calculation module, used to calculate the similarity between the query vector and the high-dimensional index vectors of each song audio data in the melody-free feature library, to obtain a corresponding similarity sequence, wherein the high-dimensional index vectors in the melody-free feature library are extracted by the feature extraction model from each song audio data with melody-free information; and a melody-free filtering processing module, used to compare whether the similarity of each song audio data in the similarity sequence is lower than a preset threshold. When the similarity of all song audio data is lower than the preset threshold, the similarity calculation module continues to run; otherwise, the other modules are terminated.

[0193] In an extended embodiment, the song indexing device further includes: a query acquisition module for acquiring query audio data to be compared; a query indexing module for calling the feature extraction model to determine the high-dimensional index vector corresponding to the query audio data; a query calculation module for calculating the similarity between the high-dimensional index vector of the query audio data and the high-dimensional index vector of the song audio data; and a similarity comparison module for determining whether the similarity exceeds a preset threshold, and determining that the query audio data and the song audio data constitute similar songs when the similarity exceeds the preset threshold.

[0194] In an optional embodiment, the index processing module 1400 is configured to concatenate the output feature vectors of each branch network into the same high-dimensional index vector or to distribute them into multiple high-dimensional index vectors and store them in the song feature library.

[0195] To address the aforementioned technical problems, embodiments of this application also provide computer equipment. For example... Figure 12The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store a sequence of control information. When the computer-readable instructions are executed by the processor, the processor can implement a song indexing method. The processor of the computer device provides computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions, which, when executed by the processor, can cause the processor to execute the song indexing method of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0196] In this embodiment, the processor is used to execute... Figure 11 The system contains the specific functions of each module and its sub-modules. The memory stores the program code and various data required to execute these modules or sub-modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules / sub-modules in the song indexing device of this application. The server can call the server's program code and data to execute the functions of all sub-modules.

[0197] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the song indexing method of any embodiment of this application.

[0198] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0199] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0200] In summary, this application improves the feature extraction model for song audio data, thereby enhancing the representation and learning capabilities of deep semantic information in song audio data. The deep semantic information obtained can achieve more accurate and efficient results in serving song querying, retrieval, and matching. It can serve various downstream tasks such as song recognition by listening, humming recognition, cover song recognition, and song copyright comparison, thus improving the comprehensive service capabilities of online music platforms.

[0201] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in the prior art that are similar to those disclosed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0202] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A song indexing method, characterized in that, Includes the following steps: Encode the audio information in the song's audio data to obtain the corresponding encoded information; The encoded information is input into a shared network of a feature extraction model that has been trained to convergence. Multiple convolutional blocks in the shared network sequentially perform multi-level feature extraction on the encoded information to obtain intermediate feature information that has extracted the deep semantic information of the song audio data. The intermediate feature information is input into two or more branch networks of the feature extraction model. After feature extraction by multiple convolutional blocks in each branch network, the intermediate feature information is converted into a corresponding output feature vector. The deep semantic information contained in the output feature vectors of each branch network is different. The output feature vectors of each branch network of the feature extraction model are used as high-dimensional index vectors representing the deep semantic information of the song audio data. The step of inputting the intermediate feature information into two or more branch networks of the feature extraction model includes any two or more of the following steps: The intermediate feature information is input into the first branch network for extracting global information. In the first branch network, multiple convolutional blocks are used to extract features from the intermediate feature information to obtain global feature information. The global feature information is then pooled into a global output feature vector. The intermediate feature information is input into the second branch network for extracting local information. In the second branch network, multiple convolutional blocks are used to extract the intermediate feature information and then pool it by channel to obtain the channel output feature vector of the song audio data. The intermediate feature information is input into a third branch network for extracting local information. In the third branch network, multiple convolutional blocks are used to extract features from the intermediate feature information and then pooling is performed according to frequency bands to obtain the frequency band output feature vector of the song audio data.

2. The song indexing method according to claim 1, characterized in that: When the first branch network performs the pooling operation, it uses mean pooling and / or max pooling to obtain one or two global output feature vectors accordingly. When the second branch network performs the pooling operation, it uses mean pooling for one or more channels to obtain one or more channel output feature vectors accordingly. When the third branch network performs the pooling operation, it uses mean pooling for one or more frequency bands to obtain one or more frequency band output feature vectors accordingly.

3. The song indexing method according to claim 1, characterized in that, In the step of encoding the audio information in the song audio data, the audio information is any one of the time-spectrum information, Mel spectrum information, CQT filter information, pitch contour information, and Chroma feature information of the song audio data.

4. The song indexing method according to claim 1, characterized in that, In the shared network, at least one of the convolutional blocks applies an attention module to extract key information from the song's audio data. The attention module is either a spatial attention module or a channel attention module.

5. The song indexing method according to claim 1, characterized in that, The convolutional block is a convolutional block that has applied residual convolution.

6. The song indexing method according to claim 5, characterized in that, The convolutional block is used to perform the following steps: The information in the input is subjected to convolutional transformation to obtain transformation feature information; The transformed feature information is combined into spliced ​​feature information after being processed by instance normalization and batch normalization respectively, and the spliced ​​feature information is activated and output. The concatenated feature information of the activation output is processed through multiple convolution operations and batch normalization to obtain residual information; The residual information is superimposed onto the input information to activate the output.

7. The song indexing method according to any one of claims 1 to 6, characterized in that, The training process of the feature extraction model includes the following iterative training steps: A training sample is retrieved from the training set, and the encoding information of the audio information of the training sample is determined. The training sample is pre-collected song audio data, which is a complete song or a segment thereof. The encoded information is input into the feature extraction model and trained to obtain the corresponding output feature vectors. Each output feature vector is classified and predicted to map to the corresponding classification label. The loss value of the feature extraction model is calculated using the supervision label corresponding to the training sample and the classification label, and the feature extraction model is updated according to the gradient based on the loss value; Determine whether the loss value has reached a convergence state. If it has not reached the preset threshold, call the next training sample in the training set to continue iterative training of the feature extraction model until the feature extraction model is trained to a convergence state.

8. The song indexing method according to any one of claims 1 to 6, characterized in that, After the step of using the output feature vectors of each branch network of the feature extraction model as high-dimensional index vectors representing the deep semantic information of the song audio data, the following steps are included: In response to a query request for audio data, the feature extraction model is invoked to extract the corresponding high-dimensional index vector as the query vector. The similarity between the query vector and multiple high-dimensional index vectors in the preset song feature library is calculated to obtain a similarity data sequence. The high-dimensional index vectors stored in the song feature library are obtained by the feature extraction model by extracting the corresponding song audio data. The song audio data corresponding to the maximum similarity exceeding a preset threshold in the similarity data sequence is identified as the similar songs of the query audio data.

9. The song indexing method according to claim 8, characterized in that, Before the step of calculating the similarity between the query vector and multiple high-dimensional index vectors in the preset song feature library, the following steps are included: Calculate the similarity between the query vector and the high-dimensional index vector of each song audio data in the no-melody feature library to obtain the corresponding similarity sequence. The high-dimensional index vector in the no-melody feature library is extracted by the feature extraction model from each song audio data without melody information. The similarity of each song audio data in the similarity sequence is compared to whether it is lower than a preset threshold. If the similarity of all song audio data is lower than the preset threshold, the subsequent steps continue; otherwise, the subsequent steps are terminated.

10. The song indexing method according to any one of claims 1 to 6, characterized in that, After the step of using the output feature vectors of each branch network of the feature extraction model as high-dimensional index vectors representing the deep semantic information of the song audio data, the following steps are included: Obtain the audio data to be compared; The feature extraction model is invoked to determine the high-dimensional index vector corresponding to the queried audio data; Calculate the similarity between the high-dimensional index vector of the queried audio data and the high-dimensional index vector of the song audio data; Determine whether the similarity exceeds a preset threshold. If it exceeds the preset threshold, determine that the queried audio data and the song audio data constitute a similar song.

11. The song indexing method according to any one of claims 1 to 6, characterized in that, In the step of using the output feature vectors of each branch network of the feature extraction model as high-dimensional index vectors representing the deep semantic information of the song audio data, the output feature vectors of each branch network are concatenated into the same high-dimensional index vector or distributed into multiple high-dimensional index vectors and stored in the song feature library.

12. A song indexing device, characterized in that, include: The encoding processing module is used to encode the audio information in the song's audio data to obtain the corresponding encoded information; The common extraction module is used to input the encoded information into the common network of the feature extraction model that has been trained to convergence. The encoded information is then subjected to multi-level feature extraction through multiple convolutional blocks in the common network to obtain intermediate feature information that has extracted the deep semantic information of the song audio data. The branch extraction module is used to input the intermediate feature information into two or more branch networks of the feature extraction model. After the intermediate feature information is extracted by multiple convolutional blocks in each branch network, it is converted into a corresponding output feature vector. The deep semantic information contained in the output feature vectors of each branch network is different. The index processing module is used to use the output feature vectors of each branch network of the feature extraction model as high-dimensional index vectors representing the deep semantic information of the song audio data. The branch extraction module includes any two or more of the following sub-modules: The first branch submodule is used to input the intermediate feature information into the first branch network for extracting global information. In the first branch network, multiple convolutional blocks are used to extract features from the intermediate feature information to obtain global feature information, and the global feature information is pooled into a global output feature vector. The second branch submodule is used to input the intermediate feature information into the second branch network for extracting local information. In the second branch network, multiple convolutional blocks are used to extract the intermediate feature information and then pool it by channel to obtain the channel output feature vector of the song audio data. The third branch submodule is used to input the intermediate feature information into the third branch network for extracting local information. In the third branch network, multiple convolutional blocks are used to extract features from the intermediate feature information and then pooling is performed according to the frequency band to obtain the frequency band output feature vector of the song audio data.

13. A computer device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 11, which, when invoked by a computer, executes the steps included in the corresponding method.

15. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 11.

Images