Song matching method and apparatus, device, medium, and product

Abstract

The application discloses a song matching method and device, equipment, medium and product, and the method comprises the following steps: obtaining the encoding information corresponding to the audio data of the to-be-matched karaoke melody submitted by a client; using a feature extraction model trained to a convergence state to extract a high-dimensional index vector representing the multi-scale deep semantic information of the to-be-matched karaoke melody according to the encoding information; calculating the similarity between the high-dimensional index vector and the high-dimensional index vector representing the multi-scale deep semantic information of each melody segment extracted by the feature extraction model in the melody feature library, and screening out a target melody segment satisfying a preset condition; and pushing a target song containing the target melody segment in a song library to a client device. Through the above process, the song search service can be quickly and efficiently realized, and the user can find a target song similar to the to-be-matched karaoke melody.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of music information retrieval, and in particular to a song matching method and a corresponding device, computer equipment, computer readable storage medium, and computer program product. BACKGROUND

[0002] When a user can only hum a certain segment (usually only a few seconds to tens of seconds) of a song, but does not know the song name, singer name, or lyrics, a traditional text keyword-based retrieval scheme is ineffective, and therefore a hum recognition technique is needed to search for a target song matching the user's hummed melody. The basic principle of the hum recognition technique is to automatically match a corresponding song according to the similarity of the hummed melody of the user to the melodies of songs in a song library.

[0003] Specifically, traditional hum-to-song techniques based on melody matching mostly rely on MIDI music melodies as target song samples for searching and matching, and have the following disadvantages:

[0004] 1. Low noise resistance and high misjudgment rate of the melody extraction algorithm:

[0005] Due to the limitations of the melody extraction algorithm, it is generally only used for the extraction of pure vocals, and the melody extraction accuracy of audio with accompaniment is very low and cannot meet the needs of practical applications. Therefore, when building a library, the MIDI data of a song is generally used as a song melody library; and if the hummed segment of the user contains a lot of noise, the accuracy of the extracted melody will be affected, leading to matching failure or misjudgment.

[0006] 2. Small song melody library and low recognition rate:

[0007] Due to the high cost of song MIDI production, the song melody library cannot be made large-scale; in addition, melody matching usually needs to calculate the distance through dynamic programming algorithms such as DTW, and the operation amount is large, so the library size is usually small; due to the small size of the song melody library, many songs cannot be recognized, and the recognition rate is low.

[0008] In summary, the technical solutions related to user hum recognition lack universal applicability, have low recognition accuracy, and are low in recognition efficiency, so the present applicant tries to explore more effective technical solutions. SUMMARY

[0009] The primary purpose of the present application is to solve at least one of the above problems and provide a song matching method and a corresponding device, computer equipment, computer readable storage medium, and computer program product.

[0010] To meet the various purposes of the present application, the present application adopts the following technical solutions:

[0011] A song matching method is provided for adapting to one of the purposes of the present application, comprising the following steps:

[0012] Obtaining the encoding information corresponding to the audio data of the karaoke melody submitted by the client and to be matched;

[0013] Using the feature extraction model trained to a convergent state to extract a high-dimensional index vector representing the multi-scale deep semantic information of the karaoke melody to be matched according to the encoding information;

[0014] Calculating the similarity between the high-dimensional index vector and the high-dimensional index vector representing the multi-scale deep semantic information of each melody segment extracted by the feature extraction model in the melody feature library, and screening out a target melody segment whose similarity meets a preset condition;

[0015] Pushing a target song containing the target melody segment in the song library to the client device.

[0016] In a deepened embodiment, the step of obtaining the encoding information corresponding to the audio data of the karaoke melody submitted by the client and to be matched comprises the following steps:

[0017] Receiving a song matching request submitted by the client and obtaining the audio data of the karaoke melody to be matched specified by the request;

[0018] Detecting whether the audio data contains human voice singing information, and terminating subsequent execution if it does not;

[0019] Formatting the audio data into audio data between a minimum preset time length and a maximum preset time length;

[0020] Encoding the audio data to obtain corresponding encoding information.

[0021] In an extended embodiment, before the step of calculating the similarity between the high-dimensional index vector and the high-dimensional index vector representing the multi-scale deep semantic information of each melody segment extracted by the feature extraction model in the melody feature library, the following steps are included:

[0022] Calling the non-melody feature library to verify the validity of the high-dimensional index vector of the karaoke melody to be matched, and determining its validity when the high-dimensional index vector and the high-dimensional index vector of the non-melody song segment in the non-melody feature library do not constitute a similarity, and then continuing to execute; the high-dimensional index vector in the non-melody feature library is a feature representation of a non-melody song segment extracted by the feature extraction model.

[0023] In the deepened embodiment, the calling of the a cappella melody feature library is used to verify the validity of the high-dimensional index vector of the a cappella melody to be matched. When the high-dimensional index vector does not form a similarity with the high-dimensional index vector of the a cappella song segment in the a cappella melody feature library, the execution is continued after determining the validity, including the following steps:

[0024] The a cappella melody feature library is called, and the a cappella melody feature library stores a plurality of preset high-dimensional index vectors corresponding to a cappella song segments. The high-dimensional index vector is obtained by extracting the encoding information of the corresponding a cappella song segment by the feature extraction model.

[0025] The similarity between the high-dimensional index vector of the a cappella melody to be matched and each high-dimensional index vector in the a cappella melody feature library is calculated respectively to obtain a corresponding similarity value.

[0026] It is judged whether there is an a cappella song segment with a similarity value exceeding a preset threshold. If there is, it is determined that the a cappella melody to be matched does not contain a song melody and the execution is terminated. If not, it is determined that the a cappella melody to be matched contains a song melody and the subsequent steps are continued.

[0027] In the deepened embodiment, the calculation of the similarity between the high-dimensional index vector and the high-dimensional index vector representing the multi-scale deep semantic information of each melody segment extracted by the feature extraction model in the melody feature library screens out target melody segments with a similarity satisfying a preset condition, including the following steps:

[0028] The melody feature library is called to obtain the high-dimensional index vector corresponding to each melody segment. The high-dimensional index vector is a single high-dimensional vector representing the deep semantic information of different scales of a melody segment as a whole.

[0029] The similarity between the high-dimensional index vector of the a cappella melody to be matched and each high-dimensional index vector in the melody feature library is calculated respectively to obtain a corresponding similarity sequence.

[0030] The melody segment with a similarity value exceeding a preset threshold in the similarity sequence is screened out, and it is determined as a target melody segment with a similarity satisfying a preset condition.

[0031] In the expanded embodiment, the method includes the following steps for constructing the melody feature library:

[0032] The audio data of the candidate song is obtained from the song library. According to the timestamp of the lyrics file of the candidate song, the audio data is segmented into audio data corresponding to a plurality of melody segments to be verified.

[0033] The audio data of each melody segment to be verified is encoded to obtain its encoding information.

[0034] For each melody segment to be verified, the feature extraction model is called to extract deep semantic information of multiple scales of the encoding information thereof, and a high-dimensional index vector thereof is obtained accordingly;

[0035] For each melody segment to be verified, the similarity between the high-dimensional index vector thereof and the high-dimensional index vectors of each non-melody song segment in the preset non-melody feature library is calculated, and a corresponding similarity value is obtained;

[0036] For each melody segment to be verified, it is judged whether there is a non-melody song segment with a similarity value exceeding a preset threshold, and if so, the melody segment to be verified is discarded, and if not, the mapping relationship data between the melody segment to be verified and the high-dimensional index vector thereof is added to the melody feature library.

[0037] In specific embodiments, when the feature extraction model is called, the following steps are performed:

[0038] The multiple convolution blocks in the shared network of the trained feature extraction model are used to sequentially perform multi-level feature extraction on the encoding information, and intermediate feature information extracted from the deep semantic information of the encoding information is obtained;

[0039] After the multiple convolution blocks in the two or more branch networks of the feature extraction model are used to perform feature extraction on the intermediate feature information at different scales, the intermediate feature information is converted into output feature vectors at corresponding scales, and the deep semantic information contained in the output feature vectors of each branch network is different;

[0040] The output feature vectors of each branch network are output as the high-dimensional index vector by the feature extraction model.

[0041] In a deepened embodiment, after the multiple convolution blocks in the two or more branch networks of the feature extraction model are used to perform feature extraction on the intermediate feature information at different scales, the intermediate feature information is converted into output feature vectors at corresponding scales, including any two or more of the following steps:

[0042] The multiple convolution blocks in the first branch network are used to perform feature extraction on the intermediate feature information to obtain global feature information, and the global feature information is pooled into an output feature vector at a global scale;

[0043] After the multiple convolution blocks in the second branch network are used to perform feature extraction on the intermediate feature information, the intermediate feature information is divided into multiple parts according to channels and pooled, and an output feature vector at a channel scale is obtained accordingly;

[0044] After the multiple convolution blocks in the third branch network are used to perform feature extraction on the intermediate feature information, the intermediate feature information is divided into multiple parts according to frequency bands and pooled, and an output feature vector at a frequency band scale is obtained accordingly.

[0045] In a preferred embodiment, when the first branch network performs the pooling operation, mean pooling and / or max pooling operation is used to obtain one or two output feature vectors of the global scale; when the second branch network performs the pooling operation, mean pooling operation is used for single or multiple channels to obtain one or more output feature vectors of the channel scale; and when the third branch network performs the pooling operation, mean pooling operation is used for single or multiple frequency bands to obtain one or more output feature vectors of the frequency band scale.

[0046] In a further embodiment, when the convolution block is called, the following steps are performed:

[0047] Convolution transformation is performed on the information in the input to obtain transformed feature information;

[0048] The transformed feature information is respectively subjected to instance normalization and batch normalization processing, and then combined into spliced feature information, and the spliced feature information is activated and output;

[0049] The activated and output spliced feature information is subjected to multiple convolution operations and batch normalization processing to obtain residual information;

[0050] The residual information is superimposed on the information in the input to activate and output.

[0051] In an optional embodiment, in the shared network, at least one of the convolution blocks applies an attention module for extracting key information from the audio data of the song segment, and the attention module is a spatial attention module or a channel attention module.

[0052] In an optional embodiment, the source of the encoding information is any one of the time-frequency spectrum information, the mel spectrum information, the CQT filter information, the loudness contour information, and the Chroma feature information of the corresponding audio data.

[0053] In an extended embodiment, the training process of the feature extraction model includes the following steps of iterative training:

[0054] A training sample is called from the training set, and the encoding information of the training sample is determined, the training sample being a song segment with melody, which contains background music or does not contain background music, and the melody therein being a human voice singing or a MIDI voice simulation;

[0055] The encoding information is input into the feature extraction model to implement training, so as to obtain corresponding output feature vectors;

[0056] Each of the output feature vectors is subjected to classification prediction to map out a corresponding classification label;

[0057] Calculate a loss value of the feature extraction model by using the supervised label corresponding to the training sample and the classification label, and perform gradient update on the feature extraction model according to the loss value;

[0058] Determine whether the loss value reaches a preset threshold, and when the loss value does not reach the preset threshold, continue to perform iterative training on the feature extraction model by using a next training sample in the training set until the loss value reaches the preset threshold.

[0059] A song matching device is provided for one of the purposes of the present application, comprising a segment encoding module, a semantic extraction module, a target matching module, and a song pushing module, wherein the segment encoding module is configured to obtain encoding information corresponding to audio data of a karaoke melody submitted by a client; the semantic extraction module is configured to extract a high-dimensional index vector representing deep semantic information of multiple scales of the karaoke melody to be matched by using a feature extraction model trained to a convergent state according to the encoding information; the target matching module is configured to calculate a similarity between the high-dimensional index vector and a high-dimensional index vector representing deep semantic information of multiple scales of each melody segment extracted by the feature extraction model from a melody feature library, and to filter out a target melody segment whose similarity satisfies a preset condition; and the song pushing module is configured to push a target song containing the target melody segment in a song library to a client device.

[0060] A computer device is provided for one of the purposes of the present application, comprising a central processing unit and a memory, and the central processing unit is configured to invoke a computer program stored in the memory to execute steps of a song matching method described in the present application.

[0061] A computer readable storage medium is provided for another purpose of the present application, which stores a computer program implemented according to the song matching method in the form of computer readable instructions, and when the computer program is invoked and run by a computer, steps included in the method are executed.

[0062] A computer program product is provided for another purpose of the present application, comprising computer program / instructions, which when executed by a processor, implement steps of the method described in any one of the embodiments of the present application.

[0063] Compared with the prior art, the present application has the following advantages:

[0064] Firstly, the application obtains the encoding information of the audio data of the to-be-matched a cappella melody according to the encoding, obtains the high-dimensional index vector representing the deep semantic information of the style-invariant feature of the to-be-matched a cappella melody by means of the pre-trained feature extraction model, and then matches the similar target melody segment of the to-be-matched a cappella melody in the melody feature library, and determines the target song containing the target melody segment in the song library according to the target melody segment. Since the high-dimensional index vector of the melody segment in the melody feature library is also extracted by using the same feature extraction model, the deep semantic representation of the corresponding audio data of the melody segment is realized on different semantic scales, and therefore, the to-be-matched a cappella melody can be matched in the semantic manner, so that the matching of the similar melody segment of the to-be-matched a cappella melody is accurately realized in the semantic manner, and finally the target song containing the a cappella melody is found based on the a cappella melody. The end-to-end model architecture solves the technical problem of humming recognition of the client device.

[0065] Secondly, compared with the traditional humming recognition scheme in which the melody library is generally composed of MIDI corresponding to a song, or a cappella corresponding to a song, or a separated vocal of a song, resulting in high cost of producing MIDI corresponding to a song, incomplete collection of a cappella corresponding to a song, time-consuming and easy to damage the vocal in the step of separating the vocal of a song, etc., the humming recognition scheme of the application is composed of a song segment, which represents the main melody in the song. The form of the song segment is rich, and the subsequent semantic-based matching is simple and convenient, and can achieve good recognition effect, and can quickly enter any song.

[0066] Furthermore, since the application realizes multi-scale feature extraction of the deep semantic information of the audio data of the song segment in the feature extraction model, the obtained high-dimensional index vector has better representation ability, such as global feature information, significant feature information, channel feature information, frequency band feature information, etc. of the audio data of the song segment, so as to realize more effective indexing of the corresponding audio data of the song segment, and to obtain more accurate and efficient matching effect. Even if the a cappella melody contains a certain amount of noise or can be compatible, the matching accuracy of humming recognition can be improved.

[0067] In addition, when the application is based on the end-to-end representation learning ability, supplemented by a retrieval and matching mechanism, obvious scale effect can be achieved, and can be deployed in the background of an online music service platform to realize a standardized interface, and then serve the needs of various application scenarios, provide comprehensive and multi-purpose open services, and improve the economic advantage of music information retrieval of the platform. BRIEF DESCRIPTION OF DRAWINGS

[0068] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings of which:

[0069] Figure 1 Flowchart of a typical embodiment of the song matching method of the present application;

[0070] Figure 2 Flowchart of the process of obtaining encoding information in an embodiment of the present application;

[0071] Figure 3 Flowchart of an extended embodiment of the song matching method of the present application;

[0072] Figure 4 Flowchart of the specific process of filtering the to-be-matched a cappella melody using the melody-free feature library in an embodiment of the present application;

[0073] Figure 5 Flowchart of the specific process of performing similar melody segment matching on the to-be-matched a cappella melody using the melody-containing feature library in an embodiment of the present application;

[0074] Figure 6 Flowchart of the process of constructing the melody-containing feature library in an embodiment of the present application;

[0075] Figure 7 Flowchart of the process of running the feature extraction model in an embodiment of the present application;

[0076] Figure 8 Network architecture diagram of the feature extraction model in an embodiment of the present application;

[0077] Figure 9 Network architecture diagram of the feature extraction model in another embodiment of the present application;

[0078] Figure 10 Flowchart of the working process of the residual convolution block employed in the feature extraction model of the present application;

[0079] Figure 11 Flowchart of the process of training the feature extraction model of the present application;

[0080] Figure 12 Principle block diagram of the classification model accessed by the feature extraction model of the present application in the training stage;

[0081] Figure 13 Principle block diagram of the song matching device of the present application;

[0082] Figure 14 Structural diagram of a computer device employed by the present application. DETAILED DESCRIPTION

[0083] Embodiments of the present application are described in detail below with reference to the attached drawing figures, wherein like or similar constituent elements or features can be denoted by the same or similar reference signs throughout the drawings. The embodiments described below are merely exemplary for the purpose of explanation and are not to be construed as limiting the present application.

[0084] It should be understood that the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. It should be further understood that the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” and the like, when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or intervening elements can be present. Also, the use of “connected” or “coupled” herein also includes wireless connection or wireless coupling. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0085] Those skilled in the art will appreciate that unless otherwise indicated, as used herein, all terms, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0086] Those skilled in the art will appreciate that the term "client", "terminal", "terminal device" as used herein encompasses devices that are solely wireless signal receivers, devices that are wireless signal receivers with no transmit capability, and devices that are both receivers and transmitters capable of bi-directional communication over a bi-directional communication link. Such devices can include cellular or other communication devices with or without a multi-line display, a plurality of push-to-talk buttons, and / or a numeric keypad. Such devices can also include Personal Communications Service (PCS) devices, Personal Digital Assistants (PDAs), cellular telephone / PDA combinations, Internet / Intranet access devices, and / or other devices that are capable of receiving wireless signals. The term "client", "terminal", "terminal device" as used herein can be portable, transportable, installed in a vehicle (aeronautical, maritime, and / or land-based), or adapted to be operated locally and / or in a distributed manner on Earth and / or in any other location in space. The term "client", "terminal", "terminal device" as used herein can also be a communication terminal, an Internet access terminal, a music / video playing terminal, such as a PDA, a Mobile Internet Device (MID), and / or a mobile phone with music / video playing function, a smart television, a set-top box, and / or the like.

[0087] The term "server", "client", "service node", and the like as used herein refers to hardware that has the equivalent capability of a personal computer, and is essentially an electronic device with a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device, and the like necessary components disclosed by the Von Neumann principle. A computer program is stored in the memory, the central processing unit calls the program stored in the external memory into the memory to run, executes the instructions in the program, and interacts with the input and output devices, thereby completing a specific function.

[0088] It should be noted that the concept of "server" in the present application can also be extended to the case of a server cluster. According to the principle of network deployment understood by those skilled in the art, the servers should be logically divided, and in physical space, these servers can be independent of each other but can be called through an interface, or can be integrated into a physical computer or a computer cluster. Those skilled in the art should understand this variation and should not be restricted by the implementation of the network deployment of the present application.

[0089] One or more technical features of the present application, unless explicitly specified, can be deployed on a server for implementation and accessed by a client remotely calling an online service interface provided by the server, or can be directly deployed and run on a client for implementation.

[0090] The neural network model referred to or possibly referred to in the present application, unless explicitly specified, can be deployed on a remote server and remotely called by a client, or can be deployed on a client with sufficient device capability for direct calling. In some embodiments, when it runs on a client, its corresponding intelligence can be obtained through transfer learning to reduce the requirement for client hardware running resources and avoid excessive occupation of client hardware running resources.

[0091] The various data involved in the present application, unless explicitly specified, can be remotely stored on a server or stored on a local terminal device, as long as it is suitable for being called by the technical solutions of the present application.

[0092] Those skilled in the art should know that the various methods of the present application, although based on the same concept and described to present commonality among them, are independently executable unless otherwise specified. Similarly, for each embodiment disclosed in the present application, it is based on the same inventive concept, and therefore, for the same conceptually expressed concepts, and although the conceptually expressed concepts are different, they should be understood as equivalent.

[0093] Unless it is explicitly stated that the embodiments disclosed in the present application are mutually exclusive, the technical features involved in each embodiment can be combined flexibly to construct new embodiments, as long as such combination does not deviate from the spirit of the present application and can meet the needs of the prior art or solve some deficiencies in the prior art. For this variation, those skilled in the art should know.

[0094] The song matching method of the present application can be programmed as a computer program product, deployed in a server for implementation, whereby the client can access the interface opened by the computer program product running in the form of a web program or an application program, and realize human-computer interaction with the process of the computer program product through a graphical user interface.

[0095] Referring to Figure 1 The song matching method of the present application, in its typical embodiment, comprises the following steps:

[0096] Step S1100, obtaining the encoding information corresponding to the audio data of the to-be-matched a cappella melody submitted by the client:

[0097] The technical solution of the present application can be deployed in the server of an online music platform, and open to a large number of platform users to provide a service of searching for a song by humming a song, specifically, to provide a service of humming song recognition. In the humming song recognition business, a user can record a melody segment formed by human voice singing on his / her client, obtain the corresponding audio data, and submit it to the server as a to-be-matched a cappella melody. The server finds a song with the same melody according to the similarity of the melody. In general, the audio data of the to-be-matched a cappella melody can also be obtained by separating the human voice singing part from a video file or a song file. According to this variation, those skilled in the art can further flexibly expand the service capability of the technical solution of the present application in searching for a song by humming a song.

[0098] The melody segment referred to in the present application is essentially a song segment with only melody information but no accompaniment music information. The song segment referred to in the present application can be generally pre-specified as a song segment with a total audio duration falling within a specific duration range.

[0099] After the audio data of the to-be-matched a cappella melody submitted by the user reaches the server, the server will encode it accordingly to obtain the corresponding encoding information. In the encoding process, adaptive processing can be performed on the audio data of the to-be-matched a cappella melody according to specific circumstances.

[0100] The audio data of the to-be-matched a cappella melody submitted to the server can be audio data in any format such as MP3, WMA, M4A, WAV, etc., or audio data obtained by separating audio from various video files. The audio data of the to-be-matched a cappella melody is usually composed of multiple speech data packets in the time domain. On this basis, the speech data packets are subjected to corresponding transformation processing according to the specific encoding information type, and the corresponding encoding information can be obtained.

[0101] The coding information is mainly related information describing the style-invariant features in the audio data of the song segment, and can be of various types, including but not limited to time-frequency spectrum information, mel spectrum information, CQT filter information, level contour information, Chroma feature information, etc. extracted from the speech data packets of the audio data. Such information can be encoded using corresponding algorithms to obtain the corresponding type of coding information. In this application, any one of the above types of coding information can be used to implement feature extraction in this application. In practice, it is recommended to encode the CQT filter information measured optimally to obtain the coding information.

[0102] Those skilled in the art understand that the above various coding information can be encoded using corresponding algorithms. In the encoding process, the audio data of the song segment needs to be pre-emphasized, framed and windowed, etc. before being analyzed in the time or frequency domain, i.e. speech signal analysis. The purpose of pre-emphasis is to enhance the high frequency part of the speech signal to make the spectrum smooth; general pre-emphasis is achieved through a first-order high-pass filter. Before analyzing the speech signal, it also needs to be framed, usually setting the length of each frame of the speech signal to 20 ms, considering the frame shift factor, there can be 10 ms overlap between adjacent two frames. In order to realize framing, the speech signal can be windowed to achieve this. Different window selection will affect the results of speech signal analysis, and the relatively common one is to use the Hamming window (Hamm) corresponding window function to implement the windowing operation.

[0103] On the basis of completing the pre-processing required for speech signal analysis of the audio data of the song segment, further analysis in the time and frequency domains can be performed to achieve coding and obtain corresponding coding information:

[0104] For the time-frequency spectrum information, the speech data of each speech data packet in the time domain is pre-emphasized, framed, windowed, and transformed into the frequency domain by short-time Fourier transform (STFT) to obtain the data corresponding to the spectrogram, thereby constituting the time-frequency spectrum information.

[0105] The mel spectrum information can be obtained by filtering the time-frequency spectrum information using a mel scale filter bank. Similarly, by taking the logarithm of the mel spectrum information and performing DCT transform, the corresponding mel cepstrum information is obtained, which is also applicable. It can be understood that the mel spectrum information and its mel cepstrum information can better describe the style-invariant features in the song, such as pitch, tone, timbre, etc.

[0106] For the CQT filter information, in music, all sounds are composed of 12 equal temperament octaves, i.e. 12 equal temperament, corresponding to the 12 semitones in an octave on the piano. The frequency ratio between adjacent semitones is 2 1 / 12It is obvious that two octaves of the same pitch class, the high octave is twice the frequency of the low octave. Therefore in music, the sound is exponentially distributed, but the audio spectrum obtained by Fourier transform is linearly distributed, and the frequency points of the two cannot be one-to-one corresponding, which will cause the estimated value of some scale frequency to produce error. Therefore, the CQT time-frequency transform algorithm can be used to replace the Fourier transform means to analyze the speech. CQT, Constant Q Transform, refers to the filter bank whose center frequency is exponentially distributed, and the filter bandwidth is different, but the center frequency and the bandwidth ratio is a constant Q. The difference between it and Fourier transform is that the frequency of its spectrum is not linear, but based on log2, and the filter window length can be changed according to the different spectral line frequencies to obtain better performance. Since the CQT is the same as the distribution of the scale frequency, the amplitude value of the music signal at each note frequency can be directly obtained by calculating the CQT spectrum of the music signal, which is more perfect for music signal processing. Therefore, the embodiment recommends using this information for corresponding coding to obtain corresponding coding information as the input of the neural network model of the application.

[0107] The pitch class profile information includes PCP (Pitch Class Profile) and HPCP (Harmonic Pitch Class Profile), which can be used to extract the corresponding pitch sequence from the audio data of the song segment, and then convert the pitch sequence into a melody profile sequence after normalization, merging and segmentation, and then convert the standard pitch difference generated by the standard pitch into a corresponding feature representation. The coding information constructed based on the pitch class profile information has good robustness to environmental noise.

[0108] The Chroma feature information is the general term of Chroma Vector and Chromagram. Chroma Vector is a vector containing 12 elements, which represent the energy of 12 pitch classes in a period of time (such as 1 frame). The energy of the same pitch class in different octaves is accumulated, and the Chromagram is a sequence of Chroma Vector. Specifically, after the short-time Fourier transform of the audio data packet of the song segment is converted from the time domain to the frequency domain, some noise reduction processing is performed, and then the tuning is performed; the absolute time is converted into frames according to the length of the selected window, and the energy of each pitch in each frame is recorded as a pitch spectrum; on the basis of the pitch spectrum, the energy (in loudness) of the notes of the same time, the same pitch class and different octaves is superimposed into the element of the same pitch class in the Chroma Vector to become the Chromagram. The data corresponding to the Chromagram is the Chroma feature information.

[0109] Any of the above specific encoding information can be used as input of the feature extraction model of the present application. In order to facilitate the processing of the feature extraction model, the encoding information can be organized according to a certain preset format. For example, the encoding information corresponding to each speech packet is organized as a row vector, and the row vectors of each speech data packet of the entire audio data to be encoded are organized in rows in time sequence to obtain a two-dimensional matrix as the complete encoding information thereof. And so on, which can be adapted to the preset of the feature extraction model, and can be implemented flexibly by those skilled in the art.

[0110] It should be noted that the encoding principle referred to here is adopted to adapt to the input required by the feature extraction model, and thus is applicable to all audio data processed by the feature extraction model for representation learning. For example, in the present application, it is not only applicable to the audio data of the a cappella melody to be matched, but also applicable to the processing of the audio data of the training samples in the training stage of the feature extraction model. Similarly, it is also applicable to the processing of the audio data of the song segments included in the melody-free feature library and the melody-containing feature library of the present application. Those skilled in the art can understand this.

[0111] Step S1200, extracting a high-dimensional index vector representing the deep semantic information of multiple scales of the a cappella melody to be matched from the encoding information by using the feature extraction model trained to a convergent state:

[0112] The feature extraction model for extracting deep semantic information of song segments implemented based on a convolutional neural network model in the present application is pre-trained to a convergent state. After training, it learns the ability to extract deep semantic information of multiple scales of the audio data of the song segments according to the encoding information, realizes representation learning of the style-invariant features of the corresponding audio data of the song segments, and thus can be used for query, retrieval, matching and other needs between songs.

[0113] The feature extraction model of the present application is implemented to extract deep semantic information of multiple scales of the same audio data, and represent these deep semantic information as a single or multiple high-dimensional index vectors, so as to realize feature representation of the audio data from multiple different aspects and / or different angles. The high-dimensional index vector is essentially a high-dimensional vector, which plays an indexing role of the encoding information of the corresponding audio data at the semantic level. These different scales include global scale based on encoding information, or frequency band scale, channel scale, etc. for feature extraction. For a song segment, selecting deep semantic information of two or more arbitrary scales corresponding to the encoding information thereof to represent as a high-dimensional index vector can realize multi-scale deep semantic information feature representation of the corresponding song segment.

[0114] After the feature extraction model is trained to converge, a service interface is opened for calling the technical solution of the embodiment, and the encoded information of the to-be-matched a cappella melody is fed into the feature extraction model. The feature extraction model extracts features based on the encoded information to obtain a high-dimensional index vector corresponding to the to-be-matched a cappella melody.

[0115] It should be understood that, since the feature extraction model can extract deep semantic information of a song segment from multiple scales, when the deep semantic information of different scales is converted into the high-dimensional index vector, different organization forms can be used, for example, the high-dimensional index vector is represented as a single high-dimensional vector, and generally the single high-dimensional vector represents the deep semantic information of a song segment as a whole; or the high-dimensional index vector is represented as a plurality of discrete high-dimensional vectors according to the scale correspondence, and each high-dimensional vector corresponds to a scale. Regardless of the organization form, a person skilled in the art can flexibly organize the high-dimensional vectors according to the actual needs of the scale semantic information to facilitate the calling of the representation data of the overall deep semantic information of the song.

[0116] For this step, the feature extraction model extracts features from the encoded information of the to-be-matched a cappella melody, and finally a high-dimensional index vector corresponding to the to-be-matched a cappella melody is obtained, which can be used for subsequent similar matching.

[0117] Step S1300, calculate the similarity between the high-dimensional index vector and the high-dimensional index vector representing the multi-scale deep semantic information of each melody segment extracted by the feature extraction model in the melody feature library, and screen out target melody segments whose similarity satisfies a preset condition:

[0118] The present application pre-constructs a melody feature library. The construction process of the melody feature library is disclosed in detail in the corresponding embodiments of the present application. In summary, the melody feature library is used to store the mapping relationship data between a large number of songs with a main melody, i.e. melody segments, and their corresponding high-dimensional index vectors. These melody segments can be songs with melody but without background music, i.e. pure vocal singing parts, or can have background music; can be in MIDI audio form, or can be in pure a cappella or humming recording form. The high-dimensional index vectors in the melody feature library are obtained by the feature extraction model of the present application extracting corresponding melody songs. These melody songs can be pre-collected from the song library, for example, by using the time stamp in the lyrics corresponding to the song, combining a specific time range, and cutting out the melody segment corresponding to the lyrics in the song, thereby obtaining each corresponding melody song. In addition, of course, they can also be manually identified and collected. Generally, the melody referred to in the present application is the main melody of a song.

[0119] In an exemplary online music service platform, a song library is prepared, which stores a large amount of audio data corresponding to songs, and can provide melody segments for the melody feature library. Therefore, for the to-be-matched a cappella melody, the melody segments pointed to by the melody feature library are candidate song segments in the search process, and these candidate song segments can correspond to the corresponding target song in the song library.

[0120] The melody feature library can apply a preset similarity calculation formula to calculate the similarity between the to-be-matched a cappella melody and each melody segment based on the high-dimensional index vector of the to-be-matched a cappella melody and the high-dimensional index vector corresponding to each melody segment in the melody feature library. The similarity calculation formula can use any algorithm suitable for calculating the similarity distance between data, such as cosine similarity algorithm, Euclidean distance algorithm, Pearson coefficient algorithm, Jaccard similarity algorithm, and nearest neighbor search algorithm. After similarity calculation, the similarity sequence between the high-dimensional index vector of the to-be-matched a cappella melody and the high-dimensional index vector of each melody segment in the melody feature library is obtained. In the similarity sequence, the similarity values of the to-be-matched a cappella melody and each melody segment in the melody feature library are stored.

[0121] After determining the similarity sequence corresponding to the to-be-matched a cappella melody, the elements in the similarity sequence can be further filtered according to a preset condition. For example, a preset threshold value, which can be an empirical threshold value or an experimental threshold value, can be used to filter the similarity values in the similarity sequence, and all elements with similarity values exceeding the preset threshold value can be filtered out. If the element exceeding the preset threshold value is 0, it indicates that there is no similar melody segment in the melody feature library that is similar to the to-be-matched a cappella melody. If multiple similarity values are obtained after filtering, the target melody segment corresponding to the largest similarity value can be selected as the similar melody segment corresponding to the to-be-matched a cappella melody according to the constraint of the preset condition, that is, the target melody segment obtained by humming recognition.

[0122] Step S1400, pushing the target song containing the target melody segment in the song library to the client device:

[0123] So far, according to the client device submitted or specified to be matched with the a cappella melody, the application determines its target melody segment similar in semantics, and further calls the audio data corresponding to the song to which the target melody segment belongs from the song library, obtains the access link of the audio data of the song, encapsulates it into a playable page or playable data containing the link, and pushes it to the corresponding client device for further access by the user. As can be seen, the user only needs to submit or specify the corresponding to-be-matched a cappella melody to obtain the corresponding similar song, realizing the purpose of humming recognition.

[0124] In other embodiments to be disclosed later in this application, there are various changes in the process of humming recognition, which will not be described here. According to the introduction of the typical embodiment, it can be understood that the implementation of the application has rich advantages, including but not limited to the following aspects:

[0125] Firstly, the application encodes the audio data of the to-be-matched a cappella melody to obtain its encoding information, obtains a high-dimensional index vector representing the deep semantic information of its style-invariant features by means of a pre-trained feature extraction model in a converged state, and then matches a target melody segment similar to the to-be-matched a cappella melody in the melody feature library. Then, according to the target melody segment, the target song containing the target melody segment in the song library is determined. Since the high-dimensional index vectors of the melody segments in the melody feature library are also extracted by the same feature extraction model, the deep semantic representation of the audio data of the melody segments is realized on different semantic scales, so that the to-be-matched a cappella melody can be matched in semantics. Thus, the matching of the similar melody segment of the to-be-matched a cappella melody is accurately realized according to the semantics, and finally the target song containing the a cappella melody is found based on the humming. The end-to-end model architecture solves the technical problem of humming recognition of the client device.

[0126] Secondly, compared with the traditional humming recognition scheme in which the melody library is generally composed of MIDI corresponding to the song, or a cappella corresponding to the song, or separated vocal of the song, resulting in high cost of producing MIDI corresponding to the song, incomplete collection of a cappella corresponding to the song, time-consuming and easy to damage the vocal of the song in the process of separating the vocal of the song, the humming recognition scheme of the application is composed of song segments, which represent the main melody in the song. However, the form of the song segment is rich, and the subsequent semantic-based matching is simple and convenient, and can achieve good recognition effect, and can quickly enter any song.

[0127] Furthermore, since the application realizes multi-scale feature extraction of the deep semantic information of the audio data of the song segment in the feature extraction model it adopts, the high-dimensional index vector obtained can have more representation ability, such as representing the global feature information, significant feature information, channel feature information, frequency band feature information, etc. of the audio data of the song segment, thereby realizing more effective indexing of the audio data of the corresponding song segment, and performing the search task of the melody segment on this basis can obtain more accurate and efficient matching effect, even if the a cappella melody contains a certain amount of noise or can be compatible, thereby improving the matching accuracy of the humming recognition.

[0128] In addition, when the application is based on end-to-end representation learning ability, supplemented by a retrieval matching mechanism, it can achieve obvious scale effect, can be deployed in the background of an online music service platform to realize a standardized interface, and then serve the needs of various application scenarios, provide comprehensive multi-purpose open services, and improve the economic advantage of music information retrieval of the platform.

[0129] Please refer to Figure 2 In a deepened embodiment, the step S1100 of obtaining the encoding information corresponding to the audio data of the a cappella melody to be matched submitted by the client includes the following steps:

[0130] Step S1110, receiving the song matching request submitted by the client, and obtaining the audio data of the a cappella melody to be matched specified by the request:

[0131] In this embodiment, the user records or specifies audio data formed by an a cappella melody in the song matching page displayed on the client device, and if necessary, the audio data can be previously specified by the application program to have a preset time length, and then triggers the song matching request of the audio data as the audio data required by the a cappella melody to be matched, and submits the request to the server of the application open humming music recognition service. The server analyzes the request to obtain the audio data of the a cappella melody to be matched.

[0132] Step S1120, detecting whether the audio data contains human singing information, and if not, terminating subsequent execution:

[0133] To improve the matching accuracy, the server can preprocess the audio data it receives, such as detecting whether the audio data contains a human voice singing melody part through a VAD logic module. If the audio data does not contain a human voice singing melody part, the subsequent execution of the present application can be terminated, and the corresponding notification can be directly returned to the client device. When it is confirmed that there is audio data corresponding to the human voice singing melody part, the subsequent steps can be continued. The VAD logic module can be implemented using various existing technologies known to those skilled in the art, and it is recommended to use an end-to-end pre-trained neural network model to the convergence state for implementation.

[0134] Step S1130, format the audio data to be between a minimum preset time length and a maximum preset time length:

[0135] Although the length of the audio data submitted by the user can be specified in the client, and the user is required to submit a song segment within the expected time length range, in order to expand the compatibility of the audio data submitted by the user, the server side can further preprocess the audio data submitted by the user to achieve further specification through this step.

[0136] Specifically, the server side can process the audio data to be matched to the karaoke melody to a preset specific time length range, such as an interval of 10 seconds to 15 seconds. The corresponding audio data can be determined to contain a melody part, i.e., a human voice singing part, by means of the detection means of the previous step. If the total length of the audio data submitted by the user fails to reach the minimum preset time length of the specific time length range, the matching work of processing the audio data can be terminated. If the total length of the audio data submitted by the user exceeds the maximum preset time length of the specific time length range, the corresponding specific time length range can be appropriately truncated in length as the audio data for encoding.

[0137] Step S1140, encode the audio data to obtain corresponding encoding information.

[0138] For audio data that has been processed by human voice detection and formatting, the encoding principle described in the foregoing of the present application can be used to encode it to obtain corresponding encoding information. As described previously, it is recommended to use CQT filtering information to construct the encoding information of the audio data to be matched to the karaoke melody.

[0139] The present embodiment realizes the filtering of part of the audio data without human voice and the specification of the audio data by preprocessing the audio data to be matched to the karaoke melody, avoids the corresponding online service frequently responding to invalid audio data to be matched to the karaoke melody, and can improve the matching accuracy of the humming recognition, and also saves the system overhead of the backend service.

[0140] Please refer to Figure 3, in an extended embodiment, before the step S1300 of calculating the similarity between the high-dimensional index vector and the high-dimensional index vector representing the multi-scale deep semantic information of each melody segment extracted by the feature extraction model, the method comprises the following steps:

[0141] In step S1000, the high-dimensional index vector of the to-be-matched a cappella melody is verified by calling the non-melody feature library. When the high-dimensional index vector does not form a similarity with the high-dimensional index vector of the non-melody song segment in the non-melody feature library, it is determined to be valid, and then the subsequent steps are continued to be executed. The high-dimensional index vector in the non-melody feature library is a feature representation of the non-melody song segment extracted by the feature extraction model.

[0142] The present application pre-constructs a non-melody feature library for storing the mapping relationship data between a large number of non-melody song segments and their corresponding high-dimensional index vectors, wherein the high-dimensional index vectors are obtained by the feature extraction model of the present application extracting the corresponding non-melody song segments. These non-melody song segments can be obtained by collecting non-melody songs in advance. The corresponding non-melody songs can be pure music accompaniment songs, or non-melody prelude, transition, ending, etc. in a melody song, and the like, which can be detected and intercepted by public identification technology, or obtained by manual processing.

[0143] In order to reduce the misjudgment rate of matching similar melody segments for the to-be-matched a cappella melody, the similarity between the high-dimensional index vector of the to-be-matched a cappella melody and the high-dimensional index vector of each non-melody song segment in the non-melody feature library is calculated, and then it is determined whether there is a non-melody song segment similar to the to-be-matched a cappella melody according to the similarity value. If there is, it means that the to-be-matched a cappella melody does not contain a song melody, so there is no need to continue the subsequent steps, and the corresponding client device can be directly notified. If not, it means that the to-be-matched a cappella melody contains a song melody, and the high-dimensional index vector thereof can be determined as a to-be-matched vector, and then the to-be-matched vector is used to match the target melody segment similar to the to-be-searched a cappella melody.

[0144] Similarly, when calculating the similarity, the cosine similarity algorithm, the Euclidean distance algorithm, the Pearson coefficient algorithm, the Jaccard similarity algorithm, the nearest neighbor search algorithm, etc. can be used to realize any algorithm suitable for calculating the similarity distance between data, which can be flexibly implemented by those skilled in the art.

[0145] The embodiment obtains the high-dimensional index vector representing the deep semantic information of the style-invariant feature of the to-be-matched a cappella melody according to the encoding information obtained by encoding the to-be-matched a cappella melody, discriminates whether the to-be-matched a cappella melody belongs to a melodyless song segment according to the melodyless feature library for storing the high-dimensional index vectors of melodyless song segments, determines the high-dimensional index vector of the to-be-matched a cappella melody as the to-be-matched vector when the to-be-matched a cappella melody is determined to have a song melody, and then matches a target melody segment similar to the to-be-matched a cappella melody in the melody feature library. The purpose of searching and matching only the audio data with melody can be achieved by filtering out the redundant information of melodyless songs, and the precision of the search and matching result is improved by the cooperation of the two.

[0146] Please refer to Figure 4 In the deepened embodiment, the step S1000 of calling the melodyless feature library to verify the validity of the high-dimensional index vector of the to-be-matched a cappella melody is performed only when the high-dimensional index vector and the high-dimensional index vectors of the melodyless song segments in the melodyless feature library are not similar, and the step S1000 includes the following steps.

[0147] The step S1001 of calling the melodyless feature library, the melodyless feature library stores a plurality of high-dimensional index vectors corresponding to preset melodyless song segments, and the high-dimensional index vectors are obtained by the feature extraction model extracting the encoding information of the corresponding melodyless song segments.

[0148] According to the embodiments disclosed in the foregoing, the melodyless feature library is constructed in advance, and the melodyless feature library stores a plurality of high-dimensional index vectors corresponding to melodyless song segments. The melodyless song segments are usually the accompaniment part before, in or after a song, or pure music. The melodyless feature library can be called for the need of filtering the to-be-matched a cappella melody.

[0149] The step S1002 of calculating the similarity between the high-dimensional index vector of the to-be-matched a cappella melody and each high-dimensional index vector in the melodyless feature library respectively to obtain the corresponding similarity value.

[0150] In the embodiment, the high-dimensional index vector of the to-be-matched a cappella melody and the high-dimensional index vectors of the song segments in the melodyless feature library are all single high-dimensional vectors representing the multi-scale deep semantic information of the corresponding audio data. In order to determine whether the two are matched, various similarity algorithms can be applied to realize the calculation of the data distance between the two. As described above, the cosine similarity algorithm, the Euclidean distance algorithm, the Pearson coefficient algorithm, the Jaccard similarity algorithm, the nearest neighbor search algorithm and the like can be used to realize the algorithm suitable for calculating the similarity distance between data, and the skilled in the art can flexibly implement it.

[0151] After the similarity calculation, the similarity sequence between the high-dimensional index vector of the to-be-matched a cappella melody and the high-dimensional index vector of each melodyless song segment in the melodyless feature library is obtained, and the similarity sequence stores the similarity values of the to-be-matched a cappella melody and each melodyless song segment in the melodyless feature library.

[0152] In step S1003, it is determined whether there is a melodyless song segment with a similarity value exceeding a preset threshold. If yes, it is determined that the to-be-matched a cappella melody does not contain a song melody, and the execution is terminated. If no, it is determined that the to-be-matched a cappella melody contains a song melody, and the subsequent steps are continued.

[0153] To determine whether the to-be-matched a cappella melody is melodyless audio data, a preset threshold can be set to compare with the similarity values corresponding to the song segments in the melodyless feature library. It can be understood that if the similarity value between a song segment in the melodyless feature library and the to-be-matched a cappella melody is greater than the preset threshold, the to-be-matched a cappella melody can be regarded as a melodyless song segment. Otherwise, if the similarity value between a song segment in the melodyless feature library and the to-be-matched a cappella melody is less than the preset threshold, the to-be-matched a cappella melody can be regarded as a song segment with melody. According to this principle, only the maximum similarity value in the similarity sequence is compared with the preset threshold. When the former is greater than the latter, it is determined that the to-be-matched a cappella melody is a melodyless song segment, and the subsequent steps can be terminated. The user can be simply notified. When the former is less than the latter, it is determined that the to-be-matched a cappella melody is a song segment with melody. Accordingly, the high-dimensional index vector of the to-be-matched a cappella melody can be determined as the to-be-matched vector for subsequent matching.

[0154] In this embodiment, the similarity algorithm is used to calculate the data distance between the high-dimensional index vectors of the to-be-matched a cappella melody and the melodyless song segments in the melodyless feature library, to determine and filter out melodyless audio data, and to determine the high-dimensional index vector of the audio data with melody as the to-be-matched vector. The filtering operation of the front end is realized for the background service, so that the accuracy of the search and comparison of the audio data by the back-end service is improved, and the system overhead of the back-end service is saved, and the response to redundant and invalid information is avoided.

[0155] Please refer to Figure 5 In the deepened embodiment, step S1300 of calculating the similarity between the high-dimensional index vector and the high-dimensional index vector representing the multi-scale deep semantic information of each melody segment extracted by the feature extraction model from the melody feature library, and screening the target melody segment with a similarity satisfying a preset condition, includes the following steps.

[0156] Step S1310, call the melody feature library to obtain the high-dimensional index vector corresponding to each melody segment, the high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of different scales of a melody segment as a whole:

[0157] According to the embodiments disclosed in the foregoing, the melody feature library is constructed in advance, in which the high-dimensional index vectors corresponding to a plurality of melodic song segments are stored, the melodic song segments are usually the parts corresponding to the lyrics in a song, and the melody feature library is called for the need of matching the a cappella melody to be matched.

[0158] Step S1320, respectively calculate the similarity between the high-dimensional index vector of the a cappella melody to be matched and each high-dimensional index vector in the melody feature library, and obtain the corresponding similarity sequence:

[0159] In the embodiment, the high-dimensional index vector of the a cappella melody to be matched and the high-dimensional index vector of the melody segment in the melody feature library are both single high-dimensional vectors that represent the multi-scale deep semantic information of the corresponding song segment as a whole. In order to determine whether they are matched, various similarity algorithms can be applied to realize by calculating the data distance between the two. As described above, any algorithm suitable for calculating the similarity distance between data can be used, such as cosine similarity algorithm, Euclidean distance algorithm, Pearson coefficient algorithm, Jaccard similarity algorithm, and neighbor search algorithm, etc., which can be implemented flexibly by those skilled in the art.

[0160] After the similarity calculation, the similarity sequence between the high-dimensional index vector of the a cappella melody to be matched and the high-dimensional index vector of each melody segment in the melody feature library is obtained, and the similarity sequence stores the similarity values of the a cappella melody to be matched and each melody segment in the melody feature library.

[0161] Step S1330, screen out the melody segments with similarity values exceeding the preset threshold value in the similarity sequence, and determine them as the target melody segments whose similarity satisfies the preset condition:

[0162] In order to determine whether there is a target melody segment similar to the a cappella melody to be matched in the melody feature library, a preset threshold value can be used for judgment, which can be an empirical threshold value or an experimental threshold value and can be determined flexibly by those skilled in the art. According to this, all elements in the similarity sequence are compared with the preset threshold value one by one, and all elements higher than the preset threshold value are determined. When there is one or more elements with similarity values exceeding the preset threshold value, generally, the melody segment corresponding to the element with the largest similarity value is determined as the target melody segment similar to the a cappella melody to be matched.

[0163] The embodiment matches the to-be-matched vector of the to-be-matched a cappella melody with the high-dimensional index vector of the melody feature library to determine a target melody segment similar to the to-be-matched a cappella melody, thereby completing the service loop and enabling a user submitting the to-be-matched a cappella melody to obtain a desired similar melody segment, and realizing the traceability or similar song matching of the to-be-matched a cappella melody.

[0164] Please refer to Figure 6 In an extended embodiment, the song matching method of the application comprises the following steps for constructing the melody feature library:

[0165] In step S2100, audio data of a candidate song is obtained from a song library, and a plurality of audio data corresponding to to-be-verified melody segments are segmented from the audio data according to the time stamps of the lyrics file of the candidate song:

[0166] Still taking an online music service platform as an example, the platform has a song library containing a large number of songs, which can be used as candidate songs for constructing the melody feature library in the embodiment.

[0167] In order to scientifically segment each candidate song to obtain a plurality of corresponding song segments, a preset time length is set, which can be a fixed value or a range value. For the latter, the preset time length is usually a specific time length range between a minimum preset time length and a maximum preset time length, for example, any value between 10 seconds and 15 seconds.

[0168] In the case of adapting the preset time length to a specific time length range, the time stamps of a plurality of target lyrics falling within the specific time length range can be calculated according to the time stamps of the continuous multiple lyrics in the lyrics of each candidate song, and the audio data segments corresponding to the song segments singing the target lyrics are cut from the candidate song according to the time stamps of the target lyrics. Accordingly, a plurality of melody segments can be obtained from each candidate song.

[0169] Further, considering the similarity of the melody and background music of the same lyrics in a song, the plurality of melody segments of each candidate song can also be de-duplicated according to the lyrics, that is, only one of the melody segments with the same content in each candidate song can be retained. Thus, each candidate song has a plurality of melody segments corresponding thereto, and the mapping relationship data between each melody segment and the candidate song to which it belongs is flexibly stored and represented for subsequent calling.

[0170] This step separates a plurality of melody segments from the candidate song at the algorithm level, which is a technical implementation, and it is best to further verify and confirm the separated melody segments, so the melody segments at this time are regarded as to-be-verified melody segments.

[0171] In an optional embodiment, a step can be added for vocal separation of the melody segments to be verified. The candidate songs in the song library usually contain accompaniment background music. In order to facilitate matching of the vocal singing melody part, the audio data corresponding to each of the melody segments to be verified can be processed to eliminate the background music, thereby eliminating the chord accompaniment and the like, and obtaining the content of the relatively pure vocal singing part. In essence, the background music elimination of the melody segments to be verified is to separate the melody part content corresponding to the vocals by using the vocal separation technology. The vocal separation technology is well known to those skilled in the art, and can be implemented by using various mature tools, algorithms, and neural network models, which will not be described here.

[0172] Step S2200, encoding the audio data of each melody segment to be verified to obtain its encoded information:

[0173] The melody segments to be verified obtained through the previous step need to extract the deep semantic information of each melody segment to be verified. Therefore, the feature extraction model of the present application which has been pre-trained to a convergent state is still used to process each melody segment to be verified. For this purpose, the audio data of the corresponding melody segment to be verified needs to be encoded to obtain the corresponding encoded information. The principle and process of encoding have been disclosed in the foregoing embodiments of the present application, which will not be described here.

[0174] Step S2300, for each melody segment to be verified, calling the feature extraction model to extract the deep semantic information of multiple scales of the encoded information thereof, and correspondingly obtaining its high-dimensional index vector:

[0175] After determining the encoded information of a melody segment to be verified, the feature extraction model can be called to extract the features of the encoded information. Under the action of the feature extraction model, the deep semantic information of multiple scales of the corresponding melody segment to be verified is extracted according to the encoded information, and finally organized into a high-dimensional index vector. As mentioned above, the present application recommends that the deep semantic information of multiple scales is spliced to represent a single high-dimensional vector as the high-dimensional index vector.

[0176] Step S2400, for each melody segment to be verified, calculating the similarity between its high-dimensional index vector and the high-dimensional index vectors of each melodyless song segment in the preset melodyless feature library, and obtaining the corresponding similarity value:

[0177] In order to improve the accuracy of the library, the high-dimensional index vectors of each melodyless song segment stored in the melodyless feature library which has been pre-constructed are used to calculate the similarity between the high-dimensional index vectors of the melodyless song segments and the high-dimensional index vectors of the melody segments to be verified, and obtain the corresponding similarity sequence. The similarity sequence stores the similarity values corresponding to each melodyless song segment.

[0178] Step S2500, for each melody segment to be checked, it is judged whether there is a non-melody song segment with a similarity value exceeding a preset threshold, if there is, the melody segment to be checked is discarded, if not, the mapping relationship data of the melody segment to be checked and its high-dimensional index vector is added to the melody characteristic library:

[0179] Finally, according to the principle disclosed in the foregoing embodiments, it is judged whether there is a non-melody song segment with a similarity value exceeding a preset threshold according to whether the maximum similarity value in the similarity sequence is higher than the preset threshold, if there is, it means that the melody segment to be checked is highly suspected to be a non-melody song segment, which can be discarded and not stored in the library; otherwise, it means that the melody segment to be checked is highly suspected to be a melody song segment, which can be added to the melody characteristic library. In this way, the melody characteristic library is continuously enriched, and the construction of the library is completed, which can be used to match similar target melody segments for the a cappella melody to be matched in the present application.

[0180] The present embodiment further discloses the construction process of the melody characteristic library of the present application. As can be seen from the present embodiment, the song segments of the melody characteristic library can be selected from the song library of an online music service platform, and the mapping relationship between the melody segments and the corresponding candidate songs of the song library is established. Accordingly, when it is necessary to match similar songs (melody segments) for the a cappella melody to be searched, the index feature vectors of the melody segments in the melody characteristic library can be used for similarity matching, and the similar songs in the song library can be determined according to the similar melody segments matched. In this process, since the melody segments in the melody characteristic library have been filtered and selected by means of the non-melody characteristic library, they are generally melody song segments, and therefore the accuracy of the similarity values obtained by the algorithm is higher when searching and matching, and misjudgment is less likely to occur.

[0181] Please refer to Figure 7 In specific embodiments, when the feature extraction model is called, the following steps are performed:

[0182] Step S3100, a plurality of convolution blocks in the shared network of the feature extraction model trained to a convergent state are used to sequentially perform multi-level feature extraction on the encoding information, and intermediate feature information in which deep semantic information of the encoding information is extracted is obtained:

[0183] The feature extraction model is constructed based on the multi-branch idea of the present application, and can be flexibly deformed to meet the requirements of different embodiments of the present application. In typical embodiments of the feature extraction model, for example, Figure 8As shown in the principle block diagram of the feature extraction model, the feature extraction model is composed of a shared network and a plurality of branch networks, wherein the shared network includes a plurality of convolution blocks for progressively extracting deep semantic information of the encoding information to obtain intermediate feature information; the plurality of branch networks respectively perform extraction of different types of deep semantic information based on the intermediate feature information to obtain corresponding output feature information. Each of the plurality of branch networks includes a part of the same structure, which includes a plurality of convolution blocks for progressively extracting deep semantic information, and after the last convolution block outputs, different processing can be performed according to different functions of each branch network.

[0184] The convolution block can be implemented by a convolution layer based on CNN and RNN, and preferably a convolution block based on residual convolution principle. In order to realize the function of context grooming so as to extract key information in the audio data of the song segment, an attention mechanism can be applied in any one of the convolution blocks, and a corresponding attention module is added, which is specifically a spatial attention module (SAM) or a channel attention module (CAM). In the reinforced embodiment, an instance normalization operation (IN) and a batch normalization (BN) operation are applied in the convolution block to divide the information input therein into two parts, one part performs instance normalization operation to learn style-invariant features, and the other part performs batch normalization operation to realize normalization. Therefore, the so-called IBN architecture is applied. By applying this architecture, style-highly-diversified music attributes such as notes, rhythms, and timbres can be learned, while retaining version information.

[0185] Accordingly, it is not difficult to understand that the feature extraction model is adapted to different application scenarios, different branch networks are enabled, and the training set is preselected to train the feature extraction model to a convergent state, so as to obtain corresponding feature extraction capability, thereby being suitable for performing tasks corresponding to the application scenarios to extract output feature information corresponding to the audio data of the song segment from the encoding information of the audio data of the song segment input therein. The training process of the feature extraction model will be given in the exemplary embodiments of the present application, which will not be described here.

[0186] In this step, in the architecture as shown in Figure 8 After the encoding information is progressively subjected to feature extraction by the plurality of convolution blocks of the shared network, especially after the key information is extracted by the last convolution block, the intermediate feature information in which the key information of the encoding information is extracted can be obtained, and the intermediate feature information is divided into a plurality of paths and output to the plurality of branch networks, so as to perform extraction of deep semantic information from different angles in each branch network.

[0187] After step S3200, the intermediate feature information is converted into output feature vectors of corresponding scales after different scale feature extraction by multiple convolution blocks in two or more branch networks in the feature extraction model, and the output feature vectors of each branch network contain different deep semantic information:

[0188] As described above, Figure 8 In the architecture shown, each branch network can be flexibly selected and combined, and therefore the specific architecture obtained by combination can determine how many branch networks are present. The intermediate feature information output by the shared network is input into each branch network for further feature extraction processing.

[0189] According to Figure 8 In the architecture shown, each branch network belonging to the same structural part includes two convolution blocks, and the output of the extracted features after feature extraction of the two convolution blocks in turn can adapt to the specific structure of different branch networks for different processing.

[0190] Specifically, different branch networks adapt to the different deep semantic information extracted by themselves and can be processed differently in non-same structural parts. For example, one branch network can be subjected to maximum pooling or average pooling output, one branch network can be subjected to Dropout layer random redundancy feature discard and then maximum pooling output, the intermediate feature information output by the last convolution block in another branch network can be subjected to equal channel segmentation and then average pooling output, the intermediate feature information output by the last convolution block in another branch network can be subjected to equal frequency band segmentation and then average pooling output, and so on. Through various processing of the feature information output by the last convolution block, output feature information containing different deep semantic information can be obtained, which describes the deep semantic information of the audio data of the song segment from different scales, including global information and various local information of the audio data of the song segment, such as global information abstracting the significant features of the encoding information of the audio data of the song segment, local information abstracting the channel or frequency band features of the encoding information of the audio data of the song segment, and so on. Accordingly, multiple output feature information different in representation can be obtained, which can be independently called or arbitrarily combined for use as needed.

[0191] In the present application, the output feature information output by each branch network is normalized into an output feature vector representation, therefore, the multiple branch networks can correspondingly obtain multiple output feature vectors, each output feature vector represents the deep semantic information of the audio data of the song segment in different aspects or different scales, and the deep semantic information contained in each output feature vector is different from each other.

[0192] In use, usually two or more branch networks are used to obtain two or more output feature vectors, so as to use two or more deep semantic information to represent the audio data of the song segment, for example, the output feature vector used to represent the global information of the audio data of the song segment can be used in combination with the output feature vector used to represent the channel information of the audio data of the song segment, or the output feature vector used to represent the global information of the audio data of the song segment can be used in combination with the output feature vector used to represent the frequency band information of the audio data of the song segment, or the output feature vector used to represent the channel information of the audio data of the song segment can be used in combination with the output feature vector used to represent the frequency band information of the audio data of the song segment, or all the output feature vectors can be used in combination. And so on, which can be called as needed by those skilled in the art.

[0193] Step S3300, output the output feature vector of each branch network as the high-dimensional index vector by the feature extraction model:

[0194] The output feature vectors obtained by each branch network can be finally converted into a high-dimensional index vector for storage or direct use. The high-dimensional index vector is a high-dimensional vector used to index the audio data of the corresponding song segment. Since each branch network has normalized its output feature information into an output feature vector, the high-dimensional index vector can be processed flexibly depending on the specific use of the feature extraction model. For example, for application requirements that are only for storage and separate calling, each output feature vector can be stored as a corresponding high-dimensional index vector in the melody-free feature library and the melody-containing feature library for on-demand calling of the high-dimensional index vectors output by different branch networks for retrieval, query, matching, etc. For the specific task of humming recognition, all output feature vectors output by all structured branch networks can be sequentially spliced to obtain a single high-dimensional index vector, which can be stored, for example, in the melody-free feature library or the melody-containing feature library, or for matching a humming melody, the obtained high-dimensional index vector is directly used for subsequent similar melody segment matching. Thus, the representation learning of the audio data of the song segment is realized through the high-dimensional index vector. In this application, for the convenience of the high-dimensional index vectors of the song segments stored in the melody-free feature library and the melody-containing feature library, and the high-dimensional index vector of the humming melody to be matched, the form of a single high-dimensional vector spliced from multiple output feature vectors can be used.

[0195] According to the principles disclosed above, the melody-free feature library and the melody-containing feature library can be prepared for part or all of the songs in the song library of an online music service platform according to the process of the present embodiment. By applying each step of the present embodiment to the audio data of the song segments of each corresponding song in the song library, the high-dimensional index vectors corresponding to the audio data of each song segment can be obtained, and these high-dimensional index vectors can be stored in association with the corresponding song segments to construct the feature library. If a song segment has no song melody information, its high-dimensional index vector can be added to the melody-free feature library, and if a song segment has song melody information, its high-dimensional index vector can be added to the melody-containing feature library. The high-dimensional index vectors in the feature library can be directly called for retrieval, query, matching, etc.

[0196] In addition to the various application modes disclosed in the present application, the mining and utilization based on the high-dimensional index vectors obtained in the present application can have various different uses, which can be flexibly used by those skilled in the art according to the principles disclosed herein, without affecting the embodiment of the present application.

[0197] Through the above introduction of the execution process of the feature extraction model and its network architecture, it can be understood that the present embodiment contains very rich beneficial effects, including but not limited to the following aspects:

[0198] Firstly, the feature extraction model encodes the audio data of the song segment to obtain the style-invariant features of the audio data of the song segment, and then extracts intermediate feature information from the encoded information through a shared network. On the basis of the intermediate feature information, deep semantic information of the audio data of the song segment is extracted from different angles through multiple branch networks to obtain corresponding output feature information. Finally, the output feature information is taken as a high-dimensional index vector corresponding to the audio data of the song segment, and the end-to-end representation learning of the audio data of the song segment is completed.

[0199] Secondly, since the shared network and the multiple branch networks are combined in the feature extraction model, multi-scale feature extraction of the deep semantic information of the audio data of the song segment is realized, so that the obtained high-dimensional index vector has more representation ability, such as representing the global feature information, significant feature information, channel feature information, frequency band feature information, etc. of the audio data of the song segment, thereby realizing more effective indexing of the corresponding audio data of the song segment. On this basis, the downstream processing such as retrieval, query and matching of the audio data of the song segment can obtain more accurate and efficient matching effect, and can be used in various application scenarios such as cover song recognition, song recognition, humming recognition and song copyright judgment.

[0200] In addition, the output feature vectors obtained by the multiple branch networks of the feature extraction model can be combined into a single high-dimensional index vector, or can be used independently as different high-dimensional index vectors. According to the required deep semantic information, it is flexible to determine the use and flexible to use. When processing the representation learning of a large amount of audio data of song segments, obvious scale effect can be achieved. It can be deployed in the background of an online music service platform to realize a standardized interface, thereby serving the needs of various application scenarios, providing comprehensive multi-purpose open services, and improving the economic advantages of music information retrieval of the platform.

[0201] In a deepened embodiment, after the step S3200, the multiple convolution blocks in two or more branch networks of the feature extraction model are used to extract features of different scales from the intermediate feature information, and then convert them into output feature vectors of corresponding scales, including any two or more of the following steps:

[0202] Step S3210, the multiple convolution blocks in the first branch network are used to extract features from the intermediate feature information to obtain global feature information, and the global feature information is pooled into an output feature vector of a global scale:

[0203] Figure 8 In the first branch network exemplarily given, after the intermediate feature information is extracted by two convolution blocks of the same structure as other branch networks, the output of the last convolution block is divided into two paths, one of which directly performs a mean pooling operation to obtain its overall feature information, and the other of which randomly discards part of the time-frequency region information through a Dropout layer and then extracts significant feature information in the global through a maximum pooling operation. Thus, two global output feature vectors are correspondingly output. According to the architecture, in the model training stage, on the one hand, the generalization ability of the model to audio with local time-frequency domain changes such as segment deletion and segment insertion in the audio data of the song segment is improved, and on the other hand, the model overfitting is prevented to some extent. In addition, one of the two global output feature vectors captures the overall feature and the other captures the significant feature, which improves the recognition ability of the model.

[0204] In step S3220, the intermediate feature information is extracted by a plurality of convolution blocks in the second branch network, and then divided into multiple parts for pooling according to channels, and a channel-scale output feature vector is correspondingly obtained:

[0205] Since the feature information output by each convolution block is usually represented in the form of “channel number * frequency band number * frame number”, the division processing can be performed according to the channel number. Figure 8 In the second branch network exemplarily given, after the intermediate feature information is extracted by two convolution blocks of the same structure as other branch networks, the output of the last convolution block is divided into multiple paths, for example, two paths, and then respectively passes through a 1*1 convolution layer, and after mean pooling, the channel output feature information corresponding to the two parts of channels is obtained. In this process, the two channel branches focus on capturing local features of audio, and for audio with large differences in arrangement, a large amount of information is overwhelmed by strong noise or other interference sounds, and a feature representation can be established from a few local significant common features.

[0206] In step S3230, the intermediate feature information is extracted by a plurality of convolution blocks in the third branch network, and then divided into multiple parts for pooling according to frequency bands, and a frequency band-scale output feature vector is correspondingly obtained:

[0207] Figure 8In the third branch network exemplarily given, after the intermediate feature information is extracted by two convolutional blocks with the same structure as other branch networks step by step, the output of the last convolutional block is subjected to mean pooling and then is divided by frequency band to divide into multiple paths, for example, two paths, and after mean pooling, the frequency band output feature information corresponding to two parts of frequency band is obtained. In this process, each frequency band branch focuses on extracting the feature information of the corresponding frequency band, and has a significant effect on resisting the band-selective weakening of the harsh sound pickup environment, balancing the contribution of high and low frequency information in feature composition, resisting the addition or deletion of content in a fixed range of frequency band (such as adding or reducing a drum sound) or strong interference in a fixed frequency band range.

[0208] It can be understood that the multiple output feature vectors obtained in the same branch network can also be further processed into the same output feature vector by splicing or mean pooling. Those skilled in the art can flexibly implement it.

[0209] In the embodiment, the rich branch network extracts multi-aspect and multi-scale feature information from the audio data of the song segment, so that the obtained output feature vector can obtain rich deep semantic information representation, which not only represents the global information and significant information of the audio data of the song segment, but also represents the relevant local information of the audio data of the song segment according to the channel and frequency band. Considering that the intermediate feature information has captured the key information of the audio data of the song segment under the action of the shared network, the embodiment realizes the indexing value of the audio data of the song segment from multiple aspects, and when the high-dimensional index vector obtained subsequently is used for query, retrieval and matching, the precision in all aspects can be improved.

[0210] Since the embodiment can capture deep semantic information of the audio data of the song segment from multiple aspects, it is suitable for feature extraction of the audio data of the song segment, can effectively mine deep semantic information of the short-length audio data of the song segment, and make the information represented by the corresponding high-dimensional index vector more rich and accurate, and can have a more accurate matching effect.

[0211] Please refer to Figure 9 On the basis of the previous embodiment, the network structure of the feature extraction model of the present application is improved. It can be seen that, Figure 9 The difference between the network architecture in Figure 8 and the network architecture in Figure 9In the first branch network, the output of the last convolution block is directly subjected to maximum pooling to obtain a global output feature vector, which captures the significant feature information of the encoding information of the audio data of the song segment. In the second branch network, the output of the last convolution block is divided into four parts corresponding to the feature information of the channels, and the feature information of each part corresponding to the channels is subjected to mean pooling processing, and then the corresponding output feature vector is obtained by re-pasting. It is not difficult to understand that the output feature vector obtained by the segmentation and construction of the local branch can learn more optimal local feature information.

[0212] Similarly, as a replacement for the equal division of the channels in the second branch network, the output of the last convolution block can also be divided according to the frequency band dimension, and four parts of the frequency band corresponding feature information are obtained, and then the mean pooling processing is performed, and then the corresponding output feature vector is obtained by re-pasting. Figure 9

[0213] The embodiment exemplarily shows the modification based on the network architecture shown in the figure, which is relatively light. According to this example, it is not difficult to understand that the focus of the creative spirit of the present application is the flexible combination of the plurality of branch networks. Based on the principles disclosed in the present application, according to the characteristics of the multi-scale deep semantic information possessed by the output feature vectors obtained by the branch networks, different feature extraction models can be selected according to different specific purposes, and the other embodiments of the present application can be transformed to meet the needs such as humming recognition, song recognition, cover recognition, and infringement comparison. Figure 8 Please refer to

[0214] In a further embodiment, when the convolution block is called, the following steps are performed: Figure 10 Step S4100, performing convolution transformation on the information in the input to obtain transformed feature information:

[0215] Each convolution block in the feature extraction model of the present application is subjected to a 1*1 convolution kernel for convolution operation on the information in the input, whether it is the encoding information or the intermediate feature information output by the previous convolution block.

[0216] Step S4200, combining the transformed feature information after instance normalization and batch normalization processing to obtain splicing feature information, and activating and outputting the splicing feature information:

[0217]

[0218] ​​After the first convolution, an instance batch normalization layer (IN) is applied to process the transformed feature information. The transformed feature information is divided into two parts, one batch normalization block (BN) is used to perform batch normalization processing on half of the channels, and the other channels are subjected to instance normalization processing by applying an instance normalization layer. The latter allows the corresponding convolution block to capture the style-invariant features of the audio data of the song segment, thereby enabling better utilization of the song representation with style diversity in a single data.

[0219] Step S4300, after the activation output of the spliced feature information is subjected to multiple convolution operations and batch normalization processing, residual information is obtained:

[0220] The activated and output spliced feature information is further subjected to convolution operations by multiple convolution layers to further extract features. Each such convolution layer is followed by a batch normalization layer for normalization processing and output. The last convolution layer uses a 1*1 convolution kernel to implement, so as to avoid the attenuation of the representation learning ability of the entire feature extraction model after multiple instance normalization processes by multiple convolution blocks. Accordingly, the final output feature information is the residual information in the residual convolution process.

[0221] Step S4400, the residual information is superimposed on the information in the input and activated and output.

[0222] Finally, according to the residual convolution principle, the transformed feature information obtained by the first convolution is superimposed with the residual information and then activated and output, so as to obtain the intermediate feature information output after the residual convolution operation of the current convolution block.

[0223] In this embodiment, the convolution block required for constructing the feature extraction model of the application is constructed by applying residual convolution combined with instance batch normalization operation. The residual convolution network is improved based on the basic model of the Resnet series, and the IBN architecture is superimposed. The feature extraction model built in this way is easier to train and can achieve more accurate feature extraction effect, especially for the feature extraction of the audio data of the song segment.

[0224] Please refer to Figure 11 In the extended embodiment, the training process of the feature extraction model includes the following iterative training steps:

[0225] Step S5100, a training sample is called from the training set, and the encoding information of the training sample is determined. The training sample is a song segment with melody, which contains background music or does not contain background music, and the melody is in the form of human voice singing or MIDI voice simulation:

[0226] Those skilled in the art can understand that different training sets for training the feature extraction model can be constructed to adapt to different downstream tasks, such as humming recognition tasks, each training set containing a sufficient amount of training samples, and each training sample being provided with a corresponding supervision label.

[0227] The training samples can be pre-collected by those skilled in the art, and each training sample is audio data of a song segment. These audio data can be melodic audio data generated in MIDI sound simulation form, melodic audio data generated by human voice tracks in karaoke singing, and melodic audio data corresponding to the melody part sung by human voice segmented from a song file, etc. Song segments of the same main melody in different forms can contain background music or not. Different singing versions of the same song segment prepared in the aforementioned different forms can be combined into the same category, i.e., corresponding to the same supervision label, to enhance the generalization ability of model classification.

[0228] Subsequently, these training samples can also be used to make the melodic feature library and the non-melodic feature library required by the present application, for example:

[0229] In one embodiment, each song in the song library of an online music service platform is segmented into multiple song segments according to a specific time length range based on the time stamp of the lyrics of each song. Some of the song segments are melodic song segments, and the other part is non-melodic song segments. Subsequently, after the feature extraction model of the present application is trained to a convergent state, the feature extraction model is used to extract features from the two parts of song segments to obtain their corresponding high-dimensional index vectors. The non-melodic song segments and their high-dimensional index vectors are stored in the non-melodic feature library of the present application, and the melodic song segments and their high-dimensional index vectors are stored in the melodic feature library of the present application. The construction of the first and melodic feature libraries is completed respectively. The melodic song segments obtained in this way can be used as training samples of the present embodiment.

[0230] In another improved embodiment, song segments that do not meet the specific time length range can be deleted to improve the accuracy of the samples required for model training. In other embodiments of the present application, the high-dimensional index vectors of certain candidate song segments can be extracted using the feature extraction model of the present application, the similarity between the high-dimensional index vectors of the candidate song segments and the high-dimensional index vectors of the song segments in the pre-constructed non-melodic feature library is calculated, and a decision is made as to whether the candidate song segments are melodic song segments based on the similarity. Melodic song segments and their high-dimensional index vectors are added to the melodic feature library.

[0231] For the training samples in the training set, the encoding information corresponding to the audio data of the song segments can be prepared in advance for the convenience of model training, or the corresponding encoding information can be obtained in real time when the audio data of each song segment is called for training of the feature extraction model. For the specific encoding principle, refer to the corresponding process disclosed in the foregoing for processing.

[0232] Step S5200, input the encoding information into the feature extraction model to train it to obtain corresponding respective output feature vectors:

[0233] In the training process of a training sample, the encoding information corresponding to the training sample is output to the feature extraction model for feature extraction. For the principle of feature extraction, refer to the description of the principle of the feature extraction model in the foregoing embodiments, which will not be described here. In this process, the representation learning of the training sample is realized by the feature extraction model to obtain respective output feature vectors.

[0234] Step S5300, respectively, for each of the output feature vectors, classification prediction is performed to map out the corresponding classification label:

[0235] In this application, the training task of the feature extraction model is understood as a classification task, therefore, by connecting the output feature vectors of the feature extraction model to the corresponding prepared classification model, investigating the classification results of each classification model, and using the corresponding supervision label for supervision, the training of the model can be implemented. Based on this principle, in the training phase, a classification model is connected to the output end of each output feature vector of each branch network of the feature extraction model implemented by any embodiment of the application.

[0236] The classification model adopts the structure as shown in Figure 12 The classification model adopts the structure as shown in

[0237] The classifier in the classification model can be constructed by a multi-classifier implemented by a Softmax function, or by a multi-classifier implemented by an AM-Softmax function that can enhance the intra-class compactness and expand the inter-class sparsity. The latter obviously has better classification advantage.

[0238] Step S5400, calculate the loss value of the feature extraction model using the supervision label corresponding to the training sample and the classification label, and perform gradient update on the feature extraction model according to the loss value:

[0239] In the classification model, the batch normalization layer is adopted to balance the triplet loss and the cross-entropy classification loss, and subsequently, the triplet loss can be calculated by the batch normalization layer, and the cross-entropy classification loss can be calculated by the full connection layer, and the optimization of the output feature vector can be realized by comprehensively combining the two losses.

[0240] Accordingly, after the training sample is predicted with the corresponding classification label, the loss value between the supervision label and the classification label can be calculated according to the corresponding supervision label, and then the gradient update is performed on the feature extraction model according to the loss value, the weight parameters of each link of the whole model are corrected, and the model is converged.

[0241] Since there are multiple branch networks, each branch network can have multiple outputs of output feature vectors, and there are multiple classification models accordingly. Therefore, when calculating the loss value, a weighted method can be adopted, that is, the triplet loss and the classification loss in each classification model are weighted and summed to obtain the loss value corresponding to each output feature vector, and then the loss values corresponding to the output feature vectors are weighted and summed to obtain the final loss value, and the gradient update is performed on the whole feature extraction model according to the loss value.

[0242] Step S5500, determining whether the loss value reaches a preset threshold, and when the loss value does not reach the preset threshold, the next training sample in the training set is called to continue the iterative training of the feature extraction model until the loss value reaches the preset threshold:

[0243] For the loss value calculated for each training sample, it is determined whether it is infinitely close to 0 value or whether it reaches the preset threshold, and when it meets these determination conditions, it can be determined that the feature extraction model has been trained to a converged state, and accordingly, the training of the model can be terminated, and the feature extraction model can be put into the production stage. If the convergence state is not reached, the next training sample in the training set can be called to continue the iterative training of the feature extraction model until the feature extraction model is trained to a converged state.

[0244] The training principle and process of the feature extraction model of the present application are disclosed in the embodiment. As can be seen from the embodiment, the feature extraction model can learn the ability of extracting the corresponding output feature vector from the encoded information of the audio data of the song segment by training the feature extraction model with the prepared training set, the effective representation learning of the deep semantic information of the audio data of the song segment can be realized, and the output feature vectors of multiple scales of the audio data of the same song segment can be jointly trained, the training efficiency is higher, the model function is more abundant, and when the model is put into the production stage, the deep semantic information corresponding to multiple scales of the audio data of the same song segment can be quickly obtained.

[0245] The classification model of the embodiment can balance the gradient update of the model of the triplet loss and the classification loss due to the use of the batch normalization layer and the multi-classifier implemented by the AM-Softmax function, can make the model be trained to converge more quickly, and the trained model can better perform effective representation learning on the deep semantic information of the audio data of the song segment. When the output feature vector is combined and used as needed, the feature information of the audio data of the song segment can be more effectively represented, and a more efficient matching effect can be achieved.

[0246] The embodiment also embodies the scalability and compatibility of the feature extraction model of the application in application. Specifically, the embodiment allows the feature extraction model to serve different downstream tasks by training the feature extraction model using training samples corresponding to different downstream tasks, so that the feature extraction model can serve different downstream tasks. Therefore, it is a relatively basic improvement and has relatively good economic utility.

[0247] Please refer to Figure 13 The song matching device provided by the application is functionally deployed according to the song matching method of the application, and includes a segment encoding module 1100, a semantic extraction module 1200, a target matching module 1300, and a song pushing module 1400. The segment encoding module 1100 is configured to obtain encoding information corresponding to audio data of a karaoke melody to be matched submitted by a client. The semantic extraction module 1200 is configured to extract a high-dimensional index vector representing deep semantic information of the karaoke melody to be matched in multiple dimensions by using a feature extraction model trained to a convergent state according to the encoding information. The target matching module 1300 is configured to calculate the similarity between the high-dimensional index vector and a high-dimensional index vector representing deep semantic information of each melody segment in multiple dimensions extracted by the feature extraction model from a melody feature library, and to filter out a target melody segment satisfying a preset condition. The song pushing module 1400 is configured to push a target song containing the target melody segment in a song library to a client device.

[0248] In a deepened embodiment, the segment encoding module 1100 includes a request analysis sub-module configured to receive a song matching request submitted by a client and obtain audio data of a karaoke melody to be matched specified by the request. A voice detection sub-module is configured to detect whether the audio data contains human voice singing information, and if not, terminate subsequent execution. A format processing sub-module is configured to format the audio data into audio data between a minimum preset time length and a maximum preset time length. An audio encoding sub-module is configured to encode the audio data to obtain corresponding encoding information.

[0249] In an extended embodiment, the song matching device further comprises a filtering processing module, configured to call the non-melody feature library to verify the validity of the high-dimensional index vector of the to-be-matched a cappella melody, and continue to execute only when the high-dimensional index vector of the to-be-matched a cappella melody is determined to be valid after the high-dimensional index vector of the to-be-matched a cappella melody does not form a similarity with the high-dimensional index vector of the non-melody song segment in the non-melody feature library. The high-dimensional index vector of the non-melody song segment in the non-melody feature library is obtained by the feature extraction model extracting the feature representation of the non-melody song segment.

[0250] In a deepened embodiment, the filtering processing module comprises a non-melody calling submodule, configured to call the non-melody feature library, wherein the non-melody feature library stores a plurality of high-dimensional index vectors corresponding to a plurality of preset non-melody song segments, and the high-dimensional index vector is obtained by the feature extraction model extracting the encoding information of the corresponding non-melody song segment; a non-melody calculating submodule, configured to respectively calculate the similarity between the high-dimensional index vector of the to-be-matched a cappella melody and each high-dimensional index vector in the non-melody feature library, and obtain a corresponding similarity value; and a filtering judgment submodule, configured to determine whether there is a non-melody song segment with a similarity value exceeding a preset threshold, and terminate execution if the to-be-matched a cappella melody does not contain a song melody, and continue to execute subsequent steps if the to-be-matched a cappella melody contains a song melody.

[0251] In a deepened embodiment, the target matching module 1300 comprises a melody calling submodule, configured to call the melody feature library to obtain a high-dimensional index vector corresponding to each melody segment, wherein the high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of different scales of a melody segment as a whole; a melody calculating submodule, configured to respectively calculate the similarity between the high-dimensional index vector of the to-be-matched a cappella melody and each high-dimensional index vector in the melody feature library, and obtain a corresponding similarity sequence; and a segment screening submodule, configured to screen out a melody segment with a similarity value exceeding a preset threshold in the similarity sequence, and determine the melody segment as a target melody segment with a similarity satisfying a preset condition.

[0252] In an extended embodiment, the song matching device of the present application comprises the following modules for constructing the melody feature library: a song segment cutting module for obtaining audio data of a candidate song from a song library, and cutting the audio data into audio data corresponding to a plurality of melody segments to be verified according to time stamps of a lyrics file of the candidate song; an audio independent coding module for coding the audio data of each melody segment to be verified to obtain coding information thereof; an audio semantic extraction module 1200 for calling the feature extraction model to extract deep semantic information of multiple scales of the coding information of each melody segment to be verified, and correspondingly obtaining a high-dimensional index vector thereof; a segment filtering calculation module for calculating, for each melody segment to be verified, a similarity between the high-dimensional index vector thereof and high-dimensional index vectors of each non-melody song segment in the preset non-melody feature library, and obtaining a corresponding similarity value; and a segment library judgment module for judging, for each melody segment to be verified, whether there is a non-melody song segment with a similarity value exceeding a preset threshold, and if so, discarding the melody segment to be verified, and if not, adding mapping relationship data of the melody segment to be verified and the high-dimensional index vector thereof to the melody feature library.

[0253] In a specific embodiment, when the feature extraction model is called, it has the following structure: a shared extraction module for sequentially performing multi-level feature extraction on the coding information by using a plurality of convolution blocks in a shared network of the feature extraction model trained to a convergent state to obtain intermediate feature information in which deep semantic information of the coding information is extracted; a branch extraction module for converting output feature vectors of different scales after performing feature extraction on the intermediate feature information by using a plurality of convolution blocks in two or more branch networks of the feature extraction model, the output feature vectors of each branch network containing different deep semantic information; and an output processing module for outputting the output feature vectors of each branch network as the high-dimensional index vector by the feature extraction model.

[0254] In a deepened embodiment, the branch extraction module comprises any two or more of the following sub-modules: a first extraction sub-module for performing feature extraction on the intermediate feature information by using a plurality of convolution blocks in a first branch network to obtain global feature information, and pooling the global feature information into an output feature vector of a global scale; a second extraction sub-module for performing feature extraction on the intermediate feature information by using a plurality of convolution blocks in a second branch network, and then performing pooling on multiple parts obtained by cutting along channels to correspondingly obtain an output feature vector of a channel scale; and a third extraction sub-module for performing feature extraction on the intermediate feature information by using a plurality of convolution blocks in a third branch network, and then performing pooling on multiple parts obtained by cutting along frequency bands to correspondingly obtain an output feature vector of a frequency band scale.

[0255] In a preferred embodiment, when performing the pooling operation, the first branch network uses mean pooling and / or max pooling operation to obtain one or two output feature vectors of the global scale; when performing the pooling operation, the second branch network uses mean pooling operation for single or multiple channels to obtain one or more output feature vectors of the channel scale; and when performing the pooling operation, the third branch network uses mean pooling operation for single or multiple frequency bands to obtain one or more output feature vectors of the frequency band scale.

[0256] In a further embodiment, when the convolution block is invoked, it has a structure comprising: a convolution transformation unit for performing convolution transformation on the input information to obtain transformed feature information; a parallel normalization unit for performing instance normalization and batch normalization on the transformed feature information respectively, combining the normalized information into spliced feature information, and activating and outputting the spliced feature information; a residual calculation module for obtaining residual information by performing multiple convolution operations and batch normalization on the activated and output spliced feature information; and an activation output unit for superimposing the residual information on the input information and activating and outputting the information.

[0257] In an optional embodiment, in the shared network, at least one of the convolution blocks applies an attention module to extract key information from the audio data of the song segment, and the attention module is a spatial attention module or a channel attention module.

[0258] In an optional embodiment, the source of the encoded information is any one of the time-frequency spectrum information, the mel spectrum information, the CQT filter information, the loudness contour information, and the Chroma feature information of the corresponding audio data.

[0259] In an extended embodiment, the song matching device of this application includes a training structure for iteratively training the feature extraction model. This structure includes: a sample retrieval module for retrieving a training sample from the training set and determining the encoding information of the training sample, wherein the training sample is a melody song fragment, which may or may not contain background music, and the melody is in the form of vocals or MIDI onomatopoeia; a representation learning module for inputting the encoding information into the feature extraction model to train it and obtain corresponding output feature vectors; a classification prediction module for performing classification prediction on each of the output feature vectors to map them to corresponding classification labels; a gradient update module for calculating the loss value of the feature extraction model using the supervision label corresponding to the training sample and the classification label, and performing gradient updates on the feature extraction model based on the loss value; and an iterative decision module for determining whether the loss value reaches a preset threshold. If the preset threshold is not reached, the next training sample in the training set is retrieved to continue iterative training of the feature extraction model until the loss value reaches the preset threshold.

[0260] To address the aforementioned technical problems, embodiments of this application also provide computer equipment. For example... Figure 14 The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store a sequence of control information. When the computer-readable instructions are executed by the processor, the processor can implement a song matching method. The processor of the computer device provides computing and control capabilities, supporting the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When these computer-readable instructions are executed by the processor, the processor can execute the song matching method of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0261] In this embodiment, the processor is used to execute... Figure 13The specific functions of each module and its sub-modules in the above embodiment are described in detail in the foregoing description. The memory stores the program codes and various data required for executing the above modules or sub-modules. The network interface is used for data transmission between the user terminal and the server. The memory in the above embodiment stores the program codes and data required for executing all modules / sub-modules of the song matching device of the present application, and the server can call the program codes and data of the server to execute the functions of all sub-modules.

[0262] The present application also provides a storage medium storing computer readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the song matching method of any embodiment of the present application.

[0263] The present application also provides a computer program product comprising computer programs / instructions, which, when executed by one or more processors, implement the steps of the method described in any embodiment of the present application.

[0264] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of each method. The storage medium can be a computer readable storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0265] In summary, the present application uses a feature extraction model to realize multi-scale deep semantic information representation learning of song segments, obtains a high-dimensional index vector, and performs similar song segment matching based on the high-dimensional index vector. This can serve the query, retrieval, and matching of songs, and can play a more accurate and efficient role in serving various downstream tasks such as humming and singing recognition. Based on the submitted a cappella melody, the target song is matched, and the comprehensive service capability of the online music platform is improved.

[0266] A person of ordinary skill in the art can understand that the steps, measures, and schemes in the various operations, methods, and processes discussed in the present application can be alternated, changed, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in the present application can also be alternated, changed, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art with the various operations, methods, and processes disclosed in the present application can also be alternated, changed, rearranged, decomposed, combined, or deleted.

[0267] The above merely describes some embodiments of the present application, and it should be pointed out that, for those skilled in the art, some improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A song matching method, characterized in that, Includes the following steps: Obtain the encoding information corresponding to the audio data of the a cappella melody to be matched submitted by the client; A feature extraction model trained to convergence is used to extract a high-dimensional index vector representing deep semantic information at multiple scales of the a cappella melody to be matched, based on the encoded information. Calculate the similarity between the high-dimensional index vector and the high-dimensional index vector representing the multi-scale deep semantic information of each melody segment extracted by the feature extraction model in the melody feature library, and filter out the target melody segments whose similarity meets the preset conditions. Push the target song containing the target melody fragment from the music library to the client device; When the feature extraction model is invoked, the following steps are performed: Multiple convolutional blocks in the shared network of the feature extraction model, trained to convergence, are used sequentially to perform multi-level feature extraction on the encoded information to obtain intermediate feature information containing deep semantic information of the encoded information; Multiple convolutional blocks in two or more branch networks of the feature extraction model are used to extract features from the intermediate feature information at different scales, and then converted into output feature vectors of corresponding scales. The deep semantic information contained in the output feature vectors of each branch network is different; The feature extraction model outputs the output feature vectors of each branch network as the high-dimensional index vector. The training process of the feature extraction model includes the following iterative training steps: A training sample is retrieved from the training set, and the encoding information of the training sample is determined. The training sample is a melody song fragment, which may or may not contain background music, and the melody is in the form of vocals or MIDI sound effects. The encoding information is input into the feature extraction model for training to obtain corresponding output feature vectors. Classification prediction is performed on each output feature vector to map it to a corresponding classification label. The loss value of the feature extraction model is calculated using the supervision label corresponding to the training sample and the classification label, and gradient updates are performed on the feature extraction model based on the loss value. It is determined whether the loss value reaches a preset threshold. If the preset threshold is not reached, the next training sample in the training set is retrieved to continue iterative training of the feature extraction model until the loss value reaches the preset threshold.

2. The song matching method according to claim 1, characterized in that, The step of obtaining the encoding information corresponding to the audio data of the a cappella melody to be matched submitted by the client includes the following steps: Receive a song matching request submitted by the client and obtain the audio data of the a cappella melody to be matched as specified in the request; The system checks whether the audio data contains human vocal information; if not, it terminates the subsequent execution. The audio data is formatted to be between the minimum preset duration and the maximum preset duration. The audio data is encoded to obtain the corresponding encoding information.

3. The song matching method according to claim 1, characterized in that, Before the step of calculating the similarity between the high-dimensional index vector and the high-dimensional index vector representing the multi-scale deep semantic information of each melody segment extracted by the feature extraction model in the melody feature library, the following steps are included: The validity of the high-dimensional index vector of the a cappella melody to be matched is verified by calling the no-melody feature library. If the high-dimensional index vector is not similar to the high-dimensional index vector of the no-melody song fragment in the no-melody feature library, it is determined to be valid before the process continues. The high-dimensional index vector in the no-melody feature library is the feature representation of the no-melody song fragment extracted by the feature extraction model.

4. The song matching method according to claim 3, characterized in that, The process of calling the no-melody feature library to verify the validity of the high-dimensional index vector of the a cappella melody to be matched, and determining its validity only after the high-dimensional index vector does not form a similarity with the high-dimensional index vector of the no-melody song segment in the no-melody feature library, includes the following steps: The feature library for songs without melody is invoked. The feature library stores multiple preset high-dimensional index vectors corresponding to song segments without melody. The high-dimensional index vectors are obtained by the feature extraction model by extracting the encoding information of the corresponding song segments without melody. Calculate the similarity between the high-dimensional index vector of the a cappella melody to be matched and each high-dimensional index vector in the no-melody feature library, and obtain the corresponding similarity values. Determine if there is a song segment without melody whose similarity value exceeds a preset threshold. If it exists, determine that the a cappella melody to be matched does not contain the song melody and terminate the execution. If it does not exist, determine that the a cappella melody to be matched contains the song melody and continue to execute the subsequent steps.

5. The song matching method according to claim 1, characterized in that, The calculation of the similarity between the high-dimensional index vector and the high-dimensional index vector representing the multi-scale deep semantic information of each melody segment extracted by the feature extraction model in the melody feature library, and the selection of target melody segments whose similarity meets the preset conditions, includes the following steps: The melody feature library is invoked to obtain the high-dimensional index vectors corresponding to each melody segment. The high-dimensional index vector is a single high-dimensional vector that represents the deep semantic information of a melody segment at different scales as a whole. Calculate the similarity between the high-dimensional index vector of the a cappella melody to be matched and each high-dimensional index vector in the melody feature library to obtain the corresponding similarity sequence. Melody segments with similarity values ​​exceeding a preset threshold are selected from the similarity sequence and identified as target melody segments whose similarity meets the preset conditions.

6. The song matching method according to claim 1, characterized in that, This method includes the following steps for constructing a melody-based feature library: The audio data of candidate songs is obtained from the music library. Based on the timestamp of the lyrics file of the candidate songs, the audio data is divided into multiple audio data corresponding to the melody segments to be verified. The audio data of each melody segment to be verified is encoded to obtain its encoding information; For each melody segment to be verified, the feature extraction model is invoked to extract deep semantic information of its encoded information at multiple scales, thereby obtaining its high-dimensional index vector. For each melody segment to be verified, calculate the similarity between its high-dimensional index vector and the high-dimensional index vector of each melody-free song segment in the preset melody-free feature library, and obtain the corresponding similarity value. For each melody segment to be verified, determine whether there is a melody-less song segment with a similarity value exceeding a preset threshold. If so, discard the melody segment to be verified; otherwise, add the mapping relationship data between the melody segment to be verified and its high-dimensional index vector to the melody feature library.

7. The song matching method according to any one of claims 1 to 6, characterized in that, The step of extracting intermediate feature information at different scales by using multiple convolutional blocks from two or more branches of the feature extraction model and then converting it into an output feature vector of the corresponding scale includes any two or more of the following steps: The intermediate feature information is extracted by using multiple convolutional blocks in the first branch network to obtain global feature information, and the global feature information is pooled into a global scale output feature vector. After extracting features from the intermediate feature information using multiple convolutional blocks in the second branch network, the network is divided into multiple parts by channel and pooled to obtain the corresponding channel-scale output feature vector. After extracting features from the intermediate feature information using multiple convolutional blocks in the third branch network, the network is divided into multiple parts according to frequency band and pooled to obtain the corresponding output feature vector at the frequency band scale.

8. The song matching method according to claim 7, characterized in that: When the first branch network performs the pooling operation, it uses mean pooling and / or max pooling to obtain one or two output feature vectors at the global scale. When the second branch network performs the pooling operation, it uses mean pooling for one or more channels to obtain one or more output feature vectors at the channel scale. When the third branch network performs the pooling operation, it uses mean pooling for one or more frequency bands to obtain one or more output feature vectors at the frequency band scale.

9. The song matching method according to any one of claims 1 to 6, characterized in that, When the convolutional block is invoked, the following steps are performed: The information in the input is subjected to convolutional transformation to obtain transformation feature information; The transformed feature information is combined into spliced ​​feature information after being processed by instance normalization and batch normalization respectively, and the spliced ​​feature information is activated and output. The concatenated feature information of the activation output is processed through multiple convolution operations and batch normalization to obtain residual information; The residual information is superimposed onto the input information to activate the output.

10. The song matching method according to any one of claims 1 to 6, characterized in that, In the shared network, at least one of the convolutional blocks applies an attention module to extract key information from the audio data of a song segment. The attention module is either a spatial attention module or a channel attention module.

11. The song matching method according to any one of claims 1 to 6, characterized in that, The source of the encoded information is any one of the following: time-spectrum information, Mel spectrum information, CQT filter information, pitch contour information, and Chroma feature information of the corresponding audio data.

12. A song matching device, characterized in that, include: The segment encoding module is used to obtain the encoding information corresponding to the audio data of the a cappella melody to be matched submitted by the client; The semantic extraction module is used to extract a high-dimensional index vector representing deep semantic information at multiple scales of the a cappella melody to be matched based on the encoded information using a feature extraction model that has been trained to convergence. The target matching module is used to calculate the similarity between the high-dimensional index vector and the high-dimensional index vector representing the multi-scale deep semantic information of each melody segment extracted by the feature extraction model in the melody feature library, and to filter out target melody segments whose similarity meets the preset conditions. The song push module is used to push target songs containing the target melody fragment from the music library to the client device; When the feature extraction model is invoked, the following steps are performed: Multiple convolutional blocks in the shared network of the feature extraction model, trained to convergence, are used sequentially to perform multi-level feature extraction on the encoded information to obtain intermediate feature information containing deep semantic information of the encoded information; Multiple convolutional blocks in two or more branch networks of the feature extraction model are used to extract features from the intermediate feature information at different scales, and then converted into output feature vectors of corresponding scales. The deep semantic information contained in the output feature vectors of each branch network is different; The feature extraction model outputs the output feature vectors of each branch network as the high-dimensional index vector. The training process of the feature extraction model includes the following iterative training steps: A training sample is retrieved from the training set, and the encoding information of the training sample is determined. The training sample is a melody song fragment, which may or may not contain background music, and the melody is in the form of vocals or MIDI sound effects. The encoding information is input into the feature extraction model for training to obtain corresponding output feature vectors. Classification prediction is performed on each output feature vector to map it to a corresponding classification label. The loss value of the feature extraction model is calculated using the supervision label corresponding to the training sample and the classification label, and gradient updates are performed on the feature extraction model based on the loss value. It is determined whether the loss value reaches a preset threshold. If the preset threshold is not reached, the next training sample in the training set is retrieved to continue iterative training of the feature extraction model until the loss value reaches the preset threshold.

13. A computer device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 11, which, when invoked by a computer, executes the steps included in the corresponding method.

15. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 11.

Images