Asymmetric deep hashing method, network and network training method for massive music retrieval

Abstract

An asymmetric deep hashing method, network and network training method for massive music retrieval belong to the intersection of multimedia information technology and machine learning. In order to solve the problem of massive music retrieval, the binary representation of the music data to be retrieved and the binary representation of each music data in the music database are obtained, and the binary representation of the music data of the same category in the music database tends to the quantized center distribution of the music category, and the quantized center is the binary representation of the music category in the music database; the Hamming distance between the binary representation of the music data to be retrieved and the binary representation of each music data in the music database is calculated; according to the Hamming distance, the category label of the music data in the music library with the closest Hamming distance to the binary representation of the music data to be retrieved is used as the category label of the music to be retrieved, which has the effect of improving the deep hashing binary representation capability under a small-scale training data set.

Description

Technical Field

[0001] The present invention belongs to the intersection of multimedia information technology and machine learning, and relates to an asymmetric deep hashing method for massive music retrieval. Background Art

[0002] With the rapid development of technologies such as the internet, mobile internet, and the Internet of Things, we have entered the Information Age. Online multimedia data is exploding at an unprecedented rate. Applications based on this data, such as online audio-visual, social networking software, and online shopping, are becoming increasingly popular, influencing and enriching people's daily lives and creating significant social and economic value. Music, as a key multimedia data type, has evolved through the eras of columnar records, films, cassettes, and CDs, and is now entering a new digital age. This has led to an unprecedented boom in personal cover music, music communities, and short video apps, with an increasing amount of digital music being available online.

[0003] In the era of big data on the Internet, an efficient retrieval algorithm based on music content is very necessary for music-related applications. First, it can improve the current mainstream text-based music retrieval model and provide more personalized retrieval services for commercial digital music service providers such as Kugou Music and Baidu Music, such as humming retrieval and music-based music detection. Secondly, it can provide music creators with more practical originality inspection and copyright protection methods. Nowadays, ordinary people can become music creators, and how to quickly determine the similarity between the content of their creations and the vast number of existing musical works is very useful. Finally, it can provide a more convenient music addition service for multimedia production, such as matching different music in a large number of game scenes or adding background music when making short videos. The most convenient way is to match the most suitable music according to the usage scenario from the massive music library.

[0004] However, in real-world applications, music data often exhibits high-dimensionality, multi-source, and heterogeneous characteristics. Efficient indexing and retrieval of this high-dimensional data directly impacts the user experience of these music multimedia applications and is the core of efficient music content-based retrieval algorithms. Converting massive amounts of music data into binary code for storage and retrieval has been a popular topic recently. This is primarily due to two factors: first, binary representation requires very little storage space; second, similarity between binary codes can be calculated using Hamming distance or multi-hash table indexing, resulting in highly efficient retrieval.

[0005] Of course, how to improve the representation ability of binary data, that is, whether the data in the binary space can maintain the neighbor relationship between the original space, is the key to hashing the data. To this end, researchers have proposed a series of hashing methods for general vector data, including audio data. For example, according to whether the label information of the sample data is used, it can be divided into unsupervised hashing methods and supervised hashing methods; according to the original data similarity maintenance strategy, it can be divided into distance matrix maintenance methods, binary / triplet maintenance methods, sorting maintenance methods, etc. With the rapid development of deep learning technology in recent years, supervised hashing technology based on deep networks can greatly improve the representation ability of hash binary, showing outstanding strength beyond traditional hashing methods, and has attracted the interest of more and more scholars. However, its development also has some bottlenecks and problems.

[0006] First, in supervised hashing methods based on deep networks, the objective function needs to measure the representation ability of the output binary code for the original input sample. A common strategy is to construct an error term through the network output of music similarity pairs (two samples with the same label) or music triplets (two of the three samples have the same label). However, this similarity measurement has flaws. First, for the entire database, the music binary / triplet is only a local similarity representation, which limits the representation ability of the hash binary. Some scholars have proposed to improve it by using the sorting relationship of the multitudes, but the algorithm complexity is very high. Second, when the database is large, the number of combinations of its multitudes is of a very large magnitude. In practical applications, it is often impossible to traverse all multitudes. At the same time, there will be a multitude imbalance problem. For example, the number of similar music sample pairs is much smaller than the number of dissimilar music sample pairs, which also limits the performance of this type of hashing method.

[0007] Secondly, most deep hashing methods use a symmetric binary generation mechanism, meaning that the binary representations of both the training data and the database data are generated by passing the data through the same neural network. The performance of such methods often depends on the amount of training data: the larger the amount of training data, the stronger the binary representation and generation capabilities of the trained deep neural network. For example, when constructing an error function based on music sample tuples, the large number of sample tuples and the fact that each pair of music samples must be trained through a neural network restrict the number of training samples that can be selected in practice. For massive music datasets in reality, the aforementioned symmetric deep hashing methods are clearly unable to fully utilize the information in the massive datasets due to the limited size of the training datasets, making it difficult to produce high-quality binary representations.

[0008] In summary, in the current network environment with massive music data, effective music data storage and retrieval requires the support of an efficient deep hashing method with strong representation capabilities. Summary of the Invention

[0009] In order to overcome the above-mentioned shortcomings, the present invention aims at the problem of massive music retrieval and proposes an asymmetric deep hashing method based on center similarity, with the aim of enhancing the hashing method's ability to use supervision information of massive music databases and improving the representation of binary representation for original music data and neighbor retrieval performance.

[0010] In one aspect, the present invention provides an asymmetric deep hashing method for massive music retrieval in one embodiment, comprising:

[0011] Obtaining binary representations of the music data to be retrieved and binary representations of each music data in a music database, wherein the binary representations of the music data of the same category in the music database tend to be distributed around a quantized center of the music category, the quantized center being the binary representation of the music category in the music database;

[0012] Calculating the Hamming distance between the binary representation of the music data to be retrieved and the binary representation of each music data in the music database;

[0013] According to the Hamming distance, the category label of the music data in the music library having the closest Hamming distance to the binary representation of the music data to be retrieved is used as the category label of the music to be retrieved.

[0014] In an asymmetric deep hashing method for massive music retrieval in one embodiment of the present invention, the average value of the quantized center Hamming distance between any two of the music categories is at least k / 2, where k represents the number of bits of the generated binary representation.

[0015] In an asymmetric deep hashing method for massive music retrieval in one embodiment of the present invention, the method for obtaining the quantization center includes:

[0016] For a given number of bits k in binary representation, the Hadamard matrix H of order k is k,k and -H k,k Represents 2k quantization centers, and 2k quantization centers are represented by the matrix L = [H k,k ;-H k,k ];

[0017] If the number of music categories in the music database is t≤2k, the first t rows of the matrix L are taken as the binary representation of the music category; otherwise, t-2k quantization centers are generated through Bernoulli distribution, and the t-2k quantization centers are represented as the matrix H1. The matrix L and the matrix H1 are combined to obtain the binary representation of the music category in the music database.

[0018] In an asymmetric deep hashing method for massive music retrieval in one embodiment of the present invention, s matrices H1 are generated through Bernoulli distribution, and the matrix H1 corresponding to the maximum average value of the Hamming distance between the quantization centers of any two of the music categories is selected as the t-2k quantization centers.

[0019] In a second aspect, the present invention provides an asymmetric deep hashing network for massive music retrieval in one embodiment, which is used to implement the method, including:

[0020] Convolutional networks, used to extract features from music data;

[0021] A multilayer perceptron is used to generate binary representations of the features of the music data.

[0022] In an embodiment of the present invention, an asymmetric deep hashing network for massive music retrieval is provided, wherein the loss function of the asymmetric deep hashing network for massive music retrieval includes:

[0023] A first constraint, wherein the first constraint is used to constrain the network output to be consistent when the category labels of the music in the training music set and the music in the music database are the same, and to constrain the distance between the network output and the binary representation of the music in the database to be k bits of the binary representation when the category labels of the music in the training music set and the music in the music database are different;

[0024] A second constraint, wherein the second constraint is used to constrain the output of the network to be near the quantization center of the music category to which the input music belongs;

[0025] The third constraint is used to constrain the binary representation of the music in the music database to be near the quantization center of the music category to which the music belongs.

[0026] In an embodiment of the present invention, the asymmetric deep hashing network for massive music retrieval is used, and the loss function is expressed as

[0027]

[0028] in:

[0029] L1(Θ,B)=∑ ij (tanh(f(x i ;Θ)) T b j -k*s ij ) 2 , L1(Θ,B) represents the hash loss, tanh(f(x i ;Θ)) represents the output of the network, x irepresents the spectrum representation of the i-th training music, Θ represents the network parameters, b j represents the binary representation of the jth music in the music database, k represents the bit length of the binary representation, a ij Represents the similarity between music i and j, the similarity matrix S of the training music set and the music database mn : represents the label of the i-th training music, l j Indicates the label of the j-th database music;

[0030] L2(Θ)=∑ i (tanh(f(x i ;Θ))-c(x i )) 2 , L2(Θ) represents the central quantization loss of training music, c(x i ) represents the quantized center of the i-th training music, x i represents the spectrum representation of the i-th training music;

[0031] L3(B)=∑ i (b i -c(b i )) 2 , L3(B) represents the central quantization loss of database music, b i represents the binary representation of the music in the i-th database, c(b i ) represents the quantitative center of the music in the i-th database;

[0032] λ1 is the first hyperparameter of the loss function, λ2 is the second hyperparameter of the loss function, B represents the binary representation of the entire music database, and n represents the number of music in the database.

[0033] On the third aspect, in one embodiment of the present invention, a training method for an asymmetric deep hashing network for massive music retrieval is provided. The asymmetric deep hashing network for massive music retrieval obtains optimized network parameters Θ and a binary representation B of the music database through an iterative process of alternating optimization during training. In each iterative process, the method includes:

[0034] Fix the binary representation B of the music database, optimize the network parameters Θ, and fix the network parameters Θ, optimize the binary representation B of the music database, until the iteration termination condition is met.

[0035] In one embodiment of the present invention, a training method for an asymmetric deep hashing network for massive music retrieval is provided. The binary representation B of the fixed music database and the method for optimizing the network parameter Θ include:

[0036] S1. First, in the music database<Y,L> Randomly select music training set

[0037] S2. Construct the similarity matrix S between the training music set and the music database mn : represents the label of the i-th training music, l j represents the label of the j-th database music,

[0038] S3. Calculate the gradient of L(Θ,B):

[0039]

[0040] in: represents the gradient of L(Θ,B), a i =tanh(f(x i ;Θ)) represents x i The output after passing through the network, x i represents the spectrum representation of the i-th training music, Θ represents the network parameters, b j represents the binary representation of the jth music in the music database, k represents the bit length of the binary representation, s ij represents the similarity between music i and j, λ1 represents the first hyperparameter of the loss function, c(x i ) represents the quantized center of the i-th training music, ⊙ represents the Hadamard product, that is, the multiplication of the elements at corresponding positions;

[0041] S4. Use back propagation technology to train network parameters Θ, where a i =tanh(f(x i ;Θ)),tanh(f(x i ;Θ)) represents the output of the network.

[0042] In one embodiment of the present invention, a training method for an asymmetric deep hashing network for massive music retrieval is provided. The method of fixing the network parameter Θ and optimizing the binary representation B of the music database includes:

[0043] S1. Expand L1(Θ,B) and L3(B) to obtain in Where: A represents the network output of the entire training subset, B represents the binary representation of the entire music database, k represents the bit length of the binary representation, S represents the similarity matrix between the training subset and the music database, tr(.) represents the trace of the matrix, λ2 represents the second hyperparameter of the loss function, const represents a constant, b n Represents the binary representation of the nth database music;

[0044] S2. Iterate the matrix B column by column to obtain the solution of L(Θ,B): in This formula represents the calculation method of the w-th column of matrix B, where w means only taking the w-th column of the current matrix. It means taking all columns except w from the current matrix, where sign(.) represents the sign function.

[0045] The beneficial effects of the present invention are:

[0046] (1) The hash results have stronger representation power. The center similarity-based method improves the discrimination of binary representations of different music categories. By constructing the optimization target through music center and asymmetric hashing method, the binary representation power obtained is better than the traditional method based on music multi-tuples.

[0047] (2) The hashing method is more efficient. The hashing method training process fully utilizes the label information of all the music in the database. While obtaining the deep network parameters, it also obtains the binary representation of the music in the database. This method is more suitable for solving the problem of improving the binary representation capability of deep hashing in small-scale training datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is the overall module design diagram of the present invention.

[0049] Figure 2 This is an application diagram of the deep hashing method based on center similarity of the present invention.

[0050] Figure 3 This is a diagram of the network training and music library binary representation generation process of the present invention. DETAILED DESCRIPTION

[0051] The embodiments of the present invention are described in detail below.

[0052] Example 1: The present invention combines the advantages of center similarity quantification and deep neural networks to perform hash representation on massive music data. This process is supervised, and then music retrieval is realized in the Hamming space. The basic idea is to generate a discriminative binary code for the category identifier of the massive music library, and then constrain the training data and database data to be close to the binary code of their category identifier through a deep neural network. Based on this idea, the present invention first generates a binary code for the category identifier of the music library based on the Hadamard matrix as the hash center of the music data; secondly, a deep neural network is constructed to perform feature representation on the music data, and the training loss function is constructed by combining the category identifiers of the training data and the database data with the obtained hash center; finally, the training data is selected, and the network is trained through the above-mentioned network and loss function to obtain the binary representation of the trained network and database data.

[0053] For the music data to be retrieved, the trained network can be used to obtain its binary representation. The Hamming distance between the data and all binary codes in the music database is then calculated, and the music with the smallest distance is returned as the query result. If the music database is large, such as millions of music codes, multiple hash tables can be used to index the music database to further accelerate the query process. Figure 1 The overall module design diagram of the present invention is shown. In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is as follows:

[0054] An asymmetric deep hashing method for massive music retrieval Figure 1 As can be seen from the figure, the implementation of the present invention is divided into two stages: an offline stage and an online stage. In the offline stage, the main task is to generate a binary representation of the music category identifier. Based on this category center and the deep network, the feature learning and loss function of the music are designed. Finally, a training dataset is selected to train the network and generate a binary representation of the database data. In the online stage, the input test music is passed through the deep network obtained in the offline stage to generate a binary representation. The Hamming distance between the binary representation and the database data is calculated to achieve retrieval of massive music.

[0055] Based on these two phases, the asymmetric deep hashing method consists of four modules: a module for generating a music-centric binary code, a module for constructing a deep network and loss function, a module for training the network and generating the music binary code, and a module for massive music retrieval based on binary representation. The first three modules are offline, while the last one is online. These modules can be designed based on specific massive music retrieval application scenarios.

[0056] The Music Center binary representation generation module is primarily responsible for generating binary representations for the category labels in the music database, based on a given binary bit length. The binary representations must meet certain discrimination requirements. This module utilizes two binary generation methods: one based on the classic Hadamard matrix and one based on Bernoulli distribution sampling.

[0057] The deep network and loss function construction module primarily designs a deep network to extract features from the input music, constructs a similarity matrix based on the labels of the database music, and defines a loss function for network training based on the binary representation of the music center. The loss function in this module includes the approximation error of the similarity matrix between the training music and the database music, and the quantization error of each relative to the binary representation of the center.

[0058] The network training and music binary representation generation module primarily selects a training music set and, based on the deep network and loss function, designs the network parameter training and binary generation processes. This module trains the network parameters and generates the binary representation of the music database through an iterative process of alternating optimization. Each iteration consists of two parts: first, the binary representation of the database music is fixed, and the network parameters are trained through backpropagation. Then, the network parameters are fixed and the binary representation of the music database is optimized until the iteration termination condition is met.

[0059] The main task of the binary-representation-based music retrieval module is to obtain the binary representation of the music to be queried using the trained deep hash network, calculate and sort the Hamming distance with the binary code of the music database, and return the nearest one or more music data.

[0060] The characteristics of this method that distinguish it from existing methods are:

[0061] (1) This invention combines the advantages of center similarity quantification and asymmetric hashing technology. By binary-representing the category labels of database music, the hash center of the music data is obtained. Then, based on the hash center, a similarity matrix is constructed using the labels of the training music and the database music. A hash map is defined through a deep neural network, which constrains the hash results of the training music and the database music to be located near the binary center of their labels, thereby improving the accuracy of the binary representation of the music and the performance of nearest neighbor retrieval.

[0062] (2) This invention proposes a novel optimization objective for an asymmetric deep hashing method. Based on the binary representation of the category center and a specific deep network form, a new objective loss function is constructed by defining the similarity error between the binary representations of the training music and the database music, the approximation error of the training music to its category center, and the approximation error of the database music to its category center. An iterative solution method with alternating optimization is also proposed. In each iteration, the parameters of the deep network and the binary representation of the database music are alternately optimized.

[0063] Example 2: An asymmetric deep hashing method for massive music retrieval, referring to Figure 2 The method is implemented by four modules: a music center binary code generation module, a deep network and loss function construction module, a network training and music binary code generation module, and a massive music retrieval module based on binary representation. Specifically:

[0064] (1) Music Center Binary Code Generation Module

[0065] For a given music database and its category labels represents the i-th music, l i Represents its category label. Similar to most music data processing methods, y i Refers to the spectral representation of the original music, such as Mel-frequency cepstral coefficients (MFCC). This spectral representation is not necessary, but it can improve the performance of subsequent music processing algorithms. The goal of the hashing method is to find a mapping f:R d →{0,1} k , so that any music data y, b = f (y) is the binary representation of the music data y with a length of k, then the entire music library Y is converted into its binary representation In an ideal hash map f, for two music pieces with the same label, their binary representations should be close to the same. The music data is the spectrum representation data of the music.

[0066] Since the traditional music multi-tuple method cannot well reflect the similarity relationship of the data as a whole, this paper proposes that we can first find the quantization center for the category label by using the quantitative method of center similarity. Each category corresponds to a quantization center, which is the binary representation of the category label. Then, we use the mapping f to constrain each music data to tend to the quantization center of its category label, thereby finding a suitable mapping f. The quantization center C is expressed as t is the number of category labels, that is, the number of music categories in the music database. To maintain good discrimination between different quantization centers (binary representations of different category labels), the ideal quantization center C should generally satisfy the requirement that the average Hamming distance between all two different quantization centers is at least k / 2. In a music database, music of the same category is distributed near the quantization center corresponding to its category. Quantization centers distributed farther apart cause music of different categories to be spatially distributed farther apart, making it less likely to be confused and maintaining good discrimination.

[0067] That is, the following formula (1) is satisfied.

[0068] avg({d H (c i ,c j )|1≤i,j≤t,i>j})≥k / 2 (1)

[0069] Among them: avg{} means finding the average, d H Represents the distance between any two quantization centers. i is greater than j to avoid repeated access, such as d H (c1,c2) and d H (c2,c1) is a case of repeated acquisition.

[0070] Algorithm 1 describes the generation of binary representations of music centers, while Algorithm 2 describes network training and the generation of binary representations. Algorithm 2 constrains music of the same category to be distributed near its center to improve the accuracy of the trained model. However, this results in each music center being associated with a large number of music pieces. If the Hamming distance of the music centers is not restricted, it may lead to confusion between music categories, resulting in reduced model accuracy.

[0071]

[0072] Algorithm Description:

[0073] In order to generate the quantization center C that satisfies formula (1), this module adopts the following two methods. Method 1: With the help of Hadamard matrix, a k×k dimensional Hadamard matrix H k,k =[h1;…;h k ] satisfies any two rows to be orthogonal, i.e. <h i ,h j >=0, and d H (h i ,h j )=k / 2. k,kThe generation of can be done by Sylvester recursive iterative algorithm. Method 2: Since the dimension k of Hadamard matrix must be a multiple of 4, the Hadamard matrix method is invalid for the generation of binary numbers with a bit length that is not a multiple of 4. In this case, this module uses the sampling method based on Bernoulli distribution to generate the quantization center C. That is, for c i For each bit of , a number l∈(0,1] is randomly generated. If l≤0.5, the current bit is 0, otherwise, the current bit is 1. Based on the above two generation methods, the generation process of the binary representation of the music center is shown in Algorithm 1.

[0074] Specifically, for a given binary bit length k, the second line of Algorithm 1 uses the k-order Hadamard matrix H k,k and -H k,k , we obtain 2k quantization centers that satisfy formula (1), represented by matrix L. If the number of music categories t≤2k, the algorithm takes the first t rows of matrix L as the binary representation of the music category (row 4). Otherwise, the algorithm uses the second method to generate t-2k quantization centers H1 with the help of Bernoulli distribution, and together with matrix L, it serves as the binary representation of the music category (rows 7 and 8). To reduce the randomness brought by the Bernoulli distribution, in practice, the algorithm generates s H1 and selects the average value of the Hamming distance of all different quantization centers, that is, formula (1), and the corresponding H1 when it reaches the maximum value.

[0075] (2) Construction modules of deep networks and loss functions

[0076] The main task of supervised hashing methods is to find a mapping f such that, for an input piece of music y, the resulting mapping b = f(y) approaches the binary representation of its class identifier. Based on this, this module has two main tasks: first, how to define the mapping f so that it effectively extracts the features of the input music; second, how to design an efficient loss function that fully utilizes the supervised information in the music database.

[0077] As a typical nonlinear mapping, deep neural networks, especially convolutional networks, have demonstrated powerful feature extraction capabilities in multimedia applications. This module's deep network architecture also uses a convolutional network as the music feature extraction layer. Currently, a variety of deep networks are commonly used in audio processing, such as AlexNet, Inception V3, ResNet50, VGGish, and TLWeak. In practice, this module selects the VGGish network for music feature extraction. Furthermore, this module uses a three-layer multilayer perceptron (MLP) as the subsequent hashing layer to generate a hash representation of the music features. The number of neurons is 512, 256, and k, respectively, where k is the given hash bit length. The final layer of the network uses the tanh(.) activation function, which keeps the output values close to -1 and +1.

[0078] In terms of loss function design, different from the traditional loss function based on the multi-tuple of training samples, this module adopts an asymmetric loss function construction strategy. Generally selected from database music, that is, According to the label of the training music And the label L of the database music, we can get the similarity matrix S of the training music and the database music mn .

[0079]

[0080] in: represents the label of the i-th training music, l j Represents the label of the j-th database music.

[0081] Based on the similarity matrix S mn and the binary representation of the category identifier, the loss function L(Θ,B) of the deep network is constructed as shown in formula (3), Θ is the network parameter, and B is the binary representation of the database music Y.

[0082]

[0083] Where: L1(Θ,B)=∑ ij (tanh(f(x i ;Θ)) T b j -k*s ij ) 2 , L1(Θ,B) represents the hash loss, that is, the first constraint, tanh(f(x i ;Θ)) represents the output of the network, x i represents the spectrum representation of the i-th training music, Θ represents the network parameters, b j represents the binary representation of the jth music in the music database, k represents the bit length of the binary representation, s ij Represents the similarity between music i and j, the similarity matrix S of the training music set and the music database mn : represents the label of the i-th training music, l j Represents the label of the j-th database music.

[0084] L2(Θ)=∑ i (tanh(f(x i ;Θ))-c(x i )) 2 , L2(Θ) represents the central quantization loss of training music, c(x i) represents the quantized center of the i-th training music, x i represents the spectrum representation of the i-th training music.

[0085] L3(B)=∑ i (b i -c(b i )) 2 , L3(B) represents the central quantization loss of database music, b i represents the binary representation of the music in the i-th database, c(b i ) represents the quantitative center of the i-th database music.

[0086] λ1 is the first hyperparameter of the loss function, λ2 is the second hyperparameter of the loss function, B represents the binary representation of the entire music database, and n represents the number of music in the database. ij (tanh(f(x i ;Θ)) T b j -k*s ij ) 2 , the network parameters Θ need to be estimated through training data. Obviously, if the training music has x i and database music j As with the category label, L1(Θ,B) constrains the network output tanh(f(x i ;Θ)) and y j The binary representation of b j Try to make them consistent, otherwise try to make the distance between them k. L2(Θ)=∑ i (tanh(f(x i ;Θ))-c(x i )) 2 In order to constrain the output of the network tanh(f(x i ;Θ)) in its category identification binary code c(x i ). L3(B)=∑ i (b i -c(b i )) 2 In order to constrain the binary representation of the database music in its category identification binary code c(b i ). λ1 and λ2 are two hyperparameters of the loss function and can be determined through experiments.

[0087] (3) Network training and music binary code generation module

[0088] Based on the network constructed in the previous module and the loss function formula (3), this module obtains the optimized network parameters Θ and the binary representation B of the music database through an iterative process of alternating optimization. In each iteration, there are two main steps: first, the binary representation B of the music database is fixed and the network parameters Θ are optimized; second, the network parameters Θ are fixed and the binary representation B of the music database is optimized until the iteration termination condition is met. The overall execution process of module three is shown in Figure 3 .

[0089] Algorithm 2 gives the network training and the generation process of the binary representation of music, where T is the number of network training times, i.e., the size of the epoch. In each epoch, first, fix B and optimize Θ, see lines 3-6 of Algorithm 2. At this time, the third term of the loss function formula (3) is fixed, and only L(Θ, B) = L1(Θ, B) + λ1L2(Θ) needs to be optimized. This module first calculates the value of the loss function in the database.<Y,L> Randomly select a training set According to formula (2), the similarity matrix S (row 3) is constructed. In practice, the gradient of L(Θ, B) is calculated by the following formula (4), and the network parameters Θ are trained using back propagation technology, where α i =tanh(f(x i ;Θ)),tanh(f(x i ;Θ)) represents the output of the network.

[0090]

[0091] in: represents the gradient of L(Θ,B), a i =tanh(f(x i ;Θ)) represents x i The output after passing through the network, x i represents the spectrum representation of the i-th training music, Θ represents the network parameters, b j represents the binary representation of the jth music in the music database, k represents the bit length of the binary representation, a ij represents the similarity between music i and j, λ1 represents the first hyperparameter of the loss function, c(x i ) represents the quantized center of the i-th training music, and ⊙ represents the Hadamard product, which is the multiplication of the elements at corresponding positions. Then, fix Θ and optimize B, as shown in lines 7-9 of Algorithm 2. At this time, the second term of the loss function formula (3) is fixed, and only L(Θ,B)=L1(Θ,B)+λ2L3(B) needs to be optimized. Expanding L1(Θ,B) and L3(B) yields in Where: A represents the network output of the entire training subset, B represents the binary representation of the entire music database, k represents the bit length of the binary representation, S represents the similarity matrix between the training subset and the music database, tr(.) represents the trace of the matrix, λ2 represents the second hyperparameter of the loss function, const represents a constant, b n Indicates the binary representation of the nth database music. The solution of L(Θ,B) can be obtained by iteratively solving the matrix B column by column using formula (5).

[0092]

[0093] Formula (5) gives the calculation method of the w-th column of matrix B, where w means taking only the w-th column of the current matrix. Indicates taking all columns except w from the current matrix. sign(.) indicates the sign function.

[0094]

[0095] (4) Massive music retrieval module based on binary representation

[0096] After the above module obtains the deep network parameters Θ and the binary representation B of the music database Y, this module builds a music retrieval application based on binary representation. For the input music to be retrieved q, its spectral coefficients are first calculated. The module uses the Mel frequency cepstral coefficients m(q), and then calculates its binary representation a through formula (6).

[0097] a=sign(f(m(q);Θ)) (6) Finally, calculate the Hamming distance between a and all binaries in database B. Sort by Hamming distance and return the closest one or more music data as the query result.

[0098] If the binary database B is large, this module uses the multi-hash table indexing technology to further improve the query speed. Simply put, before querying, the k bits are randomly divided into t mutually disjoint bit groups {T1, T2, ..., T t}, then through each bit group T i The music in the database is indexed separately, resulting in t hash table indexes. When searching for a piece of music q, the corresponding index items are found in each of the t hash tables. These index items are combined to form a candidate set of nearest neighbors for the query. The candidate set is then sorted by Hamming distance and the query result is returned. Multiple hash table indexes can accelerate the nearest neighbor query process for binary codes, both theoretically and practically.

[0099] The present invention is used to solve the problem of massive music retrieval, converting music data into binary codes for efficient representation and retrieval. First, a binary representation is generated for the category identifier of the music library, which serves as the hash center of the music data; second, a deep neural network is constructed to perform feature extraction and hash mapping on the music, and the loss function of the deep network is constructed by combining the category identifier of the training set and the music database with the hash center obtained above; finally, training data is selected, and asymmetric network training is performed using the above network and loss function to obtain the binary code of the trained deep network and the music database. For the image to be retrieved, its binary code is obtained through the trained network, and its nearest neighbors in the binary music library are calculated based on the Hamming distance and returned. The present invention combines the advantages of central similarity quantification and asymmetric hashing technology. First, a discriminative hash center is generated for the category identifier of the massive music library, and then the training data and database data are constrained by the deep neural network to be close to the hash center of their category identifier. Compared with existing methods, the binary representation obtained by the present invention has stronger descriptive ability and higher retrieval accuracy.

[0100] As described above, the deep hashing method described in the present invention is implemented based on the following four modules: a module for generating a music center binary code, a module for constructing a deep network and loss function, a module for training the network and generating the music binary code, and a module for retrieving massive music data based on binary representation. This method combines the advantages of both center similarity quantification and asymmetric hashing techniques. It first generates discriminative binary codes for the category identifiers of a massive music library, and then uses a deep neural network to constrain the training data and database data to be close to the binary codes of these category identifiers. Compared to existing methods, this method produces a more descriptive binary representation and achieves higher retrieval accuracy.

[0101] First, a binary representation is generated for the category identifier of the music library, which serves as the hash center of the music data. Second, a deep neural network is constructed to perform feature extraction and hash mapping on the music data. The loss function of the deep network is constructed by combining the category identifiers of the training set and the music database with the hash center obtained previously. Finally, training data is selected and asymmetric network training is performed using the above network and loss function to obtain the binary representation of the trained network and database data. For the test music, its binary representation is obtained through the trained network.

[0102] The Music Center binary code generation module is primarily responsible for generating binary representations of the music database's category labels, based on a given binary bit length. The binary representations must meet certain discrimination requirements. This module employs two binary generation methods: one based on the classic Hadamard matrix and one based on Bernoulli distribution sampling.

[0103] The deep network and loss function construction module primarily designs a deep network to extract features from the input music, constructs a similarity matrix based on the labels of the database music, and defines a loss function for network training based on the binary representation of the music center. The loss function in this module includes the approximation error of the similarity matrix between the training music and the database music, and the quantization error of each relative to the binary representation of the center.

[0104] The network training and music binary code generation module primarily selects a training music set and, based on the deep network and loss function, designs the network parameter training and binary code generation process. This module trains the network parameters and generates the binary representation of the music database through an iterative process of alternating optimization. Each iteration consists of two parts: first, the binary representation of the database music is fixed, and the network parameters are trained through backpropagation. Then, the network parameters are fixed and the database binary representation is optimized until the iteration termination condition is met.

[0105] The binary-representation-based massive music retrieval module primarily uses the trained hash network and the binary representation of the music database to rapidly retrieve massive amounts of music. Given a piece of music to be retrieved, this module uses the hash network to obtain its binary representation, then sorts the music in the database by Hamming distance, returning the closest piece or pieces of music.

[0106] In the deep network and loss function construction module, the central similarity quantization is introduced into asymmetric deep hashing for the first time, and a novel loss function is constructed, including the approximation error of the similarity matrix between the training music and the database music, and their respective quantization errors relative to the central binary representation.

[0107] The network training and music binary representation generation module provides an approximate solution to the optimization objective proposed in this invention. Because the loss function for the optimization objective includes both network parameters and the binary representation of the music library, optimizing both parameters simultaneously is not feasible. Therefore, this invention proposes an iterative method of alternating optimization, which has achieved excellent results and high efficiency in practice.

[0108] The above-described invention is an asymmetric deep hashing method for massive music retrieval. However, for those skilled in the art, without departing from the concept of the invention, other methods of binarizing category identifiers can be introduced into the music center binary code generation module to improve the representation ability of the music binary center; in the network training and music binary representation generation module, more novel and efficient loss functions can also be tried to obtain a more accurate music binary representation. In addition, for the present invention, several improvements and variations can be made for other application scenarios of music data, such as music recognition and generation, as well as other types of multimedia data representations.

[0109] Based on the above embodiments, an embodiment of the present application further provides a computer program, which, when executed on a computer, enables the computer to execute the method provided in the above embodiments.

[0110] Based on the above embodiments, an embodiment of the present application further provides a computer storage medium, in which a computer program is stored. When the computer program is executed by a computer, the computer executes the method provided in the above embodiments.

[0111] The storage medium may be any available medium that can be accessed by a computer. By way of example and not limitation, computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer.

[0112] Based on the above embodiments, an embodiment of the present application further provides a chip, which is used to read a computer program stored in a memory to implement the method provided in the above embodiments.

[0113] Based on the above embodiments, an embodiment of the present application provides a computer program product, which implements the methods provided in the above embodiments when the computer program product is run on an electronic device.

[0114] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0115] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0118] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. An asymmetric deep hashing method for massive music retrieval, characterized by: include Obtaining binary representations of the music data to be retrieved and binary representations of each music data in a music database, wherein the binary representations of the music data of the same category in the music database tend to be distributed around a quantized center of the music category, the quantized center being the binary representation of the music category in the music database; Calculating the Hamming distance between the binary representation of the music data to be retrieved and the binary representation of each music data in the music database; According to the Hamming distance, the category label of the music data in the music library having the closest Hamming distance to the binary representation of the music data to be retrieved is used as the category label of the music to be retrieved; The method for obtaining the quantitative center includes: For a given number of binary representations , Hadamard matrix of order and express A quantitative center, The quantization center is represented as a matrix ; If the number of music categories in the music database , take the matrix Before The rows serve as binary representations of music categories; Otherwise, regenerate via Bernoulli distribution A quantitative center, The quantization center is represented as a matrix ,matrix and matrix The combination obtains the binary representation of the music categories in the music database.

2. The asymmetric deep hashing method for massive music retrieval according to claim 1 is characterized in that: The average value of the quantized center Hamming distance between any two of the music categories is at least , Indicates the number of bits in the resulting binary representation.

3. The asymmetric deep hashing method for massive music retrieval according to claim 1 is characterized in that: Generate s matrices through Bernoulli distribution , select the matrix that maximizes the average value of the Hamming distance between the quantization centers of any two music categories As A quantitative center.

4. An asymmetric deep hashing system for massive music retrieval, characterized in that: An asymmetric deep hashing network for massive music retrieval is included for implementing the method of claim 1, wherein the asymmetric deep hashing network for massive music retrieval includes Convolutional networks, used to extract features from music data; A multilayer perceptron is used to generate binary representations of the features of the music data.

5. The asymmetric deep hashing system for massive music retrieval according to claim 4, wherein: The loss function of the asymmetric deep hashing network for massive music retrieval includes: The first constraint is used to constrain the network output to be consistent when the category labels of the music in the training music set and the music in the music database are the same, and to constrain the distance between the network output and the binary representation of the music in the music database to be the number of bits in the binary representation when the category labels of the music in the training music set and the music in the music database are different. ; A second constraint, wherein the second constraint is used to constrain the output of the network to be near the quantization center of the music category to which the input music belongs; The third constraint is used to constrain the binary representation of the music in the music database to be near the quantization center of the music category to which the music belongs.

6. The asymmetric deep hashing system for massive music retrieval according to claim 5, wherein: The loss function is expressed as in: , represents the hash loss, represents the output of the network, represents the spectrum representation of the i-th training music, represents the network parameters, represents the binary representation of the jth music in the music database, k represents the bit length of the binary representation, Represents the similarity between music i and j, the similarity matrix of the training music set and the music database represents the label of the i-th training music, Indicates the label of the j-th database music; in: represents the center quantization loss of training music, represents the quantized center of the i-th training music, represents the spectrum representation of the i-th training music; in: represents the central quantization loss of the database music, represents the binary representation of the i-th database music, represents the quantitative center of the music in the i-th database; is the first hyperparameter of the loss function, It is the second hyperparameter of the loss function, B represents the binary representation of the entire music database, and n represents the number of music in the database.

7. A method for constructing an asymmetric deep hashing system for massive music retrieval based on claim 4, characterized in that: The asymmetric deep hashing network for massive music retrieval obtains optimized network parameters through an iterative process of alternating optimization during training. and binary representation of the music database , in each iteration process, including: Binary representation of a fixed music database , optimize network parameters and fixed network parameters , optimizing the binary representation of music databases , until the iteration termination condition is met.

8. The method for constructing an asymmetric deep hashing system for massive music retrieval according to claim 7, wherein: Binary representation of the fixed music database , optimize network parameters The methods include: S1. First, in the music database Randomly select music training set , S2. Construct the similarity matrix between the training music set and the music database represents the label of the i-th training music, represents the label of the j-th database music, S3. Calculation Gradient: in: express The gradient, The output after passing through the network is represents the spectrum representation of the i-th training music, represents the network parameters, represents the binary representation of the jth music in the music database, k represents the bit length of the binary representation, represents the similarity between music i and j, represents the first hyperparameter of the loss function, represents the quantized center of the i-th training music, Represents the Hadamard product, that is, the multiplication of elements in corresponding positions; S4. Use back-propagation technology to train network parameters ,in , Represents the output of the network.

9. The method for constructing an asymmetric deep hashing system for massive music retrieval according to claim 8, wherein: The fixed network parameters , optimizing the binary representation of music databases The methods include: S1.Yes and Expand to get: in , A represents the network output of the entire training subset, B represents the binary representation of the entire music database, k represents the bit length of the binary representation, S represents the similarity matrix between the training subset and the music database, (.) represents the trace of the matrix, represents the second hyperparameter of the loss function, Represents a constant, Represents the binary representation of the nth database music S2. Pair Matrix Column-by-column iteration to obtain Solution: in , which represents the matrix No. Column calculation method, where Indicates that only the current matrix is taken List, Indicates removing from the current matrix All columns except columns, where: Represents a symbolic function.

Images