Training methods for audio search models, audio search methods, and related products.

Abstract

This application relates to a training method for an audio search model, an audio search method, an apparatus, a computer device, a storage medium, and a computer program product. By inputting the user's search operation data and feature vectors, the user's feature vector, and the audio feature vectors of multiple audio files into the decomposer module, tree model cross-validation module, and deep neural network module of the audio search model to be trained, a model search score is obtained. Based on the similarity between the search score and the model search score, the audio search model is trained. During actual searching, the user's search text is input into the user's corresponding audio search model, and multiple audio files are sorted based on the obtained search scores to obtain a sequence of audio search results. Compared to traditional audio search based on deep neural networks, this solution improves audio search accuracy by using an audio search model that includes a decomposer module, a deep neural network module, and a tree model cross-validation module to search for audio that matches the user's search intent.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a training method for an audio search model, an audio search method, an apparatus, a storage medium, and a computer program product. Background Technology

[0002] With the rapid development of digital and mobile music, users can now listen to music through various music streaming apps. When using these apps, users need to search for specific songs, and the apps need to analyze the user's input keywords and other search information to retrieve and rank the results. Currently, the ranking of audio search results is typically determined using deep neural networks. However, this method often fails to accurately reflect the user's search intent.

[0003] Therefore, current audio search methods suffer from poor search accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a training method, audio search method, device, computer equipment, computer-readable storage medium, and computer program product for an audio search model that can improve search accuracy, in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for training an audio search model, the method comprising:

[0006] Obtain multiple audio files and their respective search scores from the audio search results sequence obtained by the user's audio search; obtain the search operation data and post-search operation data corresponding to the user's audio search.

[0007] Based on the search operation data and the post-search operation data, the search operation feature vector and the post-search operation feature vector are obtained;

[0008] The post-search operation feature vector, the user feature vector, and the audio feature vectors of each of the multiple audios are input into the decomposer module of the audio search model to be trained. The search operation data is input into the tree model cross module of the audio search model to be trained. The search operation feature vector is input into the deep neural network module of the audio search model to be trained.

[0009] Based on the module search scores of the decomposer module, the tree model cross module, and the deep neural network module for the multiple audios, the model search score of the audio search model to be trained for the multiple audios is obtained.

[0010] Based on the similarity between the search score corresponding to each audio and the search score of the corresponding model, the model parameters of the audio search model to be trained are adjusted until the model training conditions are met to obtain the audio search model.

[0011] In one embodiment, obtaining the model search score of the audio search model to be trained for the plurality of audios based on the module search scores of the decomposition machine module, the tree model cross module, and the deep neural network module for the plurality of audios includes:

[0012] The decomposer module outputs the first module score for each audio based on the post-search operation feature vector, the audio feature vector corresponding to each audio, and the user's user feature vector.

[0013] Obtain the second module score corresponding to each audio element output by the deep neural network module based on the feature vector of the search operation;

[0014] Obtain the third module score corresponding to each audio item output by the tree model cross module based on the search operation data;

[0015] Based on the concatenation of the scores from the first module, the second module, and the third module, the model search scores of the audio search model to be trained for each of the multiple audio samples are obtained.

[0016] In one embodiment, the step of obtaining the first module score corresponding to each audio, output by the decomposer module based on the post-search operation feature vector, the audio feature vector corresponding to each audio, and the user's user feature vector, includes:

[0017] The user's audio preference information for each audio is determined by the decomposition module, the post-search operation feature vector, and the user feature vector.

[0018] The first module score corresponding to each audio is determined based on the audio preference information.

[0019] In one embodiment, obtaining the second module score corresponding to each audio element output by the deep neural network module based on the feature vector of the search operation includes:

[0020] The deep neural network module and the search operation feature vector are used to determine the user's operation type information for each audio and the operation frequency information for each operation type.

[0021] The second module score corresponding to each audio is determined based on the operation type information and the operation frequency information of each operation type information.

[0022] In one embodiment, obtaining the third module scores corresponding to each audio element output by the tree model cross module based on the search operation data includes:

[0023] The tree model cross module is calculated and learned based on the search operation data of multiple users to determine the leaf nodes in the tree model cross module;

[0024] Obtain the node score of each leaf node output by the tree model cross module based on the search operation data of multiple users;

[0025] The third module score, representing the users' preferences for each audio file, is obtained by summing the scores of multiple nodes.

[0026] In one embodiment, adjusting the model parameters of the audio search model to be trained based on the similarity between the search score corresponding to each audio and the corresponding model search score, until the model training conditions are met to obtain the audio search model, includes:

[0027] Obtain a first audio feature vector whose similarity to the model search score of the audio and the search score corresponding to the audio is greater than or equal to a preset similarity threshold;

[0028] Obtain a second audio feature vector whose similarity to the model search score of the audio is less than a preset similarity threshold;

[0029] Based on the first audio feature vector and the second audio feature vector, determine the function value of the hinge loss function in the audio search model to be trained;

[0030] If the function value is greater than the preset function threshold, adjust the model parameters of the audio search model to be trained, and return to the steps of inputting the post-search operation feature vector, the user's user feature vector and the audio feature vectors of the multiple audios into the decomposer module of the audio search model to be trained, inputting the search operation data into the tree model cross module of the audio search model to be trained, and inputting the search operation feature vector into the deep neural network module of the audio search model to be trained;

[0031] If the function value is greater than the preset function threshold, the current audio search model to be trained is taken as the trained audio search model.

[0032] In one embodiment, obtaining the search operation feature vector and the post-search operation feature vector based on the search operation data and the post-search operation data includes:

[0033] The search operation data is one-hot encoded and discretized to obtain the corresponding search operation feature vector;

[0034] The user's post-search operation data for multiple audio files is discretized and feature-mapped to obtain the post-search operation feature vector.

[0035] In one embodiment, the search operation data includes input operations when a user searches for audio and / or editing operations on the audio search result sequence, and the post-search operation data includes the user's singing behavior on the audio.

[0036] Secondly, this application provides an audio search method, the method comprising:

[0037] Receive the search text entered by the user;

[0038] The search text is input into the user's audio search model; the audio search model is trained based on the method described above.

[0039] Obtain multiple audio files and their respective search scores output by the audio search model;

[0040] The multiple audio files are sorted according to their respective search scores to obtain an audio search result sequence.

[0041] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0042] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0043] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0044] The aforementioned audio search model training method, audio search method, device, computer equipment, storage medium, and computer program product acquire multiple audio files and their respective search scores from the audio search result sequence obtained by the user's audio search. The user's post-search operation feature vector, the user's user feature vector, and the audio feature vectors of each audio file are input into the decomposer module of the audio search model to be trained. The user's search operation data is input into the tree model cross-referencing module of the audio search model to be trained. The search operation feature vector corresponding to the user's search operation data is input into the deep neural network module of the audio search model to be trained. Based on the module search scores of the decomposer module, tree model cross-referencing module, and deep neural network module for each audio file, the model search score of the audio search model to be trained for the multiple audio files is obtained. Then, based on the similarity between the search score corresponding to each audio file and the corresponding model search score, the model parameters of the audio search model to be trained are adjusted until the model training conditions are met, thus obtaining the audio search model. Therefore, in actual searching, the user's input search text can be input into the user's corresponding audio search model, and the multiple audio files and their respective search scores output by the audio search model can be obtained. The multiple audio files are then sorted according to their respective search scores to obtain the audio search result sequence. Compared to traditional audio search results determined solely by deep neural networks, this solution improves the accuracy of audio search by using an audio search model that includes a decomposer module, a deep neural network module, and a tree model cross-module to accurately search for audio that matches the user's search intent. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the training method of an audio search model in one embodiment;

[0046] Figure 2 This is a schematic diagram of the structure of an audio search model in one embodiment;

[0047] Figure 3 This is a flowchart illustrating an audio search method in one embodiment;

[0048] Figure 4 This is a structural block diagram of a training device for an audio search model in one embodiment;

[0049] Figure 5 This is a structural block diagram of an audio search device in one embodiment;

[0050] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. Detailed Implementation

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] In one embodiment, such as Figure 1 As shown, a training method for an audio search model is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server, including the following steps:

[0053] Step S202: Obtain multiple audio files and their respective search scores from the audio search result sequence obtained by the user's audio search, and obtain the search operation data and post-search operation data corresponding to the user's audio search.

[0054] User search can refer to a user's search behavior for audio on a terminal. For example, the terminal can be a mobile phone, and the audio can be a song. A mobile phone may have a karaoke app installed, allowing the user to search for songs, thus generating a user search behavior. When a user searches for audio, they receive a sequence of audio search results, which may contain multiple audio files and their respective search scores. The sorting of the multiple audio files in the audio search results sequence can be a sorting method that aligns with the user's actual intent; that is, the sorting method can serve as a reference sample, and the multiple audio files in the audio search results sequence can also be audio samples. The sorting method of the multiple audio files can be determined based on the search score of each audio file; for example, a higher search score results in a higher ranking. Users can generate corresponding operation data during the audio search process, and the terminal can acquire the search operation data generated by the user during the audio search and the post-search operation data. The search operation data can be a series of operations performed by the user on the search interface of the audio search software when using the terminal to search for audio. For example, in one embodiment, the search operation data may include the user's input operations on the search interface when searching for audio, such as the user entering text on the search interface. Furthermore, the terminal 102 can also obtain the user's input search query based on this input operation. The search operation data may also include the user's editing operations on the sequence of audio results found during the audio search. For example, after searching for audio, the user may obtain a sequence of audio search results containing multiple audio tracks. The user can click, remove, or favorite each audio track in the sequence, and the terminal 102 can record the user's search operation data. The post-search operation data may include the user's singing behavior on the audio. For example, if the searched audio is an instrumental track, the post-search operation data may be the user's act of recording and singing a specific instrumental track after finding multiple instrumental tracks. The terminal can record the user's recording behavior as post-search operation data.

[0055] Step S204: Based on the search operation data and the post-search operation data, obtain the search operation feature vector and the post-search operation feature vector.

[0056] The search operation data can be data generated by the user during the audio search process, while the post-search operation data can be the user's singing behavior towards the searched audio. After acquiring the search operation data and post-search operation data, the terminal can vectorize them respectively to obtain search operation feature vectors and post-search operation feature vectors. The aforementioned search operation data and post-search operation data can include multiple sparse features and multiple continuous features. Sparse features refer to features that are sparsely represented, while continuous features are features that are relatively dense compared to sparse features. For example, in the aforementioned search operation data, the user's editing operations such as clicking, saving, and removing in the audio search result sequence can be used as sparse features, the user's input operations in the search interface can be used as continuous features, and the user's singing behavior towards the audio can also be used as a continuous feature. The terminal 102 can process the sparse features accordingly to map them into dense vectors.

[0057] For example, in one embodiment, obtaining search operation feature vectors and post-search operation feature vectors based on search operation data and post-search operation data includes: performing one-hot encoding and discretization on the search operation data to obtain the corresponding search operation feature vectors; and discretizing and feature mapping on the post-search operation data of multiple audio files to obtain post-search operation feature vectors. In this embodiment, the terminal can perform one-hot encoding and discretization on the user-generated search operation data to obtain the search operation feature vectors corresponding to the search operation data. The terminal can also discretize and feature map the post-search operation data of multiple audio files to obtain post-search operation feature vectors, meaning that the post-search operation feature vectors can include multiple vectors of the aforementioned audio files used as samples. Specifically, the terminal 102 can perform one-hot encoding on multiple sparse features in the search operation data, and obtain dense feature embedding vectors through feature mapping. The terminal can also discretize the continuous features such as the post-search operation data and the user-input search statements, and then obtain dense feature embedding vectors of these continuous features through feature mapping, which serve as the post-search operation feature vectors and the search operation feature vectors corresponding to the user-input search statements. The terminal improves the robustness of the model by discretizing the continuous features mentioned above. That is, if there are n input sparse features and n continuous features, then there will also be n feature embedding vectors.

[0058] Step S206: Input the post-search operation feature vector, the user's user feature vector, and the audio feature vectors of each of the multiple audio files into the decomposer module of the audio search model to be trained; input the search operation data into the tree model cross module of the audio search model to be trained; and input the search operation feature vector into the deep neural network module of the audio search model to be trained.

[0059] The terminal can train an audio search model using the user's post-search operation feature vector, search operation data, search operation feature vector, user feature vector, and audio feature vectors of multiple audio samples. The user's feature vector can be a user profile containing relevant feature information, such as publicly available user identity information. The audio feature vectors can be the vectorized audio feature vectors of each of the multiple audio samples. Specifically, the multiple audio samples can be multiple accompaniments, and the terminal can vectorize each accompaniment to obtain its audio feature vector. The audio search model to be trained can be a DeepFM model with an added vector outer product feature layer, such as... Figure 2 As shown, Figure 2 This is a schematic diagram of an audio search model in one embodiment. The model includes a tree model crossover module, an FM (Factorization Machine) module, and a DNN (Deep Neural Network) module. The tree model crossover module can learn the audio preferences of multiple users to determine the degree of preference among most users for a given search query for each audio item. The FM module can capture low-order cross-features between features, primarily addressing the issue of feature combination in cases of sparse data. The DNN module, as shown... Figure 2 The Deep side of the model can include a multilayer perceptron, such as embedding layers, average pooling layers, and merging. The DNN module can be used to capture high-order cross features between features. The terminal can input the above vectors and data into different modules of the audio search model to be trained. For example, the terminal can input the post-search operation feature vector, the user's user feature vector, and the audio feature vectors of multiple audio files into the decomposition module of the audio search model to be trained, input the search operation data into the tree model cross module of the audio search model to be trained, and input the search operation feature vector into the deep neural network module of the audio search model to be trained. Specifically, the FM module can be used to determine the individual user's preference information for each audio file, the DNN module can be used to determine the operation type and operation frequency information of each audio file in the audio search result sequence for an individual user, and the tree model cross module can be used to learn the preference information of most users for each audio file.

[0060] Step S208: Based on the module search scores of the decomposer module, the tree model cross module, and the deep neural network module for multiple audios, obtain the model search score of the audio search model to be trained for multiple audios.

[0061] The terminal inputs the various feature vectors and data into different modules of the audio search model to be trained. Based on the scores output by each module, it determines the final search score for each audio track. For example, the decomposer module, tree model cross-referencing module, and deep neural network module in the audio search model can each output their respective module search scores learned from the features or data corresponding to each audio track. Thus, the terminal receives three module search scores. Based on these three module search scores, the terminal obtains the model search score for multiple audio tracks. Alternatively, the terminal can concatenate the module search scores to obtain the model search score for each audio track. The model search score characterizes the degree to which the audio track corresponds to the user's search intent.

[0062] Step S210: Adjust the model parameters of the audio search model to be trained based on the similarity between the search score corresponding to each audio and the corresponding model search score, until the model training conditions are met to obtain the audio search model.

[0063] Each audio file can serve as a sample for training an audio search model. Each audio file can have a corresponding known search score, which reflects the user's search intent. In other words, the audio search results sequence ordered based on this search score reflects the user's search intent. The model search score is the score obtained by the audio search model after learning from the features and data generated during the user's audio search. The terminal can adjust the model parameters of the audio search model based on the similarity between the search score and the corresponding model search score for each audio file until the training conditions are met, resulting in a well-trained audio search model. The terminal can perform pairwise learning on the audio search model using a set loss function. Pairwise learning is a ranking learning method that uses binary pairs of samples to learn a classifier to solve the ranking problem. This avoids the issue of different feature ranges under different search statements, allowing features of different accompaniments under the same search statement to be compared pairwise, improving the model's ability to predict ranking scenarios. Specifically, the loss function mentioned above can be hinge loss. Hinge loss, as a loss function, is usually used in the maximum margin algorithm, while hinge loss is specifically used for binary classification problems.

[0064] In the aforementioned audio search method, multiple audio files and their respective search scores are obtained from the audio search results sequence obtained by the user's audio search. The post-search operation feature vectors corresponding to the user's post-search operation data, the user's user feature vector, and the audio feature vectors of each audio file are input into the decomposer module of the audio search model to be trained. The user's search operation data is input into the tree model cross-referencing module of the audio search model to be trained. The search operation feature vectors corresponding to the user's search operation data are input into the deep neural network module of the audio search model to be trained. Based on the module search scores of the decomposer module, tree model cross-referencing module, and deep neural network module for each audio file, the model search score of the audio search model to be trained for the multiple audio files is obtained. Then, based on the similarity between the search score corresponding to each audio file and the corresponding model search score, the model parameters of the audio search model to be trained are adjusted until the model training conditions are met, thus obtaining the audio search model. Therefore, in actual searching, the user's input search text can be input into the user's corresponding audio search model, and the multiple audio files and their respective search scores output by the audio search model can be obtained. The multiple audio files are then sorted according to their respective search scores to obtain the audio search results sequence. Compared to traditional audio search results determined solely by deep neural networks, this solution improves the accuracy of audio search by using an audio search model that includes a decomposer module, a deep neural network module, and a tree model cross-module to accurately search for audio that matches the user's search intent.

[0065] In one embodiment, the model search score of the audio search model to be trained for multiple audios is obtained based on the module search scores of the decomposer module, the tree model cross module, and the deep neural network module for each audio. This includes: obtaining the first module score for each audio output by the decomposer module based on the post-search operation feature vector, the audio feature vector corresponding to each audio, and the user's user feature vector; obtaining the second module score for each audio output by the deep neural network module based on the search operation feature vector; obtaining the third module score for each audio output by the tree model cross module based on the search operation data; and obtaining the model search score of the audio search model to be trained for each audio based on the concatenation result of the first module score, the second module score, and the third module score.

[0066] In this embodiment, the terminal can input different feature vectors and data into different modules of the audio search model to be trained, thereby obtaining module search scores output by different modules. The audio search model to be trained may include a decomposer module, a tree model crossover module, and a deep neural network module. The terminal can obtain the first module score for each audio, output by the decomposer module based on the post-search operation feature vector, the audio feature vectors corresponding to each audio, and the user's user feature vector. The terminal can obtain the second module score for each audio, output by the deep neural network based on the search operation feature vector. The terminal can also obtain the third module score for each audio, output by the tree model crossover module based on the search operation data. The terminal can then concatenate the first, second, and third module scores, and based on the concatenation result, obtain the model search score of the audio search model to be trained for each audio. Each module in the audio search model to be trained can determine its module search score for each audio from different perspectives. Specifically, in the model input step, the terminal inputs the various vectors and data into the three parts of the audio search model to be trained, namely the inner product FM, DNN, and tree model cross module, as described above. These three modules share input features. The FM and DNN modules operate on the input feature embedding vectors in the same way as the DeepFM model. The inner product FM module can characterize the linear and second-order cross relationships between features, the DNN module can characterize the non-linear and higher-order implicit relationships between features, and the tree model cross module can determine the cross score through multiple leaf nodes. During the concatenation process, the terminal can concatenate the module search scores output by the FM, DNN, and tree model modules, and connect each to an output layer neuron to obtain the prediction results of the samples during training, which serve as the model search score.

[0067] Through this embodiment, the terminal can learn from various input vectors and feature data from different perspectives based on multiple modules in the audio search model to be trained, and obtain the model's search score for each audio based on the module search score output by each module. This ensures the accuracy and precision of the search while satisfying personalization, enabling popular online content and original backing tracks to be effectively ranked, preventing the ranking from being lost due to personal preferences, ensuring the authority of the search, and improving the accuracy of audio search.

[0068] In one embodiment, obtaining the first module score for each audio output by the decomposer module based on the post-search operation feature vector, the audio feature vector corresponding to each audio, and the user's user feature vector includes: determining the user's audio preference information for each audio through the decomposer module, the post-search operation feature vector, and the user feature vector; and determining the first module score for each audio based on the audio preference information.

[0069] In this embodiment, the terminal can obtain a first module score based on the decomposition machine module in the audio search model to be trained. The terminal can input the post-search operation feature vector and the user feature vector into the decomposition machine module. The terminal can then use this module to capture low-order cross-features, determine the user's audio preference information for each audio track, and subsequently determine the first module score for each audio track based on this preference information. For example, the post-search operation feature could be the user's singing behavior on the audio, and the user feature vector could be a vector representing the user profile. The terminal can use the decomposition machine module to analyze the cross-features between the user profile and the user's singing behavior on each audio track to determine the user's preference information for each audio track, thereby obtaining the first module score.

[0070] In this embodiment, the terminal can capture low-order cross features through the decomposition machine module to determine the first module score. The terminal can then determine the model search score for each audio based on the first module score, thereby improving the accuracy of the user's audio search.

[0071] In one embodiment, obtaining the second module score corresponding to each audio based on the search operation feature vector output by the deep neural network module includes: determining the user's operation type information and operation frequency information for each audio through the deep neural network module and the search operation feature vector; and determining the second module score corresponding to each audio based on the operation type information and operation frequency information for each operation type.

[0072] In this embodiment, the terminal can determine the second module score based on the deep neural network module in the audio search model to be trained. The terminal can input the search operation feature vector into the deep neural network module of the audio search model to be trained. The terminal can analyze the search operation feature vector based on the deep neural network module to determine the user's operation type information and the operation frequency information of each operation type for each audio. Therefore, the terminal can determine the second module score corresponding to each audio based on the operation type information and the operation frequency information of each operation type through the deep neural network module. The operation type information can be different types of operations performed by the user on each audio in the audio search result sequence, such as clicking, removing from the queue, and adding to favorites. Each operation has a certain operation frequency, such as the frequency of each operation within a specific time period. Therefore, the terminal can obtain the user's search operation feature vector based on the above operation information. The terminal can determine the user's operation type information and operation frequency information for each audio through the deep neural network module and the search operation feature vector, thereby capturing high-order cross features between these features in the deep neural network and determining the second module score corresponding to each audio.

[0073] Through this embodiment, the terminal can capture high-order cross features through a deep neural network module to determine the second module score. The terminal can then determine the model search score for each audio based on the second module score, thereby improving the accuracy of the user's audio search.

[0074] In one embodiment, obtaining the third module score corresponding to each audio output by the tree model cross module based on search operation data includes: performing computational learning on the tree model cross module based on search operation data from multiple users to determine the leaf nodes in the tree model cross module; obtaining the node score of each leaf node output by the tree model cross module based on search operation data from multiple users; and obtaining the third module score representing the preference information of multiple users for each audio based on the sum of the multiple node scores.

[0075] In this embodiment, the terminal can determine the third module score based on the tree model cross-module in the audio search model to be trained. The terminal can input search operation data from multiple users into the tree model cross-module, perform computational learning through the tree model cross-module, and determine each leaf node in the tree model cross-module. Each leaf node can output a corresponding node score. The terminal can then obtain the node score of each leaf node output by the tree model cross-module based on the search operation data from multiple users, and can obtain the third module score representing the preferences of multiple users for each audio based on the sum of multiple node scores. The aforementioned search operation data can be various behavioral features generated by users when performing audio searches, such as inputting characters in the search box, editing operations on various audio files in the audio search result sequence, etc. The terminal can analyze the search operation data from multiple users using the tree model cross-module to obtain various associations of low-order features. Specifically, the terminal can input user search behavior features, such as search operation data, into the tree model cross-module, and calculate the split nodes through a tree model algorithm. The terminal inputs the features into the tree model described above, sequentially traversing each node of each tree to obtain the values ​​of each leaf node. The terminal then sums the values ​​obtained from the leaf nodes of each tree to obtain the module cross-score, which is the third module score mentioned above. In other words, the terminal can determine the degree of preference for each audio file among most users based on the tree model's cross-module analysis.

[0076] Through this embodiment, the terminal can determine various relationships between low-order features through the tree model cross module, and determine the third module score that represents the degree of preference of multiple users for each audio. Thus, the terminal can determine the model search score of each audio based on the third module score, thereby improving the accuracy of users when searching for audio.

[0077] In one embodiment, the model parameters of the audio search model to be trained are adjusted based on the similarity between the search score corresponding to each audio and the corresponding model search score until the model training conditions are met, resulting in an audio search model. This includes: obtaining a first audio feature vector where the similarity between the model search score of the audio and the search score corresponding to the audio is greater than or equal to a preset similarity threshold; obtaining a second audio feature vector where the similarity between the model search score of the audio and the model search score corresponding to the audio is less than a preset similarity threshold; determining the function value of the hinge loss function in the audio search model to be trained based on the first and second audio feature vectors; if the function value is greater than a preset function threshold, adjusting the model parameters of the audio search model to be trained, and returning to the steps of inputting the post-search operation feature vector, the user's user feature vector, and the audio feature vectors of multiple audios into the decomposer module of the audio search model to be trained, inputting the search operation data into the tree model cross-multiplication module of the audio search model to be trained, and inputting the search operation feature vector into the deep neural network module of the audio search model to be trained; if the function value is greater than a preset function threshold, using the current audio search model to be trained as the trained audio search model.

[0078] In this embodiment, the terminal can train the audio search model to be trained using a set hinge loss function. The terminal can train the audio search model multiple times. After each training iteration, the terminal can obtain the model search score for each audio file output by the audio search model. The terminal can obtain the similarity between the model search score of each audio file and the search score of the corresponding reference sample for each audio file. The terminal can then use the audio feature vectors corresponding to audio files with similarity greater than or equal to a preset similarity threshold as the first audio feature vector, and the audio feature vectors corresponding to audio files with similarity less than the preset similarity threshold as the second audio feature vector. There can be multiple first and second audio feature vectors. The terminal can determine the hinge loss function value in the audio search model to be trained based on the first and second audio feature vectors. If the function value is greater than a preset threshold, the terminal can determine that the training is incomplete. The terminal can then adjust the model parameters of the audio search model to be trained and return to the steps of inputting the post-search operation feature vector, the user's user feature vector, and the audio feature vectors of multiple audio files into the decomposer module of the audio search model to be trained, inputting the search operation data into the tree model cross-validation module of the audio search model to be trained, and inputting the search operation feature vector into the deep neural network module of the audio search model to be trained, thereby performing the next training. If the function value is greater than the preset threshold, the terminal can determine that the model accuracy is sufficient, and the terminal can use the current audio search model to be trained as the trained audio search model.

[0079] In this model, the first audio feature vector represents the model output that matches the user's search intent (positive sample), and the second audio feature vector represents the model output that does not match the user's search intent (negative sample). The terminal can determine whether the model has been trained successfully using pairwise learning and a hinge loss function. The audio feature vector can be the vector corresponding to each audio track. The pairwise algorithm does not focus on accurately predicting the relevance between each audio track; it primarily concerns the order of two audio tracks, making it closer to the concept of ranking compared to pointwise algorithms. In pairwise learning, ranking algorithms are typically transformed into classifying audio pairs. The classification result determines which audio track is more relevant, and the learning objective is to reduce misclassified audio pairs. In a perfect model, all audio pairs are correctly classified. That is, the terminal uses the audio search model to output results for both positive and negative samples, and then uses a hinge loss function to make the model as separable as possible between positive and negative samples. In the actual prediction stage, the terminal can directly output the final score for the current sample based on the output layer of the audio search model, thus predicting the model search score for each audio track.

[0080] Through this embodiment, the terminal can learn pairwise to avoid the problem of differences in the range of feature calculation under different search statements, improve the model's ability to solve actual prediction and ranking scenarios, and enable the features of different accompaniments under the same search statement to be compared pairwise, thereby improving the model's ability to solve actual prediction and ranking scenarios and improving the search accuracy of the audio search model.

[0081] In one embodiment, an application example is provided. In this embodiment, the terminal adds a vector outer product feature layer based on the DeepFM model. The model input consists of multiple sparse features that are one-hot encoded, and after feature mapping, a dense feature embedding vector is obtained. Then, the dense features are concatenated to obtain the embedding vector layer. The audio search model inputs each vector from the embedding vector layer into the FM, DNN, and outer product feature modules, respectively. The FM and DNN processes the features in the same way as DeepFM. The tree model cross module of the outer product feature module performs computation and learning on the original continuous and offline feature input modules to obtain the various split nodes of the tree model. The terminal can obtain the final third module score of the tree model by substituting the features into the trained tree model and adding the scores of its leaf nodes. The terminal can concatenate the module scores of the three modules, capturing low-order cross features through the FM part, capturing high-order cross features through the DNN part, and utilizing the various correlations of low-order features captured by the tree model cross module to obtain a combined model search score. Furthermore, the audio search model uses hinge loss as the loss function for pairwise learning, avoiding the problem of different feature ranges under different search statements. This allows features of different accompaniments under the same statement to be compared pairwise, improving the model's ability to solve real-world prediction and ranking scenarios.

[0082] Specifically, the aforementioned audio can be a backing track, the user's post-search action data can be the user's recording and singing behavior on the backing track, the user features can be a user profile, and the search action data can be the user's action behavior features during the search. During training, the terminal can convert features such as the user's recording and singing behavior flow, user profile, and backing track profile into embedding vectors: {BehaviorEmb1,BehaviorEmb2,…,BehaviorEmb3}, {UserProfileEmb1,UserProfileEmb2,…,UserProfileEmb3}…{DocEmb1,DocEmb2,…,DocEmb3}. Here, BehaviorEmb represents the user's recording and singing behavior vector, UserProfileEmb represents the user profile vector, and DocEmb represents the backing track profile vector. The backing track profile can include various attribute information of the backing track, such as composer, lyricist, and singer information.

[0083] The terminal can also extract user operation behavior features during searches, obtaining {fea1, fea2,…, fea3}, where fea represents various behavioral data generated by the user when searching for audio. The terminal can perform one-hot encoding and discretization of the above operation behaviors and map them into embedded vectors. The conversion formula is as follows: {fea1, fea2,…, fea3}=>{BEmb1,BEmb2,…,BEmb3}, where BEmb represents the vector corresponding to the user's operation behavior features. The terminal can input the vectors corresponding to the user's recording behavior flow, user profile, and accompaniment profile into the decomposition machine module to obtain the output fm_logit=FM(BehaviorEmb,UserProfileEmb,…,DocEmb), which is the score of the first module. The terminal can input the {BEmb1,BEmb2,…,BEmb3} into the deep neural network module to obtain the output deep_logit=DeepModel(BEmb), which is the score of the second module. The terminal can also input the original operation behavior features into the tree model cross-cutting module to obtain the output xgb_logit = TreeModel(fea), which is the score of the third module. The terminal can fuse the outputs of the three to obtain the final model search score: Output=CalModel(deep_logit,fm_logit,xgb_logit).

[0084] Through the above embodiments, the terminal can accurately search for audio that matches the user's search intent using an audio search model that includes a decomposition machine module, a deep neural network module, and a tree model cross module. This ensures that the search results are personalized while also taking into account user preferences and search accuracy, allowing users to quickly find original versions and popular online backing tracks. This improves the accuracy of audio search and, by combining the scores of the three modules, determines which part influences the ranking results, thus improving interpretability.

[0085] In one embodiment, such as Figure 3 As shown, an audio search method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server, including the following steps:

[0086] Step S302: Receive the search text entered by the user.

[0087] The search text can be the text entered by the user when performing an audio search on the terminal. For example, if a user searches for "song 1" in a music search software, then "song 1" can be used as the search text. The terminal can receive the search text entered by the user.

[0088] Step S304: Input the search text into the user's audio search model; the audio search model is trained based on the above-described audio search model training method.

[0089] After receiving the search text, the terminal can input the audio search model corresponding to the user. This audio search model is trained using the aforementioned training method. Specifically, the audio search model can be a user-specific model used to determine each user's audio preferences, thereby accurately searching for audio that matches the user's search intent. Specifically, the terminal can obtain the user-specific audio search model, determine search scores based on the search text and various data generated during the user's search, and output the search scores for each audio track. For example, if a user searches for song 1 and their user profile indicates a liking for singer 2, the terminal can increase the search scores for all live audio recordings of song 1 sung by singer 2, ranking them higher. However, the user may not be able to find the original singer of song 1 and might want to listen to the original version for comparison, but their personalized data shows a weak preference for singer 1. In this case, the tree model cross-referencing module in the aforementioned audio search model can increase the search scores for the original and authoritative versions based on the behavior of most users, ranking them higher and ensuring the authority and accuracy of the search results.

[0090] Step S306: Obtain multiple audio files and their respective search scores output by the audio search model.

[0091] The audio search model outputs search scores for each audio file. The terminal can obtain multiple audio files and their respective search scores from the audio search model, thus generating multiple search scores. Each search score can be obtained by concatenating search scores from multiple modules.

[0092] Step S308: Sort the multiple audio files according to their respective search scores to obtain an audio search result sequence.

[0093] The terminal can sort multiple audio files based on their respective search ratings to obtain a sequence of audio search results. For example, the terminal can sort multiple audio files from highest to lowest search rating and display the sorted audio search results sequence on the terminal's search results page, allowing users to view each search result.

[0094] In the aforementioned audio search method, multiple audio files and their respective search scores are obtained from the audio search results sequence obtained by the user's audio search. Based on the user's post-search operation data, the corresponding post-search operation feature vectors, the user's user feature vector, and the audio feature vectors of the multiple audio files, an audio search model is trained, incorporating a decomposer module, a tree model cross-validation module, and a deep neural network module. Thus, during actual searching, the user's search text can be input into the corresponding audio search model, and the multiple audio files and their respective search scores output by the model can be obtained. The multiple audio files are then sorted according to their search scores to obtain the audio search results sequence. Compared to traditional audio search results determined solely by deep neural networks, this solution, through an audio search model incorporating a decomposer module, a deep neural network module, and a tree model cross-validation module, accurately searches for audio files that match the user's search intent, improving the accuracy of audio search.

[0095] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0096] Based on the same inventive concept, this application also provides a training device and an audio search device for implementing the audio search method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more audio search device embodiments provided below can be found in the limitations of the audio search method described above, and will not be repeated here.

[0097] In one embodiment, such as Figure 4 As shown, a training device for an audio search model is provided, including: an acquisition module 500, a vectorization module 502, an input module 504, a concatenation module 506, and an adjustment module 504, wherein:

[0098] The acquisition module 500 is used to acquire multiple audio files and their respective search scores from the audio search results sequence obtained by the user's audio search, and to acquire the search operation data and post-search operation data corresponding to the user's audio search.

[0099] The vectorization module 502 is used to obtain the search operation feature vector and the post-search operation feature vector based on the search operation data and the post-search operation data.

[0100] The input module 504 is used to input the post-search operation feature vector, the user's user feature vector, and the audio feature vectors of multiple audios into the decomposer module of the audio search model to be trained, input the search operation data into the tree model cross module of the audio search model to be trained, and input the search operation feature vector into the deep neural network module of the audio search model to be trained.

[0101] The splicing module 506 is used to obtain the model search score of the audio search model to be trained for multiple audios based on the module search scores of the decomposition machine module, the tree model cross module and the deep neural network module for multiple audios.

[0102] The adjustment module 508 is used to adjust the model parameters of the audio search model to be trained based on the similarity between the search score corresponding to each audio and the search score of the corresponding model, until the model training conditions are met and the audio search model is obtained.

[0103] In one embodiment, the splicing module 506 is specifically used to obtain the first module score for each audio output by the decomposer module based on the post-search operation feature vector, the audio feature vector corresponding to each audio, and the user's user feature vector; obtain the second module score for each audio output by the deep neural network module based on the search operation feature vector; obtain the third module score for each audio output by the tree model cross module based on the search operation data; and obtain the model search score for each audio by the audio search model to be trained based on the splicing result of the first module score, the second module score, and the third module score.

[0104] In one embodiment, the splicing module 506 is specifically used to determine the user's audio preference information for each audio by using the decomposer module, the post-search operation feature vector, and the user feature vector; and to determine the first module score corresponding to each audio based on the audio preference information.

[0105] In one embodiment, the splicing module 506 is specifically used to determine the user's operation type information and operation frequency information of each audio through a deep neural network module and a search operation feature vector; and to determine the second module score corresponding to each audio based on the operation type information and operation frequency information of each operation.

[0106] In one embodiment, the splicing module 506 is specifically used to perform computational learning on the tree model cross module based on the search operation data of multiple users, determine the leaf nodes in the tree model cross module; obtain the node score of each leaf node output by the tree model cross module based on the search operation data of multiple users; and obtain the third module score representing the preference information of multiple users for each audio based on the sum of the multiple node scores.

[0107] In one embodiment, the adjustment module 508 is specifically used to obtain a first audio feature vector whose similarity to the model search score of the audio is greater than or equal to a preset similarity threshold; obtain a second audio feature vector whose similarity to the model search score of the audio is less than a preset similarity threshold; determine the function value of the hinge loss function in the audio search model to be trained based on the first and second audio feature vectors; if the function value is greater than the preset function threshold, adjust the model parameters of the audio search model to be trained, and return to the steps of inputting the post-search operation feature vector, the user's user feature vector, and the audio feature vectors of multiple audios into the decomposer module in the audio search model to be trained, inputting the search operation data into the tree model cross module in the audio search model to be trained, and inputting the search operation feature vector into the deep neural network module in the audio search model to be trained; if the function value is greater than the preset function threshold, use the current audio search model to be trained as the trained audio search model.

[0108] In one embodiment, the vectorization module 502 is specifically used to perform one-hot encoding and discretization on the search operation data to obtain the corresponding search operation feature vector; and to discretize and feature map the user's post-search operation data for multiple audio files to obtain the post-search operation feature vector.

[0109] In one embodiment, such as Figure 5 As shown, an audio search device is provided, including: a receiving module 600, a search module 602, an output module 604, and a sorting module 606, wherein:

[0110] The receiving module 600 is used to receive the search text input by the user.

[0111] The search module 602 is used to input the search text into the user's audio search model; the audio search model is trained based on the above-described audio search model training method.

[0112] Output module 604 is used to obtain multiple audio files and their respective search scores output by the audio search model.

[0113] The sorting module 606 is used to sort multiple audio files according to their respective search scores to obtain a sequence of audio search results.

[0114] Each module in the aforementioned audio search device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0115] In one embodiment, a computer device is provided, which may be a terminal or related product, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a training method for an audio search model and an audio search method. The display screen can be an LCD screen or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0116] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0117] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described audio search model training method and audio search method.

[0118] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described audio search model training method and audio search method.

[0119] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described audio search model training method and audio search method.

[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0121] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A training method for an audio search model, characterized in that, The method includes: Obtain multiple audio files and their respective search scores from the audio search results sequence obtained by the user's audio search; obtain the search operation data and post-search operation data corresponding to the user's audio search. Based on the search operation data and the post-search operation data, the search operation feature vector and the post-search operation feature vector are obtained; The post-search operation feature vector, the user feature vector, and the audio feature vectors of each of the multiple audios are input into the decomposer module of the audio search model to be trained. The search operation data is input into the tree model cross module of the audio search model to be trained. The search operation feature vector is input into the deep neural network module of the audio search model to be trained. Based on the concatenation of the first module score, second module score, and third module score corresponding to each audio, the audio search model obtains the model search score for each of the multiple audios; the first module score is obtained by the decomposer module based on the post-search operation feature vector, the audio feature vector corresponding to each audio, and the user feature vector; the second module score is obtained by the deep neural network module based on the search operation feature vector; and the third module score is obtained by the tree model cross-validation module based on the search operation data. Based on the similarity between the search score corresponding to each audio and the search score of the corresponding model, the model parameters of the audio search model to be trained are adjusted until the model training conditions are met to obtain the audio search model.

2. The method according to claim 1, characterized in that, The steps for obtaining the score of the first module include: The user's audio preference information for each audio is determined by the decomposition module, the post-search operation feature vector, and the user feature vector. The first module score corresponding to each audio is determined based on the audio preference information.

3. The method according to claim 1, characterized in that, The steps for obtaining the score for the second module include: The deep neural network module and the search operation feature vector are used to determine the user's operation type information for each audio and the operation frequency information for each operation type. The second module score corresponding to each audio is determined based on the operation type information and the operation frequency information of each operation type information.

4. The method according to claim 1, characterized in that, The steps for obtaining the score of the third module include: The tree model cross module is calculated and learned based on the search operation data of multiple users to determine the leaf nodes in the tree model cross module; Obtain the node score of each leaf node output by the tree model cross module based on the search operation data of multiple users; The third module score, representing the users' preferences for each audio file, is obtained by summing the scores of multiple nodes.

5. The method according to claim 1, characterized in that, The step of adjusting the model parameters of the audio search model to be trained based on the similarity between the search score corresponding to each audio and the search score of the corresponding model, until the model training conditions are met, to obtain the audio search model, includes: Obtain a first audio feature vector whose similarity to the model search score of the audio and the search score corresponding to the audio is greater than or equal to a preset similarity threshold; Obtain a second audio feature vector whose similarity to the model search score of the audio is less than a preset similarity threshold; Based on the first audio feature vector and the second audio feature vector, determine the function value of the hinge loss function in the audio search model to be trained; If the function value is greater than the preset function threshold, adjust the model parameters of the audio search model to be trained, and return to the steps of inputting the post-search operation feature vector, the user's user feature vector and the audio feature vectors of the multiple audios into the decomposer module of the audio search model to be trained, inputting the search operation data into the tree model cross module of the audio search model to be trained, and inputting the search operation feature vector into the deep neural network module of the audio search model to be trained; If the function value is greater than the preset function threshold, the current audio search model to be trained is taken as the trained audio search model.

6. The method according to claim 1, characterized in that, The step of obtaining the search operation feature vector and the post-search operation feature vector based on the search operation data and the post-search operation data includes: The search operation data is one-hot encoded and discretized to obtain the corresponding search operation feature vector; The user's post-search operation data for multiple audio files is discretized and feature-mapped to obtain the post-search operation feature vector.

7. The method according to any one of claims 1 to 6, characterized in that, The search operation data includes the user's input operations when searching for audio and / or editing operations on the audio search result sequence, and the post-search operation data includes the user's singing behavior on the audio.

8. An audio search method, characterized in that, The method includes: Receive the search text entered by the user; The search text is input into the user's audio search model; the audio search model is trained based on the method described in any one of claims 1 to 7. Obtain multiple audio files and their respective search scores output by the audio search model; The multiple audio files are sorted according to their respective search scores to obtain an audio search result sequence.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images