Method, device, equipment and storage medium for retrieving audio

Abstract

The present application discloses a method, device, equipment and storage medium for retrieving audio, belonging to the field of machine learning technology. The method comprises: obtaining a first audio; determining the category of the first audio based on a pre-trained classification model, wherein the category comprises a first category or a second category, the first category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio meets a preset condition, and the second category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio does not meet the preset condition; if the category of the first audio is the first category, then retrieving a second audio that matches the first audio; if the category of the first audio is the second category, then controlling the output of a retrieval failure prompt message. The present application can save the process of retrieving audio with high noise, as well as the process of users playing and listening to the song audios in the retrieval results one by one, thereby saving the time of the above process.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a method, apparatus, device and storage medium for retrieving audio. Background Art

[0002] With the development of information technology and network technology, the size of music libraries has grown exponentially, and the rapid retrieval of music songs has become a new research hotspot. Traditional music retrieval methods are mainly based on text keyword retrieval, and relevant song audio is retrieved based on text information such as song names, singers, genres, lyrics, etc. entered by users. In order to facilitate user retrieval, a new music retrieval method has emerged. This method can retrieve the audio hummed by the user, and then retrieve the corresponding song audio in the music library when the user cannot provide the exact song name, singer, etc. text information.

[0003] In the related art, a user starts a music application and displays the main interface of the music application on a terminal. When the user clicks a humming search button on the main interface, the terminal receives an instruction to record humming audio and starts recording humming audio. When the terminal finishes recording the humming audio, the terminal sends the recorded humming audio to a server. The server receives the recorded humming audio, performs a search based on the recorded humming audio, obtains song audio that matches the humming audio, generally song audio whose similarity exceeds a certain threshold, and sends these song audio to the terminal. After obtaining these song audios, the terminal displays these song audios on the terminal. The user can find song audios that meet the user's needs by playing these song audios on the terminal.

[0004] In the process of implementing this application, the inventors found that the related technology has at least the following problems:

[0005] When a certain audio contains a lot of noise, the search results obtained based on the audio may not contain the song audio that meets the user's needs, but the user will play and listen to the multiple song audios in the search results one by one, so as to find the song audio that meets the user's needs in the search results. When the user does not find the song audio that meets the user's needs in the search results, the user will re-record the humming audio and search based on the re-recorded humming audio. Therefore, the process of searching for the audio with a lot of noise, and the process of the user playing the song audios in the search results one by one but not getting the song audio that the user wants to find, will greatly waste the user's time. Summary of the invention

[0006] The embodiments of the present application provide a method, device, equipment and storage medium for retrieving audio, which can solve the problem of wasting the user's time in the process of retrieving audio with high noise and the process of the user playing and listening to the search results obtained based on the audio with high noise one by one. The technical solution is as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for retrieving audio, the method comprising:

[0008] Get the first audio;

[0009] Determine a category of the first audio based on a pre-trained classification model, wherein the category includes a first category or a second category, the first category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio meets a preset condition, and the second category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio does not meet the preset condition;

[0010] If the category of the first audio is the first category, retrieving the second audio matching the first audio;

[0011] If the category of the first audio is the second category, the control output indicates that the retrieval failed.

[0012] Optionally, determining the category of the first audio based on a pre-trained classification model includes:

[0013] Filter out a target audio segment whose volume value in the first audio is greater than a preset volume value;

[0014] Obtain the pitch line of each target audio segment;

[0015] The pitch line of each target audio segment is input into a pre-trained classification model, and the category of the first audio is output.

[0016] Optionally, the pre-trained classification model includes a conversion module, a self-attention learning module, a sub-classification module and a processing module; the step of inputting the pitch line of each target audio segment into the pre-trained classification model and outputting the category of the first audio includes:

[0017] Input the pitch line of each target audio segment into the conversion module in the pre-trained classification model to obtain the pitch sequence of each target audio segment after dimension alignment;

[0018] Input the pitch sequence of each target audio segment after dimension alignment into the self-attention learning module in the pre-trained classification model to obtain the pitch sequence of each target audio segment after weight adjustment;

[0019] Input the pitch sequence of each target audio segment after weight adjustment into the sub-classification module in the pre-trained classification model to obtain the category of each target audio segment;

[0020] The category of each target audio segment is input into a processing module in the pre-trained classification model to obtain the category of the first audio.

[0021] Optionally, the step of inputting the category of each target audio segment into a processing module in the pre-trained classification model to obtain the category of the first audio includes:

[0022] The category of each target audio segment is input into the processing module in the pre-trained classification model, so that the processing module determines the total duration of each category based on the category of each target audio segment and the duration of each target audio segment and determines the category with the longest total duration as the category of the first audio.

[0023] Optionally, the retrieving the second audio that matches the first audio includes:

[0024] Based on the pitch line of each target audio segment, a second audio matching the first audio is retrieved.

[0025] Optionally, retrieving a second audio that matches the first audio based on the pitch line of each target audio segment includes:

[0026] For each pitch line of the target audio segment, determine, from pre-stored reference pitch lines of various audios, a plurality of reference pitch lines that meet a matching condition with the pitch line of the target audio segment, and obtain at least one reference pitch line of the target audio segment;

[0027] Among the reference pitch lines of all target audio segments, determine the target reference pitch line that appears most frequently;

[0028] The audio of the target reference pitch line is determined as the second audio that matches the first audio.

[0029] Optionally, retrieving a second audio that matches the first audio based on the pitch line of each target audio segment includes:

[0030] Setting the pitch lines of the other audio segments of the first audio segment except the target audio segment to 0;

[0031] Based on the pitch lines of each target audio segment and the other audio segments, a second audio matching the first audio is retrieved.

[0032] Optionally, determining the category of the first audio based on a pre-trained classification model includes:

[0033] Filter out a target audio segment whose volume value in the first audio is greater than a preset volume value;

[0034] The target audio segment is input into a pre-trained classification model, and the category of the first audio is output.

[0035] Optionally, the pre-trained classification model includes a feature extraction module, a self-attention learning module, a sub-classification module and a processing module; inputting the target audio segment into the pre-trained classification model and outputting the category of the first audio includes:

[0036] Inputting the target audio segment into the feature extraction module in the pre-trained classification model to obtain feature information corresponding to the target audio segment;

[0037] Inputting the feature information into the self-attention learning module in the pre-trained classification model, and outputting the feature information after weight adjustment;

[0038] Input each of the weight-adjusted feature information into a sub-classification module in the pre-trained classification model, and output the category of each target audio segment;

[0039] The category of each target audio segment is input into a processing module in the pre-trained classification model to obtain the category of the first audio.

[0040] In a second aspect, an embodiment of the present application provides a device for retrieving audio, the device comprising:

[0041] An acquisition module, configured to acquire a first audio;

[0042] a determination module configured to determine a category of the first audio based on a pre-trained classification model, wherein the category includes a first category or a second category, the first category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio meets a preset condition, and the second category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio does not meet the preset condition;

[0043] a retrieval module configured to retrieve a second audio matching the first audio if the category of the first audio is the first category;

[0044] The prompt module is configured to control the output of retrieval failure prompt information if the category of the first audio is the second category.

[0045] Optionally, the determining module is configured to:

[0046] Filter out a target audio segment whose volume value in the first audio is greater than a preset volume value;

[0047] Obtain the pitch line of each target audio segment;

[0048] The pitch line of each target audio segment is input into a pre-trained classification model, and the category of the first audio is output.

[0049] Optionally, the pre-trained classification model includes a conversion module, a self-attention learning module, a sub-classification module and a processing module; the determination module is configured to:

[0050] Input the pitch line of each target audio segment into the conversion module in the pre-trained classification model to obtain the pitch sequence of each target audio segment after dimension alignment;

[0051] Input the pitch sequence of each target audio segment after dimension alignment into the self-attention learning module in the pre-trained classification model to obtain the pitch sequence of each target audio segment after weight adjustment;

[0052] Input the pitch sequence of each target audio segment after weight adjustment into the sub-classification module in the pre-trained classification model to obtain the category of each target audio segment;

[0053] The category of each target audio segment is input into a processing module in the pre-trained classification model to obtain the category of the first audio.

[0054] Optionally, the determining module is configured to:

[0055] The category of each target audio segment is input into the processing module in the pre-trained classification model, so that the processing module determines the total duration of each category based on the category of each target audio segment and the duration of each target audio segment and determines the category with the longest total duration as the category of the first audio.

[0056] Optionally, the retrieval module is configured to:

[0057] Based on the pitch line of each target audio segment, a second audio matching the first audio is retrieved.

[0058] Optionally, the retrieval module is configured to:

[0059] For each pitch line of the target audio segment, determine, from pre-stored reference pitch lines of various audios, a plurality of reference pitch lines that meet a matching condition with the pitch line of the target audio segment, and obtain at least one reference pitch line of the target audio segment;

[0060] Among the reference pitch lines of all target audio segments, determine the target reference pitch line that appears most frequently;

[0061] The audio of the target reference pitch line is determined as the second audio that matches the first audio.

[0062] Optionally, the retrieval module is configured to:

[0063] Setting the pitch lines of the other audio segments of the first audio segment except the target audio segment to 0;

[0064] Based on the pitch lines of each target audio segment and the other audio segments, a second audio matching the first audio is retrieved.

[0065] Optionally, the determining module is configured to:

[0066] Filter out a target audio segment whose volume value in the first audio is greater than a preset volume value;

[0067] The target audio segment is input into a pre-trained classification model, and the category of the first audio is output.

[0068] Optionally, the pre-trained classification model includes a feature extraction module, a self-attention learning module, a sub-classification module and a processing module; the determination module is configured to:

[0069] Inputting the target audio segment into the feature extraction module in the pre-trained classification model to obtain feature information corresponding to the target audio segment;

[0070] Inputting the feature information into the self-attention learning module in the pre-trained classification model, and outputting the feature information after weight adjustment;

[0071] Input each of the weight-adjusted feature information into a sub-classification module in the pre-trained classification model, and output the category of each target audio segment;

[0072] The category of each target audio segment is input into a processing module in the pre-trained classification model to obtain the category of the first audio.

[0073] In a third aspect, an embodiment of the present application provides a computer device, comprising a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the operations performed by the method for retrieving audio as described in the first aspect.

[0074] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the instruction is loaded and executed by a processor to implement the operations performed by the method for retrieving audio as described in the first aspect.

[0075] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0076] The method provided in the embodiment of the present application detects the category of the first audio through a pre-trained classification model; when the category of the first audio is the first category, it is determined that the noise contained in the first audio is small, and a search can be performed based on the first audio, and a second audio matching the first audio is retrieved. When the category of the first audio is the second category, it is determined that the noise contained in the second audio is large, and the control output prompt message of the retrieval failure is output. In the present application, when the category of the first audio is the first category, the accuracy of the retrieval results obtained based on the first audio is high, and the user can quickly find the song audio that meets his needs in the retrieval results; when the category of the first audio is the second category, because the accuracy of its retrieval is not high, the first audio is not retrieved, thereby omitting the process of retrieving the audio with large noise, and the process in which the user plays and listens to the retrieval results obtained based on the audio with large noise one by one but cannot get the song audio he wants to find, thereby saving time in the above process. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0078] Figure 1 It is a schematic diagram of an implementation environment for retrieving audio provided in an embodiment of the present application;

[0079] Figure 2 is a flow chart of a method for retrieving audio provided in an embodiment of the present application;

[0080] Figure 3 is a schematic diagram of audio retrieval provided in an embodiment of the present application;

[0081] Figure 4 is a schematic diagram of audio retrieval provided in an embodiment of the present application;

[0082] Figure 5 is a schematic diagram of audio retrieval provided in an embodiment of the present application;

[0083] Figure 6 is a schematic diagram of the structure of an audio retrieval device provided in an embodiment of the present application;

[0084] Figure 7 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0085] Figure 8 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0087] Figure 1 is a schematic diagram of an implementation environment for retrieving audio provided by an embodiment of the present application, such as Figure 1 As shown, the implementation environment may include: a server 101 and a terminal 102 .

[0088] Server 101 can be a single server or a server cluster composed of multiple servers. Server 101 can be at least one of a cloud computing platform and a virtualization center, which is not limited in the embodiments of the present application. Server 101 can be used to receive a search request sent by a terminal, which carries a first audio, and can also be used to retrieve a second audio that matches the first audio. Of course, the server 101 can also include other functional servers to provide more comprehensive and diversified services.

[0089] Terminal 102 may be at least one of a smart phone, a game console, a desktop computer, a tablet computer, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, and a laptop computer. Terminal 102 is connected to server 101 via a wired network or a wireless network, and terminal 102 installs and runs an application that can perform retrieval based on recorded audio. Terminal 102 may be used to record the first audio after receiving a recording instruction of the first audio triggered by a user; may also be used to determine the category of the first audio; may also be used to send a retrieval request to the server when it is detected that the category of the first audio is the first category; may also be used to control the output of a retrieval failure prompt message when it is detected that the category of the first audio is the second category.

[0090] The method provided in the embodiment of the present application can be applied to music applications, and can also be applied to other applications for retrieving audio. Taking the music application as an example, the user opens the music application on the terminal, and the main interface of the music application is displayed on the terminal. When the user clicks the button for retrieving humming audio on the main interface of the music application, the terminal receives a recording instruction and starts recording the user's humming audio. When the terminal finishes recording the user's humming audio, the classification model pre-set on the terminal identifies the humming audio and determines the category of the humming audio. When the category of the humming audio is the first category, the terminal sends a retrieval request carrying the humming audio to the server. When the server receives the retrieval request sent by the terminal, the server obtains the humming audio in the retrieval request, retrieves the song audio matching the humming audio, and sends the song identifier of the retrieved song audio as a retrieval result to the terminal, wherein the retrieved song audio is generally multiple. When the terminal receives the retrieval result sent by the server, the multiple song identifiers in the retrieval result are displayed on the interface of the terminal. When the category of the humming audio is the second category, the terminal can display a prompt message of retrieval failure on the display interface of the terminal to remind the user to re-record the humming audio. In the present application, when the category of the humming audio is the first category, the accuracy of the search results obtained based on the humming audio is high, and the user can quickly find the song audio that meets his needs in the search results. When the category of the first audio is the second category, since the search accuracy is not high, the first audio is not searched, and the process of searching for audio with large noise and the process of the user playing the song audio one by one in the search results are omitted, thereby saving the user's time.

[0091] The present application embodiment provides a method for retrieving audio. Figure 2 The flowchart of a method for retrieving audio provided in the embodiment of the present application is used as an example for explanation. Figure 2 As shown, the method comprises the following steps:

[0092] Step 201: Acquire first audio.

[0093] The first audio is the audio hummed by the user.

[0094] In implementation, after the user opens the music application of the terminal, the terminal displays the main interface of the music application. When the user clicks the audio search button on the main interface, the terminal receives the audio search instruction and starts recording the audio of the user humming. When the terminal finishes recording the audio of the user humming, the first audio is obtained.

[0095] Alternatively, the user starts a music application, and the terminal displays the main interface of the music application. The user can click on the search entry on the main interface, and the audio search interface is displayed. On the audio search interface, the user can click on the record button on the audio search interface. After the terminal receives the recording instruction, the terminal starts recording the audio of the user humming. When the terminal finishes recording the audio of the user humming, the terminal obtains the first audio.

[0096] Step 202: Determine the category of the first audio based on a pre-trained classification model.

[0097] Among them, the category of the first audio includes the first category or the second category, the first category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio meets the preset condition, and the second category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio does not meet the preset condition. It should be noted that the preset relationship is obtained through machine learning, and the preset relationship, such as the volume relationship between the noise audio and the human voice audio, may be the same or different for different audios. The present application provides the following methods for determining the category of the first audio, as follows:

[0098] 1. Determine a first implementation manner of a first audio type.

[0099] A target audio segment whose volume value in the first audio is greater than a preset volume value is screened out; a pitch line of the target audio segment is obtained; and the pitch line of the target audio segment is input into a pre-trained classification model to obtain a category of the first audio.

[0100] The preset volume value may be a fixed volume value preset by a technician, for example, -60 decibels. Alternatively, it may be set according to the actual situation of the first audio, that is, the volume value corresponding to each audio frame in the first audio is determined, the volume value corresponding to each audio frame in the first audio is added, and the volume value is divided by the number of audio frames included in the first audio to obtain the average volume corresponding to the first audio, and the average volume value is multiplied by a preset ratio to determine the preset volume value of the first audio.

[0101] Since there will be pauses when people hum songs, only the noise within the time interval will be recorded during the pause time. Therefore, in the implementation, the volume value of each audio frame in the first audio is obtained, and for the volume value of each audio frame, the audio frames with volume values ​​lower than the preset volume value (i.e., audio frames containing only noise) are filtered out, and based on each filtered audio frame and the time corresponding to each filtered audio frame, the audio frames adjacent in time are connected to obtain at least one target audio segment. Then, the target audio segment in the pulse coded PCM format is sampled to obtain the pitch value of each audio frame, and the pitch value of the audio frame is smoothed, and the pitch values ​​within 1 semitone are connected to obtain the pitch line of the target audio segment. The pitch line of each target audio segment is input into the pre-trained classification model, and the category of the first audio is output. It should be noted that the pitch line of the target audio segment includes multiple pitch lines formed by pitch values ​​within 1 semitone.

[0102] Then, the category of each target audio segment is obtained based on the pitch line of each target audio segment and the sub-classification module in the pre-trained classification model; the total duration of each category is determined based on the category of each target audio segment and the duration of each target audio segment; and the category with the longest total duration is determined as the category of the first audio.

[0103] When determining the category of the target audio segment, the dimensions of the pitch sequences of each target audio segment can be aligned. Specifically, according to the pitch line of each target audio segment, the pitch sequence corresponding to each target audio segment is obtained. The pitch sequence can be considered as a vector, and the dimension of the pitch sequence can be the length of the vector. If the dimension of the pitch sequence corresponding to each target audio segment is different, the pitch sequence corresponding to each target audio segment is processed, and the pitch sequence dimensions of each target audio segment after processing are the same, that is, the pitch sequence after the dimension of each target audio segment is aligned is obtained. The pitch sequence after the dimension alignment corresponding to each target audio segment is input into the sub-classification model in the pre-trained classification model, and the category of each target audio segment is output. If the dimension of the pitch sequence corresponding to each target audio segment is the same, it is directly input into the sub-classification model in the pre-trained classification model, and the category of each target audio segment is output.

[0104] It should be noted that in the embodiment of the present application, the classification model is divided into a conversion module, a sub-classification module and a processing module. The conversion module is used to determine the pitch sequence of each target audio segment, and process each pitch sequence to obtain a pitch sequence after dimension alignment, wherein the dimension of each dimension alignment pitch sequence is the same. The sub-classification module is used to determine the category of the target audio segment. The processing module is used to add the duration of the target audio segments corresponding to the same category to obtain the total duration of each category, and determine the category with the longest corresponding total duration. For example, the category of the target audio segment 1 in the first audio is the first category, and the corresponding duration is 5 seconds; the category of the target audio segment 2 is the second category, and the corresponding duration is 7 seconds; the category of the target audio segment 2 is the first category, and the corresponding duration is 6 seconds. Since the total duration of the first category is 11 seconds and the total duration of the second category is 7 seconds, the total duration of the first category is greater than the total duration of the second category, therefore, the category of the first audio is the first category.

[0105] The training method of the classification model may be: according to the above method for obtaining the pitch line of at least one target audio segment in the first audio, each sample audio in the sample audio set is processed to obtain the pitch line of at least one target sample audio segment of each sample audio. The technician marks the category of each sample audio in the sample audio set. For example, if the technician believes that the noise of the sample audio is smaller than the human voice when listening to the sample audio, the benchmark category of the sample audio is set to the first category; if the technician believes that the noise of the sample audio is larger than the human voice when listening to the sample audio, the benchmark category of the sample audio is set to the second category, and the benchmark category of each sample audio is obtained by this method. In this way, the pitch line and benchmark category of at least one target sample audio segment of a sample audio can be used as a training sample to form a training sample set. A training sample is randomly selected from the training sample set, and the pitch line of at least one target sample audio segment corresponding to the training sample is input into the classification model, and the category of the sample audio is output. The category and the benchmark category of the training sample are input into the loss function to obtain the loss information, and the weight parameter of the classification model is adjusted based on the loss information, thereby completing a training process. The classification model is then trained based on other training samples in the training sample set until the preset training process is completed. After the preset training process is completed, a pre-trained classification model is obtained.

[0106] When dividing the first audio into multiple target audio segments, the method provided in the embodiment of the present application filters out audio frames with a volume smaller than a preset value, that is, removes audio segments containing only noise, that is, audio segments corresponding to pauses when the user hums the audio, thereby improving the accuracy of the pre-trained classification model in identifying the category of the first audio.

[0107] In this implementation, the classification model may further include a self-attention learning module. The pitch sequence of each target audio segment after dimension alignment is input into the self-attention learning module to obtain the pitch sequence of each target audio segment after weight adjustment. The pitch sequence of each target audio segment after weight adjustment is input into the sub-classification module to obtain the category of each target audio segment.

[0108] Specifically, the pitch sequence of the target audio segment can be adjusted using the adjustment weights in the trained self-attention learning module. It should be noted that before adjusting the target audio segment, the adjustment weights in the self-attention learning module can be adaptively adjusted according to the actual situation of the target audio segment. For the sake of description, the adjustment weights before adjustment can be referred to as initial adjustment weights, and the adjustment weights after adjustment can be referred to as target adjustment weights.

[0109] A structure of a self-attention learning module is as follows Figure 3 As shown, after the pitch sequence after dimension alignment is input into the self-attention learning module, the self-attention learning module adds the pitch sequence after dimension alignment and the initial adjustment weight, performs activation function 1, convolution processing, activation function 2 and sampling processing on the summed result to obtain the target adjustment weight a, and multiplies the target adjustment weight a with the pitch sequence after dimension alignment to obtain the pitch sequence after weight adjustment. It should be noted that the pitch sequence after dimension alignment and the initial adjustment weight have the same dimension, so the two can be added together.

[0110] Alternatively, after the dimensionally aligned pitch sequence corresponding to the target audio segment is input into the self-attention learning module, the dimensionally aligned pitch sequence and the initial adjustment weight are weighted summed based on the pre-set first weight and second weight, and the result of the weighted summation is subjected to activation function 1, convolution processing, activation function 2 and sampling processing to obtain the target adjustment weight a, and the target adjustment weight a is multiplied with the dimensionally aligned pitch sequence to obtain the weighted adjusted pitch sequence.

[0111] The above-mentioned classification model including the conversion module, the sub-classification module, the processing module and the self-attention learning module is a trained model. During the training process, the weight parameters of the self-attention learning module and the weight parameters of the sub-classification module are adjusted according to the loss information, so as to obtain the trained self-attention learning module and the sub-classification module. Among them, the weight parameters in the self-attention learning module include the initial adjustment weight, the coefficient in the activation function 1, the weight parameter in the convolution processing and the coefficient in the activation function 2. It should be noted that the technician can pre-set the initial adjustment weight before training, and continuously adjust the initial adjustment weight during multiple training processes. After completing the training, the initial adjustment weight of the self-attention learning module can be obtained.

[0112] In an embodiment of the present application, the weight of each pitch sequence is adjusted through a self-attention learning module, and then after the weight-adjusted pitch sequence is input into a sub-classification module, the sub-classification module will focus its attention on the part with a larger corresponding weight, thereby making the sub-classification module classify the target audio segment more accurately.

[0113] 2. Determine a second implementation manner of the first audio type.

[0114] Filter out a target audio segment whose volume value is greater than a preset volume value in the first audio; input the target audio segment into a pre-trained classification model to obtain a category of the first audio. Different from the previous implementation, what is input into the classification model is not the pitch line of the target audio segment, but the target audio segment is directly input into the classification model for classification.

[0115] The classification model includes a feature extraction module, a self-attention learning module, a sub-classification module and a processing module. Therefore, the target audio segment can be input into the feature extraction module to obtain feature information corresponding to the target audio segment; the feature information is input into the self-attention learning module to output the feature information after weight adjustment; each feature information after weight adjustment is input into the sub-classification module to output the category of each target audio segment; the processing module determines the category of the first audio based on the category of each target audio segment.

[0116] Among them, the feature extraction module is used to extract feature information of the target audio segment, and the weights of the extracted multiple feature information are equal. The self-attention learning module is used to adjust the weight of the feature information corresponding to each target audio segment. The sub-classification module can be a neural network module, and the neural network module performs convolution processing on the input feature information after weight adjustment, and then determines the category of the target audio segment corresponding to the feature information after weight adjustment, wherein the sub-classification module can be an SVM module. It should be noted that the self-attention learning module in this implementation is similar to the self-attention learning module in the first implementation, except that the input of the self-attention learning module in this implementation is the feature information of each target audio segment.

[0117] The characteristic information may be the fundamental frequency of the audio or the envelope of the audio.

[0118] For example, a densenet (dense convolutional network) model can be used to determine the probability that the first audio belongs to a certain category. The process is to input a certain audio segment into a feature block (feature extraction module) and obtain the output of the feature block. Input the output of the feature block into the first dense block (dense convolution module) to obtain the output of the first denseblock. Among them, there are n dense layers in the first dense block, and the input of each dense layer is the concatenation of the outputs of all previous layers. Input the output of the first dense block into the first transition block (transition module) to obtain the output of the first transition block. Among them, the transition block consists of convolution (convolution layer) and pooling (pooling layer). Input the output of the first transition block into the second dense block to obtain the output of the second dense block. Input the output of the second dense block into the third dense block to obtain the output of the third dense block. Input the output of the third dense block into the classification block (classification module) to output the score corresponding to the audio segment, wherein the classificationblock consists of pooling (pooling layer) and linear (linear layer). The obtained score is input into the prediction layer to obtain the probability that the audio segment belongs to the first category and the probability that the audio segment belongs to the second category.

[0119] In the above process, a self-attention learning module can be added before the input layer of the densenet network model to make the network model focus more on more effective audio. The principle is to input the target audio segment into the featureblock, obtain the feature information corresponding to the target audio segment, multiply the feature information corresponding to the target audio segment with the pre-trained weights, obtain the feature information after weight adjustment, and then input the processed feature information into the first dense block.

[0120] Furthermore, each feature information after weight adjustment is input into a subclassification module in a pre-trained classification model, and the probability that the target audio segment corresponding to each feature information after weight adjustment is of the first category and the probability of the second category are output; the processing module can add the probability that the target audio segment corresponding to each feature information after weight adjustment is of the first category to determine the total probability corresponding to the first category; add the probability that the target audio segment corresponding to each feature information after weight adjustment is of the second category to determine the total probability corresponding to the second category; when the probability corresponding to the first category is greater than the probability of the second category, the category of the first audio is determined to be the first category; when the probability of the second category is greater than the probability of the first category, the category of the first audio is determined to be the second category.

[0121] 3. Determine a third implementation manner of the first audio type.

[0122] The first audio is directly input into a pre-trained classification model, and the category of the first audio is output.

[0123] The classification model can be trained by taking a sample audio and a reference category corresponding to the sample audio as a training sample, and forming a training sample set with multiple training samples. The sample audio in the training sample is input into the classification model to output the category of the sample audio, and a training process is completed based on the category and the reference category, so that a pre-trained classification model is obtained after multiple trainings.

[0124] It should be noted that the classification model in this implementation is similar to that in the second implementation, that is, the audio is directly input into the classification model, so the relevant description can be found in the above description and will not be repeated here.

[0125] 4. Determine a fourth implementation manner of the first audio type.

[0126] A pitch line of the first audio is obtained, and the pitch line of the first audio is input into a pre-trained classification model to obtain a category of the first audio.

[0127] It should be noted that the classification model in this implementation is similar to the first implementation, that is, both require input of pitch lines into the classification model, so the relevant description can be found in the above description and will not be repeated here.

[0128] 5. Determine a fifth implementation manner of the first audio type.

[0129] After obtaining multiple target audio segments from the first audio, the pitch lines of the other audio segments of the first audio segment except the target audio segment are set to 0; the pitch line of each target audio segment and the pitch lines of the other audio segments are input into a pre-trained classification model, and the category corresponding to the first audio is output. In this implementation, the training samples of the pre-trained classification model include the pitch lines of the target sample audio segment of the sample audio and the pitch lines of the other sample audio segments, and the sample audio is annotated with the corresponding category.

[0130] It should be noted that the classification model in this implementation is similar to the classification model in the first implementation, and will not be repeated here.

[0131] In an embodiment of the present application, before training the above-mentioned classification module, data cleaning can be performed on the sample audios in the sample audio set, thereby removing duplicate sample audios and damaged sample audios in the sample audio set, so that each sample audio in the sample audio set after data cleaning is a valid audio.

[0132] Step 203: If the category of the first audio is the first category, retrieve the second audio that matches the first audio.

[0133] Among them, the second audio is a song audio whose matching degree with the first audio satisfies the matching condition, and the number of the second audio is generally multiple.

[0134] In implementation, when the terminal determines that the category of the first audio is the first category, the terminal generates a retrieval request containing the first audio, and sends the retrieval request to the server. When the server receives the retrieval request sent by the terminal, the server obtains the first audio in the retrieval request, retrieves the second audio that matches the first audio, determines the song identifier corresponding to the second audio, and sends the song identifier corresponding to the second audio to the terminal. After receiving the song audio corresponding to the second audio, the terminal displays the song identifier of the second audio on the display interface. When the user clicks the play button on the display interface, the terminal receives the play instruction, obtains the song identifier of the play instruction, and sends the play request carrying the song identifier to the server. When the server receives the play request, the server obtains the song identifier in the play request, determines the song audio of the song identifier, and sends the song audio of the song identifier to the terminal. The terminal receives the song audio sent by the server and plays the song audio.

[0135] The song identifier of the second audio is used to uniquely identify the identifier of the second audio, and the song audio of the found song identifier and the second audio corresponding to the song identifier may be the same audio.

[0136] It should be noted that if the category of the first audio is the first category, it means that the volume of the noise audio and the volume of the human voice audio in the first audio are less than the preset conditions. It can be understood that the noise audio in the first audio has little effect on the retrieval process based on the first audio, and can even be ignored.

[0137] Optionally, based on the pitch line of each target audio segment, a second audio matching the first audio is retrieved. Optionally, for the pitch line of each target audio segment, multiple reference pitch lines that meet the matching condition with the pitch line of the target audio segment are determined from the pre-stored reference pitch lines of each audio, to obtain at least one reference pitch line of the target audio segment; among the reference pitch lines of all target audio segments, the target reference pitch line with the largest number of occurrences is determined; and the audio of the target reference pitch line is determined as the second audio matching the first audio. The matching condition is that the melodies of the song audios corresponding to the two pitch lines are the same.

[0138] For example, Figure 4 As shown, the first audio is divided into target audio segment 1, target audio segment 2, target audio segment 3, target audio segment 4 and target audio segment 5, and pitch line 1 of target audio segment 1, pitch line 2 of target audio segment 2, pitch line 3 of target audio segment 3, pitch line 4 of target audio segment 4 and pitch line 5 of target audio segment 5 are determined. The reference pitch lines that meet the matching condition with pitch line 1 are reference pitch line 1, reference pitch line 3 and reference pitch line 4, the reference pitch lines that meet the matching condition with pitch line 2 are reference pitch line 2, the reference pitch lines that meet the matching condition with pitch line 3 are reference pitch line 1, reference pitch line 2 and reference pitch line 4, the reference pitch lines that meet the matching condition with pitch line 4 are reference pitch line 3 and reference pitch line 4, and the reference pitch lines that meet the matching condition with pitch line 5 are reference pitch line 1 and reference pitch line 4. Reference pitch line 1 appears 3 times, reference pitch line 2 appears 2 times, reference pitch line 3 appears 2 times, and reference pitch line 4 appears 4 times. Reference pitch line 4 appears the most times, and reference pitch line 4 can be used as the target reference pitch line.

[0139] It should be noted that, in one case, multiple target reference pitch lines that match the pitch lines of all target audio segments can be determined, where the reference pitch line can be the pitch line corresponding to the audio of a song, or the pitch line corresponding to the audio of a song over a period of time.

[0140] Optionally, the pitch lines of other audio segments of the first audio segment except the target audio segment are set to 0; based on the pitch lines of each target audio segment and other audio segments, a second audio matching the first audio segment is retrieved. The other audio segments are audio segments containing only noise audio, and the duration of the other audio segments is much shorter than the duration of the target audio segment. For example Figure 5As shown, the first audio includes target audio segment 1, target audio segment 2 and target audio segment 3, as well as other audio segment 1 and other audio segment 2. The pitch lines of other audio segment 1 and other audio segment 2 are set to 0 respectively.

[0141] Step 204: If the category of the first audio is the second category, the control outputs a retrieval failure prompt message.

[0142] It should be noted that if the category of the first audio is the second category, it means that the volume of the noise audio in the first audio and the volume of the human voice audio are greater than the preset ratio. It can be understood that the noise audio in the first audio has a greater impact on the retrieval process based on the first audio, and the server cannot perform retrieval based on the first audio.

[0143] In implementation, if the category of the first audio is the second category, the terminal controls the output of retrieval failure prompt information, where the retrieval failure prompt information is used to indicate that the first audio contains relatively large noise.

[0144] Optionally, a pre-trained classification model can be set in a server and searched in the server. After the terminal completes recording the first audio, the terminal sends an audio classification request to the server, and the audio classification request carries the first audio recorded by the terminal. The server receives the audio classification request sent by the terminal and obtains the first audio in the audio classification request. Based on the classification model pre-trained in the server, the category of the first audio is determined. If the category of the first audio is the first category, the server retrieves the second audio that matches the first audio, obtains the song identifier of the second audio, and sends the song identifier of the second audio as a search result to the terminal; after the terminal receives the search result, it obtains the song identifier in the search result and displays it on the display interface of the terminal. If the category of the first audio is the second category, the server sends a search failure prompt message to the terminal. Based on the search failure prompt message sent by the server, the terminal displays a search failure prompt message on the display interface, wherein the prompt message also includes the reason for the search failure being that the noise included in the first audio is large.

[0145] It should be noted that the classification model can be trained in the server or in the terminal, and the terminal then sends the pre-trained classification model to the terminal.

[0146] Optionally, the server can send a pre-trained classification model to the terminal and set the pre-trained classification model in the terminal. At the same time, the pitch line library of each audio can also be pre-set in the terminal, and then the second audio matching the first audio can be retrieved through the terminal, wherein the pitch line library also stores the song identification of each pitch line. After the terminal completes recording the first audio, the category of the first audio is determined by the pre-trained classification model set in the terminal. When the category of the first audio is the first category, the terminal can determine multiple pitch lines that meet the matching conditions with the pitch lines of the first audio based on the pitch lines of the first audio and the pitch line library, and then determine the song identification of the pitch lines that meet the matching conditions, and display it on the display interface of the terminal. If the category of the first audio is the second category, the terminal controls the output of a retrieval failure prompt message.

[0147] The present application detects the category of the first audio through a pre-trained classification model; when the category of the first audio is the first category, it is determined that the noise contained in the first audio is small, and a search can be performed based on the first audio, and a second audio matching the first audio is retrieved. When the category of the first audio is the second category, it is determined that the noise contained in the second audio is large, and the control output prompt message of the retrieval failure is output. In the present application, when the category of the first audio is the first category, the accuracy of the retrieval results obtained based on the first audio is high, and the user can quickly find the song audio that meets his needs in the retrieval results; when the category of the first audio is the second category, the first audio is not retrieved, and the process of retrieving the audio with large noise and the process of the user playing the retrieval results one by one are omitted, thereby saving the user's time.

[0148] The present application embodiment provides a device for retrieving audio. Figure 6 The schematic diagram of the structure of an audio retrieval device provided in the embodiment of the present application is used as an example for explanation. Figure 6 As shown, the device includes an acquisition module 610, a determination module 620, a retrieval module 630 and a prompt module 640. If the device is a terminal, the determination module and / or the retrieval module can be integrated into the terminal, or can be deployed separately from the terminal, specifically deployed on a server:

[0149] The acquisition module 610 is configured to acquire the first audio;

[0150] The determination module 620 is configured to determine the category of the first audio based on a pre-trained classification model, wherein the category includes a first category or a second category, the first category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio meets a preset condition, and the second category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio does not meet the preset condition;

[0151] A retrieval module 630 is configured to retrieve a second audio that matches the first audio if the category of the first audio is the first category;

[0152] The prompt module 640 is configured to control the output of a retrieval failure prompt message if the category of the first audio is the second category.

[0153] Optionally, the determining module 620 is configured to:

[0154] Filter out a target audio segment whose volume value in the first audio is greater than a preset volume value;

[0155] Obtain the pitch line of each target audio segment;

[0156] The pitch line of each target audio segment is input into a pre-trained classification model, and the category of the first audio is output.

[0157] Optionally, the pre-trained classification model includes a conversion module, a self-attention learning module, a sub-classification module and a processing module; the determination module 620 is configured to:

[0158] Input the pitch line of each target audio segment into the conversion module in the pre-trained classification model to obtain the pitch sequence of each target audio segment after dimension alignment;

[0159] Input the pitch sequence of each target audio segment after dimension alignment into the self-attention learning module in the pre-trained classification model to obtain the pitch sequence of each target audio segment after weight adjustment;

[0160] Input the pitch sequence of each target audio segment after weight adjustment into the sub-classification module in the pre-trained classification model to obtain the category of each target audio segment;

[0161] The category of each target audio segment is input into a processing module in the pre-trained classification model to obtain the category of the first audio.

[0162] Optionally, the determining module 620 is configured to:

[0163] The category of each target audio segment is input into the processing module in the pre-trained classification model, so that the processing module determines the total duration of each category based on the category of each target audio segment and the duration of each target audio segment and determines the category with the longest total duration as the category of the first audio.

[0164] Optionally, the retrieval module 630 is configured to:

[0165] Based on the pitch line of each target audio segment, a second audio matching the first audio is retrieved.

[0166] Optionally, the retrieval module 630 is configured to:

[0167] For each pitch line of the target audio segment, determine, from pre-stored reference pitch lines of various audios, a plurality of reference pitch lines that meet a matching condition with the pitch line of the target audio segment, and obtain at least one reference pitch line of the target audio segment;

[0168] Among the reference pitch lines of all target audio segments, determine the target reference pitch line that appears most frequently;

[0169] The audio of the target reference pitch line is determined as the second audio that matches the first audio.

[0170] Optionally, the retrieval module 630 is configured to:

[0171] Setting the pitch lines of the other audio segments of the first audio segment except the target audio segment to 0;

[0172] Based on the pitch lines of each target audio segment and the other audio segments, a second audio matching the first audio is retrieved.

[0173] Optionally, the determining module 620 is configured to:

[0174] Filter out a target audio segment whose volume value in the first audio is greater than a preset volume value;

[0175] The target audio segment is input into a pre-trained classification model, and the category of the first audio is output.

[0176] Optionally, the pre-trained classification model includes a feature extraction module, a self-attention learning module, a sub-classification module and a processing module; the determination module 620 is configured to:

[0177] Inputting the target audio segment into the feature extraction module in the pre-trained classification model to obtain feature information corresponding to the target audio segment;

[0178] Inputting the feature information into the self-attention learning module in the pre-trained classification model, and outputting the feature information after weight adjustment;

[0179] Input each of the weight-adjusted feature information into a sub-classification module in the pre-trained classification model, and output the category of each target audio segment;

[0180] The category of each target audio segment is input into a processing module in the pre-trained classification model to obtain the category of the first audio.

[0181] It should be noted that: the device for retrieving audio provided in the above embodiment only uses the division of the above functional modules as an example when retrieving audio. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device for retrieving audio provided in the above embodiment and the method embodiment for retrieving audio belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0182] Figure 7 700 is a schematic diagram of a terminal structure provided by an embodiment of the present application. The terminal 700 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 700 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0183] Typically, the terminal 700 includes a processor 701 and a memory 702 .

[0184] The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0185] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one instruction, which is used to be executed by the processor 701 to implement the method for retrieving audio provided in the method embodiment of the present application.

[0186] In some embodiments, the terminal 700 may further optionally include: a peripheral device interface 703 and at least one peripheral device. The processor 701, the memory 702 and the peripheral device interface 703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 703 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 704, a touch display screen 705, a camera assembly 706, an audio circuit 707, a positioning assembly 708 and a power supply 709.

[0187] The peripheral device interface 703 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 701 and the memory 702. In some embodiments, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 701, the memory 702, and the peripheral device interface 703 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0188] The radio frequency circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 704 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 704 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 704 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0189] The display screen 705 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 705 is a touch display screen, the display screen 705 also has the ability to collect touch signals on the surface or above the surface of the display screen 705. The touch signal can be input to the processor 701 as a control signal for processing. At this time, the display screen 705 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 705 can be one, and the front panel of the terminal 700 is set; in other embodiments, the display screen 705 can be at least two, which are respectively set on different surfaces of the terminal 700 or are folded; in some other embodiments, the display screen 705 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 700. Even, the display screen 705 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 705 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0190] The camera assembly 706 is used to capture images or videos. Optionally, the camera assembly 706 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0191] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 701 for processing, or input them into the radio frequency circuit 704 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 700. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0192] Positioning component 708 is used to locate the current geographical location of terminal 700 to implement navigation or LBS (Location Based Service). Positioning component 708 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system or the European Union's Galileo system.

[0193] The power supply 709 is used to power various components in the terminal 700. The power supply 709 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 709 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0194] In some embodiments, the terminal 700 further includes one or more sensors 710 , including but not limited to: an acceleration sensor 711 , a gyroscope sensor 712 , a pressure sensor 713 , a fingerprint sensor 714 , an optical sensor 715 , and a proximity sensor 716 .

[0195] The acceleration sensor 711 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 700. For example, the acceleration sensor 711 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 701 can control the touch screen 705 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 711. The acceleration sensor 711 can also be used for collecting game or user motion data.

[0196] The gyro sensor 712 can detect the body direction and rotation angle of the terminal 700, and the gyro sensor 712 can cooperate with the acceleration sensor 711 to collect the user's 3D actions on the terminal 700. The processor 701 can implement the following functions based on the data collected by the gyro sensor 712: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0197] The pressure sensor 713 can be set on the side frame of the terminal 700 and / or the lower layer of the touch display screen 705. When the pressure sensor 713 is set on the side frame of the terminal 700, it can detect the user's holding signal of the terminal 700, and the processor 701 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 713. When the pressure sensor 713 is set on the lower layer of the touch display screen 705, the processor 701 controls the operability controls on the UI interface according to the user's pressure operation on the touch display screen 705. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0198] The fingerprint sensor 714 is used to collect the user's fingerprint, and the processor 701 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 714, or the fingerprint sensor 714 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 701 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 714 can be set on the front, back, or side of the terminal 700. When a physical button or a manufacturer logo is set on the terminal 700, the fingerprint sensor 714 can be integrated with the physical button or the manufacturer logo.

[0199] The optical sensor 715 is used to collect the ambient light intensity. In one embodiment, the processor 701 can control the display brightness of the touch display screen 705 according to the ambient light intensity collected by the optical sensor 715. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 705 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 705 is reduced. In another embodiment, the processor 701 can also dynamically adjust the shooting parameters of the camera assembly 706 according to the ambient light intensity collected by the optical sensor 715.

[0200] The proximity sensor 716, also called a distance sensor, is usually arranged on the front panel of the terminal 700. The proximity sensor 716 is used to collect the distance between the user and the front of the terminal 700. In one embodiment, when the proximity sensor 716 detects that the distance between the user and the front of the terminal 700 is gradually decreasing, the processor 701 controls the touch display screen 705 to switch from the screen-on state to the screen-off state; when the proximity sensor 716 detects that the distance between the user and the front of the terminal 700 is gradually increasing, the processor 701 controls the touch display screen 705 to switch from the screen-off state to the screen-on state.

[0201] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the terminal 700, and the terminal 700 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0202] Figure 8 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 801 and one or more memories 802, wherein the memory 802 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 801 to implement the methods provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.

[0203] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the method of retrieving audio in the above embodiment. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0204] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0205] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for retrieving audio, characterized in that: The method comprises: Get the first audio; Based on a pre-trained classification model, the category of the first audio is determined, wherein the category includes a first category or a second category, the first category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio meets a preset condition, and the second category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio does not meet the preset condition; wherein, the determination of the category of the first audio based on the pre-trained classification model includes: screening out target audio segments whose volume values ​​in the first audio are greater than a preset volume value; obtaining the pitch line of each target audio segment; wherein the pre-trained classification model includes a conversion module, a self-attention learning module, a sub-classification module and a processing module; inputting the pitch line of each target audio segment into the conversion module in the pre-trained classification model to obtain a pitch sequence of each target audio segment after dimension alignment; inputting the pitch sequence of each target audio segment after dimension alignment into the self-attention learning module in the pre-trained classification model to obtain a pitch sequence of each target audio segment after weight adjustment; The pitch sequence of each target audio segment after weight adjustment is input into the sub-classification module in the pre-trained classification model to obtain the category of each target audio segment; the category of each target audio segment is input into the processing module in the pre-trained classification model to obtain the category of the first audio; or, the target audio segment whose volume value is greater than the preset volume value in the first audio is screened out; the pre-trained classification model includes a feature extraction module, a self-attention learning module, a sub-classification module and a processing module; the target audio segment is input into the feature extraction module in the pre-trained classification model to obtain the feature information corresponding to the target audio segment; the feature information is input into the self-attention learning module in the pre-trained classification model to output the feature information after weight adjustment; each of the feature information after weight adjustment is input into the sub-classification module in the pre-trained classification model to output the category of each target audio segment; the category of each target audio segment is input into the processing module in the pre-trained classification model to obtain the category of the first audio; If the category of the first audio is the first category, retrieving the second audio matching the first audio; If the category of the first audio is the second category, the control output indicates that the retrieval failed.

2. The method according to claim 1, characterized in that The step of inputting the category of each target audio segment into a processing module in the pre-trained classification model to obtain the category of the first audio includes: The category of each target audio segment is input into the processing module in the pre-trained classification model, so that the processing module determines the total duration of each category based on the category of each target audio segment and the duration of each target audio segment and determines the category with the longest total duration as the category of the first audio.

3. The method according to claim 1, characterized in that: The retrieving a second audio that matches the first audio comprises: Based on the pitch line of each target audio segment, a second audio matching the first audio is retrieved.

4. The method according to claim 3, characterized in that The method of retrieving a second audio that matches the first audio based on the pitch line of each target audio segment comprises: For each pitch line of the target audio segment, determine, from pre-stored reference pitch lines of various audios, a plurality of reference pitch lines that meet a matching condition with the pitch line of the target audio segment, and obtain at least one reference pitch line of the target audio segment; Among the reference pitch lines of all target audio segments, determine the target reference pitch line that appears most frequently; The audio of the target reference pitch line is determined as the second audio that matches the first audio.

5. The method according to claim 3, characterized in that: The method of retrieving a second audio that matches the first audio based on the pitch line of each target audio segment comprises: Setting the pitch lines of the other audio segments of the first audio segment except the target audio segment to 0; Based on the pitch lines of each target audio segment and the other audio segments, a second audio matching the first audio is retrieved.

6. A device for retrieving audio, characterized in that: The device comprises: An acquisition module, configured to acquire a first audio; A determination module is configured to determine the category of the first audio based on a pre-trained classification model, wherein the category includes a first category or a second category, the first category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio meets a preset condition, and the second category is used to indicate that the volume relationship between the noise audio and the human voice audio in the first audio does not meet the preset condition; wherein the determination module is used to: screen out target audio segments in the first audio whose volume values ​​are greater than a preset volume value; obtain the pitch line of each target audio segment; wherein the pre-trained classification model includes a conversion module, a self-attention learning module, a sub-classification module and a processing module; input the pitch line of each target audio segment into the conversion module in the pre-trained classification model to obtain a pitch sequence after dimension alignment of each target audio segment; input the dimensionally aligned pitch sequence of each target audio segment into the self-attention learning module in the pre-trained classification model to obtain a pitch sequence after weight adjustment of each target audio segment; and input the weight of each target audio segment into the self-attention learning module in the pre-trained classification model to obtain a pitch sequence after weight adjustment of each target audio segment. The adjusted pitch sequence is input into the sub-classification module in the pre-trained classification model to obtain the category of each target audio segment; the category of each target audio segment is input into the processing module in the pre-trained classification model to obtain the category of the first audio; or, the determination module is used to screen out the target audio segment whose volume value is greater than the preset volume value in the first audio; the pre-trained classification model includes a feature extraction module, a self-attention learning module, a sub-classification module and a processing module; the target audio segment is input into the feature extraction module in the pre-trained classification model to obtain the feature information corresponding to the target audio segment; the feature information is input into the self-attention learning module in the pre-trained classification model to output the feature information after weight adjustment; each of the feature information after weight adjustment is input into the sub-classification module in the pre-trained classification model to output the category of each target audio segment; the category of each target audio segment is input into the processing module in the pre-trained classification model to obtain the category of the first audio; a retrieval module configured to retrieve a second audio matching the first audio if the category of the first audio is the first category; The prompt module is configured to control the output of retrieval failure prompt information if the category of the first audio is the second category.

7. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the operation performed by the method for retrieving audio according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the operation performed by the method for retrieving audio as described in any one of claims 1 to 5.

Images