Audio recognition method and device, computer device and computer readable storage medium

Abstract

The application discloses an audio recognition method and device, computer equipment and a computer readable storage medium, and applies to the technical field of computers. The method comprises the following steps: inputting to-be-recognized audio data into an audio recognition model to obtain an audio fingerprint of the to-be-recognized audio data output by the audio recognition model; wherein the audio recognition model is obtained based on contrast learning of first training audio data and second training audio data and the first training audio data and third training audio data; the audio identifiers of the first training audio data and the second training audio data are the same; the audio identifiers of the first training audio data and the third training audio data are different; target audio fingerprints satisfying a preset condition are determined from an audio fingerprint library according to the audio fingerprint of the to-be-recognized audio data; and an identification result is determined according to the target audio fingerprints, wherein the identification result comprises an audio identifier corresponding to the to-be-recognized audio data. Through the method, the accuracy of audio recognition can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to an audio recognition method, an audio recognition device, a computer device and a computer readable storage medium. BACKGROUND

[0002] With the increasing number of songs, when a user hears a song but does not know the song name, the user can record a music segment heard by the user through a terminal device, and identify the song to which the music segment belongs through a song recognition function in the terminal device. For example, the user can start a song recognition function through a music application installed in the terminal device, so that the terminal device can record a music segment heard by the user, and identify the recorded music segment. At present, the recognition accuracy of the song recognition function is low. SUMMARY

[0003] The embodiments of the present application provide an audio recognition method, an audio recognition device, a computer device and a computer readable storage medium, which can improve the accuracy of audio recognition.

[0004] In a first aspect, an embodiment of the present application provides an audio recognition method, which comprises:

[0005] inputting to-be-recognized audio data into an audio recognition model to obtain an audio fingerprint of the to-be-recognized audio data output by the audio recognition model; wherein the audio recognition model is obtained based on first training audio data and second training audio data, and contrast learning of the first training audio data and third training audio data, the audio identification of the first training audio data and the second training audio data is the same; the audio identification of the first training audio data and the third training audio data is different;

[0006] determining, from an audio fingerprint library, a target audio fingerprint that satisfies a preset condition with the audio fingerprint of the to-be-recognized audio data;

[0007] determining a recognition result according to the target audio fingerprint, the recognition result comprising an audio identification corresponding to the to-be-recognized audio data.

[0008] It can be seen that the audio recognition model obtained based on contrast learning can extract an audio fingerprint with a higher similarity for audio data of the same audio identification, and extract an audio fingerprint with a lower similarity for audio data of different audio identifications, so that subsequent recognition can be more convenient, and the recognition accuracy can be improved.

[0009] In an implementation manner, the method further comprises:

[0010] The initial audio recognition model is trained by using a training sample set to obtain target loss information; the training sample set includes a positive sample set and a negative sample set, the positive sample set includes the first training audio data and the second training audio data, and the negative sample set includes the third training audio data.

[0011] In a case where the target loss information meets a training end condition, the initial audio recognition model is determined as the audio recognition model.

[0012] In an implementation manner, the first training audio data is original audio data, and the second training audio data is audio data obtained by performing data enhancement processing on the first training audio data; or, the first training audio data and the second training audio data are audio data obtained by performing different data enhancement processing on original audio data corresponding to the first training audio data.

[0013] In an implementation manner, the first training audio data is original audio data, and the second training audio data is cover audio data of the first training audio data; or,

[0014] The first training audio data and the second training audio data are cover audio data of original audio data corresponding to the first training audio data.

[0015] In an implementation manner, the training of the initial audio recognition model by using the training sample set to obtain the target loss information includes:

[0016] The training sample set is input into the initial audio recognition model to obtain audio fingerprints of the first training audio data, the second training audio data, and the third training audio data output by the initial audio recognition model;

[0017] The target loss information is determined according to a first similarity and a second similarity; the first similarity is a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data, the second similarity is a sum of a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data and a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the third training audio data.

[0018] In an implementation manner, the method further includes:

[0019] The first similarity is determined according to a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data.

[0020] The second similarity is determined according to a sum of a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data and a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the third training audio data.

[0021] In an implementation manner, the method further includes:

[0022] playing original audio data corresponding to the audio identifier of the to-be-identified audio data.

[0023] In an implementation manner, the method further includes:

[0024] displaying, on an audio identification interface, an audio identifier corresponding to the to-be-identified audio data.

[0025] In a second aspect, an embodiment of the present application provides an audio identification device, which includes:

[0026] an input unit configured to input to-be-identified audio data into an audio identification model to obtain an audio fingerprint of the to-be-identified audio data output by the audio identification model; wherein the audio identification model is obtained based on first training audio data and second training audio data and contrast learning of the first training audio data and third training audio data, the audio identifier of the first training audio data is the same as that of the second training audio data, and the audio identifier of the first training audio data is different from that of the third training audio data;

[0027] a determination unit configured to determine, from an audio fingerprint library, a target audio fingerprint that satisfies a preset condition with the audio fingerprint of the to-be-identified audio data;

[0028] the determination unit is further configured to determine, according to the target audio fingerprint, an identification result, the identification result including an audio identifier corresponding to the to-be-identified audio data.

[0029] In an implementation manner, the audio identification device further includes:

[0030] a training unit configured to train an initial audio identification model by using a training sample set to obtain target loss information; the training sample set including a positive sample set and a negative sample set, the positive sample set including the first training audio data and the second training audio data, and the negative sample set including the third training audio data;

[0031] the determination unit is further configured to determine, in a case where the target loss information satisfies a training end condition, the initial audio identification model as the audio identification model.

[0032] In an implementation manner, the first training audio data is original audio data, and the second training audio data is audio data obtained by performing data enhancement processing on the first training audio data; or

[0033] The first training audio data and the second training audio data are audio data obtained by performing different data enhancement processing on original audio data corresponding to the first training audio data.

[0034] In an implementation manner, the first training audio data is original audio data, and the second training audio data is cover audio data of the first training audio data; or

[0035] The first training audio data and the second training audio data are cover audio data of original audio data corresponding to the first training audio data.

[0036] In an implementation manner, when the training unit trains the initial audio recognition model by using the training sample set to obtain target loss information, the training unit can be specifically configured to:

[0037] input the training sample set into the initial audio recognition model to obtain audio fingerprints of the first training audio data, the second training audio data, and the third training audio data output by the initial audio recognition model;

[0038] determine the target loss information according to a first similarity and a second similarity; the first similarity is a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data, the second similarity is a sum of a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data and a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the third training audio data.

[0039] In an implementation manner, the determination unit is further configured to determine the first similarity according to a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data.

[0040] The determination unit is further configured to determine the second similarity according to a sum of a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data and a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the third training audio data.

[0041] In an implementation manner, the audio recognition apparatus further includes:

[0042] The playing unit is configured to play original audio data identified by the audio identifier corresponding to the to-be-identified audio data.

[0043] In an implementation manner, the audio identification device further includes:

[0044] The display unit is configured to display the audio identifier corresponding to the to-be-identified audio data on the audio identification interface.

[0045] In a third aspect, an embodiment of the present application provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the audio identification method.

[0046] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program. When the computer program is read and executed by a processor of a computer device, the computer device executes the audio identification method.

[0047] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. The computer instructions are stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium. The processor executes the computer instructions, so that the computer device executes the audio identification method.

[0048] In the method provided in the present application, the audio identification model obtained based on contrast learning can extract audio fingerprints with high similarity from audio data of the same audio identifier. By extracting the audio fingerprint of the to-be-identified audio data, and comparing the extracted audio fingerprint with each audio fingerprint extracted by the audio identification model in the audio fingerprint library, an identification result with high accuracy is obtained, and the recognition effect of song recognition is improved. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0050] Figure 1 is a structural schematic diagram of an audio identification system provided by an embodiment of the present application;

[0051] Figure 2 is a flowchart of an audio identification method provided by an embodiment of the present application;

[0052] Figure 3is a user interface schematic diagram of an audio recognition result provided by an embodiment of the present application;

[0053] Figure 4 is a user interface schematic diagram of playing recognized audio provided by an embodiment of the present application;

[0054] Figure 5 is an architecture schematic diagram of an audio recognition method provided by an embodiment of the present application;

[0055] Figure 6 is a flow schematic diagram of another audio recognition method provided by an embodiment of the present application;

[0056] Figure 7 is an architecture schematic diagram of data enhancement processing of original audio data provided by an embodiment of the present application;

[0057] Figure 8 is an architecture schematic diagram of processing of cover audio data provided by an embodiment of the present application;

[0058] Figure 9 is a structure schematic diagram of an audio recognition device provided by an embodiment of the present application;

[0059] Figure 10 is a structure schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work fall within the scope of protection of the present application.

[0061] It should be noted that the “first”, “second” and the like described in the embodiments of the present application are only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the technical features or implicitly indicating the number of the technical features indicated. Therefore, the technical features limited by “first”, “second” can explicitly or implicitly include at least one of the features.

[0062] Before the embodiments of the present application are further described in detail, the terms and terms involved in the embodiments of the present application are explained, and the terms and terms involved in the embodiments of the present application are applicable to the following explanations.

[0063] 1、Audio fingerprint: refers to the compressed representation of the audio features of an audio data, similar to a person's fingerprint can uniquely represent a person, audio fingerprint can uniquely represent a song. For example, the audio data can be converted into time-frequency, and the frequency spectrum diagram of the audio data is obtained. The local peak points of the energy of the frequency spectrum diagram are more representative for a song. Therefore, a peak point sequence is extracted from a music segment, and the sequence can uniquely represent a music segment, so that the peak point sequence is used to uniquely identify a music segment or a song.

[0064] 2、Timestamp file: refers to the audio data and the corresponding time node. For example, the lyrics timestamp file refers to the audio data of a song and the corresponding word time node (word timestamp) and / or sentence time node (sentence timestamp) of the lyrics text of the song, which may, for example, include the start time and end time of each sentence of the lyrics. For another example, the melody timeline file refers to the start and end time nodes (timestamps) of a melody, such as the start time and end time of the chorus part.

[0065] 3、Cover: refers to re-singing a song that has been published and sung by others according to one's own style, including rewording and re-arranging. Cover recognition refers to identifying a song similar to the original song in lyrics and arrangement. Since cover involves re-arrangement and re-singing, the audio data of the cover has a greater impact on the spectral peak points, and the recognition effect is poor when using spectral peak as audio fingerprint.

[0066] 4、Vector mapping (embedding): vector mapping is a distributed representation method, that is, the original input data is distributedly represented as a linear combination of a series of features, that is, a large sparse vector can be mapped to a low-dimensional space that preserves semantic relationships. The embedding vector can also be referred to as an embedding feature. If the distance between two embedding features is far, it can be indicated that the objects corresponding to the embedding features have a far difference in meaning. If the distance between two embedding features is close, it can be indicated that the objects corresponding to the embedding features have a far difference in meaning. For example, if the distance between the embedding features extracted from two audio data is close, it can be indicated that the two audio data are similar, and may be different music segments of the same song.

[0067] 5、Contrastive Learning: Contrastive Learning is a form of unsupervised learning, and the training sample does not carry any label. The role of contrastive learning is to train the model to pull the distance of each positive sample closer, and to pull the distance of each positive sample and each negative sample farther away, so as to determine through the model that the positive sample is of the same class, and the positive sample and the negative sample are not of the same class. Wherein, the distance can be the cosine distance of the feature vector in the feature space, also known as the cosine similarity, and the distance can be used to measure the difference between the feature vectors. For example, the positive sample can be audio data with the same audio identifier, that is, audio data of the same song, and the negative sample can be audio data with different audio identifiers, that is, audio data of different songs. Therefore, the model trained through contrastive learning can pull the distance of the audio data of the same song closer, and pull the distance of the audio data of different songs farther away, so as to determine the audio identifier corresponding to the closest audio data in the audio fingerprint library of the audio data to be identified, that is, obtain the recognition result.

[0068] In the embodiments of the present application, technologies such as artificial intelligence (AI) and machine learning are involved, wherein:

[0069] Artificial intelligence is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to design and implement principles and methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, automatic driving, intelligent transportation, etc.

[0070] Machine Learning (ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0071] At present, by performing time-frequency conversion on audio data, a spectrogram of the audio data is obtained, and local peaks of energy in the spectrogram are used as audio fingerprints of the audio data. However, due to the influence of cover songs and noise in real scenes on spectral peaks, the recognition accuracy is low.

[0072] Based on the above problems, the embodiment of the present application proposes an audio recognition method, which can be applied to a song recognition scene. The song recognition scene refers to a scene of accurate retrieval from sound to sound. Song recognition is an important way for song dissemination and new song promotion. The audio recognition method proposes that the to-be-recognized audio data is input into an audio recognition model to obtain an audio fingerprint of the to-be-recognized audio output by the audio recognition model, and then a target audio fingerprint satisfying a preset condition with the audio fingerprint of the to-be-recognized audio data is determined from an audio fingerprint library, and an identification result is determined according to the target audio fingerprint, the identification result including an audio identifier corresponding to the to-be-recognized audio data. The audio recognition model is obtained based on comparative learning of first training audio data and second training audio data of the same audio identifier, and first training audio data and third training audio data of different audio identifiers. The audio recognition model obtained by comparative learning can well classify audio data of the same audio identifier and different audio identifiers, thereby improving the effect of audio recognition and improving the accuracy of recognition.

[0073] In a specific implementation, the audio recognition method mentioned above can be executed by a computer device, which can be a terminal device or a server. The terminal device can be, for example, a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart vehicle, and the like, but is not limited thereto. The server can be, for example, a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution servers (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0074] Alternatively, the audio recognition method mentioned above can be executed by a terminal device and a server together. For example, see Figure 1 , Figure 1 is an architecture diagram of an audio recognition system provided by an embodiment of the present application. As shown in Figure 1 , the terminal device 101 can first obtain the to-be-recognized audio data and send the to-be-recognized audio data to the server 102. Correspondingly, after receiving the to-be-recognized audio data, the server 102 can input the to-be-recognized audio data into an audio recognition model, obtain an audio fingerprint of the to-be-recognized audio data output by the audio recognition model, determine a target audio fingerprint from an audio fingerprint library that satisfies a preset condition with the audio fingerprint of the to-be-recognized audio data, and further determine a recognition result according to the target audio fingerprint, the recognition result including an audio identifier corresponding to the to-be-recognized audio data. Optionally, the server 102 can also send the recognition result of the to-be-recognized audio data to the terminal device 101, so that the terminal device 101 can display the recognition result of the to-be-recognized audio data, can also voice broadcast the recognition result of the to-be-recognized audio data, and can also play original audio data identified by the audio identifier corresponding to the to-be-recognized audio data.

[0075] Optionally, the audio recognition model mentioned above can be trained by a training device, as shown in Figure 1 The training device 103 is a computer device other than a terminal device and a server, which can be a terminal device or a server. The training device 103 can obtain an initial audio recognition model, train the initial audio model, and deploy the trained audio recognition model in the server 102 mentioned above. Optionally, the audio recognition model mentioned above can also be trained by the server 102 mentioned above, that is, the training device 103 and the server 102 are the same device.

[0076] Through the embodiments of the present application, the audio recognition model obtained based on contrast learning can extract audio fingerprints with high similarity from audio data of the same audio identifier. By extracting the audio fingerprint of the to-be-identified audio data, the extracted audio fingerprint is compared with each audio fingerprint extracted by the audio recognition model in the audio fingerprint library, so that the accuracy of the recognition result is high.

[0077] It can be understood that the system architecture diagram described in the embodiments of the present application is used to more clearly illustrate the technical solutions of the embodiments of the present application, and does not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of system architecture and the appearance of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0078] In the specific embodiments of the present application, related data such as to-be-identified audio and to-be-identified spectrum graph are involved. When the above embodiments of the present application are applied to specific products or technologies, the related data all need to obtain the permission or consent of the related object, and the collection, use and processing of the related data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0079] Based on the above description, the audio recognition method proposed by the embodiments of the present application will be further described below in combination with the flowchart shown in Figure 2 In the embodiments of the present application, the audio recognition method is mainly described by taking the computer device mentioned above as an example. Please refer to Figure 2 The audio recognition method can specifically include steps 201-203. Wherein:

[0080] 201, the computer device inputs the to-be-identified audio data into the audio recognition model to obtain the audio fingerprint of the to-be-identified audio data output by the audio recognition model.

[0081] In the embodiments of the present application, the to-be-identified audio data refers to audio data that needs to be identified, which can be, for example, audio data of a complete song, audio data of a music segment in a song, or audio data of a recording, such as a recording of a user humming a song, a recording of a user singing or adapting the song, etc. The to-be-identified audio data can be local audio data of the computer device, such as audio data stored locally by the computer device, audio data recorded and stored locally by the computer device, or audio data sent to the computer device by other devices, which are not limited by the present application.

[0082] The audio recognition model refers to a model pre-trained for recognizing audio data. The audio recognition model can be used to extract features of the audio data to be recognized. The extracted features of the audio data can be referred to as an audio fingerprint, which can be used for subsequent recognition. The audio fingerprint refers to a compressed representation of audio features. Different audio data has different features, so the audio fingerprint can be used to represent the audio data to be recognized, so that the audio data to be recognized is recognized based on the audio fingerprint. For audio data, the audio fingerprint can be a local energy peak point of an audio signal in a frequency spectrum graph, which can represent the audio features of the song. The audio fingerprint can also be an embedding feature of the audio data extracted by the audio recognition model.

[0083] In a possible implementation, the audio recognition model used to extract the audio fingerprint of the audio data to be recognized is obtained through contrastive learning of the first training audio data and the second training audio data having the same audio identifier, and the first training audio data and the third training audio data having different audio identifiers. The audio identifier can be used to identify an audio data and distinguish it from other audio data, for example, a song name. In the embodiment of the present application, the positive sample can be audio data having the same audio identifier (for example, the first training audio data and the second training audio data described above), and the negative sample can be audio data having different audio identifiers (for example, the third training audio data described above), so as to achieve the purpose of recognition.

[0084] In a possible implementation, in the process of recognizing an audio data to be recognized, the audio data to be recognized can be input into the audio recognition model, and the audio fingerprint of the audio data to be recognized is extracted by the audio recognition model, that is, the features of the audio data to be recognized are extracted, and then the audio fingerprint of the audio data to be recognized is searched in the pre-extracted audio fingerprints of each audio data, so as to determine the recognition result of the audio data to be recognized according to the searched audio fingerprint. It should be noted that the audio fingerprint of each audio data and the audio fingerprint of the audio data to be recognized are both extracted by the audio recognition model, so the audio fingerprints extracted by the same audio recognition model can be searched for higher similarity as the searched audio data when searching.

[0085] Optionally, in the embodiments of the present application, the original audio data can be input into the audio recognition model to obtain the audio fingerprint of the original audio data output by the audio recognition model, or the features extracted from the original audio data can be input into the audio recognition model, and the audio fingerprint of the original audio data is obtained by processing the extracted features by the audio recognition model. The present application does not make any limitation thereon, and the specific implementation scenario is determined. It can be understood that the object input into the audio recognition model is different, and the training sample set is also different.

[0086] 202. The computer device determines a target audio fingerprint from the audio fingerprint library that meets a preset condition with the audio fingerprint of the to-be-identified audio data.

[0087] In the embodiments of the present application, the audio fingerprint library is a database storing audio fingerprints of a plurality of audio data. In the song recognition scenario, the plurality of audio fingerprints in the audio fingerprint library are respectively audio fingerprints of songs in a song library, or audio fingerprints of at least one segment of each song in the song library. Each audio fingerprint in the audio fingerprint library can carry an audio identifier, so that the computer device can determine the recognition result according to the found audio fingerprint. The preset condition can be that the similarity is higher than a set threshold, and the similarity between the target audio fingerprint and the audio fingerprint of the to-be-identified audio data is higher than the set threshold. The preset condition can also be that the target audio fingerprint is the most similar, i.e., the highest similarity, to the audio fingerprint of the to-be-identified audio data in the audio fingerprint library. The preset condition is taken as an example in the present application, that is, the similarity between the target audio fingerprint and the audio fingerprint of the to-be-identified audio data is the highest in the audio fingerprint library. In the embodiments of the present application, the similarity higher than the threshold can be that the cosine distance is less than the threshold.

[0088] In a possible implementation, the computer device can input the original audio data in the song library into the audio recognition model before identifying the to-be-identified audio data. The original audio data refers to audio data that has not been processed, which can be audio data of a song or audio data of multiple songs. The original audio data can also be audio data of a music segment of a song. The audio fingerprint of the original audio data in the song library is extracted by the audio recognition model, and the audio fingerprint is added to the audio fingerprint library. The audio fingerprint of the original audio data in the song library and the audio fingerprint of the to-be-identified audio data are extracted by the audio recognition model, so that the extracted audio fingerprints with the same audio identifier are more similar, and the extracted audio fingerprints with different audio identifiers are more different. For example, the similarity of the extracted audio fingerprints with the same audio identifier is higher, and the similarity of the extracted audio fingerprints with different audio identifiers is lower.

[0089] In a possible implementation, after obtaining the audio fingerprint of the audio data to be recognized output by the audio recognition model, a target audio fingerprint that meets a preset condition with the audio fingerprint of the audio data to be recognized can be determined from the audio fingerprint library. The preset condition can be that the audio fingerprint in the audio fingerprint library has the highest similarity with the audio fingerprint of the audio data to be recognized. Specifically, the computer device can calculate the similarity of the audio fingerprint of the audio data to be recognized with each audio fingerprint in the audio fingerprint library, obtain a calculation result, and determine the audio fingerprint with the highest similarity in the calculation result as the target audio fingerprint. Specifically, the cosine distance of the audio fingerprint of the audio data to be recognized, i.e., the extracted feature vector (such as an embedding feature), with each audio fingerprint (embedding feature) in the audio fingerprint library is calculated, and then the audio fingerprint with the shortest cosine distance is determined as the target audio fingerprint.

[0090] It can be understood that the target audio fingerprint is also an audio fingerprint of an audio data extracted by the audio recognition model, and the target audio fingerprint has the highest similarity (such as the shortest cosine distance) with the audio fingerprint of the audio data to be recognized. Therefore, the target audio fingerprint can be used as a query result for querying the audio fingerprint library, and then the recognition result can be determined according to the target audio fingerprint.

[0091] 203. The computer device determines and outputs a recognition result according to the target audio fingerprint, and the recognition result includes an audio identifier corresponding to the audio data to be recognized.

[0092] In the embodiments of the present application, in the scenario of song recognition, the recognition result can include the audio identifier of the audio data to be recognized. Taking the audio identifier as the song name as an example, the recognition result can indicate which specific song the audio data to be recognized belongs to. Since the audio fingerprint that is most similar (has the shortest cosine distance) to the audio fingerprint of the audio data to be recognized is the target audio fingerprint, the audio identifier carried by the target audio fingerprint can be used as the audio identifier of the audio data to be recognized.

[0093] Optionally, each audio data in the audio fingerprint library can carry audio information in addition to the audio identifier, and the audio information can include, for example, singer information, album information, lyrics and composition information, arrangement information, producer information, and the like, which are not limited in the present application. Further, the recognition result can further include at least one of the singer information, the album information, the lyrics and composition information, the arrangement information, and the producer information of the target audio fingerprint.

[0094] In a possible implementation, after determining the recognition result, the computer device can output the recognition result, for example, display the recognition result of the audio data to be recognized in a user interface of music software, or play the original audio data identified by the audio identifier corresponding to the audio data to be recognized. Please refer toFigure 3 , Figure 3 is a user interface schematic diagram of an audio recognition result provided by an embodiment of the present application. As shown in Figure 3 , the present application takes the interface of a computer device as a terminal device as an example for illustration, Figure 3 the left side is a user interface of song listening and music recognition in music software, and the computer device can record a piece of audio data to be recognized and perform recognition while displaying Figure 3 the user interface on the left side. Further, the computer device displays Figure 3 the user interface on the right side, i.e., an audio recognition interface, as shown in Figure 3 , in which the audio identification (e.g., a song name) and audio information of the audio data to be recognized can be displayed.

[0095] Further, please refer to Figure 4 , as shown in Figure 4 , the user can click the song display area of the recognition result in the audio recognition interface, and the computer device enters a song playing page when detecting the trigger operation (e.g., clicking the song display area of the recognition result) for the recognition result, as shown in Figure 4 , the song playing page can include the image of the song, the audio identification of the song, and the audio information, and exemplarily, as shown in Figure 4 , the singer information, the composition / lyrics information, the album information, the arrangement information, and the lyrics information, etc. can be displayed, which are not limited by the present application. In the song playing page, the original audio data of the audio identification can be played. Further, the user can pause the playing or switch to play other songs.

[0096] Please refer to Figure 5 , Figure 5 is a schematic diagram of an architecture of an audio recognition method provided by an embodiment of the present application, as shown in Figure 5 ​As shown, it is divided into two processes, one is a library building process, and one is a retrieval process. In the library building process, the original audio data in the song library can be input into the audio recognition model, and the audio fingerprints of the original audio data in the song library are extracted by the audio recognition model, wherein the original audio data can be multiple audio data, each audio data carries an audio identifier, and the extracted audio fingerprints are added to the audio fingerprint library. In the retrieval process, the audio data to be identified is obtained, and the audio data to be identified is input into the audio recognition model, the audio fingerprint of the audio data to be identified is extracted by the audio recognition model, and the audio fingerprint library is searched, and the target audio fingerprint meeting the preset condition is searched as the search result. The preset condition can be that the similarity of the audio fingerprint of the audio data to be identified and the target audio fingerprint is the highest (the cosine distance is the shortest). Further, the identification result of the audio data to be identified is determined according to the audio identifier carried by the target audio fingerprint, so as to achieve the purpose of song recognition.

[0097] In the embodiment of the present application, by inputting the audio data into the audio recognition model, the audio fingerprint of the audio data is extracted, which is more robust and can better cope with more complex noise environments and better identify cover and adapted works.

[0098] The method described in the embodiment of the present application first inputs the audio data to be identified into the audio recognition model to obtain the audio fingerprint of the audio data to be identified output by the audio recognition model, then determines the target audio fingerprint meeting the preset condition with the audio fingerprint of the audio data to be identified from the audio fingerprint library, and finally determines the identification result according to the target audio fingerprint, wherein the identification result includes the audio identifier corresponding to the audio data to be identified. The audio recognition model is obtained based on the comparison learning of the first training audio data and the second training audio data with the same audio identifier, and the first training audio data and the third training audio data with different audio identifiers. The audio recognition model obtained based on the comparison learning can extract audio fingerprints with high similarity for audio data with the same audio identifier. By extracting the audio fingerprint of the audio data to be identified and comparing the extracted audio fingerprint with each audio fingerprint extracted by the audio recognition model in the audio fingerprint library, an identification result with high accuracy is obtained, and the identification effect of song recognition is improved.

[0099] Please refer to Figure 6 Another flowchart of an audio recognition method disclosed in the embodiment of the present application is shown. The audio recognition method can be executed by a computer device, which can be a server 102 in an audio recognition system or a training device 103 in the audio recognition system. The embodiment is mainly used to explain the process of training an initial audio recognition model. The audio recognition method can specifically include steps 601-602. Wherein:

[0100] 601、The computer device trains the initial audio recognition model by using the training sample set to obtain target loss information.

[0101] In the embodiments of the present application, the execution subject of the computer device and the training device is taken as an example for description, the training sample set is a sample set used for training the initial audio recognition model, and all samples in the training sample set do not carry any label. The initial audio recognition model can be a model used for extracting a feature vector of audio data, and the initial audio recognition model can be a convolutional neural network (Convolutional Neural Network, CNN) such as Sample CNN, or other types of models, which are not limited in the present application. The target loss information can include a loss function of the model.

[0102] Specifically, the training sample set can include first training audio data, second training audio data, and third training audio data. The first training audio data, the second training audio data, and the third training audio data can each be one audio data, or can each include multiple audio data. For example, the training audio data can be audio data of a complete song, audio data of a music segment of a song, or audio data of multiple music segments of a song. The first training audio data and the second training audio data have the same audio identifier, and the first training audio data and the third training audio data have different audio identifiers. For example, the first training audio data and the second training audio data are audio data of two music segments of the same song, and the first training audio data and the third training audio data are audio data of music segments in different songs.

[0103] In a possible implementation, in order to enable the trained audio recognition model to pull audio data of the same audio identifier closer, that is, to classify them into one category, and to pull music segments of different audio identifiers farther, that is, not to classify them into one category. Therefore, the training sample set can include a positive sample set and a negative sample set. The training samples in the positive sample set are audio data of the same audio identifier, for example, the positive sample set can include the first training audio data and the second training audio data. The training samples in the negative sample set can be other audio data except the audio identifier corresponding to the first training audio data in the positive sample set, for example, the negative sample set can include the third training audio data.

[0104] In a possible implementation, since the training samples in the training sample set do not carry labels, the original audio data identified by the audio identifier can be processed by data augmentation, so as to determine the original audio data and the enhanced audio data obtained by performing different data augmentation on the original audio data as a class, that is, the audio identifier of the original audio data. It can be understood that the audio data corresponding to each audio identifier can each be a class. When the original audio data and the enhanced audio data of a certain audio identifier are used as a positive sample set, the original audio data and the enhanced audio data of other audio identifiers form a negative sample set.

[0105] In a possible implementation, the first training audio data can be original audio data of a certain audio identifier, and the second training audio data is audio data obtained by performing data augmentation on the first training audio data. Similarly, the second training audio data can be original audio data of a certain audio identifier, and the first training audio data is audio data obtained by performing data augmentation on the second training audio data.

[0106] In another possible implementation, the first training audio data and the second training audio data are audio data obtained by performing data augmentation on the original audio data corresponding to the first training audio data, that is, the first training audio data and the second training audio data are two audio data in the enhanced audio data obtained by performing data augmentation on the original audio data of the audio identifier.

[0107] In a possible implementation, the manner of performing data augmentation on the original audio data includes one or more of the following: speed change processing, pitch change processing, flip processing, noise adding processing, clipping processing, and other processing manners, which are not limited in the present application. The original audio data can include time domain features and frequency domain features. The speed change processing refers to changing the speed of the original audio data, which can be specifically obtained by upsampling or downsampling the time sequence features and the frequency domain features of the original audio data. The pitch change processing refers to changing the pitch of the original audio data, which can be specifically obtained by processing the frequency domain features of the original audio data. The flip processing can also refer to changing the pitch of the original audio data, which can be specifically obtained by flipping the frequency domain features. For example, the inverse of each spectral energy value in the frequency domain features can be taken. The noise adding processing refers to adding noise to the original audio data, for example, superimposing a preset noise frequency domain feature on the frequency domain features of the original audio data. The preset noise can be Gaussian noise or other noise, which is not limited in the present application. The clipping processing can be random clipping of the original audio data to change the length of the original audio data.

[0108] Optionally, the original audio data can be complete audio data of an audio identifier, or can be partial audio data of an audio identifier. For example, the original audio data can be complete audio data of a song. When the original audio data is subjected to data enhancement processing, a timestamp file corresponding to the original audio data can be obtained first, and the original audio data is subjected to slicing processing, and each slice is subjected to data enhancement processing, that is, the audio data of each music segment is subjected to data enhancement processing. The timestamp file can include a slicing scheme, that is, can include the time nodes (timestamps) of the start and end of each segment, so that the original audio data can be sliced according to the timestamp file. For example, the timestamp file is a lyric timestamp file, the original audio data can be sliced according to the timestamps of the start and end of each lyric, or can be sliced according to the timestamps of the start and end of each lyric segment (multiple lyrics).

[0109] Please refer to Figure 7 , Figure 7 is a process schematic diagram for performing data enhancement processing on original audio data provided by the embodiments of the present application. In the embodiments of the present application, the original audio data can be directly subjected to enhancement processing, Figure 7 For example, the original audio data is complete audio data, the original audio data can be sliced, that is, cut into multiple segments of original audio data, that is, multiple slices of original audio data are obtained. The multiple slices of original audio data are subjected to data enhancement processing, for example Figure 7 As shown, the enhanced audio data 1 can be obtained by speed processing, the enhanced audio data 2 can be obtained by pitch processing, the enhanced audio data 3 can be obtained by flipping processing, the enhanced audio data 4 can be obtained by noise adding processing, and the enhanced audio data 5 can be obtained by cutting processing, such as random cutting, on the original audio data which is not sliced. It can be understood that the first training audio data and the second training audio data can be two audio data from the original audio data and the enhanced audio data. Through data enhancement, the model can learn real noise scene transformation, and the extracted audio fingerprint is more robust.

[0110] In a possible implementation, the present application introduces cover audio data in the training sample set, and the cover audio data is used as one of the enhanced audio data. If the first training audio data is original audio data, the second training audio data can be cover audio data of the first training audio data, that is, the first training audio data is original audio data, for example, audio data of the original song, and the second training audio data is cover audio data of the song.

[0111] In another possible implementation, the first training audio data and the second training audio data are both cover audio data of original audio data identified by the audio identifier of the first training audio data, that is, the first training audio data and the second training audio data are both audio data of two different cover versions of the same song, or two music clips of the same cover version. By introducing cover audio data in the training sample, the fingerprint feature can be expanded with the increasing cover versions while maintaining uniqueness, enabling the model to learn the transformation of adaptation scenarios and cover scenarios, so that the extracted audio fingerprint can adapt to changes in song versions, and the extracted audio fingerprint is more robust, which is of great significance in the current era of adaptation works.

[0112] Please refer to Figure 8 , Figure 8 is an architecture schematic diagram provided by an embodiment of the present application for processing cover audio data. In the embodiment of the present application, the original audio data can be directly enhanced, Figure 8 For example, the original audio data is complete audio data, the original audio data can be sliced, that is, cut into multiple original audio data clips, that is, multiple original audio data slices are obtained. And n cover audio data and the timestamp file corresponding to each cover audio data are obtained, for example Figure 8 n cover timestamps in the cover timestamp file. The cover timestamp file can be a lyrics timestamp or other timestamp, for example, the cover audio data is identified to obtain the identified vocals, and the timestamp of each lyrics sung by the vocals is used as the timestamp corresponding to the cover audio data, which is not limited by the present application. The n cover audio data and the original audio data have the same audio identifier. Each cover audio data can also be sliced to obtain multiple music clips of each cover audio data, and the multiple music clips of each cover audio data obtained are determined as training samples in the positive sample set.

[0113] In a possible implementation, the training sample set is input into the initial audio recognition model to obtain the audio fingerprint of the first training audio data, the audio fingerprint of the second training audio data, and the audio fingerprint of the third training audio data in the training sample set output by the initial audio recognition model, and then the target loss information is determined according to the first similarity and the second similarity. The first similarity is the similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data, the second similarity is the sum of the similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data, and the similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the third training audio data.

[0114] It can be understood that the first similarity can be a similarity between the original audio data and the enhanced audio data in the positive sample set in the training sample set, and the second similarity can be a sum of similarities between a certain positive sample in the training sample set and any training sample in the training sample set. Specifically, the computer device can determine the first similarity according to a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data; and determine the second similarity according to the sum of the cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data and the cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the third training audio data. The target loss information of the initial audio recognition model can include a loss function of the initial audio recognition model, which can be as shown in formula 1:

[0115]

[0116] In formula 1, l i,j represents the loss function of the initial audio recognition model, the numerator is a cosine distance between the audio fingerprint of the original audio data corresponding to a certain audio identifier and the audio fingerprint of the enhanced audio data or the cover audio data. z i is the audio fingerprint of the original audio data corresponding to a certain audio identifier, z j is the audio fingerprint of the enhanced audio data or the cover audio data of the audio identifier, sim(z i ,z j ) represents the cosine distance between z i and z j , and τ is a constant that controls the range of the loss function.

[0117] The denominator is the sum of the cosine distance between the audio fingerprint of the original audio data corresponding to a certain audio identifier and the audio fingerprint of the original audio data and the enhanced audio data or the cover audio data corresponding to the audio identifier, and the cosine distance between the audio fingerprint of a certain audio data selected from the original audio data or the enhanced audio data or the cover audio data corresponding to other audio identifiers.

[0118] In formula 1, k is the current training sample label, z k is the audio fingerprint of the audio data (which can be original audio data, enhanced audio data, or cover audio data) corresponding to a certain audio identifier. In formula 1, the number of total training samples is 2N, and among the 2N training samples, there can be N positive samples and N negative samples. It can be understood that z i may be a positive sample selected from the N positive samples, z k may be a training sample selected from the 2N training samples, z kThe positive sample can also be a negative sample. In the selection process, the values of i and k can be equal, that is, the cosine distance between z i and z k is calculated. 1[k≠i] represents that the value is 1 when k≠i, and the value is 0 when k=i, and the sum of the cosine distances between a certain positive sample and each sample is obtained.

[0119] It can be understood that by inputting the training sample set into the initial audio recognition model, a target loss information such as a value of a loss function is determined, in the case where it is determined that the target loss information such as the value of the loss function does not satisfy the training end condition, the computer device can adjust the parameters of the initial audio recognition model according to the target loss information, obtain an adjusted audio recognition model, and can use the training sample set to train the adjusted audio recognition model again, and obtain a target loss information such as a value of a loss function again, and determine whether the training end condition is satisfied according to the value of the loss function again, in the case where the training end condition is not satisfied, the above-mentioned adjusted audio recognition model is adjusted, and the twice-adjusted audio recognition model is trained again, until the target loss information obtained satisfies the training end condition, that is, the value of the loss function satisfies the training end condition. And through the way of training the initial audio recognition model provided in the present application, through task-based learning, the task of model training is to compare whether the two input audio data belong to the same song or a certain music segment of the same song, which can improve the effect of extracting audio fingerprints by the audio recognition model.

[0120] 602、In the case where the above-mentioned target loss information satisfies the training end condition, the computer device determines the initial audio recognition model obtained by the last adjustment as the above-mentioned audio recognition model.

[0121] In the embodiments of the present application, the training end condition can be that the value of the loss function in the target loss information is within a preset threshold range, such as the minimum value of the loss function, and the training end condition can also be that the difference between the loss function values obtained by consecutive N times of iterative training is less than a preset difference threshold value for more than or equal to a preset number threshold value. The present application does not limit this, and it can be determined according to the use scenario. Then the audio recognition model that satisfies the training end condition can be determined as the trained audio recognition model in the case where the target loss information satisfies the training end condition.

[0122] Optionally, the model parameters of the initial audio recognition model can be adjusted by gradient descent. When the model parameters are updated using gradient descent, the gradient of the loss function is calculated, and the model parameters are iteratively updated according to the gradient to gradually converge the initial audio recognition model to improve the accuracy of the classification of the model.

[0123] Specifically, the network structure of the trained audio recognition model can be as shown in Table 1:

[0124] Table 1

[0125]

[0126]

[0127] The audio recognition model in Table 1 is only an example and does not limit the audio recognition model in the embodiments of the present application. In Table 1, layer represents the type of layer of the network structure, stride represents the stride of the convolution kernel sliding. If the stride is 1, it means that the convolution kernel slides one step after each calculation. The sliding direction can be determined by the stride. If the stride is 3, it means that the convolution kernel slides 3 steps after each calculation. The output is the scale of the output after this processing multiplied by the dimension. The scale can be determined by the input audio data, for example, 19683 described above. The dimension is determined by the convolution layer and the pooling layer, for example, 128, 256, 512 described above. Conv 3-128 represents a convolution layer in which the convolution kernel size is 3x1 and the number is 128, i.e., the output channel number is 128. Maxpool 3 can represent a maximum pooling layer with a size of 3x1.

[0128] For example, the scale of the input audio data is 19683. After the convolution layer of the first layer conv3-128, the output feature vector is 19683x128. After the convolution layer of the second layer conv3-128, the output feature vector is 19683x128, and after the maximum pooling layer, the output feature vector is 6561x128. Similarly, after the convolution layer of the third layer conv3-128, the output feature vector is 6561x128, and after the maximum pooling layer, the output feature vector is 2187x128. Similarly, after processing by a convolution layer and then by a maximum pooling layer, the output feature vector of the last convolution layer Conv 1-512 is 1x512, and after the dropout layer, the random dropout rate is 0.5, which is a set value, so that the output feature vector of 1x512 is obtained.

[0129] wherein the number of network parameters can be shown as Equation 2:

[0130] parameters = (w x h x fprevious + 1) x fcurrent Equation 2

[0131] wherein parameters is the number of parameters, w and h are the width and height of the convolution kernel of the current convolution layer, fprevious is the number of the convolution kernel of the previous layer, plus 1 is because of the bias parameter, and fcurrent is the number of the convolution kernel of the current convolution layer. For example, 512 = (3 x 1 + 1) x 128; similarly, 49280 = (3 x 128 + 1) x 128, 98560 = (3 x 128 + 1) x 256; 196864 = (3 x 256 + 1) x 256; 393782 = (3 x 256 + 1) x 512; 786944 = (3 x 512 + 1) x 512; 262656 = (1 x 512 + 1) x 512.

[0132] As can be seen from the above network structure, the time domain features of the audio data are extracted through continuous max-pooling processing and down-sampling, i.e., the stride convolution module, i.e., the above step is 3 and not 1. Since the audio fingerprints extracted from the audio data in the past all need to perform frequency domain transformation processing on the audio data, part of the frequency domain features may be lost, and the time domain features of the audio data extracted by the audio recognition model ensure the integrity of the extracted features.

[0133] The method described in the embodiments of the present application first inputs the to-be-identified audio data into the audio recognition model to obtain the audio fingerprint of the to-be-identified audio data output by the audio recognition model, then determines the target audio fingerprint that satisfies a preset condition from the audio fingerprint library, and finally determines the recognition result according to the target audio fingerprint, wherein the recognition result includes the audio identifier corresponding to the to-be-identified audio data. The audio recognition model is obtained based on contrast learning of the first training audio data and the second training audio data with the same audio identifier, and the first training audio data and the third training audio data with different audio identifiers. The audio recognition model obtained based on contrast learning can extract audio fingerprints with high similarity for audio data of the same audio identifier. By extracting the audio fingerprint of the to-be-identified audio data and comparing the extracted audio fingerprint with each audio fingerprint extracted by the audio recognition model in the audio fingerprint library, a recognition result with high accuracy can be obtained, and the recognition effect of song recognition is improved.

[0134] Based on the above audio recognition method, the embodiments of the present application provide an audio recognition device. Please refer to Figure 9FIG. 9 is a structural schematic diagram of an audio recognition device provided by an embodiment of the present application. The audio recognition device 900 can run the following units:

[0135] The input unit 901 is configured to input to-be-recognized audio data into an audio recognition model to obtain an audio fingerprint of the to-be-recognized audio data output by the audio recognition model. The audio recognition model is obtained based on first training audio data and second training audio data and contrast learning of the first training audio data and third training audio data. The audio identification of the first training audio data and the second training audio data is the same. The audio identification of the first training audio data and the third training audio data is different.

[0136] The determination unit 902 is configured to determine, from an audio fingerprint library, a target audio fingerprint that satisfies a preset condition with the audio fingerprint of the to-be-recognized audio data.

[0137] The determination unit 902 is further configured to determine and output a recognition result according to the target audio fingerprint. The recognition result includes an audio identification corresponding to the to-be-recognized audio data.

[0138] In an implementation manner, the audio recognition device 900 further includes:

[0139] The training unit 903 is configured to train an initial audio recognition model by using a training sample set to obtain target loss information. The training sample set includes a positive sample set and a negative sample set. The positive sample set includes the first training audio data and the second training audio data. The negative sample set includes the third training audio data.

[0140] The determination unit 902 is further configured to determine the initial audio recognition model as the audio recognition model in a case where the target loss information satisfies a training end condition.

[0141] In an implementation manner, the first training audio data is original audio data, and the second training audio data is audio data obtained by performing data enhancement processing on the first training audio data. Alternatively,

[0142] The first training audio data and the second training audio data are audio data obtained by performing different data enhancement processing on original audio data corresponding to the first training audio data.

[0143] In an implementation manner, the first training audio data is original audio data, and the second training audio data is cover audio data of the first training audio data. Alternatively,

[0144] The first training audio data and the second training audio data are cover audio data of original audio data corresponding to the first training audio data.

[0145] In an implementation manner, the training unit 903, when training the initial audio recognition model by using the training sample set to obtain the target loss information, can be specifically configured to:

[0146] input the training sample set into the initial audio recognition model to obtain the audio fingerprints of the first training audio data, the second training audio data and the third training audio data output by the initial audio recognition model;

[0147] determine the target loss information according to the first similarity and the second similarity, the first similarity is a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data, the second similarity is a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data, and a sum of a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the third training audio data.

[0148] In an implementation manner, the determination unit 902 is further configured to determine the first similarity according to a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data.

[0149] The determination unit 902 is further configured to determine the second similarity according to a sum of the cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data and a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the third training audio data.

[0150] In an implementation manner, the audio recognition apparatus 900 further includes:

[0151] The playing unit 904 is configured to play original audio data identified by the audio identifier corresponding to the to-be-recognized audio data.

[0152] In an implementation manner, the audio recognition apparatus 900 further includes:

[0153] The display unit 905 is configured to display the audio identifier corresponding to the to-be-recognized audio data on an audio recognition interface.

[0154] According to another embodiment of the present application, Figure 9The units in the audio recognition apparatus shown can be combined into one or several other units respectively or entirely, or some of the units can be further split into multiple units with smaller functions to constitute, which can achieve the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The units are divided based on logical functions, and in actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the audio recognition apparatus can also include other units, and in actual applications, these functions can also be assisted by other units, and can be implemented by multiple units in cooperation.

[0155] According to another embodiment of the present application, the audio recognition apparatus as shown in Figure 2 or Figure 6 the steps involved in the corresponding method shown in Figure 10 the audio recognition apparatus as shown in

[0156] In the method proposed in the present application, first, the audio data to be identified is input into the audio recognition model to obtain the audio fingerprint of the audio data to be identified output by the audio recognition model, then the target audio fingerprint satisfying the preset condition with the audio fingerprint of the audio data to be identified is determined from the audio fingerprint library, and finally the recognition result is determined according to the target audio fingerprint, where the recognition result includes the audio identifier corresponding to the audio data to be identified. The audio recognition model is obtained based on the comparison learning of the first training audio data and the second training audio data with the same audio identifier, and the first training audio data and the third training audio data with different audio identifiers. The audio recognition model obtained based on the comparison learning can extract audio fingerprints with higher similarity for audio data with the same audio identifier. By extracting the audio fingerprint of the audio data to be identified, and comparing the extracted audio fingerprint with each audio fingerprint extracted by the audio recognition model in the audio fingerprint library, an accurate recognition result with higher accuracy is obtained, and the recognition effect of song recognition is improved.

[0157] Based on the description of the method embodiments and the device embodiments, the present application further provides a computer device. Please refer to Figure 10The computer device 1000 at least includes a processor 1001, a communication interface 1002, and a computer storage medium 1003. The processor 1001, the communication interface 1002, and the computer storage medium 1003 can be connected through a bus or other means. The computer storage medium 1003 can be stored in a memory 1004 of the computer device 1000, and the computer storage medium 1003 is used to store a computer program, the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the computer storage medium 1003. The processor 1001 (also called CPU (Central Processing Unit, central processor)) is the computing core and control core of the computer device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement a corresponding method flow or a corresponding function.

[0158] In one embodiment, the processor 1001 described above in the embodiments of the present application can be used to perform a series of audio recognition processing, specifically including: inputting to-be-recognized audio data into an audio recognition model to obtain an audio fingerprint of the to-be-recognized audio data output by the audio recognition model; wherein the audio recognition model is obtained based on first training audio data and second training audio data, and contrast learning of the first training audio data and third training audio data, the audio identifiers of the first training audio data and the second training audio data are the same; the audio identifiers of the first training audio data and the third training audio data are different; determining a target audio fingerprint that satisfies a preset condition from an audio fingerprint library and the audio fingerprint of the to-be-recognized audio data; determining and outputting a recognition result according to the target audio fingerprint, the recognition result including an audio identifier corresponding to the to-be-recognized audio data.

[0159] The embodiments of the present application also provide a computer storage medium (Memory), which is a memory device in a computer device, used to store programs and data. It can be understood that the computer storage medium here can include an internal storage medium in the computer device, and of course can also include an extended storage medium supported by the computer device. The computer storage medium provides a storage space, which stores an operating system of the computer device. Moreover, one or more instructions suitable for being loaded and executed by the processor 1001 are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory; optionally, it can also be at least one computer storage medium located away from the aforementioned processor.

[0160] In an embodiment, one or more instructions stored in the computer storage medium can be loaded and executed by the processor to implement the above-described method for audio recognition Figure 2 or Figure 6 corresponding steps of the method in the above-described embodiment of the method for audio recognition; in a specific implementation, one or more instructions stored in the computer storage medium are loaded and executed by the processor 1001 to perform the following steps:

[0161] inputting the to-be-recognized audio data into the audio recognition model to obtain an audio fingerprint of the to-be-recognized audio data output by the audio recognition model; the audio recognition model is obtained based on first training audio data and second training audio data and contrast learning of the first training audio data and third training audio data, the audio identifiers of the first training audio data and the second training audio data are the same, and the audio identifiers of the first training audio data and the third training audio data are different;

[0162] determining, from the audio fingerprint library, a target audio fingerprint that satisfies a preset condition with the audio fingerprint of the to-be-recognized audio data;

[0163] determining and outputting a recognition result according to the target audio fingerprint, the recognition result including an audio identifier corresponding to the to-be-recognized audio data.

[0164] In an implementation manner, the one or more instructions can also be loaded and executed by the processor to perform the following steps:

[0165] training the initial audio recognition model using a training sample set to obtain target loss information; the training sample set includes a positive sample set and a negative sample set, the positive sample set includes the first training audio data and the second training audio data, and the negative sample set includes the third training audio data;

[0166] In a case where the target loss information satisfies a training end condition, the initial audio recognition model is determined as the audio recognition model.

[0167] In an implementation manner, the first training audio data is original audio data, and the second training audio data is audio data obtained by performing data enhancement processing on the first training audio data; or, the first training audio data and the second training audio data are audio data obtained by performing different data enhancement processing on original audio data corresponding to the first training audio data.

[0168] In an implementation manner, the first training audio data is original audio data, and the second training audio data is cover audio data of the first training audio data; or,

[0169] The first training audio data and the second training audio data are cover audio data of original audio data corresponding to the first training audio data.

[0170] In an implementation manner, when the initial audio recognition model is trained by using the training sample set to obtain the target loss information, the one or more instructions can be loaded by the processor and specifically executed as follows: inputting the training sample set into the initial audio recognition model to obtain the audio fingerprints of the first training audio data, the second training audio data and the third training audio data output by the initial audio recognition model; determining the target loss information according to the first similarity and the second similarity, the first similarity being a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data, the second similarity being a sum of a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data and a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the third training audio data.

[0171] In an implementation manner, the one or more instructions can be further loaded by the processor and executed as follows:

[0172] determining the first similarity according to a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data;

[0173] determining the second similarity according to a sum of a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data and a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the third training audio data.

[0174] In an implementation manner, the one or more instructions can be further loaded by the processor and executed as follows: playing original audio data identified by the audio identifier corresponding to the to-be-recognized audio data.

[0175] In an implementation manner, the one or more instructions can be further loaded by the processor and executed as follows:

[0176] displaying the audio identifier corresponding to the to-be-recognized audio data on the audio recognition interface.

[0177] In the method provided in the present application, first, the to-be-recognized audio data is input into an audio recognition model to obtain an audio fingerprint of the to-be-recognized audio data output by the audio recognition model, then a target audio fingerprint that meets a preset condition with the audio fingerprint of the to-be-recognized audio data is determined from an audio fingerprint library, and finally an identification result is determined according to the target audio fingerprint, where the identification result includes an audio identifier corresponding to the to-be-recognized audio data. The audio recognition model is obtained based on contrast learning of first training audio data and second training audio data with the same audio identifier, and first training audio data and third training audio data with different audio identifiers. The audio recognition model obtained based on contrast learning can extract an audio fingerprint with high similarity for audio data of the same audio identifier. By extracting the audio fingerprint of the to-be-recognized audio data, and comparing the extracted audio fingerprint with each audio fingerprint extracted by the audio recognition model in the audio fingerprint library, an identification result with high accuracy is obtained, and the recognition effect of song recognition is improved.

[0178] It should be noted that, according to an aspect of the present application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the method provided in the various optional manners of the audio recognition method embodiment aspects shown in the above. Figure 2 Figure 6 It should be understood that the above only discloses preferred embodiments of the present application, and of course cannot limit the scope of the rights of the present application, so the equivalent changes made according to the claims of the present application still belong to the scope covered by the present application.

[0179] The above only discloses a preferred embodiment of the present application, and of course cannot limit the scope of the rights of the present application, and those skilled in the art can understand that all or part of the processes of the above embodiments are implemented, and the equivalent changes made according to the claims of the present application still belong to the scope covered by the present application.​​​​

Claims

1. An audio recognition method, characterized by, The method comprises: inputting to-be-identified audio data into an audio recognition model for time domain feature extraction to obtain an audio fingerprint of the to-be-identified audio data output by the audio recognition model; wherein the audio recognition model is obtained based on first training audio data and second training audio data and contrast learning of the first training audio data and third training audio data, the audio identifiers of the first training audio data and the second training audio data are the same, and the audio identifiers of the first training audio data and the third training audio data are different; determining, from an audio fingerprint library, a target audio fingerprint corresponding to an audio fingerprint of the to-be-identified audio data with a similarity greater than a set threshold; or determining, from the audio fingerprint library, a target audio fingerprint corresponding to an audio fingerprint of the to-be-identified audio data with the highest similarity; determining an identification result according to the target audio fingerprint, the identification result comprising an audio identifier corresponding to the to-be-identified audio data; wherein the audio recognition model comprises a plurality of convolution layers and a plurality of maximum pooling layers, the number of the convolution layers is greater than the number of the maximum pooling layers, and the other convolution layers except a first convolution layer in the plurality of convolution layers are alternately connected with the plurality of maximum pooling layers; the inputting to-be-identified audio data into an audio recognition model for time domain feature extraction to obtain an audio fingerprint of the to-be-identified audio data output by the audio recognition model comprises: inputting the to-be-identified audio data into the first convolution layer in the plurality of convolution layers to obtain a feature vector of the to-be-identified audio data output by the first convolution layer; inputting the feature vector of the to-be-identified audio data output by the first convolution layer into a second convolution layer in the plurality of convolution layers to obtain a feature vector of the to-be-identified audio data output by the second convolution layer; inputting the feature vector of the to-be-identified audio data output by the second convolution layer into a corresponding maximum pooling layer in the plurality of maximum pooling layers according to an alternately connecting relationship between the other convolution layers and the plurality of maximum pooling layers to obtain a feature vector of the to-be-identified audio data output by the corresponding maximum pooling layer; after inputting the feature vector of the to-be-identified audio data output by the maximum pooling layer connected with the last convolution layer into the last convolution layer, determining an audio fingerprint of the to-be-identified audio data according to the feature vector of the to-be-identified audio data output by the last convolution layer.

2. The method of claim 1, wherein, The method further comprises: training an initial audio recognition model using a training sample set to obtain target loss information; the training sample set comprises a positive sample set and a negative sample set, the positive sample set comprises the first training audio data and the second training audio data, and the negative sample set comprises the third training audio data; in a case where the target loss information meets a training end condition, determining the initial audio recognition model as the audio recognition model.

3. The method of claim 1 or 2, wherein The first training audio data is original audio data, and the second training audio data is audio data obtained by performing data enhancement processing on the first training audio data. Alternatively, The first training audio data and the second training audio data are audio data obtained by performing different data enhancement processing on original audio data corresponding to the first training audio data.

4. The method of claim 1 or 2, wherein The first training audio data is original audio data, and the second training audio data is cover audio data of the first training audio data; or The first training audio data and the second training audio data are cover audio data of original audio data corresponding to the first training audio data.

5. The method of claim 2, wherein, The training of the initial audio recognition model using the training sample set to obtain target loss information includes: inputting the training sample set into the initial audio recognition model to obtain audio fingerprints of the first training audio data, the second training audio data, and the third training audio data output by the initial audio recognition model; determining the target loss information according to a first similarity and a second similarity; the first similarity is a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data, and the second similarity is a sum of a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data and a similarity between the audio fingerprint of the first training audio data and the audio fingerprint of the third training audio data.

6. The method of claim 5, wherein, The method further includes: determining the first similarity according to a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data; determining the second similarity according to a sum of a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the second training audio data and a cosine distance between the audio fingerprint of the first training audio data and the audio fingerprint of the third training audio data.

7. The method of claim 1 or 2, wherein, The method further includes: playing original audio data identified by the audio identifier corresponding to the to-be-recognized audio data.

8. The method of claim 1 or 2, wherein, The method further includes: displaying, on an audio recognition interface, the audio identifier corresponding to the to-be-recognized audio data.

9. A computer device, comprising: The computer device includes a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the audio recognition method of any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores one or more computer programs, and the one or more computer programs are adapted to be loaded and executed by the processor to execute the audio recognition method of any one of claims 1-8.

Images