Audio processing method, device, storage medium and computer program product

Abstract

The present application discloses an audio processing method, device, storage medium and computer program product, the method comprising: constructing a sample group based on a song feature library and a noise feature library; the song feature library comprises a song feature group of multiple songs, each song feature in the song feature group is an acoustic feature of a different version of the same song segment, and the noise feature in the noise feature library is an acoustic feature of a noise segment; the sample group satisfies any of the following conditions: a positive sample and a reference sample are different song features in the same song feature group, a negative sample is a noise feature, or a negative sample is a song feature selected from other song feature groups in the song feature library; or a positive sample and a reference sample are different noise features, and a negative sample is a song feature; a song recognition model is trained based on the sample group; the trained song recognition model is used to extract fingerprint features corresponding to the audio to be recognized, so as to perform song recognition on the audio to be recognized; the noise resistance performance of the model and the accuracy of song recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an audio processing method, device, storage medium and computer program product. Background Art

[0002] In the song recognition scenario, the audio to be recognized can usually be first filtered through a noise filtering model. When the audio to be recognized is determined to be non-noise, the fingerprint features corresponding to the audio to be recognized are extracted through the trained song recognition model, and then the song recognition is performed on the audio to be recognized based on the fingerprint features corresponding to the audio to be recognized. In the existing method, it is highly dependent on the accuracy of the noise filtering model. If the noise filtering model is too strict, the recognizable audio to be recognized will be filtered out. If the screening is too loose, the audio to be recognized will enter the trained song recognition model when it is noise. In addition, the trained song recognition model used in the existing method usually has low noise resistance. If the noise filtering model is too loose, the audio to be recognized will still enter the trained song recognition model when it is noise. At this time, the accuracy of the fingerprint features corresponding to the audio to be recognized extracted by the trained song recognition model is low, and then the accuracy of song recognition based on the fingerprint features corresponding to the audio to be recognized is low. Summary of the invention

[0003] The embodiments of the present application provide an audio processing method, apparatus, device, storage medium and computer program product, which can improve the noise resistance of the trained song recognition model, improve the accuracy of the fingerprint features corresponding to the audio to be recognized, and thus improve the accuracy of song recognition.

[0004] On the one hand, an embodiment of the present application provides an audio processing method, including:

[0005] Acquire a song feature library and a noise feature library; the song feature library includes multiple song feature groups of multiple songs, each song feature in the song feature group is an acoustic feature of different versions of the same song segment of the song, and a noise feature in the noise feature library is an acoustic feature of a noise segment;

[0006] A sample group is constructed based on the song feature library and the noise feature library; the sample group includes positive samples, negative samples and reference samples, and the sample group satisfies the following first condition, second condition or third condition: the first condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a noise feature selected from the noise feature library; the second condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a song feature selected from other song feature groups in the song feature library, and the other song feature groups are song feature groups of other songs other than the song corresponding to the positive sample; the third condition is that the positive sample and the reference sample are different noise features selected from the noise feature library, and the negative sample is a song feature selected from the song feature library;

[0007] Extracting fingerprint features corresponding to the positive sample, fingerprint features corresponding to the negative sample, and fingerprint features corresponding to the reference sample through a song recognition model;

[0008] The song recognition model is trained based on the fingerprint features corresponding to the positive samples, the fingerprint features corresponding to the negative samples and the fingerprint features corresponding to the reference samples to obtain a trained song recognition model; wherein the trained song recognition model is used to extract the fingerprint features corresponding to the audio to be recognized, and the fingerprint features corresponding to the audio to be recognized are used to perform song recognition on the audio to be recognized.

[0009] On the one hand, an embodiment of the present application provides an audio processing device, including:

[0010] An acquisition unit is used to acquire a song feature library and a noise feature library; the song feature library includes a plurality of song feature groups of a plurality of songs, each song feature in the song feature group is an acoustic feature of a different version of the same song segment of the song, and a noise feature in the noise feature library is an acoustic feature of a noise segment;

[0011] A processing unit, for constructing a sample group based on the song feature library and the noise feature library; the sample group includes a positive sample, a negative sample and a reference sample, and the sample group satisfies the following first condition, second condition or third condition: the first condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a noise feature selected from the noise feature library; the second condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a song feature selected from other song feature groups in the song feature library, and the other song feature groups are song feature groups of other songs other than the song corresponding to the positive sample; the third condition is that the positive sample and the reference sample are different noise features selected from the noise feature library, and the negative sample is a song feature selected from the song feature library;

[0012] The processing unit is further used to extract the fingerprint features corresponding to the positive sample, the fingerprint features corresponding to the negative sample, and the fingerprint features corresponding to the reference sample through a song recognition model;

[0013] The processing unit is also used to train the song recognition model based on the fingerprint features corresponding to the positive samples, the fingerprint features corresponding to the negative samples, and the fingerprint features corresponding to the reference samples to obtain a trained song recognition model; wherein the trained song recognition model is used to extract the fingerprint features corresponding to the audio to be recognized, and the fingerprint features corresponding to the audio to be recognized are used to perform song recognition on the audio to be recognized.

[0014] On the one hand, an embodiment of the present application provides an audio processing device, characterized in that the audio processing device includes an input interface and an output interface, and further includes:

[0015] a processor adapted to implement one or more instructions; and,

[0016] A computer storage medium storing one or more instructions, wherein the one or more instructions are suitable for being loaded by the processor and executing the above-mentioned audio processing method.

[0017] On the one hand, an embodiment of the present application provides a computer storage medium, characterized in that computer program instructions are stored in the computer storage medium, and when the computer program instructions are executed by a processor, they are used to execute the above-mentioned audio processing method.

[0018] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer storage medium; a processor of an audio processing device reads the computer program from the computer storage medium, and the processor executes the computer program, so that the audio processing device performs the above-mentioned audio processing method.

[0019] In the embodiment of the present application, a sample group can be constructed based on a song feature library and a noise feature library, and then a song recognition model can be trained based on the sample group; the trained song recognition model is used to extract fingerprint features corresponding to the audio to be recognized, and the fingerprint features corresponding to the audio to be recognized are used to perform song recognition on the audio to be recognized. Since the constructed sample group includes positive samples, negative samples and reference samples, and the sample group satisfies the following first condition, second condition or third condition: the first condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a noise feature selected from the noise feature library; the second condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a song feature selected from other song feature groups in the song feature library, and the other song feature groups are song feature groups of other songs other than the song corresponding to the positive sample; the third condition is that the positive sample and the reference sample are different noise features selected from the noise feature library, and the negative sample is a song feature selected from the song feature library; then when training the song recognition model based on the sample group, not only It is possible to learn the association between song clips of different versions of songs, the difference between song clips of different songs, the difference between song clips and noise clips, and the association between noise clips. This can improve the noise resistance of the trained song recognition model, and improve the accuracy of the fingerprint features corresponding to the audio to be identified extracted by the trained song recognition model; especially when the audio to be identified is noise or the signal-to-noise ratio of the audio to be identified is low, the accuracy of the fingerprint features corresponding to the audio to be identified is improved. In short, the song recognition scheme of the embodiment of the present application can get rid of the dependence on the noise filtering model, and directly extract the fingerprint features corresponding to the audio to be identified through the trained song recognition model, which can not only improve the accuracy of song recognition, but also speed up the recognition process and save recognition resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1is a schematic diagram of an audio processing solution provided in an embodiment of the present application;

[0022] Figure 2 It is a flowchart of an audio processing method provided in an embodiment of the present application;

[0023] Figure 3 It is a schematic diagram of a training idea of ​​a song recognition model provided in an embodiment of the present application;

[0024] Figure 4 It is a schematic diagram of a song recognition model trained based on a song library and a noise library provided in an embodiment of the present application;

[0025] Figure 5 is a flowchart of another audio processing method provided in an embodiment of the present application;

[0026] Figure 6 It is a schematic diagram of another song recognition model trained based on a song library and a noise library provided in an embodiment of the present application;

[0027] Figure 7 is a structural diagram of an audio processing device provided in an embodiment of the present application;

[0028] Figure 8 It is a structural diagram of the audio processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0031] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology (Computer Vision, CV), speech processing technology, natural language processing technology, and machine learning (Machine Learning, ML) / deep learning (Deep Learning, DL) and other major directions. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0032] Based on the machine learning technology mentioned above, an embodiment of the present application provides an audio processing solution, which can construct a sample group based on a song feature library and a noise feature library; wherein the song feature library includes multiple song feature groups of multiple songs, each song feature in the song feature group is an acoustic feature of different versions of the same song segment of the song, and a noise feature in the noise feature library is an acoustic feature of a noise segment; the constructed sample group includes positive samples, negative samples and reference samples, and the sample group satisfies the following first condition, second condition or third condition: the first condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is selected from the noise feature library. noise feature; the second condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a song feature selected from other song feature groups in the song feature library, and the other song feature groups are song feature groups of other songs other than the song corresponding to the positive sample; the third condition is that the positive sample and the reference sample are different noise features selected from the noise feature library, and the negative sample is a song feature selected from the song feature library; after constructing the sample group, the song recognition model can be trained based on the sample group; the trained song recognition model is used to extract the fingerprint features corresponding to the audio to be recognized, and the fingerprint features corresponding to the audio to be recognized are used to perform song recognition on the audio to be recognized.

[0033] In one embodiment, the song feature library includes multiple song feature groups for multiple songs, and each song feature in the song feature group is an acoustic feature of different versions of the same song segment of the song; different versions of the same song segment can constitute a song segment group, that is, the acoustic features of each song segment included in a song segment group constitute a song feature group. For example, if there are three versions of song A, namely song A (version 1), song A (version 2), and song A (version 3), then each song segment in a song segment group corresponding to song A may include: a song segment in song A (version 1), a song segment in song A (version 2), and a song segment in song A (version 3); wherein each song segment included in the song segment group should be a different version of the same song segment. Optionally, if the lyrics of each song segment in the song segment group are the same in the corresponding song, then the song segments in the song segment group can be considered to be different versions of the same song segment; for example, for the three versions of song A, there is a song segment group in which the lyrics of each song segment in the corresponding song are all the first sentence of lyrics, there is a song segment group in which the lyrics of each song segment in the corresponding song are all the second sentence of lyrics, and so on; for another example, if there are three versions of the song "Yellow River Cantata", there can be a song segment group in which the lyrics of each song segment in the corresponding song are all "the wind is roaring", there can be another song segment group in which the lyrics of each song segment in the corresponding song are all "the horse is neighing", and so on. In other words, songs of different versions can be segmented according to the lyrics content, and song segments describing the same lyrics in different versions of songs can be grouped into a song segment group, for example, song segments describing the first sentence of lyrics can be grouped into a song segment group. It can be further known that a song can correspond to at least one song segment group. For example, if different versions of song A are segmented according to the lyrics content, the song segments of song A (version 1) are represented as: A-1.1, A-1.2, A-1.3, A-1.4, the song segments of song A (version 2) are represented as: A-2.1, A-2.2, A-2.3, A-2.4, and the song segments of song A (version 3) are represented as: A-3.1, A-3.2, A-3.3, A-3.4. Then the song segment group corresponding to song A may include: {A-1.1, A-2.1, A-3.1}, {A-1.2, A-2.2, A-3.2}, {A-1.3, A-2.3, A-3.3} and {A-1.4, A-2.4, A-3.4}.

[0034] In one embodiment, after the sample group is constructed, the song recognition model is trained based on the sample group. The fingerprint features corresponding to the positive sample, the fingerprint features corresponding to the negative sample, and the fingerprint features corresponding to the reference sample can be extracted through the song recognition model, and then the song recognition model is trained based on the fingerprint features corresponding to the positive sample, the fingerprint features corresponding to the negative sample, and the fingerprint features corresponding to the reference sample. Optionally, the song recognition model can be a neural network model, which can be selected or constructed according to specific needs; for example, a convolutional neural network model can be selected, such as a residual network (Resnet), a visual geometry group network (Visual Geometry Group, VGG), and an Inception network in a convolutional neural network model, etc., which is not limited by the embodiments of the present application. Further, the fingerprint features corresponding to each sample in the sample group extracted by the song recognition model are the features output after the song recognition model performs corresponding processing on each sample in the sample group. For example, when the song recognition model is Resnet, the fingerprint features corresponding to each sample in the sample group extracted by the song recognition model are embedded features (i.e., embedding features).

[0035] In a specific implementation, the audio processing scheme proposed in the embodiment of the present application can be executed by an audio processing device, which can be a terminal device or a server; the terminal device here can include but is not limited to: computers, smart phones, tablet computers, laptops, smart home appliances, vehicle terminals, smart wearable devices, etc.; the server here can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. Further optionally, the audio processing scheme proposed in the embodiment of the present application can also be executed individually or collaboratively by other electronic devices with computing power, which is not limited by the embodiment of the present application.

[0036] See also Figure 1 , which is a schematic diagram of an audio processing solution provided in an embodiment of the present application. The audio processing device can construct a sample group based on a song feature library and a noise feature library. After the sample group is constructed, the fingerprint features corresponding to the positive samples, the fingerprint features corresponding to the negative samples, and the fingerprint features corresponding to the reference samples in the sample group can be extracted through the song recognition model, and then the song recognition model can be trained based on the fingerprint features corresponding to the positive samples, the fingerprint features corresponding to the negative samples, and the fingerprint features corresponding to the reference samples.

[0037] It is particularly important to note that in the specific implementation of this application, object-related data, such as audio to be recognized, etc., when the embodiments of this application are applied to specific products or technologies, it is necessary to obtain the object's permission or consent, and the collection, use and processing of relevant data must comply with local laws, regulations and standards.

[0038] Based on the above audio processing solution, the present application embodiment provides an audio processing method. Figure 2 A flowchart of an audio processing method provided in an embodiment of the present application. Figure 2 The audio processing method shown can be executed by an audio processing device, and can also be executed individually or collaboratively by other electronic devices with computing power. The embodiment of the present application takes an audio processing device as an example. Figure 2 The audio processing method shown may include the following steps:

[0039] S201, obtaining a song feature library and a noise feature library.

[0040] Among them, the song feature library includes multiple song feature groups of multiple songs, each song feature in the song feature group is the acoustic feature of different versions of the same song segment of the song, and different versions of the same song segment can constitute a song segment group, that is, the acoustic features of each song segment included in a song segment group constitute a song feature group, and a noise feature in the noise feature library is the acoustic feature of a noise segment.

[0041] In one embodiment, the versions of the various song segments in any song segment group are different, that is, the various song segments in any song segment group are song segments in different versions of songs. In a specific implementation, when constructing the song segment group, the audio processing device may perform song segmentation processing on songs of different versions, and construct the song segment group based on the correspondence between the song segments in the songs of different versions.

[0042] Optionally, the lyrics content described in each song segment in any song segment group is the same in the corresponding song; then, in an optional implementation, when the audio processing device performs song segmentation processing on different versions of songs and constructs song segment groups based on the correspondence between song segments in different versions of songs, it can perform song segmentation processing on different versions of songs according to the lyrics content, and group the song segments in different versions of songs that describe the same lyrics content into a song segment group.

[0043] Optionally, the duration of each song segment in any song segment group may correspond to the same; then, in an optional implementation, when the audio processing device performs song segmentation processing on songs of different versions and constructs song segment groups based on the correspondence between song segments in songs of different versions, the different versions of songs may be segmented according to preset segment durations, and song segments at corresponding positions in songs of different versions may be grouped into a song segment group; wherein the preset segment duration may be set according to specific needs, for example, it may be set to 3s, 5s, etc., and the reference starting position may be set according to specific needs when performing song segmentation processing on songs of different versions, for example, the reference starting position may be set to the starting position of the song, the reference starting position may be set to the starting position of the first line of lyrics in the song, the reference starting position may be set to the starting position of each line of lyrics in the song, and so on.

[0044] In one embodiment, each song feature in any song feature group is an acoustic feature of each song segment in the corresponding song segment group; in a specific implementation, the audio processing device can extract the acoustic features of each song segment in the song segment group as song features, and construct a song feature group corresponding to the song segment group based on each song feature; wherein the acoustic features can be selected according to specific needs, for example, the acoustic features can be Mel-frequency cepstral coefficient features, fundamental frequency features, resonance peak features, etc., which are not limited in the embodiments of the present application. Further, the audio processing device obtains a song feature library, which can include: performing song segmentation processing on songs of different versions in the song library, and constructing a song segment group based on the correspondence between song segments in different versions of songs; extracting the acoustic features of each song segment in the song segment group as song features, and constructing a song feature group corresponding to the song segment group based on each song feature, and then constructing a song feature library based on each song feature group.

[0045] In one embodiment, the audio processing device obtains a noise feature library, which may include: extracting acoustic features of each noise segment in the noise library as noise features, and constructing a noise feature library based on each noise feature; optionally, the noise segment may be a pure scene noise or a song segment with an extremely low signal-to-noise ratio; further optionally, the duration of the noise segment may be the same as the duration of the song segment.

[0046] S202, constructing a sample group based on a song feature library and a noise feature library.

[0047] Among them, the sample group includes positive samples, negative samples and reference samples, and the sample group satisfies the following first condition, second condition or third condition: the first condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a noise feature selected from a noise feature library; the second condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a song feature selected from other song feature groups in the song feature library, and the other song feature groups are song feature groups of other songs other than the song corresponding to the positive sample; the third condition is that the positive sample and the reference sample are different noise features selected from the noise feature library, and the negative sample is a song feature selected from the song feature library; wherein the number of reference samples can be set to one or more according to specific needs.

[0048] Furthermore, since the second condition indicates: the positive sample and the reference sample are different song features selected from the same song feature group, the negative sample is a song feature selected from other song feature groups in the song feature library, and the other song feature groups are song feature groups of other songs other than the songs corresponding to the positive sample; that is, the songs indicated by each song feature in the song feature group corresponding to the negative sample are different from the songs indicated by each song feature in the song feature group corresponding to the positive sample and the reference sample. For example, if the number of reference samples is 2, the song library includes song A (version 1), song A (version 2), song A (version 3), song B (version 1), song B (version 2), song B (version 3), and song B (version 4); song A corresponds to 4 song clip groups, which further correspond to 4 song feature groups, and song B corresponds to 3 song clip groups, which further correspond to 3 song feature groups; if the 4 song feature groups corresponding to song A are: song feature group A1, song feature group A2, song feature group A3, and song feature group A4, and the 3 song feature groups corresponding to song B are: song feature group B1, song feature group B2, and song feature group B3; if when constructing the sample group, the positive sample and the reference sample are different song features selected from the song feature group A1, then the negative sample can be a song feature selected from any song feature group corresponding to other songs in the song feature library except song A, for example, the negative sample can be a song feature selected from song feature groups B1, B2, or B3 corresponding to song B.

[0049] S203, extracting fingerprint features corresponding to the positive sample, fingerprint features corresponding to the negative sample, and fingerprint features corresponding to the reference sample through the song recognition model.

[0050] In one embodiment, the audio processing device extracts the fingerprint features corresponding to any sample in the sample group through a song recognition model, which may include: performing feature conversion processing on the sample through the song recognition model to obtain the fingerprint features corresponding to the sample; wherein the relevant process of performing feature conversion processing on the sample through the song recognition model is adapted to the specifically selected song recognition model.

[0051] S204, training the song recognition model based on the fingerprint features corresponding to the positive samples, the fingerprint features corresponding to the negative samples, and the fingerprint features corresponding to the reference samples to obtain a trained song recognition model.

[0052] Among them, the trained song recognition model is used to extract fingerprint features corresponding to the audio to be recognized, and the fingerprint features corresponding to the audio to be recognized are used to perform song recognition on the audio to be recognized.

[0053] In one embodiment, the audio processing device trains a song recognition model based on fingerprint features corresponding to positive samples, fingerprint features corresponding to negative samples, and fingerprint features corresponding to reference samples, which may include: determining a target fingerprint feature based on the fingerprint features corresponding to the reference sample; adjusting model parameters of the song recognition model in a direction of reducing the feature distance between the fingerprint features corresponding to the positive sample and the target fingerprint feature, and in a direction of increasing the feature distance between the fingerprint features corresponding to the negative sample and the target fingerprint feature; wherein the feature distance may be selected according to specific needs, for example, Euclidean distance, cosine distance, etc. may be selected.

[0054] In a specific implementation, when the sample group includes a reference sample, the audio processing device determines the target fingerprint feature based on the fingerprint feature corresponding to the reference sample, which may include: taking the fingerprint feature corresponding to the reference sample as the target fingerprint feature; when the sample group includes multiple reference samples, the audio processing device determines the target fingerprint feature based on the fingerprint feature corresponding to the reference sample, which may include: averaging the fingerprint features corresponding to each reference sample to obtain the target fingerprint feature; the audio processing device averages the fingerprint features corresponding to each reference sample to obtain the target fingerprint feature in order to take the fingerprint features corresponding to each reference sample in the feature space as the target fingerprint feature.

[0055] Further, the audio processing device may adjust the model parameters of the song recognition model in a direction of reducing the feature distance between the fingerprint feature corresponding to the positive sample and the target fingerprint feature, and in a direction of increasing the feature distance between the fingerprint feature corresponding to the negative sample and the target fingerprint feature; see Figure 3, which is a schematic diagram of a training idea of ​​a song recognition model provided in an embodiment of the present application. As shown in the mark 301, it can be the feature distance between the fingerprint feature corresponding to the positive sample obtained by the song recognition model and the target fingerprint feature, and as shown in the mark 302, it can be the feature distance between the fingerprint feature corresponding to the negative sample and the target fingerprint feature. Based on the above description, an embodiment of the present application provides a loss function to adjust the model parameters of the song recognition model in the direction of reducing the feature distance between the fingerprint feature corresponding to the positive sample and the target fingerprint feature, and increasing the feature distance between the fingerprint feature corresponding to the negative sample and the target fingerprint feature by adjusting the loss value of the loss function; the loss function can be shown by the following formula 1.1:

[0056]

[0057] Among them, f a represents the target fingerprint feature, f p represents the fingerprint feature corresponding to the positive sample, f n Indicates the fingerprint features corresponding to the negative samples; Represents the feature distance between the fingerprint feature corresponding to the positive sample and the target fingerprint feature, Indicates the characteristic distance between the fingerprint feature corresponding to the negative sample and the target fingerprint feature; α is a constant that can be set according to specific needs, and "+" means that when the value in the bracket is less than 0, it is taken as 0, and when it is greater than 0, it remains unchanged; the audio processing device can adjust the model parameters of the song recognition model in the direction of reducing the loss value of the loss function. Among them, the target fingerprint feature can be shown by the following formula 1.2:

[0058]

[0059] Where M represents the number of reference samples in the sample group, j is the independent variable, represents the jth reference sample in the sample group, Represents the fingerprint feature corresponding to the jth reference sample in the sample group.

[0060] Furthermore, when the song recognition model is trained in batches based on the constructed sample groups, the loss function can be shown by the following formula 2.1:

[0061]

[0062] Where N represents the number of sample groups used in each batch, that is, batch size; f i a represents the target fingerprint feature determined based on the i-th sample group among N sample groups, f i prepresents the fingerprint feature corresponding to the positive sample in the i-th sample group among N sample groups, f i n Represents the fingerprint feature corresponding to the negative sample in the i-th sample group in the N sample groups; the audio processing device can adjust the model parameters of the song recognition model in the direction of reducing the loss value of the loss function.

[0063] See also Figure 4 , which is a schematic diagram of a song recognition model training based on a song library and a noise library provided in an embodiment of the present application; the audio processing device can perform song segmentation processing on songs of different versions in the song library, and construct a song segment group based on the correspondence between song segments in different versions of songs; extract the acoustic features of each song segment in the song segment group as song features, and construct a song feature group corresponding to the song segment group based on each song feature, and then construct a song feature library based on each song feature group; and the audio processing device can extract the acoustic features of each noise segment in the noise library as noise features, and construct a noise feature library based on each noise feature. Further, the audio processing device can construct a sample group based on the song feature library and the noise feature library. After the sample group is constructed, the fingerprint features corresponding to the positive sample, the fingerprint features corresponding to the negative sample, and the fingerprint features corresponding to the reference sample in the sample group can be extracted through the song recognition model, and then the song recognition model can be trained based on the fingerprint features corresponding to the positive sample, the fingerprint features corresponding to the negative sample, and the fingerprint features corresponding to the reference sample.

[0064] In the embodiment of the present application, a sample group can be constructed based on a song feature library and a noise feature library, and then a song recognition model can be trained based on the sample group; the trained song recognition model is used to extract fingerprint features corresponding to the audio to be recognized, and the fingerprint features corresponding to the audio to be recognized are used to perform song recognition on the audio to be recognized. Since the constructed sample group includes positive samples, negative samples and reference samples, and the sample group satisfies the following first condition, second condition or third condition: the first condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a noise feature selected from the noise feature library; the second condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a song feature selected from other song feature groups in the song feature library, and the other song feature groups are song feature groups of other songs other than the song corresponding to the positive sample; the third condition is that the positive sample and the reference sample are different noise features selected from the noise feature library, and the negative sample is a song feature selected from the song feature library; then when training the song recognition model based on the sample group, not only It is possible to learn the association between song clips of different versions of songs, the difference between song clips of different songs, the difference between song clips and noise clips, and the association between noise clips. This can improve the noise resistance of the trained song recognition model, and improve the accuracy of the fingerprint features corresponding to the audio to be identified extracted by the trained song recognition model; especially when the audio to be identified is noise or the signal-to-noise ratio of the audio to be identified is low, the accuracy of the fingerprint features corresponding to the audio to be identified is improved. In short, the song recognition scheme of the embodiment of the present application can get rid of the dependence on the noise filtering model, and directly extract the fingerprint features corresponding to the audio to be identified through the trained song recognition model, which can not only improve the accuracy of song recognition, but also speed up the recognition process and save recognition resources.

[0065] Based on the above-mentioned related embodiments of the audio processing method, the present application embodiment provides another audio processing method. Figure 5 A flowchart of another audio processing method provided in an embodiment of the present application. Figure 5 The audio processing method shown can be executed by an audio processing device, and can also be executed individually or collaboratively by other electronic devices with computing power. The embodiment of the present application takes an audio processing device as an example. Figure 5 The audio processing method shown may include the following steps:

[0066] S501, obtaining a song feature library and a noise feature library.

[0067] Among them, the song feature library includes multiple song feature groups of multiple songs, each song feature in the song feature group is the acoustic feature of different versions of the same song segment of the song, and different versions of the same song segment can constitute a song segment group, that is, the acoustic features of each song segment included in a song segment group constitute a song feature group, and a noise feature in the noise feature library is the acoustic feature of a noise segment; the relevant process of step S501 is similar to the relevant process of the above-mentioned step S201, and will not be repeated here.

[0068] S502: Select multiple noise features from the noise feature library, determine a newly added noise feature based on the multiple noise features, and add the newly added noise feature to the noise feature library to update the noise feature library.

[0069] In one embodiment, the number of noise features selected from the noise feature library can be set according to specific needs; the audio processing device determines the newly added noise features based on multiple noise features, which can include: averaging the multiple noise features to obtain the newly added noise features; the audio processing device averages the multiple noise features in order to use the center point of the multiple noise features in the feature space as the newly added noise features; updating the noise feature library based on the newly added noise features can make the feature space expression of the noise features more complete, and reduce the possibility that the audio to be identified cannot be identified due to the lack of noise features when the audio to be identified is subsequently identified as a song. Further optionally, multiple noise features can be selected from the noise feature library multiple times, and newly added noise features can be determined based on the multiple noise features, so as to iteratively update the noise feature library based on the newly added noise features, and the number of iterative updates of the noise feature library can be set according to specific needs.

[0070] S503, constructing a sample group based on the song feature library and the updated noise feature library.

[0071] Among them, the sample group includes positive samples, negative samples and reference samples, and the sample group satisfies the following first condition, second condition or third condition: the first condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a noise feature selected from an updated noise feature library; the second condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a song feature selected from other song feature groups in the song feature library, and the other song feature groups are song feature groups of other songs other than the song corresponding to the positive sample; the third condition is that the positive sample and the reference sample are different noise features selected from the updated noise feature library, and the negative sample is a song feature selected from the song feature library; wherein the number of reference samples can be set to one or more according to specific needs.

[0072] S504, extracting fingerprint features corresponding to the positive sample, fingerprint features corresponding to the negative sample, and fingerprint features corresponding to the reference sample through the song recognition model.

[0073] S505, training the song recognition model based on the fingerprint features corresponding to the positive samples, the fingerprint features corresponding to the negative samples, and the fingerprint features corresponding to the reference samples to obtain a trained song recognition model.

[0074] Among them, the relevant processes from step S503 to step S505 are similar to the relevant processes from step S202 to step S204 mentioned above, and will not be repeated here.

[0075] See also Figure 6 , which is another schematic diagram of a song recognition model trained based on a song library and a noise library provided in an embodiment of the present application; the audio processing device can perform song segmentation processing on songs of different versions in the song library, and construct a song segment group based on the correspondence between song segments in different versions of songs; extract the acoustic features of each song segment in the song segment group as song features, and construct a song feature group corresponding to the song segment group based on each song feature, and then construct a song feature library based on each song feature group; and the audio processing device can extract the acoustic features of each noise segment in the noise library as noise features, and construct a noise feature library based on each noise feature. Select multiple noise features from the noise feature library, determine a newly added noise feature based on the multiple noise features, and add the newly added noise feature to the noise feature library to update the noise feature library. Furthermore, the audio processing device can construct a sample group based on the song feature library and the updated noise feature library. After the sample group is constructed, the fingerprint features corresponding to the positive samples, the fingerprint features corresponding to the negative samples, and the fingerprint features corresponding to the reference samples in the sample group can be extracted through the song recognition model, and then the song recognition model can be trained based on the fingerprint features corresponding to the positive samples, the fingerprint features corresponding to the negative samples, and the fingerprint features corresponding to the reference samples.

[0076] In one embodiment, the trained song recognition model is used to extract fingerprint features corresponding to the audio to be recognized, and the fingerprint features corresponding to the audio to be recognized are used to perform song recognition on the audio to be recognized; wherein, the audio to be recognized can be the audio that needs to be recognized for song recognition, for example, it can be the audio recorded by the object, and the segment length of the audio to be recognized can be the same as the segment length of the song segment. In a specific implementation, the audio processing device can extract the fingerprint features corresponding to each song feature in the song feature library and the fingerprint features corresponding to each noise feature in the noise feature library through the trained song recognition model; obtain the acoustic features of the audio to be recognized, and perform feature conversion processing on the acoustic features of the audio to be recognized through the trained song recognition model to obtain the fingerprint features corresponding to the audio to be recognized; based on the feature distance between the fingerprint features corresponding to the audio to be recognized and the fingerprint features corresponding to each song feature, and the feature distance between the fingerprint features corresponding to the audio to be recognized and the fingerprint features corresponding to each noise feature, perform song recognition on the audio to be recognized to obtain a song recognition result. Among them, the related process of the audio processing device extracting fingerprint features corresponding to song features, extracting fingerprint features corresponding to noise features, and extracting fingerprint features corresponding to the audio to be identified through the trained song recognition model is similar to the related process of the audio processing device extracting the fingerprint features corresponding to any sample in the sample group through the song recognition model, and will not be repeated here.

[0077] Further, the audio processing device performs song recognition on the audio to be recognized based on the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint feature corresponding to each song feature, and the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint feature corresponding to each noise feature, and obtains the song recognition result, which may include: determining the minimum feature distance from the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint feature corresponding to each song feature, and the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint feature corresponding to each noise feature; if the minimum feature distance corresponds to the song feature, the song corresponding to the song feature is determined as the song recognition result; if the minimum feature distance corresponds to the noise feature, the recognition failure prompt information is determined as the song recognition result. Among them, the feature distance can be selected according to specific needs, for example, the Euclidean distance, the Manhattan distance, etc. can be selected; the recognition failure prompt information is used to prompt the failure of song recognition for the audio to be recognized, and the content and expression form of the recognition failure prompt information can be set according to specific needs, for example, it can be text information, voice information, etc., and the embodiment of the present application is not limited.

[0078] In another feasible implementation, the audio processing device performs song recognition on the audio to be recognized based on the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to each song feature, and the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to each noise feature, to obtain a song recognition result, which may include: determining feature similarity based on the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint feature corresponding to any song feature, and determining feature similarity based on the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint feature corresponding to any noise feature; determining the maximum feature similarity from the feature similarities between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to each song feature, and the feature similarities between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to each noise feature; if the maximum feature similarity corresponds to a song feature, the song corresponding to the song feature is determined as the song recognition result; if the maximum feature similarity corresponds to a noise feature, the recognition failure prompt information is determined as the song recognition result; wherein the feature similarity is negatively correlated with the feature distance, that is, the smaller the feature distance, the greater the feature similarity.

[0079] In one embodiment, based on the trained song recognition model, the fingerprint features corresponding to each song feature in the song feature library and the fingerprint features corresponding to each noise feature in the noise feature library can be extracted to construct a first retrieval library, so that the audio processing device can perform song recognition on the audio to be recognized based on the fingerprint features corresponding to the audio to be recognized and the fingerprint features in the first retrieval library to obtain a song recognition result. It is easy to understand that the fingerprint features in the first retrieval library can be expanded based on the method of constructing the above-mentioned first retrieval library. For example, when a new song is recorded in the song library, the new song can be processed by song segmentation. For any song segment obtained after the song segmentation, the acoustic features of the song segment can be converted by the trained song recognition model to obtain the corresponding fingerprint feature, and the fingerprint feature is added to the first retrieval library.

[0080] In one embodiment, since the noise resistance performance of the trained song recognition model is improved, the trained song recognition model can be used in a post-processing process after an exact match fails in the song recognition process. Generally speaking, the song recognition process in this case can be divided into two stages; the first stage is the precise matching stage, which can extract the recognition features of the song segments corresponding to each song feature in the song feature library, and extract the recognition features of the audio to be recognized; the recognition features of the audio to be recognized are used to indicate the changes in the spectrum peaks of the audio to be recognized; the recognition features of the audio to be recognized are feature matched with the recognition features of each song segment to obtain a feature matching result; if the feature matching result indicates a successful match, the song corresponding to the successfully matched song segment is determined as the song recognition result; if the feature matching result indicates a failed match, the second stage can be entered; the second stage is to extract the fingerprint features corresponding to each song feature in the song feature library and the fingerprint features corresponding to each noise feature in the noise feature library through the trained song recognition model; obtain the acoustic features of the audio to be recognized, and perform feature conversion processing on the acoustic features of the audio to be recognized through the trained song recognition model to obtain the fingerprint features corresponding to the audio to be recognized; based on the feature distance between the fingerprint features corresponding to the audio to be recognized and the fingerprint features corresponding to each song feature, and the feature distance between the fingerprint features corresponding to the audio to be recognized and the fingerprint features corresponding to each noise feature, song recognition is performed on the audio to be recognized to obtain a song recognition result. Among them, the recognition features of the audio to be recognized are used to indicate the changes in the spectrum peaks of the audio to be recognized, and specifically can be features constructed based on the spectrum peak sequence of the audio to be recognized; the trained song recognition model is used in the post-processing process after the failure of precise matching in the song recognition process, so that the song recognition result of the audio to be recognized can be obtained after the failure of precise matching, which can improve the recall rate of song recognition.

[0081] Furthermore, a second search library can be constructed based on the extracted identification features of the song segments corresponding to each song feature in the song feature library, so that the audio processing device can perform song recognition on the audio to be recognized based on the identification features of the audio to be recognized and the identification features in the second search library, as well as based on the fingerprint features corresponding to the audio to be recognized and the fingerprint features in the first search library, to obtain a song recognition result. It is easy to understand that the identification features in the second search library can be expanded based on the method for constructing the second search library. For example, when a new song is recorded in the song library, the new song can be segmented. For any song segment obtained after the song segmentation process, the identification features of the song segment can be extracted and added to the second search library.

[0082] In an embodiment of the present application, after obtaining a song feature library and a noise feature library, multiple noise features can be selected from the noise feature library, and newly added noise features can be determined based on the multiple noise features, and the newly added noise features can be added to the noise feature library to update the noise feature library; then, a sample group can be constructed based on the song feature library and the updated noise feature library, and a song recognition model can be trained based on the sample group; the feature space expression of the noise feature can be made more complete, reducing the possibility of failing to perform song recognition on the audio to be recognized due to the lack of noise features when performing song recognition on the audio to be recognized in the subsequent process. In the process of performing song recognition on the audio to be recognized, the fingerprint features corresponding to each song feature in the song feature library and the fingerprint features corresponding to each noise feature in the noise feature library can be extracted through the trained song recognition model, and then based on the feature distance between the fingerprint features corresponding to the audio to be recognized and the fingerprint features corresponding to each song feature, and the feature distance between the fingerprint features corresponding to the audio to be recognized and the fingerprint features corresponding to each noise feature, song recognition is performed on the audio to be recognized to obtain a song recognition result; a retrieval library is constructed based on the fingerprint features corresponding to each song feature in the song feature library and the fingerprint features corresponding to each noise feature in the noise feature library, so that even when the audio to be recognized is noise, it can be accurately judged that the audio to be recognized is noise and it is not recalled, and the recognition failure information is determined as the song recognition result, which can reduce the noise false recall problem caused in the song recognition process of the audio to be recognized when the retrieval library is constructed only based on the fingerprint features corresponding to each song feature in the song feature library, that is, when the audio to be recognized is noise, the song recognition result is still determined as a certain song.

[0083] Based on the above-mentioned embodiments related to the audio processing method, the present application embodiment provides an audio processing device. Figure 7 , is a structural diagram of an audio processing device provided in an embodiment of the present application, and the audio processing device may include an acquisition unit 701 and a processing unit 702. Figure 7 The audio processing device shown can run the following units:

[0084] The acquisition unit 701 is used to acquire a song feature library and a noise feature library; the song feature library includes multiple song feature groups of multiple songs, each song feature in the song feature group is an acoustic feature of different versions of the same song segment of the song, and a noise feature in the noise feature library is an acoustic feature of a noise segment;

[0085] The processing unit 702 is used to construct a sample group based on the song feature library and the noise feature library; the sample group includes a positive sample, a negative sample and a reference sample, and the sample group satisfies the following first condition, second condition or third condition: the first condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a noise feature selected from the noise feature library; the second condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a song feature selected from other song feature groups in the song feature library, and the other song feature groups are song feature groups of other songs other than the song corresponding to the positive sample; the third condition is that the positive sample and the reference sample are different noise features selected from the noise feature library, and the negative sample is a song feature selected from the song feature library;

[0086] The processing unit 702 is further configured to extract fingerprint features corresponding to the positive sample, fingerprint features corresponding to the negative sample, and fingerprint features corresponding to the reference sample through a song recognition model;

[0087] The processing unit 702 is also used to train the song recognition model based on the fingerprint features corresponding to the positive samples, the fingerprint features corresponding to the negative samples, and the fingerprint features corresponding to the reference samples to obtain a trained song recognition model; wherein the trained song recognition model is used to extract the fingerprint features corresponding to the audio to be recognized, and the fingerprint features corresponding to the audio to be recognized are used to perform song recognition on the audio to be recognized.

[0088] In one embodiment, the processing unit 702 is further configured to:

[0089] Selecting a plurality of noise features from the noise feature library, determining a newly added noise feature based on the plurality of noise features, and adding the newly added noise feature to the noise feature library to update the noise feature library;

[0090] When the processing unit 702 constructs a sample group based on the song feature library and the noise feature library, the processing unit 702 specifically performs the following operations:

[0091] The sample group is constructed based on the song feature library and the updated noise feature library.

[0092] In one embodiment, when the processing unit 702 determines the newly added noise feature based on the multiple noise features, it specifically performs the following operations:

[0093] The multiple noise features are averaged to obtain the newly added noise feature.

[0094] In one embodiment, when the processing unit 702 trains the song recognition model based on the fingerprint features corresponding to the positive sample, the fingerprint features corresponding to the negative sample, and the fingerprint features corresponding to the reference sample, the processing unit 702 specifically performs the following operations:

[0095] Determining a target fingerprint feature based on the fingerprint feature corresponding to the reference sample;

[0096] The model parameters of the song recognition model are adjusted in a direction of reducing the feature distance between the fingerprint feature corresponding to the positive sample and the target fingerprint feature, and in a direction of increasing the feature distance between the fingerprint feature corresponding to the negative sample and the target fingerprint feature.

[0097] In one embodiment, the sample group includes a plurality of reference samples, and when the processing unit 702 determines the target fingerprint feature based on the fingerprint features corresponding to the reference samples, the processing unit 702 specifically performs the following operations:

[0098] The fingerprint features corresponding to the reference samples are averaged to obtain the target fingerprint features.

[0099] In one embodiment, the processing unit 702 is further configured to:

[0100] Extracting fingerprint features corresponding to each song feature in the song feature library and fingerprint features corresponding to each noise feature in the noise feature library through the trained song recognition model;

[0101] Acquire the acoustic features of the audio to be identified, and perform feature conversion processing on the acoustic features of the audio to be identified through the trained song recognition model to obtain fingerprint features corresponding to the audio to be identified;

[0102] Based on the feature distance between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the various song features, as well as the feature distance between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the various noise features, song identification is performed on the audio to be identified to obtain a song identification result.

[0103] In one embodiment, the processing unit 702 performs song recognition on the audio to be recognized based on the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to the various song features, and the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to the various noise features, and when obtaining the song recognition result, specifically performs the following operations:

[0104] Determine a minimum feature distance from the feature distances between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the respective song features, and the feature distances between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the respective noise features;

[0105] If the minimum feature distance corresponds to a song feature, the song corresponding to the song feature is determined as the song recognition result;

[0106] If the minimum feature distance corresponds to a noise feature, the recognition failure prompt information is determined as the song recognition result.

[0107] In one embodiment, the processing unit 702 performs song recognition on the audio to be recognized based on the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to the various song features, and the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to the various noise features, and when obtaining the song recognition result, specifically performs the following operations:

[0108] Determine the feature similarity based on the feature distance between the fingerprint feature corresponding to the audio to be identified and the fingerprint feature corresponding to any song feature, and determine the feature similarity based on the feature distance between the fingerprint feature corresponding to the audio to be identified and the fingerprint feature corresponding to any noise feature; the feature similarity is negatively correlated with the feature distance;

[0109] Determine the maximum feature similarity from the feature similarities between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the respective song features, and the feature similarities between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the respective noise features;

[0110] If the maximum feature similarity corresponds to a song feature, the song corresponding to the song feature is determined as the song recognition result;

[0111] If the maximum feature similarity corresponds to a noise feature, the recognition failure prompt information is determined as the song recognition result.

[0112] In one embodiment, the processing unit 702 is further configured to:

[0113] Extracting identification features of song segments corresponding to each song feature in the song feature library, and extracting identification features of the audio to be identified; the identification features of the audio to be identified are used to indicate changes in the spectrum peaks of the audio to be identified;

[0114] Perform feature matching processing on the identification features of the audio to be identified and the identification features of each song segment to obtain a feature matching result;

[0115] If the feature matching result indicates a successful match, the song corresponding to the successfully matched song segment is determined as the song recognition result;

[0116] If the feature matching result indicates a match failure, the fingerprint features corresponding to each song feature in the song feature library and the fingerprint features corresponding to each noise feature in the noise feature library are extracted through the trained song recognition model.

[0117] According to one embodiment of the present application, Figure 2 as well as Figure 5 The various steps involved in the audio processing method shown can be performed by Figure 7 The audio processing device shown in the figure is executed by each unit. For example, Figure 2 Step S201 shown can be performed by Figure 7 The acquisition unit 701 in the audio processing device shown in the figure is used to execute, Figure 2 Steps S202 to S204 shown in FIG. Figure 7 The processing unit 702 in the audio processing device shown in the figure is used for execution. Figure 5 Step S501 shown can be performed by Figure 7 The acquisition unit 701 in the audio processing device shown in the figure is used to execute, Figure 5 Steps S502 to S505 shown in FIG. Figure 7 The processing unit 702 in the audio processing device shown is executed.

[0118] According to another embodiment of the present application, Figure 7 The various units in the audio processing device shown can be separately or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple smaller units in function to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the audio processing device divided based on logical functions can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0119] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 2 as well as Figure 5A computer program (including program code) for each step involved in the corresponding method shown in FIG. Figure 7 The audio processing device shown in and the audio processing method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer-readable storage medium, and loaded into the above-mentioned computing device through the computer-readable storage medium and run therein.

[0120] In the embodiment of the present application, a sample group can be constructed based on a song feature library and a noise feature library, and then a song recognition model can be trained based on the sample group; the trained song recognition model is used to extract fingerprint features corresponding to the audio to be recognized, and the fingerprint features corresponding to the audio to be recognized are used to perform song recognition on the audio to be recognized. Since the constructed sample group includes positive samples, negative samples and reference samples, and the sample group satisfies the following first condition, second condition or third condition: the first condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a noise feature selected from the noise feature library; the second condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a song feature selected from other song feature groups in the song feature library, and the other song feature groups are song feature groups of other songs other than the song corresponding to the positive sample; the third condition is that the positive sample and the reference sample are different noise features selected from the noise feature library, and the negative sample is a song feature selected from the song feature library; then when training the song recognition model based on the sample group, not only It is possible to learn the association between song clips of different versions of songs, the difference between song clips of different songs, the difference between song clips and noise clips, and the association between noise clips. This can improve the noise resistance of the trained song recognition model, and improve the accuracy of the fingerprint features corresponding to the audio to be identified extracted by the trained song recognition model; especially when the audio to be identified is noise or the signal-to-noise ratio of the audio to be identified is low, the accuracy of the fingerprint features corresponding to the audio to be identified is improved. In short, the song recognition scheme of the embodiment of the present application can get rid of the dependence on the noise filtering model, and directly extract the fingerprint features corresponding to the audio to be identified through the trained song recognition model, which can not only improve the accuracy of song recognition, but also speed up the recognition process and save recognition resources.

[0121] Based on the above-mentioned audio processing method and audio processing device embodiments, the present application also provides an audio processing device. Figure 8 , is a structural diagram of an audio processing device provided in an embodiment of the present application. Figure 8The audio processing device shown may include at least a processor 801, an input interface 802, an output interface 803, and a computer storage medium 804. The processor 801, the input interface 802, the output interface 803, and the computer storage medium 804 may be connected via a bus or other means.

[0122] The computer storage medium 804 can be stored in the memory of the audio processing device. The computer storage medium 804 is used to store a computer program. The computer program includes program instructions. The processor 801 is used to execute the program instructions stored in the computer storage medium 804. The processor 801 (or CPU (Central Processing Unit)) is the computing core and control core of the audio processing device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the above-mentioned audio processing method flow or corresponding functions.

[0123] The embodiment of the present application also provides a computer storage medium (Memory), which is a memory device in an audio processing device for storing programs and data. It is understandable that the computer storage medium here can include both the built-in storage medium in the terminal and the extended storage medium supported by the terminal. The computer storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor 801 are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed random access memory (RAM) memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer storage medium located away from the aforementioned processor.

[0124] In one embodiment, the processor 801 may load and execute one or more instructions stored in the computer storage medium to implement the above-mentioned Figure 2 as well as Figure 5 In the corresponding steps of the method in the audio processing method embodiment, in a specific implementation, one or more instructions in the computer storage medium are loaded by the processor 801 and execute the following steps:

[0125] Acquire a song feature library and a noise feature library; the song feature library includes multiple song feature groups of multiple songs, each song feature in the song feature group is an acoustic feature of different versions of the same song segment of the song, and a noise feature in the noise feature library is an acoustic feature of a noise segment;

[0126] A sample group is constructed based on the song feature library and the noise feature library; the sample group includes positive samples, negative samples and reference samples, and the sample group satisfies the following first condition, second condition or third condition: the first condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a noise feature selected from the noise feature library; the second condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a song feature selected from other song feature groups in the song feature library, and the other song feature groups are song feature groups of other songs other than the song corresponding to the positive sample; the third condition is that the positive sample and the reference sample are different noise features selected from the noise feature library, and the negative sample is a song feature selected from the song feature library;

[0127] Extracting fingerprint features corresponding to the positive sample, fingerprint features corresponding to the negative sample, and fingerprint features corresponding to the reference sample through a song recognition model;

[0128] The song recognition model is trained based on the fingerprint features corresponding to the positive samples, the fingerprint features corresponding to the negative samples and the fingerprint features corresponding to the reference samples to obtain a trained song recognition model; wherein the trained song recognition model is used to extract the fingerprint features corresponding to the audio to be recognized, and the fingerprint features corresponding to the audio to be recognized are used to perform song recognition on the audio to be recognized.

[0129] In one embodiment, the processor 801 is further configured to:

[0130] Selecting a plurality of noise features from the noise feature library, determining a newly added noise feature based on the plurality of noise features, and adding the newly added noise feature to the noise feature library to update the noise feature library;

[0131] When the processor 801 constructs a sample group based on the song feature library and the noise feature library, the processor 801 specifically performs the following operations:

[0132] The sample group is constructed based on the song feature library and the updated noise feature library.

[0133] In one embodiment, when the processor 801 determines the newly added noise feature based on the multiple noise features, it specifically performs the following operations:

[0134] The multiple noise features are averaged to obtain the newly added noise feature.

[0135] In one embodiment, when the processor 801 trains the song recognition model based on the fingerprint features corresponding to the positive sample, the fingerprint features corresponding to the negative sample, and the fingerprint features corresponding to the reference sample, the processor 801 specifically performs the following operations:

[0136] Determining a target fingerprint feature based on the fingerprint feature corresponding to the reference sample;

[0137] The model parameters of the song recognition model are adjusted in a direction of reducing the feature distance between the fingerprint feature corresponding to the positive sample and the target fingerprint feature, and in a direction of increasing the feature distance between the fingerprint feature corresponding to the negative sample and the target fingerprint feature.

[0138] In one embodiment, the sample group includes a plurality of reference samples, and when the processor 801 determines the target fingerprint feature based on the fingerprint features corresponding to the reference samples, the processor 801 specifically performs the following operations:

[0139] The fingerprint features corresponding to the reference samples are averaged to obtain the target fingerprint features.

[0140] In one embodiment, the processor 801 is further configured to:

[0141] Extracting fingerprint features corresponding to each song feature in the song feature library and fingerprint features corresponding to each noise feature in the noise feature library through the trained song recognition model;

[0142] Acquire the acoustic features of the audio to be identified, and perform feature conversion processing on the acoustic features of the audio to be identified through the trained song recognition model to obtain fingerprint features corresponding to the audio to be identified;

[0143] Based on the feature distance between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the various song features, as well as the feature distance between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the various noise features, song identification is performed on the audio to be identified to obtain a song identification result.

[0144] In one embodiment, the processor 801 performs song recognition on the audio to be recognized based on the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to the various song features, and the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to the various noise features, and when obtaining the song recognition result, specifically performs the following operations:

[0145] Determine a minimum feature distance from the feature distances between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the respective song features, and the feature distances between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the respective noise features;

[0146] If the minimum feature distance corresponds to a song feature, the song corresponding to the song feature is determined as the song recognition result;

[0147] If the minimum feature distance corresponds to a noise feature, the recognition failure prompt information is determined as the song recognition result.

[0148] In one embodiment, the processor 801 performs song recognition on the audio to be recognized based on the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to the various song features, and the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to the various noise features, and when obtaining the song recognition result, specifically performs the following operations:

[0149] Determine the feature similarity based on the feature distance between the fingerprint feature corresponding to the audio to be identified and the fingerprint feature corresponding to any song feature, and determine the feature similarity based on the feature distance between the fingerprint feature corresponding to the audio to be identified and the fingerprint feature corresponding to any noise feature; the feature similarity is negatively correlated with the feature distance;

[0150] Determine the maximum feature similarity from the feature similarities between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the respective song features, and the feature similarities between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the respective noise features;

[0151] If the maximum feature similarity corresponds to a song feature, the song corresponding to the song feature is determined as the song recognition result;

[0152] If the maximum feature similarity corresponds to a noise feature, the recognition failure prompt information is determined as the song recognition result.

[0153] In one embodiment, the processor 801 is further configured to:

[0154] Extracting identification features of song segments corresponding to each song feature in the song feature library, and extracting identification features of the audio to be identified; the identification features of the audio to be identified are used to indicate changes in the spectrum peaks of the audio to be identified;

[0155] Perform feature matching processing on the identification features of the audio to be identified and the identification features of each song segment to obtain a feature matching result;

[0156] If the feature matching result indicates a successful match, the song corresponding to the successfully matched song segment is determined as the song recognition result;

[0157] If the feature matching result indicates a match failure, the fingerprint features corresponding to each song feature in the song feature library and the fingerprint features corresponding to each noise feature in the noise feature library are extracted through the trained song recognition model.

[0158] The present application provides a computer program product, which includes a computer program stored in a computer storage medium; a processor of an audio processing device reads the computer program from the computer storage medium, and the processor executes the computer program, so that the audio processing device performs the above-mentioned Figure 2 as well as Figure 5 The method embodiment shown in the figure. The computer-readable storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0159] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An audio processing method, characterized in that: include: Obtain a song feature library and a noise feature library; The song feature library includes multiple song feature groups of multiple songs, each song feature in the song feature group is an acoustic feature of different versions of the same song segment of the song, and a noise feature in the noise feature library is an acoustic feature of a noise segment; A sample group is constructed based on the song feature library and the noise feature library; the sample group includes a positive sample, a negative sample and a reference sample, and the sample group satisfies the following first condition, second condition or third condition: the first condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a noise feature selected from the noise feature library; The second condition is that the positive sample and the reference sample are different song features selected from the same song feature group, and the negative sample is a song feature selected from other song feature groups in the song feature library, and the other song feature groups are song feature groups of other songs other than the song corresponding to the positive sample; The third condition is that the positive sample and the reference sample are different noise features selected from the noise feature library, and the negative sample is a song feature selected from the song feature library; Extracting fingerprint features corresponding to the positive sample, fingerprint features corresponding to the negative sample, and fingerprint features corresponding to the reference sample through a song recognition model; The song recognition model is trained based on the fingerprint features corresponding to the positive samples, the fingerprint features corresponding to the negative samples and the fingerprint features corresponding to the reference samples to obtain a trained song recognition model; wherein the trained song recognition model is used to extract the fingerprint features corresponding to the audio to be recognized, and the fingerprint features corresponding to the audio to be recognized are used to perform song recognition on the audio to be recognized.

2. The method according to claim 1, characterized in that The method further comprises: Selecting a plurality of noise features from the noise feature library, determining a newly added noise feature based on the plurality of noise features, and adding the newly added noise feature to the noise feature library to update the noise feature library; The constructing of the sample group based on the song feature library and the noise feature library comprises: The sample group is constructed based on the song feature library and the updated noise feature library.

3. The method according to claim 2, characterized in that The determining a newly added noise feature based on the multiple noise features includes: The multiple noise features are averaged to obtain the newly added noise feature.

4. The method according to claim 1, characterized in that The training of the song recognition model based on the fingerprint features corresponding to the positive sample, the fingerprint features corresponding to the negative sample, and the fingerprint features corresponding to the reference sample includes: Determining a target fingerprint feature based on the fingerprint feature corresponding to the reference sample; The model parameters of the song recognition model are adjusted in a direction of reducing the feature distance between the fingerprint feature corresponding to the positive sample and the target fingerprint feature, and in a direction of increasing the feature distance between the fingerprint feature corresponding to the negative sample and the target fingerprint feature.

5. The method according to claim 4, characterized in that The sample group includes a plurality of reference samples, and the determining the target fingerprint feature based on the fingerprint features corresponding to the reference samples includes: The fingerprint features corresponding to the reference samples are averaged to obtain the target fingerprint features.

6. The method according to claim 1, characterized in that The method further comprises: Extracting fingerprint features corresponding to each song feature in the song feature library and fingerprint features corresponding to each noise feature in the noise feature library through the trained song recognition model; Acquire the acoustic features of the audio to be identified, and perform feature conversion processing on the acoustic features of the audio to be identified through the trained song recognition model to obtain fingerprint features corresponding to the audio to be identified; Based on the feature distance between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the various song features, as well as the feature distance between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the various noise features, song identification is performed on the audio to be identified to obtain a song identification result.

7. The method according to claim 6, characterized in that The method of performing song recognition on the audio to be recognized based on the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to the respective song features, and the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to the respective noise features, to obtain a song recognition result, includes: Determine a minimum feature distance from the feature distances between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the respective song features, and the feature distances between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the respective noise features; If the minimum feature distance corresponds to a song feature, the song corresponding to the song feature is determined as the song recognition result; If the minimum feature distance corresponds to a noise feature, the recognition failure prompt information is determined as the song recognition result.

8. The method according to claim 6, characterized in that The method of performing song recognition on the audio to be recognized based on the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to the respective song features, and the feature distance between the fingerprint feature corresponding to the audio to be recognized and the fingerprint features corresponding to the respective noise features, to obtain a song recognition result, includes: Determine the feature similarity based on the feature distance between the fingerprint feature corresponding to the audio to be identified and the fingerprint feature corresponding to any song feature, and determine the feature similarity based on the feature distance between the fingerprint feature corresponding to the audio to be identified and the fingerprint feature corresponding to any noise feature; the feature similarity is negatively correlated with the feature distance; Determine the maximum feature similarity from the feature similarities between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the respective song features, and the feature similarities between the fingerprint feature corresponding to the audio to be identified and the fingerprint features corresponding to the respective noise features; If the maximum feature similarity corresponds to a song feature, the song corresponding to the song feature is determined as the song recognition result; If the maximum feature similarity corresponds to a noise feature, the recognition failure prompt information is determined as the song recognition result.

9. The method according to claim 6, characterized in that The method further comprises: Extracting identification features of song segments corresponding to each song feature in the song feature library, and extracting identification features of the audio to be identified; the identification features of the audio to be identified are used to indicate changes in the spectrum peaks of the audio to be identified; Perform feature matching processing on the identification features of the audio to be identified and the identification features of each song segment to obtain a feature matching result; If the feature matching result indicates a successful match, the song corresponding to the successfully matched song segment is determined as the song recognition result; If the feature matching result indicates a match failure, the fingerprint features corresponding to each song feature in the song feature library and the fingerprint features corresponding to each noise feature in the noise feature library are extracted through the trained song recognition model.

10. An audio processing device, characterized in that: The audio processing device includes an input interface and an output interface, and also includes: a processor adapted to implement one or more instructions; and, A computer storage medium storing one or more instructions, wherein the one or more instructions are suitable for being loaded by the processor and executing the audio processing method according to any one of claims 1 to 9.

11. A computer storage medium, characterized in that: The computer storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, they are used to execute the audio processing method according to any one of claims 1 to 9.

12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program is used to load and execute the audio processing method according to any one of claims 1 to 9.

Images