Audio recognition method, audio recognition device, medium, product and electronic equipment

Abstract

The present disclosure relates to the technical field of computer, and provides an audio recognition method, an audio recognition device, a medium, a product and an electronic device. The audio recognition method comprises: determining candidate audios from existing audios in an audio library according to a similarity between a first audio feature vector of a to-be-recognized audio and second audio feature vectors of the existing audios; for each candidate audio, performing audio feature alignment on the to-be-recognized audio and the candidate audio according to first fingerprint information of the to-be-recognized audio and second fingerprint information of the candidate audio; and determining a same audio recognition result of the to-be-recognized audio and the candidate audio based on a result of the audio feature alignment. The present scheme can improve the recognition accuracy of the same audio.

Description

Technical Field

[0001] This disclosure relates to the field of information processing technology, and more specifically, to an audio recognition method, an audio recognition device, a computer-readable storage medium, a computer program product, and an electronic device. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure set forth in the claims, and the description herein is not acknowledged as prior art simply because it is included in this section.

[0003] Similar audio recognition can be applied to fields such as voice unlocking and identifying identical songs. Related technologies use neural network models to encode two audio files into feature vectors, and then measure the similarity between the two audio files based on the similarity between these feature vectors, thus obtaining the identification result for similar audio. Summary of the Invention

[0004] However, the audio features extracted by the relevant technologies are relatively sparse, and the accuracy of recognizing the same audio is low when the audio duration is long.

[0005] Therefore, there is a great need for an audio recognition method to improve the accuracy of recognizing the same audio.

[0006] In this context, embodiments of the present invention are intended to provide an audio recognition method, an audio recognition device, a computer-readable storage medium, a computer program product, and an electronic device.

[0007] According to a first aspect of the present disclosure, an audio recognition method is provided, comprising: determining candidate audio from existing audio based on the similarity between a first audio feature vector of an audio to be recognized and a second audio feature vector of existing audio in an audio library; for each candidate audio, performing audio feature alignment between the audio to be recognized and the candidate audio based on a first fingerprint information of the audio to be recognized and a second fingerprint information of the candidate audio; and determining a similarity recognition result between the audio to be recognized and the candidate audio based on the result of the audio feature alignment.

[0008] Optionally, the first fingerprint information includes a first set of local peak features in the spectrum of the audio to be identified, and the second fingerprint information includes a second set of local peak features in the spectrum of the candidate audio. Any local peak feature is determined based on the time and frequency of the local peak. The step of aligning the audio features of the audio to be identified and the candidate audio based on the first fingerprint information and the second fingerprint information of the candidate audio includes: generating a first inverted index based on the time information corresponding to the same frequency in the first set of local peak features; for each local peak feature in the second set of local peak features, searching for the time information corresponding to the frequency of the local peak feature in the first inverted index, and calculating a first difference between the found time information and the time of the local peak feature; and aligning the audio features of the audio to be identified and the candidate audio based on the first difference.

[0009] Optionally, the step of performing overall audio feature alignment on the audio to be identified and the candidate audio based on the first difference includes: counting the number of occurrences of different first differences in the first difference, determining a first maximum occurrence based on the maximum value among the first occurrences; if the first maximum occurrence is greater than or equal to a first preset value, taking the first difference corresponding to the first maximum occurrence as a first alignment time difference; and performing overall audio feature alignment on the audio to be identified and the candidate audio based on the first alignment time difference.

[0010] Optionally, determining the similarity recognition result between the audio to be identified and the candidate audio based on the audio feature alignment result includes: calculating a first sum of the first start time and the first alignment time difference of the candidate audio; if the relationship between the first sum and the second start time of the audio to be identified satisfies a first preset condition, obtaining the predicted end time of the audio to be identified based on the first end time and the second sum of the first alignment time difference of the candidate audio; if the relationship between the second end time of the audio to be identified and the predicted end time satisfies a second preset condition, segmenting the overall aligned candidate audio and the audio to be identified to obtain multiple first audio segment pairs, and performing local segmentation on each first audio segment pair. Audio feature alignment; based on the local audio feature alignment results of each first audio segment pair, determine the similarity recognition result between the audio to be identified and the candidate audio; wherein, the first preset condition includes the first sum value being the same as the second start time of the audio to be identified, or the first sum value being greater than the second start time and the energy of the audio segment to be confirmed being less than or equal to an energy threshold, the audio segment to be confirmed including the audio segment in the audio to be identified that starts from the second start time and has a length of the first alignment time difference; the second preset condition includes the second end time of the audio to be identified being the same as the predicted end time, or the predicted end time being less than the second end time and the absolute value of the third difference between the two being less than or equal to a fifth preset value.

[0011] Optionally, determining the similarity recognition result between the audio to be identified and the candidate audio based on the audio feature alignment result includes: segmenting the overall aligned candidate audio and the audio to be identified to obtain multiple first audio segment pairs, performing local audio feature alignment on each first audio segment pair, and determining the similarity recognition result between the audio to be identified and the candidate audio based on the local audio feature alignment result of each first audio segment pair.

[0012] Optionally, the step of segmenting the overall aligned candidate audio and the audio to be identified to obtain multiple first audio segment pairs, and performing local audio feature alignment on each first audio segment pair, includes: segmenting the candidate audio into multiple non-overlapping candidate audio segments according to a first preset duration; extracting the audio segment to be identified corresponding to the candidate audio segment from the overall aligned audio to be identified to obtain a first audio segment pair, wherein the third start time of the audio segment to be identified is less than the fourth start time of the candidate audio segment, and the third end time of the audio segment to be identified is greater than the fourth end time of the candidate audio segment; and for each first audio segment pair, performing local audio feature alignment on the candidate audio segment and the audio segment to be identified in the first audio segment pair according to the third fingerprint information of the candidate audio segment in the first audio segment pair and the fourth fingerprint information of the audio segment to be identified.

[0013] Optionally, the third fingerprint information includes a third set of local spectral features of the candidate audio segment, and the second fingerprint information includes a fourth set of local spectral peak features of the audio segment to be identified. The step of aligning the candidate audio segments and the audio segment to be identified in the first audio segment pair using the third fingerprint information of the candidate audio segments and the fourth fingerprint information of the audio segment to be identified includes: generating a second inverted index based on the time information corresponding to the same frequency in the fourth set of local spectral peak features; for each third local spectral peak feature in the third set of local spectral peak features, searching for the time information corresponding to the frequency of the third local spectral peak feature in the second inverted index, and calculating a second difference between the found time information and the time of the second local spectral peak feature; and aligning the candidate audio segments and the audio segment to be identified in the first audio segment pair using the second difference.

[0014] Optionally, the step of performing local audio feature alignment on the candidate audio segments and the audio segments to be identified in the first audio segment pair based on the second difference includes: counting the second occurrence counts of different second differences in the second difference, determining the second maximum occurrence count based on the maximum value among the second occurrence counts; if the second maximum occurrence count is greater than or equal to a second preset value, taking the second difference corresponding to the second maximum occurrence count as the second alignment time difference; and performing local audio feature alignment on the candidate audio segments and the audio segments to be identified in the first audio segment pair based on the second alignment time difference.

[0015] Optionally, determining the similarity recognition result between the audio to be identified and the candidate audio based on the local audio feature alignment result of each first audio segment pair includes: determining the first audio segment pair as the target audio segment pair when the second maximum occurrence count is greater than or equal to a second preset value; when all first audio segment pairs corresponding to the candidate audio are determined as target audio segment pairs, extracting an audio segment of the first preset duration from the audio segment to be identified in the first audio segment pair based on the alignment position of the local audio feature alignment to obtain a second audio segment; obtaining a second audio segment pair based on the second audio segment and the candidate audio segments in the first audio segment pair; determining the similarity between the candidate audio segments and the second audio segment in the second audio segment pair based on the third fingerprint information of the candidate audio segments and the fifth fingerprint information of the second audio segment in the second audio segment pair; and determining the similarity recognition result between the audio to be identified and the candidate audio based on the similarity between the candidate audio segments and the second audio segment in each second audio segment pair corresponding to the candidate audio.

[0016] Optionally, determining the similarity between the candidate audio segment and the second audio segment in the second audio segment pair based on the third fingerprint information of the candidate audio segment and the fifth fingerprint information of the second audio segment includes: determining the similarity between the candidate audio segment and the second audio segment in the second audio segment pair based on the overlap between the third fingerprint information and the fifth fingerprint information.

[0017] Optionally, determining the similarity recognition result between the audio to be identified and the candidate audio based on the similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio includes: calculating the average similarity between the candidate audio segment and the second audio segment in all second audio segment pairs corresponding to the candidate audio; and, if the average similarity is greater than or equal to a third preset value, determining the similarity recognition result between the audio to be identified and the candidate audio based on the similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio.

[0018] Optionally, determining the similarity recognition result between the audio to be identified and the candidate audio based on the similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio includes: determining that the audio to be identified and the candidate audio are the same if the similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio is greater than or equal to a fourth preset value.

[0019] Optionally, the method further includes: determining that the audio to be identified and the candidate audio are not the same if the similarity between the candidate audio segment and the second audio segment in any second audio segment pair corresponding to the candidate audio is less than the fourth preset value.

[0020] Optionally, the method further includes: outputting first prompt information for the candidate audio segment and the second audio segment whose similarity is less than the fourth preset value.

[0021] Optionally, the method further includes: if the mean is less than the third preset value, determining that the candidate audio and the audio to be identified are different.

[0022] Optionally, the method further includes: determining that the candidate audio is different from the audio to be identified when the second maximum occurrence count is less than the second preset value.

[0023] Optionally, the method further includes: determining that the audio to be identified and the candidate audio are different if the first sum value is less than the second start time.

[0024] Optionally, the method further includes: determining that the audio to be identified and the candidate audio are different if the first sum is greater than the second start time and the energy of the audio segment to be identified is greater than the energy threshold.

[0025] Optionally, the method further includes: determining that the audio to be identified and the candidate audio are different if the second end time of the audio to be identified is different from the predicted end time.

[0026] Optionally, determining that the audio to be identified and the candidate audio are different when the second end time of the audio to be identified is different from the predicted end time includes: determining that the audio to be identified and the candidate audio are different when the predicted end time is less than the second end time, or when the predicted end time is less than the second end time and the absolute value of the third difference is greater than the fifth preset value; and determining that the audio to be identified and the candidate audio are different when the predicted end time is greater than the second end time.

[0027] Optionally, the method further includes: if the first maximum occurrence count corresponding to the candidate audio is less than the first preset value, determining that the audio to be identified and the candidate audio are different.

[0028] According to a second aspect of the present disclosure, an audio recognition apparatus is provided, comprising: a candidate audio determination module configured to determine candidate audio from existing audio based on the similarity between a first audio feature vector of an audio to be recognized and a second audio feature vector of existing audio in an audio library; an audio feature alignment module configured to, for each candidate audio, perform audio feature alignment between the audio to be recognized and the candidate audio based on a first fingerprint information of the audio to be recognized and a second fingerprint information of the candidate audio; and a same audio recognition module configured to determine a similarity recognition result between the audio to be recognized and the candidate audio based on the result of the audio feature alignment.

[0029] According to a third aspect of this disclosure, a computer program product comprising instructions is provided, which, when run on a computer, causes the computer to perform the steps of the audio recognition method as described in the first aspect.

[0030] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the audio recognition method as described in the first aspect of the above embodiments.

[0031] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the audio recognition method as described in the first aspect of the above embodiments.

[0032] According to the audio recognition method, audio recognition device, computer-readable storage medium, computer program product, and electronic device of this disclosure, audio similar to the audio to be recognized is determined from an audio library as candidate audio by the similarity between audio feature vectors. Then, based on the fingerprint information of the audio to be recognized and the candidate audio, audio features are aligned between the audio to be recognized and the candidate audio. Based on the audio feature alignment result, it is determined whether the candidate audio and the audio to be recognized are the same audio. On the one hand, this disclosure further aligns audio based on audio fingerprint information on the basis of similarity recognition to identify the same audio, which can improve the accuracy of the same audio recognition. On the other hand, this disclosure first determines the candidate audio based on similarity, and only needs to perform audio feature alignment processing on the candidate audio and the audio to be recognized based on fingerprint information to obtain the recognition result. It is not necessary to perform audio feature alignment processing on all audio in the audio library and the audio to be recognized, which improves the efficiency of the same audio recognition. Furthermore, this disclosure can perform deduplication processing on the same audio by recognizing the same audio, saving storage resources. Attached Figure Description

[0033] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0034] Figure 1 A schematic diagram of an exemplary system architecture to which embodiments of the present disclosure may be applied is shown;

[0035] Figure 2 A flowchart illustrating an audio recognition method according to an exemplary embodiment of this disclosure is shown.

[0036] Figure 3 This diagram illustrates the structure of a candidate audio determination system according to an exemplary embodiment of the present disclosure.

[0037] Figure 4 This diagram illustrates a flowchart of a method for overall alignment of audio features according to an exemplary embodiment of the present disclosure.

[0038] Figure 5 This diagram illustrates the result of overall audio alignment according to an exemplary embodiment of the present disclosure.

[0039] Figure 6 A flowchart illustrating a method for determining identity recognition results according to an exemplary embodiment of this disclosure is shown.

[0040] Figure 7 This diagram illustrates a flowchart of a method for aligning local audio features according to an exemplary embodiment of the present disclosure.

[0041] Figure 8 A flowchart illustrating another method for aligning local audio features according to an exemplary embodiment of this disclosure is shown.

[0042] Figure 9 A flowchart illustrating another method for determining identity recognition results according to an exemplary embodiment of this disclosure is shown.

[0043] Figure 10 A flowchart illustrating another method for determining identity recognition results according to an exemplary embodiment of the present disclosure is shown.

[0044] Figure 11 This diagram illustrates the composition of an audio recognition device according to an exemplary embodiment of the present disclosure.

[0045] Figure 12 A schematic diagram of the structure of an electronic device in an exemplary embodiment of this disclosure is shown.

[0046] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0047] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make the invention more thorough and complete, and to fully convey the scope of the invention to those skilled in the art.

[0048] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, medium, method, or computer program product. Therefore, the present invention can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0049] According to embodiments of the present invention, an audio recognition method, an audio recognition device, a computer-readable storage medium, a computer program product, and an electronic device are provided.

[0050] In this document, any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0051] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments. Invention Overview

[0053] The inventors of this disclosure have discovered that audio recognition methods in the related art suffer from low accuracy and efficiency in recognizing the same audio.

[0054] In view of the above, the basic idea of ​​this disclosure is to provide an audio recognition method, apparatus, computer-readable storage medium, computer program product, and electronic device. This method determines candidate audio from an audio library based on the similarity between audio feature vectors, identifying audio similar to the audio to be recognized. Then, based on the fingerprint information of the audio to be recognized and the candidate audio, audio features are aligned. Based on the audio feature alignment result, it is determined whether the candidate audio and the audio to be recognized are the same audio. On the one hand, this disclosure, based on similarity recognition, further aligns audio using audio fingerprint information to identify the same audio, thus improving the accuracy of identical audio recognition. On the other hand, this disclosure first determines candidate audio based on similarity, requiring only audio feature alignment between the candidate audio and the audio to be recognized based on fingerprint information to obtain the recognition result. It does not require audio feature alignment between all audio in the audio library and the audio to be recognized, thus improving the efficiency of identical audio recognition. Furthermore, this disclosure, through identical audio recognition, can deduplicate identical audio, saving storage resources.

[0055] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below.

[0056] Application Scenarios Overview

[0057] It should be noted that the following application scenarios are shown only to facilitate understanding of the spirit and principles of the present invention, and the embodiments of the present invention are not limited in any way. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0058] In one exemplary application scenario, when a user needs to deduplicate audio in an audio library, any audio in the audio library can be used as the audio to be identified. Then, according to the method in this disclosure, the same audio as the audio to be identified can be displayed from other audio in the audio library, so as to deduplicate the identified audio.

[0059] In another exemplary application scenario, when it is necessary to determine song copyright infringement, the method disclosed herein can be used to determine whether the audio of the defendant's song is the same as that of the original song, thereby determining whether the defendant's song infringes on the copyright of the original song.

[0060] In another exemplary music scenario, when a user inputs a song fragment in a music client, the method in this disclosure can be used to identify songs in the audio library that are the same as the song fragment input by the user, thereby recommending songs to the user.

[0061] Exemplary System Architecture

[0062] First, refer to Figure 1The system architecture of the exemplary application environment of this disclosure is described.

[0063] like Figure 1 As shown, the system architecture 100 may include a terminal device 110 and a server 120. The terminal device 110 may be a smartphone, tablet, desktop computer, laptop, smart wearable device, or other similar device. The server 120 generally refers to the backend system providing services related to the audio recognition method in this exemplary embodiment, and may be a single server or a cluster of multiple servers. The terminal device 110 and the server 120 can be connected via wired or wireless communication links for data interaction.

[0064] In one exemplary embodiment, the audio recognition method described above can be executed by server 120. Correspondingly, an audio recognition device can be installed in server 120 to implement corresponding module functions. For example, when it is necessary to confirm whether the audio to be recognized is the same as existing audio in the audio library, server 120 can determine candidate audio from the existing audio based on the similarity between the first audio feature vector of the audio to be recognized and the second audio feature vector of the existing audio in the audio library. Then, for each candidate audio, audio feature alignment is performed on the audio to be recognized and the candidate audio based on the first fingerprint information of the audio to be recognized and the second fingerprint information of the candidate audio. Based on the result of the fingerprint feature alignment, the audio identical to the audio to be recognized is determined from the audio library, thus achieving the recognition of identical audio.

[0065] It should be understood that Figure 1 The number of terminal devices and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices and servers. For example, server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0066] It will be readily understood by those skilled in the art that the above application scenarios are merely illustrative and are not intended to limit the scope of this exemplary embodiment.

[0067] Exemplary methods

[0068] Figure 2 This diagram illustrates a flowchart of an audio recognition method according to an exemplary embodiment of the present disclosure, with reference to... Figure 2 The method includes:

[0069] Step S210: Based on the similarity between the first audio feature vector of the audio to be identified and the second audio feature vector of the existing audio in the audio library, candidate audio is determined from the existing audio.

[0070] Step S220: For each candidate audio, perform audio feature alignment between the audio to be identified and the candidate audio based on the first fingerprint information of the audio to be identified and the second fingerprint information of the candidate audio.

[0071] Step S230: Based on the audio feature alignment result, determine the similarity recognition result between the audio to be identified and the candidate audio.

[0072] Below, we will first provide a detailed description of the specific implementation of "step S210, determining candidate audio from the existing audio based on the similarity between the first audio feature vector of the audio to be identified and the second audio feature vector of the existing audio in the audio library".

[0073] In one exemplary implementation, the audio to be identified may include any audio that requires the same audio recognition, such as a song, instrumental music, etc.

[0074] In one exemplary implementation, candidate audio files can be identified by training a neural network model to recognize the similarity between audio files.

[0075] Taking a song as the audio to be identified as an example, a song cover model can be constructed using a neural network model. This model identifies songs identical to the song in question, thus determining candidate songs. The training data for the song cover model consists of groups of cover songs, with each song in a group being a cover of another song. First, spectral features are extracted from all songs. These spectral features can be obtained using features such as Melp or CQT (Constant-Q Transform, which captures energy characteristics at different frequencies in an audio signal). The neural network model can be a convolutional neural network or a recurrent neural network, and the training loss function can be a triplet loss function. The learning objective of the neural network is to minimize the distance between songs within a cover song group and maximize the distance between songs in different cover groups. The output of the neural network model can be a fixed-dimensional vector, such as a 512-dimensional vector. A larger dimension provides stronger expressive power but also increases computational and storage costs. Therefore, the dimension can be determined based on specific needs, and this exemplary implementation does not impose any special limitations on this.

[0076] Below, in conjunction with Figure 3 The specific implementation of step S210 will be described below. For example, Figure 3This diagram illustrates the structure of a candidate audio determination system according to an exemplary embodiment of this disclosure. (Reference) Figure 3 The system may include a database creation module 31 and a retrieval module 32.

[0077] For example, after training the neural network model, it can be used... Figure 3 The library building module 31 constructs an audio retrieval library. The purpose of library building is to construct an efficient audio retrieval library. Taking songs as the audio source and the song cover model as the neural network model mentioned above as an example, the song cover model can be used to extract high-dimensional feature vectors from the songs in the song library. Vector indexing tools such as Faiss (Facebook AISimilarity Search, an open-source library for fast and efficient vector database construction and similarity search) and Milvus (a cloud-native vector database that supports multiple index types and similarity measurement methods) provide efficient vector retrieval capabilities and support multiple index types such as inverted indexes and HNSW (Hierarchical Navigable Small World, a graph-based indexing algorithm).

[0078] After establishing the audio library, the candidate audio corresponding to the audio to be identified can be retrieved from the audio library using the retrieval module 32. Continuing with the example of a song, the song to be identified can be input into the song cover model to extract the audio feature vector of the song to be identified. This vector is then searched in the song vector library constructed by the library building module 31. For example, cosine distance can be used to measure similarity; the smaller the distance, the more similar the songs are. A similarity threshold can be set, and songs with a similarity greater than the threshold are selected as the first candidate songs. Then, the similarity of the first candidate songs is sorted in descending order, and the top N songs in the sorting results are selected as candidate songs for the song to be identified, thus obtaining a list of candidate songs.

[0079] In other words, by training a neural network model, feature vectors can be extracted from audio in the audio library using this neural network model. This is equivalent to encoding the audio into audio vectors. Then, the audio vectors are used to build an index library. The same neural network model is used to extract the feature vectors of the audio to be identified. The feature vectors of the audio to be identified are used to search in the index library. The audios indicated by the audio vectors of the N nearest neighbors in the index library are output as candidate audios for the audio to be identified.

[0080] Step S210 allows for the retrieval of a set of similar audio samples from the audio library, thereby reducing the amount of data processing required in subsequent steps S220 and S230 and improving the efficiency of identifying identical audio samples.

[0081] The following is a detailed description of the specific implementation of "step S220, for each candidate audio, performing audio feature alignment between the audio to be identified and the candidate audio based on the first fingerprint information of the audio to be identified and the second fingerprint information of the candidate audio".

[0082] In one exemplary embodiment, the first fingerprint information includes a first set of local peak features in the spectrum of the audio to be identified, and the second fingerprint information includes a second set of local peak features in the spectrum of the candidate audio. Each local peak feature is determined based on the time and frequency of the local peak. Based on this, the following... Figure 4 The specific implementation of step S220 will be further explained.

[0083] For example, Figure 4 This diagram illustrates a flowchart of a method for overall audio feature alignment according to an exemplary embodiment of this disclosure. (See reference...) Figure 4 The method may include steps S410 to S430. Wherein:

[0084] In step S410, a first inverted index library is generated based on the time information corresponding to the same frequency in the first set of local peak features of the spectrum.

[0085] For example, the fingerprint information of audio can be understood as feature points extracted from the audio spectrum. The distribution of these feature points in a two-dimensional space of time and frequency characterizes the audio content features at the signal level. Therefore, the fingerprint information constructed based on feature points has uniqueness.

[0086] In this disclosure, the local peaks of the audio spectrum can be used as feature points to obtain the fingerprint information of the audio. That is, the fingerprint information of any audio can be understood as a set of local peak features of the audio spectrum.

[0087] For example, the process of extracting fingerprint information for any audio can be as follows: First, the audio can be transformed using STFT (Scale-invariant feature transform) to obtain the audio spectrum. Specifically, the audio can be segmented into frames, for example, each frame is 20ms long with a frame shift of 10ms. A Fast Fourier Transform is performed on each frame to extract the logarithmic domain spectral features. The entire spectrum is then normalized (i.e., standardized based on the mean) to eliminate the influence of different volume levels, resulting in a two-dimensional table of time and frequency, which can be understood as the audio spectrum. Then, a filter (such as a 3x3 filter) is used to filter out local maxima. A fixed-width window is then applied to these maxima points to find the extreme points within each window as the feature point set for the audio. The information of each feature point includes (t, f), where t represents time, f represents frequency, and height represents the energy of the feature point. Typically, dozens of feature points can be selected per second of audio. The number of feature points depends on the window size. The filter size can be determined based on experience and actual conditions; this exemplary implementation does not impose any specific limitations on this. The set of these feature points is used as the fingerprint information of the audio.

[0088] For example, after obtaining the set of feature points for the audio to be identified using the method described above, a first inverted index can be generated based on this set of feature points. For instance, the frequency of a feature point can be used as the fingerprint hash key, and the set of all times that frequency appears in the feature point can be used as the value corresponding to that frequency key, thus generating the first inverted index. In other words, feature points in the set can be grouped according to the same frequency, grouping feature points with the same frequency into the same group. Then, for each group, a key-value pair can be formed using the frequency of that group as the key and all time information within that group as the value. The set of key-value pairs corresponding to all groups constitutes the first inverted index. That is, in the first inverted index, all time information corresponding to a given frequency can be retrieved based on the frequency information.

[0089] In step S420, for each second local peak feature in the second local peak feature set, the time information corresponding to the frequency of the second local peak feature is searched in the first inverted index library, and the first difference between the searched time information and the time of the second local peak feature is calculated.

[0090] For example, for each frequency in the local peak feature set of the spectrum of each candidate audio, the corresponding time information of that frequency can be found in the first inverted index. Then, the difference between the found time and the time in the feature is calculated to obtain the first time difference subset corresponding to each feature point. Taking a feature point (f1, t1) in the candidate audio as an example, the time information corresponding to f1 is found in the first inverted index, such as t2, t3, and t4. For this feature point, the first time difference subset will have three values: t2 minus t1, t3 minus t1, and t4 minus t1. Based on the set of the first time difference subsets corresponding to each feature point, the first time difference set is obtained.

[0091] In other words, after obtaining the first inverted index, for each candidate audio, the fingerprint information of the candidate audio can be used as input to search the first inverted index. The hash value f1 of time t1 for each feature point in the fingerprint information of the candidate audio is retrieved, and the corresponding time t2 in the first inverted index is found. The difference between each time t2 and t1 is calculated, resulting in a fingerprint time difference tDiff = (t2 - t1). After subtracting all feature points in the fingerprint information of the candidate audio, a set of time differences is obtained, which is the aforementioned first time difference set. That is, the set composed of each first difference value is the first time difference set.

[0092] In step S430, the audio features of the audio to be identified and the candidate audio are aligned as a whole based on the first difference.

[0093] For example, one exemplary implementation of step S430 may include: counting the number of times different first differences occur in the first differences, determining a first maximum occurrence number based on the maximum value among the first occurrence numbers; when the first maximum occurrence number is greater than or equal to a first preset value, using the first difference corresponding to the first maximum occurrence number as a first alignment time difference; and performing overall audio feature alignment on the audio to be identified and the candidate audio based on the first alignment time difference.

[0094] For example, the first difference can be grouped according to the same difference value. Then, the first number of the first difference values ​​in each group is counted. If the first number is less than a first preset value, it is determined that the audio to be identified and the candidate audio cannot be aligned as a whole, which means that the audio to be identified and the candidate audio are not the same. If the first number is greater than or equal to the first preset value, it means that the audio to be identified and the candidate audio may have an overall offset, that is, they can be aligned as a whole. Then, the first difference value corresponding to the largest first number can be used as the first alignment time difference. Based on the first alignment time difference, the audio features of the audio to be identified and the candidate audio are aligned as a whole. Based on the overall alignment, it is determined whether the audio to be identified and the candidate audio are the same audio.

[0095] The implementation method for aligning the audio features of the audio to be identified and the candidate audio based on the first alignment time difference can be as follows: add the first alignment time difference to the original start time of the candidate audio as the new start time of the candidate audio, thereby moving the candidate audio as a whole by the first alignment time difference, so as to align the candidate audio to the video to be identified, and realize the alignment of the candidate audio and the audio to be identified.

[0096] For example, Figure 5 This diagram illustrates the result of overall audio alignment according to an exemplary embodiment of this disclosure. (See reference...) Figure 5 , Figure 5 The top part of the image shows candidate audio files, and the bottom part shows the video file to be identified. Figure 5 The audio to be identified in the system has an extra audio header time compared to the candidate audio. By adding the first time difference to the original start time of the candidate audio, the new start time of the candidate audio can be obtained, thus aligning the audio to be identified and the candidate audio as a whole.

[0097] In other words, the temporal boundaries of the audio to be identified and the candidate audio can be aligned through overall audio feature alignment. That is, as mentioned earlier, a first inverted index is built using the fingerprint information of the audio to be identified. Then, the features in the fingerprint information of the candidate audio are used as input to search the first inverted index. The hash value (frequency) of the time of each feature point of the candidate audio is found in the first inverted index, and the time value t2 is obtained, thus yielding the fingerprint time difference tDiff. max =(t2-t1), after retrieving all fingerprint features of the candidate audio, a time difference set is obtained. Then, the frequency of each time difference is counted. Assuming the maximum frequency is n, the corresponding time difference is tDiff. max If n is greater than or equal to the first preset value, it means that the two audio files can be aligned as a whole, and the world boundary of the alignment pos = tDiff. maxThis indicates the time position of the candidate audio aligned with the audio to be identified. If n is less than the first preset value, it means that the two cannot be aligned as a whole, which means that the fingerprint information of the two is different, and that the two belong to different audio.

[0098] The following is a detailed description of the specific implementation of "step S230, determining the similarity recognition result between the audio to be identified and the candidate audio based on the result of audio feature alignment".

[0099] In one exemplary embodiment, after obtaining the first maximum occurrence count in step S220, a specific implementation of step S230 may include: if the first maximum occurrence count is greater than or equal to a first preset value, it can be directly determined that the audio to be identified and the candidate audio belong to the same audio; otherwise, it is determined that the audio to be identified and the candidate audio belong to different audio. While this method is more accurate than judging the same audio solely based on similarity, it cannot identify subtle differences in the audio signals, and it cannot pinpoint the location of the differences when the two are different.

[0100] Based on this, the following will be combined with Figure 6 Another specific implementation of step S230 will be described. For example, Figure 6 This diagram illustrates a flowchart of a method for determining identity recognition results according to an exemplary embodiment of this disclosure. (See reference...) Figure 6 The method may include steps S610 to S620. Wherein:

[0101] In step S610, the candidate audio and the audio to be identified after overall alignment are segmented to obtain multiple first audio segment pairs, and local audio feature alignment is performed on each first audio segment pair.

[0102] Below, in conjunction with Figure 7 An exemplary implementation of step S610 will be described. For example, Figure 7 This diagram illustrates a flowchart of a method for aligning local audio features according to an exemplary embodiment of this disclosure. (See reference...) Figure 7 The method may include steps S710 to S730. Wherein:

[0103] In step S710, the candidate audio is divided into multiple non-overlapping candidate audio segments according to the first preset duration.

[0104] The first preset duration can be customized according to needs or experience. For example, the first preset duration can be 5 seconds, 6 seconds, etc. For example, when the candidate audio is long, the first preset duration can be set slightly longer, and when the candidate audio is short, the first preset duration can be set slightly shorter. This exemplary embodiment does not impose any special limitations on this.

[0105] Taking a preset duration of 5 seconds as an example, for a candidate audio with a length greater than 5 seconds, the candidate audio can be divided into multiple overlapping 5-second segments at equal intervals to obtain multiple candidate audio segments.

[0106] In step S720, the audio segment to be identified corresponding to the candidate audio segment is extracted from the overall aligned audio to be identified to obtain the first audio segment pair.

[0107] In one exemplary implementation, the third start time of the audio segment to be identified is less than the fourth start time of the candidate audio segment, and the third end time of the audio segment to be identified is greater than the fourth end time of the candidate audio segment.

[0108] For example, the first sub-start time of each candidate audio segment can be determined based on its actual start time and a first preset duration. Then, based on each first sub-start time and the first alignment time difference, the alignment time of that first sub-start time in the audio to be identified can be found. For instance, adding the first alignment time difference to the first sub-start time yields the alignment time of each first sub-start time in the audio to be identified. For any alignment time in the audio to be identified, a video segment X seconds before and after the alignment time can be extracted, thus obtaining the audio segment to be identified corresponding to that candidate audio segment. A first audio segment pair is then obtained based on the segment pair formed by the candidate audio segment and its corresponding audio segment to be identified. X can be determined based on requirements and experience; for example, X could be 3 seconds, 4 seconds, etc. The length of the extracted audio segment to be identified and the length of the candidate audio segment can be the same or different; that is, the length of the audio segment to be identified must be greater than or equal to the length of its corresponding candidate audio segment.

[0109] For example, the candidate audio can be divided into multiple non-overlapping candidate audio segments at equal distances. Then, for each candidate audio segment, the segment whose alignment position in the audio to be identified is found is extended by X seconds to the left and right, thus obtaining the corresponding audio segment to be identified, thereby obtaining the first audio segment pair.

[0110] Slight speed changes can cause deviations in the overall alignment boundaries, leading to inaccuracies in the calculation of the similarity of some segments. Therefore, using a first audio segment pair can achieve more accurate local alignment and correct the deviations caused by speed changes. In other words, by expanding the alignment position left and right to obtain the first audio segment pair, the local alignment position can be further accurately determined, improving the accuracy of identical audio recognition.

[0111] In step S730, for each first audio segment pair, local audio feature alignment is performed on the candidate audio segment and the audio segment to be identified in the first audio segment pair based on the third fingerprint information of the candidate audio segment in the first audio segment pair and the fourth fingerprint information of the audio segment to be identified.

[0112] In one exemplary embodiment, the third fingerprint information includes a third set of local spectral features of the candidate audio segment, and the second fingerprint information includes a fourth set of local spectral peak features of the audio segment to be identified.

[0113] Based on this, the following will be combined with Figure 8 An exemplary implementation of step S730 will be described. Exemplary, Figure 8 This diagram illustrates a flowchart of another method for local audio feature alignment according to an exemplary embodiment of the present disclosure. (See reference...) Figure 8 The method may include steps S810 to S830. Wherein:

[0114] In step S810, a second inverted index library is generated based on the time information corresponding to the same frequency in the fourth spectral local peak feature set.

[0115] For example, for the fourth fingerprint information of each audio segment to be identified, a second inverted index library can be generated for each audio segment to be identified.

[0116] The generation method of the second inverted index can be referenced from the generation method of the first inverted index described above, and will not be repeated here.

[0117] In step S820, for each third spectrum local peak feature in the third spectrum local peak feature set, the time information corresponding to the frequency of the third spectrum local peak feature is searched in the second inverted index library, and the second difference between the searched time information and the time of the second spectrum local peak feature is calculated.

[0118] The specific method for determining the second difference can be referred to the specific method for determining the first difference mentioned above, and will not be repeated here.

[0119] In step S830, local audio feature alignment is performed on the candidate audio segments and the audio segments to be identified in the first audio segment pair based on the second difference.

[0120] For example, one specific implementation of step S830 may include: counting the number of times different second differences occur in the second difference, determining the second maximum occurrence number based on the maximum value among the second occurrence numbers; if the second maximum occurrence number is greater than or equal to a second preset value, using the second difference corresponding to the second maximum occurrence number as the second alignment time difference; and performing local audio feature alignment on the candidate audio segments and the audio segments to be identified in the first audio segment pair based on the second alignment time difference.

[0121] In other words, for each first audio segment pair, local audio feature alignment processing can be performed according to the method of steps S810 to S830.

[0122] For example, the specific implementation of step S830 can be referred to the relevant content of the specific implementation of step S430 above, and will not be repeated here.

[0123] For example, the method of local audio feature alignment is the same as the method of overall audio feature alignment in steps S410 to S430 above, only the segments affected are different. Local audio feature alignment is applied to align two audio segments in the first audio segment pair, while overall alignment is applied to align the candidate audio and the complete audio to be identified.

[0124] In step S620, the similarity recognition result between the audio to be identified and the candidate audio is determined based on the result of the alignment of local audio features of each first audio segment pair.

[0125] Below, in conjunction with Figure 9 A specific implementation of step S620 will be described. For example, Figure 9 A flowchart illustrating another method for determining similarity identification results according to an exemplary embodiment of this disclosure is shown. (See reference...) Figure 9 The method may include steps S910 to S940. Wherein:

[0126] In step S910, if the second maximum occurrence count is greater than or equal to the second preset value, the first audio segment pair is determined to be the target audio segment pair.

[0127] For example, for each first audio segment pair, if its corresponding second maximum occurrence count is greater than or equal to a second preset value, it is determined that the two audio segments in the first audio segment pair can be aligned, and thus the first audio segment pair is identified as the target audio segment pair; otherwise, it is determined that the two audio segments in the first audio segment pair cannot be locally aligned. When any first audio segment pair cannot be locally aligned, it can be determined that the candidate audio and the audio to be identified are different.

[0128] In step S920, when all the first audio segment pairs corresponding to the candidate audio are determined to be the target audio segment pairs, an audio segment of the first preset duration is extracted from the audio segment to be identified in the first audio segment pair based on the alignment position of the local audio feature alignment to obtain a second audio segment. Based on the second audio segment and the candidate audio segments in the first audio segment pair, a second audio segment pair is obtained.

[0129] For example, if each pair of first audio segments corresponding to the candidate audio can be locally aligned, the third start time of the candidate audio segment in each first audio segment pair can be determined based on the start time of the candidate audio segment in each first audio segment pair and the second alignment time difference of each first audio segment pair, which is then the fourth start time in the corresponding audio segment to be identified. Then, starting from the fourth start time, an audio segment of a first preset duration is extracted from the audio segment to be identified to obtain the second audio segment. A second audio segment pair is formed by combining the second audio segment and the candidate audio segments in the first audio segment pair.

[0130] In step S930, the similarity between the candidate audio segment and the second audio segment in the second audio segment pair is determined based on the third fingerprint information of the candidate audio segment and the fifth fingerprint information of the second audio segment.

[0131] For example, one specific implementation of step S930 may include: determining the similarity between the candidate audio segment and the second audio segment in the second audio segment pair based on the overlap between the third fingerprint information and the fifth fingerprint information.

[0132] In one exemplary implementation, the degree of overlap of fingerprint information can be determined based on the number of identical fingerprint features.

[0133] For example, the similarity between the two audio segments in the second audio segment pair can be calculated using the following formula (1):

[0134]

[0135] In formula (1), fconsistency represents the degree of similarity, hit_count represents the number of fingerprints that overlap between two audio segments, i.e. the number of identical fingerprint features mentioned above, ref_seg_count represents the number of fingerprint features of the candidate audio segment, and query_seg_count represents the number of fingerprint features in the second audio segment.

[0136] In other words, for any first audio segment pair, it can be assumed that the total number of fingerprints of candidate audio segments in the first audio segment pair is ref_seg_count. Based on the second alignment time difference of the aforementioned local audio features, the total number of fingerprints of the audio segment to be identified (i.e. the second audio segment mentioned above) in the first preset time period is calculated as query_seg_count. The number of overlapping fingerprints of the two audio segments is determined as hit_count. Then, the similarity between the two audio segments can be determined according to the formula (1) above.

[0137] By calculating the similarity between two audio segments, we can directly measure whether the two audio segments sound identical, thus improving the accuracy of identical audio recognition.

[0138] In step S940, the similarity recognition result between the audio to be identified and the candidate audio is determined based on the similarity between the candidate audio segment and the second audio segment in each pair of second audio segments corresponding to the candidate audio.

[0139] For example, one exemplary implementation of step S940 may include: calculating the average of the similarity between the candidate audio segment and the second audio segment in all second audio segment pairs corresponding to the candidate audio; if the average is greater than or equal to a third preset value, determining the similarity recognition result between the audio to be identified and the candidate audio based on the similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio.

[0140] For example, determining the similarity recognition result between the audio to be identified and the candidate audio based on the similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio includes: determining that the audio to be identified and the candidate audio are the same when the similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio is greater than or equal to a fourth preset value.

[0141] For example, for each candidate audio, the average similarity between the two audio segments in all its corresponding second audio segment pairs can be calculated. If a candidate audio is divided into 10 candidate audio segments, then 10 first audio segment pairs can be determined. If all 10 first audio segment pairs can be locally aligned, then 10 second audio segment pairs can be determined according to the alignment, and then the average similarity between these 10 second audio segment pairs can be calculated. If the average value is greater than or equal to the third preset value, the candidate audio and the audio to be identified are considered to have reached the required overall similarity. Then, based on the similarity of each pair of second audio segments, it can be determined whether the two are the same audio. Specifically, if the similarity of the two audio segments in each pair of second audio segments is greater than or equal to the fourth preset value, it indicates that the similarity of the two audio segments in each pair of second audio segments is relatively high, and it can be determined that the audio to be identified and the candidate audio are the same. If the similarity of the two audio segments in a pair of second audio segments is less than the fourth preset value, it indicates that the similarity of the two audio segments is low, that is, although the candidate audio and the audio to be identified show a high degree of similarity overall, there are large differences in their local parts, so the two are still different audio.

[0142] For example, if the average similarity between the candidate audio segment and the second audio segment in all pairs of second audio segments corresponding to the candidate audio is less than a third preset value, it can be directly determined that the candidate audio and the audio to be identified are not the same. Of course, if the average is greater than or equal to the third preset value, it can also be directly determined that the candidate audio and the audio to be identified are the same, without needing to obtain a similarity identification result based on the similarity of each pair of second audio segments. This exemplary embodiment does not impose any special limitations on this.

[0143] For example, another exemplary implementation of step S940 may include: determining that the audio to be identified and the candidate audio are the same if the similarity between the candidate audio segment and the second audio segment in each pair of second audio segments corresponding to the candidate audio is greater than or equal to a fourth preset value.

[0144] For example, after calculating the similarity between the two audio segments in each second audio segment pair, it is not necessary to first calculate the average similarity and then determine whether the audio to be identified and the candidate audio are the same based on whether the similarity between the two audio segments in each second audio segment pair is greater than or equal to a fourth threshold. Instead, the similarity between the two audio segments in each second audio segment pair can be directly used to determine whether the audio to be identified and the candidate audio are the same. As mentioned above, if the similarity between the two audio segments in each second audio segment pair corresponding to the candidate audio is greater than or equal to the fourth preset value, then the audio to be identified and the candidate audio are determined to be the same.

[0145] For example, as mentioned above, if the similarity between the candidate audio segment and the second audio segment in any second audio segment pair corresponding to the candidate audio is less than the fourth preset value, it is determined that the audio to be identified and the candidate audio are not the same.

[0146] In one exemplary embodiment, the audio recognition method of this disclosure may further include: outputting first prompt information for the candidate audio segment and the second audio segment whose similarity is less than the fourth preset value.

[0147] For example, the first prompt message can provide feedback to the user on the client side regarding the candidate audio segment and the second audio segment with a similarity of less than the fourth preset value, thereby helping the user to locate the reason why the candidate audio and the audio to be identified are different and the location of the difference.

[0148] In one exemplary embodiment, the audio recognition method of this disclosure may further include: determining that the candidate audio is different from the audio to be recognized when the second maximum occurrence count is less than the second preset value.

[0149] For example, if the second maximum occurrence count is less than the second preset value, it indicates that the two audio segments cannot be aligned using local audio features. This means the two audio segments have significant local differences and cannot be aligned. In other words, if there are segments in the candidate audio and the audio to be identified that cannot be aligned using local audio features, then the candidate audio and the audio to be identified are different. Alternatively, a second prompt message can be output regarding the two audio segments whose second maximum occurrence count is less than the second preset value.

[0150] Through steps S610 to S620 described above, if it is determined that the audio to be identified and the candidate audio can be aligned as a whole, the aligned candidate audio and the audio to be identified can be further segmented. Based on the segmentation result, the audio to be identified and the candidate audio can continue to undergo local audio feature alignment on the basis of overall alignment. Based on the result of local audio feature alignment, it can be determined whether the audio to be identified and the candidate audio are the same, and the segments that differ between them can be obtained based on the local feature alignment. In this way, not only can subtle differences between audio be identified, improving the accuracy of identifying identical audio, but also the differences between audio can be located, thereby assisting in improving the efficiency of subsequent manual review.

[0151] In another exemplary embodiment, after overall alignment, one or more of the following can be performed on the audio to be recognized: head missing check, head insertion check, tail insertion check, and tail missing check. After all one or more of these checks pass, local feature alignment is then performed. Based on the results of the local feature alignment, it can be determined whether the audio to be recognized and the candidate audio are the same. This not only allows for a more refined comparison of the audio, improving the accuracy of identifying identical audio, but also improves the efficiency of identifying identical audio by performing local feature alignment only after the head missing check, head insertion check, tail missing check, and tail insertion check have passed.

[0152] Below, in conjunction with Figure 10 Another specific embodiment of step S230 will be described. For example, Figure 10 A flowchart illustrating another method for determining similarity identification results according to an exemplary embodiment of this disclosure is shown. (See reference...) Figure 10 The method may include steps S1010 to S1040. Wherein:

[0153] In step S1010, the first sum of the first start time and the first alignment time difference of the candidate audio is calculated.

[0154] In one exemplary implementation, a first summation of the first start time and the first alignment time difference of the candidate audio can be calculated to obtain a new start time of the candidate audio after overall alignment. Here, the first start time can be understood as the actual start time of the candidate audio, i.e., the start time before overall alignment.

[0155] In step S1020, if the relationship between the first sum and the second start time of the audio to be identified satisfies the first preset condition, the predicted end time of the audio to be identified is obtained based on the second sum of the first end time and the first alignment time difference of the candidate audio.

[0156] In one exemplary implementation, the first preset condition includes that the first sum value is the same as the second start time of the audio to be identified, or that the first sum value is greater than the second start time and the energy of the audio segment to be confirmed is less than or equal to an energy threshold. The audio segment to be confirmed includes the audio segment in the audio to be identified that starts at the second start time and has a length equal to the first alignment time difference. The second start time can be understood as the actual start time of the audio to be identified.

[0157] In one exemplary implementation, the actual end time of the audible segment of a candidate audio can be detected based on audio energy. The predicted end time of the audio to be identified is then obtained based on the sum of the overall alignment time difference and the actual end time. Alternatively, the actual end time can be obtained directly from the end time of the candidate audio, without needing to detect the actual end time of the candidate audio based on energy.

[0158] For example, when overall alignment is possible, further checks can be performed on the audio to be identified to determine if it has missing or inserted headers. Specifically, if the first sum value differs from the start time of the audio to be identified, and the first sum value is less than the second start time of the audio to be identified, it indicates that the audio to be identified is missing an audio segment from the candidate audio between the first sum value and the second start time; that is, the audio to be identified is missing the header segment of the candidate audio. Therefore, it can be determined that the audio to be identified and the candidate audio are different. If the first sum value is greater than the second start time, it indicates that the audio to be identified has an additional audio segment from the second start time to the time corresponding to the first sum value compared to the header of the candidate audio. In other words, after overall alignment, the audio to be identified has an inserted audio segment at the header compared to the candidate audio. This inserted audio segment can then be used as the audio segment to be confirmed. If the energy of the audio segment to be confirmed is greater than or equal to the energy threshold, it indicates that the segment to be confirmed is not a silent segment. This means that the audio to be identified has an inserted audio segment at the header compared to the candidate audio, indicating that the audio to be identified and the candidate audio are different, and the difference includes the inserted audio segment at the header of the audio to be identified. If the energy of the segment to be confirmed is less than the energy threshold, it indicates that the segment to be confirmed is a silent segment. This means that the audio to be identified does not have a head insertion compared to the candidate audio, and the head insertion check passes.

[0159] Similarly, if the first sum is the same as the start time of the audio to be identified, it is determined that the audio to be identified does not have a missing head or a inserted head, that is, the missing head and inserted head checks pass.

[0160] In one exemplary implementation, if the head missing check and / or head insertion check pass after overall alignment, it can be directly determined that the audio to be identified and the candidate audio are the same.

[0161] In another exemplary embodiment, if the head insertion and head loss verifications pass, a tail loss and / or tail insertion verification can be further performed in step S1030. Based on the tail verification results, it can be determined whether the audio to be identified and the candidate audio are the same.

[0162] Next, in step S1030, if the relationship between the second end time of the audio to be identified and the predicted end time satisfies the second preset condition, the candidate audio and the audio to be identified after overall alignment are segmented to obtain multiple first audio segment pairs, and local audio feature alignment is performed on each first audio segment pair.

[0163] In one exemplary embodiment, the second preset condition includes that the second end time of the audio to be identified is the same as the predicted end time, or the predicted end time is less than the second end time and the absolute value of the third difference between the two is less than or equal to a fifth preset value. Here, the second end time can be understood as the actual end time of the audio to be identified.

[0164] For example, if the actual end time of the audio to be identified is the same as the predicted end time obtained based on the overall alignment, it means that the audio to be identified does not have any tail insertion or tail missing compared to the candidate audio, that is, the tail missing check and tail insertion check of the audio to be identified both pass.

[0165] If the predicted end time and the second end time are different, and the predicted end time is greater than the actual end time of the audio to be identified, it means that the audio to be identified is missing the tail segment of the candidate audio. This tail segment is the audio segment within the time period from the actual end time of the audio to the predicted end time. In other words, the tail missing check fails, and it can be directly determined that the candidate audio and the audio to be identified are different.

[0166] If the predicted end time and the second end time are different, and the predicted end time is much smaller than the actual end time of the audio to be identified, it means that there is redundant content at the end of the audio to be identified relative to the candidate audio, and the end insertion verification will fail.

[0167] If the predicted end time is less than the actual end time of the audio to be recognized, but the absolute value of the difference between the predicted end time and the actual end time of the audio to be recognized is less than or equal to the fifth preset value, it indicates that the predicted end time is not significantly less than the actual end time of the audio to be recognized; otherwise, it indicates that the predicted end time is significantly less than the actual end time of the audio to be recognized. If the predicted end time is less than the actual end time of the audio to be recognized but not significantly less, the tail insertion check is still considered to have passed. This takes into account the possibility that the audio may gradually fade out, i.e., the energy gradually decreases until it ends. The fade-out effect of each audio may vary, so a small time difference between the predicted end time and the actual end time is allowed. If the time difference is small, the tail insertion check of the audio to be recognized is still considered to have passed. Similarly, the above-mentioned tail missing check can also be determined as follows: if the predicted end time is significantly greater than the actual end time of the audio to be recognized, the tail missing check is determined to have failed; otherwise, the tail missing check is determined to have passed.

[0168] In one exemplary implementation, if the tail-missing check and the tail-insertion check fail, it is determined that the audio to be identified and the candidate audio are different.

[0169] In one exemplary implementation, if both the tail-deletion check and the tail-insertion check pass, it can be directly determined that the audio to be identified and the candidate audio are the same.

[0170] In another exemplary embodiment, if the tail-missing check and tail-insertion check pass, the overall aligned candidate audio and the audio to be identified can be segmented to obtain multiple first audio segment pairs. Local audio feature alignment is then performed on each first audio segment pair. Therefore, in step S1040, based on the result of the local audio feature alignment, it is determined whether the audio to be identified and the candidate audio are the same. The specific implementation of "segmenting the overall aligned candidate audio and the audio to be identified to obtain multiple first audio segment pairs, and performing local audio feature alignment on each first audio segment pair" in step S1030 can be referred to the specific implementation of the relevant content in step S610 above, and will not be repeated here.

[0171] In step S1040, the similarity recognition result between the audio to be identified and the candidate audio is determined based on the result of the local audio feature alignment of each first audio segment pair.

[0172] For example, the specific implementation of step S1040 can be referred to step S620 above, and will not be repeated here.

[0173] Through the steps S1010 to S1040 described above, the similarity between candidate audio and audio to be identified can be verified from multiple dimensions, thereby improving the accuracy of identical audio recognition.

[0174] For example, as described above, the audio recognition method in this disclosure may further include: determining that the audio to be recognized and the candidate audio are different when the first sum value is less than the second start time.

[0175] For example, if the first sum is less than the second start time, it indicates that the audio to be identified is missing a head segment compared to the candidate audio. That is, the audio segment to be identified is missing the audio segment in the candidate audio that corresponds to the time period from the first sum to the second start time. Therefore, it can be directly determined that the audio to be identified and the candidate audio are different.

[0176] For example, as described above, the audio recognition method in this disclosure may further include: determining that the audio to be recognized and the candidate audio are different when the first sum value is greater than the second start time and the energy of the audio segment to be confirmed is greater than the energy threshold.

[0177] For example, if the first sum is greater than the second start time, it means that the audio to be identified has the aforementioned audio segment to be confirmed inserted at the beginning compared to the candidate audio. If the energy of the audio segment to be confirmed is greater than the energy threshold, it means that the inserted audio segment to be confirmed is not a completely silent segment, but a segment with actual audio content. Therefore, it can be determined that the audio to be identified and the candidate audio are different.

[0178] For example, the audio recognition method in this disclosure may further include: determining that the audio to be recognized and the candidate audio are different when the second end time and the predicted end time of the audio to be recognized are different.

[0179] For example, determining that the audio to be identified and the candidate audio are different when the second end time of the audio to be identified is different from the predicted end time includes: determining that the audio to be identified and the candidate audio are different when the predicted end time is less than the second end time, or when the predicted end time is less than the second end time and the absolute value of the third difference is greater than the fifth preset value; and determining that the audio to be identified and the candidate audio are different when the predicted end time is greater than the second end time.

[0180] For example, the audio recognition method in this disclosure may further include: determining that the audio to be recognized and the candidate audio are different when the first maximum occurrence number corresponding to the candidate audio is less than the first preset value.

[0181] For example, if the first maximum occurrence count of the candidate audio is less than the first preset value, it means that the candidate audio and the audio to be identified cannot be aligned as a whole, that is, there is a large difference between the candidate audio and the audio to be identified, so it can be determined that the video to be identified and the candidate audio are different.

[0182] In one exemplary scenario of this disclosure, a two-stage detection method can improve the accuracy and efficiency of identifying identical audio. In the first stage, candidate audio is selected from the audio pairs to be compared based on the similarity of their audio feature vectors. In the second stage, for each candidate audio, its audio features are aligned with the audio to be identified. This allows for multi-dimensional verification of each audio pair consisting of a candidate audio and the audio to be identified. Based on the results of these multi-dimensional verifications, it is determined whether the two are identical, thereby identifying the audio that is identical to the audio to be identified from the candidate audio.

[0183] The aforementioned multi-dimensional verifications can include alignment confidence verification, header missing verification, header insertion verification, intermediate data verification, tail missing verification, and tail insertion verification.

[0184] The alignment confidence check can be understood as follows: if the overall fingerprint alignment fails, or the average value of the segment similarity calculated after alignment is lower than a certain threshold, then the overall similarity between the two audios is insufficient, the alignment confidence check fails, and it is directly determined that the two are not the same and the check ends; otherwise, proceed to the next step of header missing check.

[0185] Header missing verification can be understood as follows: if the fingerprint as a whole can be aligned, but the alignment start time of the candidate audio to the audio to be identified is less than the actual start time of the audio to be identified, it means that the head content of the candidate audio cannot be found in the audio to be identified. That is, the audio to be identified has missing head data compared to the candidate audio. The head verification fails, and it is directly determined that the two are different and the verification ends. Otherwise, proceed to the next step of head insertion verification.

[0186] Header insertion verification can be understood as follows: If, after overall alignment, the alignment start time of the candidate audio to the audio to be recognized is greater than the actual start time of the audio to be recognized, it means that the content of the audio to be recognized from 0 to the alignment start time is redundant compared to the candidate audio. Then, it can be checked whether this content is entirely silent. If it is not silent, it means that the head of the audio to be recognized has inserted content that is not in the candidate audio, and the head insertion verification fails. It is directly determined that the two are different and the verification ends. Otherwise, it proceeds to the next step of intermediate data verification.

[0187] Intermediate data verification can be understood as: verifying the similarity between the two audio segments in the aforementioned second audio segment pair corresponding to the candidate audio, used to determine whether the content of each local segment pair is consistent. If there are second audio segment pairs with a similarity below a certain threshold, then all second audio segment pairs with a similarity below the threshold are saved and output. If the similarity of all second audio segment pairs is greater than or equal to a certain threshold, then the intermediate data verification passes and proceeds to the next step of tail-missing data verification; otherwise, the intermediate data verification fails and the verification ends.

[0188] Tail missing verification can be understood as follows: based on the end position of the candidate audio segment in energy detection, and based on the overall alignment time difference, i.e. the first alignment time difference mentioned above and the end position of the candidate audio segment, the estimated end position of the audio segment to be identified is obtained. If the estimated end position exceeds the actual end position of the audio to be identified, it can be determined that the audio to be identified does not have the tail content of the candidate audio, i.e., the tail verification fails and the verification ends; otherwise, the next step of tail insertion verification is performed.

[0189] Tail insertion verification can be understood as follows: if the estimated end position of the audio segment to be identified does not exceed the actual end position of the audio segment to be identified, then the actual end position of the audio segment to be identified can be further detected. If the actual end position is much larger than the estimated end position, it means that there is redundant content at the end of the audio segment to be identified, and then it can be determined that the tail insertion verification of the audio segment to be identified fails, and the two are not the same; otherwise, the two are the same.

[0190] In other words, if any one of the above multiple dimensions fails the verification, it can be determined that the audio to be identified and the candidate audio are different. If the verification of each dimension passes, it can be determined that the audio to be identified and the candidate audio are the same.

[0191] It should be noted that the above-mentioned multiple dimensions of verification can be selected according to needs, and it is not that every dimension must be verified. In addition, the verification order of multiple dimensions can also be determined according to needs or experience. This exemplary implementation does not impose any special limitations on this.

[0192] This disclosure combines machine learning models and fingerprinting technology to achieve identical audio recognition. This improves recognition efficiency, while fingerprinting technology, capable of precisely comparing subtle differences in audio signals, enhances accuracy. Secondly, this disclosure employs local audio feature alignment for identical audio recognition, identifying different audio tracks and accurately locating the differences between them. This accurate location of differences improves the efficiency of subsequent manual review. Furthermore, this disclosure accurately identifies identical audio; for example, by identifying duplicate songs, the number of repeated songs in the song library can be reduced, saving storage resources. Moreover, the two-stage audio recognition method in this disclosure allows for flexible adjustment of detection dimensions to adapt to different application scenarios, demonstrating strong versatility and scalability.

[0193] Furthermore, it should be noted that the above figures are merely illustrative representations of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0194] Exemplary device

[0195] This disclosure also provides an audio recognition device through exemplary embodiments. (See reference...) Figure 11 As shown, the audio recognition device 1100 may include the following program modules: a candidate audio determination module 1110, configured to determine candidate audio from existing audio based on the similarity between a first audio feature vector of the audio to be recognized and a second audio feature vector of existing audio in an audio library; an audio feature alignment module 1120, configured to perform audio feature alignment on the audio to be recognized and the candidate audio for each candidate audio based on the first fingerprint information of the audio to be recognized and the second fingerprint information of the candidate audio; and a same audio recognition module 1130, configured to determine the sameness recognition result between the audio to be recognized and the candidate audio based on the result of the audio feature alignment.

[0196] In one exemplary embodiment, the first fingerprint information includes a first set of local peak features of the spectrum of the audio to be identified, and the second fingerprint information includes a second set of local peak features of the spectrum of the candidate audio. Any local peak feature is determined based on the time and frequency of the local peak. The audio feature alignment module 1120 can be specifically configured to: generate a first inverted index library based on the time information corresponding to the same frequency in the first set of local peak features; for each local peak feature in the second set of local peak features, search for the time information corresponding to the frequency of the local peak feature in the first inverted index library, calculate a first difference between the found time information and the time of the local peak feature; and perform overall audio feature alignment on the audio to be identified and the candidate audio based on the first difference.

[0197] In one exemplary implementation, the step of performing overall audio feature alignment between the audio to be identified and the candidate audio based on the first difference includes:

[0198] Count the number of times each of the different first differences occurs, and determine the first maximum occurrence based on the maximum value among the first occurrences;

[0199] If the first maximum occurrence count is greater than or equal to the first preset value, the first difference corresponding to the first maximum occurrence count is taken as the first alignment time difference;

[0200] Based on the first alignment time difference, the audio features of the audio to be identified and the candidate audio are aligned as a whole.

[0201] In an exemplary implementation, based on the foregoing embodiments, the same audio recognition module 1130 can be specifically configured to: calculate a first sum of the first start time and the first alignment time difference of the candidate audio; if the relationship between the first sum and the second start time of the audio to be recognized satisfies a first preset condition, obtain the predicted end time of the audio to be recognized based on the first end time and the second sum of the first alignment time difference of the candidate audio; if the relationship between the second end time of the audio to be recognized and the predicted end time satisfies a second preset condition, segment the overall aligned candidate audio and the audio to be recognized to obtain multiple first audio segment pairs, and perform local segmentation on each first audio segment pair. Audio feature alignment; based on the local audio feature alignment results of each first audio segment pair, determine the similarity recognition result between the audio to be identified and the candidate audio; wherein, the first preset condition includes the first sum value being the same as the second start time of the audio to be identified, or the first sum value being greater than the second start time and the energy of the audio segment to be confirmed being less than or equal to an energy threshold, the audio segment to be confirmed including the audio segment in the audio to be identified that starts from the second start time and has a length of the first alignment time difference; the second preset condition includes the second end time of the audio to be identified being the same as the predicted end time, or the predicted end time being less than the second end time and the absolute value of the third difference between the two being less than or equal to a fifth preset value.

[0202] In one exemplary implementation, based on the foregoing embodiments, the identical audio recognition module 1130 may be specifically configured to: segment the candidate audio and the audio to be recognized after overall alignment to obtain multiple first audio segment pairs; perform local audio feature alignment on each first audio segment pair; and determine the identity recognition result of the audio to be recognized and the candidate audio based on the result of the local audio feature alignment of each first audio segment pair.

[0203] In one exemplary implementation, based on the foregoing embodiments, the step of segmenting the overall aligned candidate audio and the audio to be identified to obtain multiple first audio segment pairs, and performing local audio feature alignment on each first audio segment pair, includes: segmenting the candidate audio into multiple non-overlapping candidate audio segments according to a first preset duration; extracting the audio segment to be identified corresponding to the candidate audio segment from the overall aligned audio to be identified to obtain a first audio segment pair, wherein the third start time of the audio segment to be identified is less than the fourth start time of the candidate audio segment, and the third end time of the audio segment to be identified is greater than the fourth end time of the candidate audio segment; and for each first audio segment pair, performing local audio feature alignment on the candidate audio segment and the audio segment to be identified in the first audio segment pair according to the third fingerprint information of the candidate audio segment in the first audio segment pair and the fourth fingerprint information of the audio segment to be identified.

[0204] In an exemplary implementation, based on the foregoing embodiments, the third fingerprint information includes a third spectral local feature set of the candidate audio segment, and the second fingerprint information includes a fourth spectral local peak feature set of the audio segment to be identified; the step of aligning the candidate audio segments and the audio segment to be identified in the first audio segment pair with local audio features based on the third fingerprint information of the candidate audio segments and the fourth fingerprint information of the audio segment to be identified in the first audio segment pair includes: generating a second inverted index library based on the time information corresponding to the same frequency in the fourth spectral local peak feature set; for each third spectral local peak feature in the third spectral local peak feature set, searching for the time information corresponding to the frequency of the third spectral local peak feature in the second inverted index library, calculating a second difference between the found time information and the time of the second spectral local peak feature; and aligning the candidate audio segments and the audio segment to be identified in the first audio segment pair with local audio features based on the second difference.

[0205] In one exemplary implementation, based on the foregoing embodiments, the step of performing local audio feature alignment on the candidate audio segments and the audio segments to be identified in the first audio segment pair according to the second difference includes: counting the second occurrence counts of different second differences in the second difference, determining the second maximum occurrence count based on the maximum value among the second occurrence counts; when the second maximum occurrence count is greater than or equal to a second preset value, using the second difference corresponding to the second maximum occurrence count as the second alignment time difference; and performing local audio feature alignment on the candidate audio segments and the audio segments to be identified in the first audio segment pair according to the second alignment time difference.

[0206] In an exemplary implementation, based on the foregoing embodiments, determining the similarity recognition result between the audio to be identified and the candidate audio based on the local audio feature alignment result of each first audio segment pair includes: determining the first audio segment pair as a target audio segment pair when the second maximum occurrence count is greater than or equal to a second preset value; when all first audio segment pairs corresponding to the candidate audio are determined to be target audio segment pairs, extracting an audio segment of the first preset duration from the audio segment to be identified in the first audio segment pair based on the alignment position of the local audio feature alignment to obtain a second audio segment; obtaining a second audio segment pair based on the second audio segment and the candidate audio segments in the first audio segment pair; determining the similarity between the candidate audio segments and the second audio segment in the second audio segment pair based on the third fingerprint information of the candidate audio segments and the fifth fingerprint information of the second audio segment in the second audio segment pair; and determining the similarity recognition result between the audio to be identified and the candidate audio based on the similarity between the candidate audio segments and the second audio segment in each second audio segment pair corresponding to the candidate audio.

[0207] In one exemplary implementation, based on the foregoing embodiments, determining the similarity between the candidate audio segments and the second audio segment in the second audio segment pair based on the third fingerprint information of the candidate audio segments and the fifth fingerprint information of the second audio segment includes:

[0208] ==Based on the overlap between the third fingerprint information and the fifth fingerprint information, the similarity between the candidate audio segment and the second audio segment in the second audio segment pair is determined.

[0209] In one exemplary implementation, based on the foregoing embodiments, determining the similarity recognition result between the audio to be identified and the candidate audio based on the similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio includes: calculating the average of the similarity between the candidate audio segment and the second audio segment in all second audio segment pairs corresponding to the candidate audio; and, if the average is greater than or equal to a third preset value, determining the similarity recognition result between the audio to be identified and the candidate audio based on the similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio.

[0210] In one exemplary implementation, based on the foregoing embodiments, determining the similarity recognition result between the audio to be identified and the candidate audio according to the similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio includes: determining that the audio to be identified and the candidate audio are the same when the similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio is greater than or equal to a fourth preset value.

[0211] In one exemplary implementation, based on the foregoing embodiments, the identical audio recognition module 1130 may be specifically configured to: determine that the audio to be recognized and the candidate audio are not the same if the similarity between the candidate audio segment and the second audio segment in any second audio segment pair corresponding to the candidate audio is less than the fourth preset value.

[0212] In one exemplary embodiment, the device may further include a first prompt information output module, which may be specifically configured to output first prompt information for the candidate audio segment and the second audio segment whose similarity is less than the fourth preset value.

[0213] In one exemplary implementation, based on the foregoing embodiments, the same audio recognition module 1130 may be specifically configured to: determine that the candidate audio and the audio to be recognized are different when the mean is less than the third preset value.

[0214] In one exemplary implementation, based on the foregoing embodiments, the same audio recognition module 1130 can be specifically configured to: determine that the candidate audio is different from the audio to be recognized when the second maximum occurrence count is less than the second preset value.

[0215] In one exemplary implementation, based on the foregoing embodiments, the same audio recognition module 1130 may be specifically configured to: determine that the audio to be recognized and the candidate audio are different when the first sum value is less than the second start time.

[0216] In one exemplary implementation, based on the foregoing embodiments, the same audio recognition module 1130 may be specifically configured to: determine that the audio to be recognized and the candidate audio are different when the first sum is greater than the second start time and the energy of the audio segment to be confirmed is greater than the energy threshold.

[0217] In one exemplary implementation, based on the foregoing embodiments, the same audio recognition module 1130 may be specifically configured to: determine that the audio to be recognized and the candidate audio are different when the second end time of the audio to be recognized is different from the predicted end time.

[0218] In one exemplary embodiment, determining that the audio to be identified and the candidate audio are different when the second end time of the audio to be identified is different from the predicted end time includes: determining that the audio to be identified and the candidate audio are different when the predicted end time is less than the second end time, or when the predicted end time is less than the second end time and the absolute value of the third difference is greater than the fifth preset value; and determining that the audio to be identified and the candidate audio are different when the predicted end time is greater than the second end time.

[0219] In one exemplary implementation, based on the foregoing embodiments, the same audio recognition module 1130 may be specifically configured to: determine that the audio to be recognized and the candidate audio are different when the first maximum occurrence number corresponding to the candidate audio is less than the first preset value.

[0220] The specific details of each part of the above-mentioned device have been described in detail in the corresponding method section of the above-mentioned implementation. For any undisclosed details, please refer to the implementation content of the method section above-mentioned implementation, and therefore will not be repeated here.

[0221] Exemplary storage media

[0222] The storage medium of an exemplary embodiment of the present invention will be described below.

[0223] In this exemplary embodiment, the above method can be implemented by a program product, such as a portable compact disc read-only memory (CD-ROM) containing program code, which can run on a device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0224] The program product may employ any combination of one or more readable media. A readable media may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0225] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0226] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RE, etc., or any suitable combination thereof.

[0227] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0228] Exemplary computer program product

[0229] Exemplary embodiments of this disclosure also provide a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the audio recognition method described above.

[0230] In one embodiment, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing the computer program, such as read-only memory, NAND flash memory, etc.

[0231] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0232] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, C++, and Python. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0233] Computer programs can be carried or transmitted via signals such as electricity, magnetism, light, electromagnetic fields, and infrared radiation. Electronic devices can convert signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code causes the electronic device to execute (more specifically, the processor of the electronic device to execute) the method steps of various exemplary embodiments of this disclosure. For example, the audio recognition method described above can be executed, which includes the following steps: determining candidate audio from existing audio based on the similarity between a first audio feature vector of the audio to be recognized and a second audio feature vector of existing audio in an audio library; for each candidate audio, performing audio feature alignment between the audio to be recognized and the candidate audio based on the first fingerprint information of the audio to be recognized and the second fingerprint information of the candidate audio; and determining the similarity recognition result between the audio to be recognized and the candidate audio based on the result of the audio feature alignment.

[0234] By executing the above method steps through a computer program, the method achieves several advantages. First, based on similarity recognition, it further aligns audio data using audio fingerprint information to identify identical audio, thus improving the accuracy of identical audio recognition. Second, by first determining candidate audio based on similarity, only the candidate audio and the audio to be identified need to be aligned using audio features based on fingerprint information to obtain the recognition result. This eliminates the need to align all audio data in the audio library with the audio to be identified, improving the efficiency of identical audio recognition. Third, by recognizing identical audio, duplicate audio can be removed, saving storage resources.

[0235] Exemplary electronic devices

[0236] refer to Figure 12 An electronic device according to an exemplary embodiment of this disclosure will be described. This electronic device is the terminal device 110 or server 120 described above. The electronic device may include a processor and a memory. The memory stores executable instructions of the processor, such as a computer program. The processor executes the executable instructions to perform the method steps of various exemplary embodiments of this disclosure. Furthermore, the electronic device may also include a display for displaying a graphical user interface.

[0237] The following is for reference. Figure 12 The electronic device is illustrated by way of a general-purpose computing device. It should be understood that... Figure 12 The electronic device 1200 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0238] like Figure 12 As shown, the electronic device 1200 may include: a processor 1210, a memory 1220, a bus 1230, an I / O (input / output) interface 1240, a network adapter 1250, and a display 1260.

[0239] Memory 1220 may include volatile memory, such as RAM 1221 and cache unit 1222, and may also include non-volatile memory, such as ROM 1223. Memory 1220 may also include one or more program modules 1224, such program modules 1224 including, but not limited to: operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. For example, program module 1224 may include the modules in the above-described apparatus.

[0240] The processor 1210 may include one or more processing units, such as an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor, and / or an NPU (Neural-Network Processing Unit).

[0241] The processor 1210 can be used to execute executable instructions stored in the memory 1220, such as the audio recognition method described above, which includes the following steps: determining candidate audio from the existing audio based on the similarity between a first audio feature vector of the audio to be recognized and a second audio feature vector of existing audio in the audio library; for each candidate audio, performing audio feature alignment between the audio to be recognized and the candidate audio based on the first fingerprint information of the audio to be recognized and the second fingerprint information of the candidate audio; and determining the similarity recognition result between the audio to be recognized and the candidate audio based on the result of the audio feature alignment.

[0242] Implementing the above method through a computer program has several advantages. First, based on similarity recognition, audio fingerprint information is used to further align audio samples to identify identical audio, thus improving the accuracy of identical audio recognition. Second, candidate audio samples are determined first based on similarity. Only the candidate audio samples and the audio to be identified need to be aligned with the fingerprint information to obtain the recognition result. This eliminates the need to align all audio samples in the audio library with the audio to be identified, improving the efficiency of identical audio recognition. Third, identical audio samples can be deduplicated through identification, saving storage resources.

[0243] Bus 1230 is used to connect different components of electronic device 1200, and may include data bus, address bus and control bus.

[0244] Electronic device 1200 can communicate with one or more external devices 1300 (e.g., keyboard, mouse, external controller, etc.) through I / O interface 1240.

[0245] Electronic device 1200 can communicate with one or more networks via network adapter 1250. For example, network adapter 1250 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication. Network adapter 1250 can communicate with other modules of electronic device 1200 via bus 1230.

[0246] The electronic device 1200 can display a graphical user interface via a display 1260, such as a graphical user interface that displays the results of audio identity recognition.

[0247] although Figure 12 Other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, may also be configured in the electronic device 1200.

[0248] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0249] It should be understood that this disclosure is not limited to the specific methods, steps, or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. Those skilled in the art will readily conceive of other embodiments based on the specific implementations provided in this disclosure. Therefore, the specific implementations provided in this disclosure are merely exemplary, and the scope and spirit of this disclosure are indicated by the claims, and should cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary technical means in the art not disclosed in this disclosure.

Claims

1. An audio recognition method, characterized in that, include: Candidate audio is determined from the existing audio based on the similarity between the first audio feature vector of the audio to be identified and the second audio feature vector of the existing audio in the audio library; For each candidate audio, audio feature alignment is performed on the audio to be identified and the candidate audio based on the first fingerprint information of the audio to be identified and the second fingerprint information of the candidate audio. Based on the results of audio feature alignment, the similarity recognition result between the audio to be identified and the candidate audio is determined; The step of aligning the audio features of the audio to be identified and the candidate audio includes: aligning the audio features of the audio to be identified and the candidate audio as a whole; The method of determining the similarity recognition result between the audio to be identified and the candidate audio based on the audio feature alignment result includes: segmenting the candidate audio and the audio to be identified after overall alignment to obtain multiple first audio segment pairs, performing local audio feature alignment on each first audio segment pair, and determining the similarity recognition result between the audio to be identified and the candidate audio based on the local audio feature alignment result of each first audio segment pair.

2. The method according to claim 1, characterized in that, The first fingerprint information includes a first set of local peak features of the spectrum of the audio to be identified, and the second fingerprint information includes a second set of local peak features of the spectrum of the candidate audio. Any local peak feature is determined based on the time and frequency of the local peak. The step of aligning the audio features of the audio to be identified and the candidate audio based on the first fingerprint information of the audio to be identified and the second fingerprint information of the candidate audio includes: A first inverted index library is generated based on the time information corresponding to the same frequency in the first set of local peak features of the spectrum. For each second local peak feature in the second local peak feature set, the time information corresponding to the frequency of the second local peak feature is searched in the first inverted index library, and the first difference between the searched time information and the time of the second local peak feature is calculated. Based on the first difference, the audio features of the audio to be identified and the candidate audio are aligned as a whole.

3. The method according to claim 2, characterized in that, The step of performing overall audio feature alignment between the audio to be identified and the candidate audio based on the first difference includes: Count the number of times each of the different first differences occurs, and determine the first maximum occurrence based on the maximum value among the first occurrences; If the first maximum occurrence count is greater than or equal to the first preset value, the first difference corresponding to the first maximum occurrence count is taken as the first alignment time difference; Based on the first alignment time difference, the audio features of the audio to be identified and the candidate audio are aligned as a whole.

4. The method according to claim 3, characterized in that, The determination of the similarity between the audio to be identified and the candidate audio based on the audio feature alignment result includes: Calculate the first sum of the first start time and the first alignment time difference of the candidate audio; If the relationship between the first sum and the second start time of the audio to be identified satisfies the first preset condition, the predicted end time of the audio to be identified is obtained based on the second sum of the first end time and the first alignment time difference of the candidate audio. When the relationship between the second end time of the audio to be identified and the predicted end time satisfies the second preset condition, the candidate audio and the audio to be identified after overall alignment are segmented to obtain multiple first audio segment pairs, and local audio feature alignment is performed on each first audio segment pair. Based on the alignment results of the local audio features of each first audio segment pair, the similarity recognition result between the audio to be identified and the candidate audio is determined; The first preset condition includes the first sum value being the same as the second start time of the audio to be identified, or the first sum value being greater than the second start time and the energy of the audio segment to be confirmed being less than or equal to the energy threshold. The audio segment to be confirmed includes the audio segment in the audio to be identified that starts from the second start time and has the first alignment time difference as its length. The second preset condition includes that the second end time of the audio to be identified is the same as the predicted end time, or the predicted end time is less than the second end time and the absolute value of the third difference between the two is less than or equal to the fifth preset value.

5. The method according to claim 1 or 4, characterized in that, The step of segmenting the overall aligned candidate audio and the audio to be identified to obtain multiple first audio segment pairs, and performing local audio feature alignment on each first audio segment pair includes: According to a first preset duration, the candidate audio is divided into multiple non-overlapping candidate audio segments; The audio segment to be identified corresponding to the candidate audio segment is extracted from the overall aligned audio to be identified to obtain a first audio segment pair, wherein the third start time of the audio segment to be identified is less than the fourth start time of the candidate audio segment, and the third end time of the audio segment to be identified is greater than the fourth end time of the candidate audio segment. For each first audio segment pair, local audio feature alignment is performed on the candidate audio segments and the audio segment to be identified in the first audio segment pair based on the third fingerprint information of the candidate audio segments in the first audio segment pair and the fourth fingerprint information of the audio segment to be identified.

6. The method according to claim 5, characterized in that, The third fingerprint information includes a third set of local spectral features of the candidate audio segment, and the second fingerprint information includes a fourth set of local peak spectral features of the audio segment to be identified. The step of aligning the candidate audio segments and the audio segment to be identified in the first audio segment pair with local audio features based on the third fingerprint information of the candidate audio segments and the fourth fingerprint information of the audio segment to be identified includes: A second inverted index library is generated based on the time information corresponding to the same frequency in the fourth set of local peak features of the spectrum; For each third spectrum local peak feature in the third spectrum local peak feature set, the time information corresponding to the frequency in the third spectrum local peak feature is searched in the second inverted index library, and the second difference between the searched time information and the time of the second spectrum local peak feature is calculated. Based on the second difference, local audio feature alignment is performed on the candidate audio segments and the audio segments to be identified in the first audio segment pair.

7. The method according to claim 6, characterized in that, The step of aligning local audio features of the candidate audio segments and the audio segments to be identified in the first audio segment pair based on the second difference includes: Count the number of times each of the different second differences occurs in the second difference, and determine the second maximum occurrence based on the maximum value among the second occurrences; If the second maximum occurrence count is greater than or equal to the second preset value, the second difference corresponding to the second maximum occurrence count is used as the second alignment time difference; Based on the second alignment time difference, local audio feature alignment is performed on the candidate audio segments and the audio segments to be identified in the first audio segment pair.

8. The method according to claim 7, characterized in that, The step of determining the similarity recognition result between the audio to be identified and the candidate audio based on the alignment result of the local audio features of each first audio segment pair includes: If the second maximum occurrence count is greater than or equal to the second preset value, the first audio segment pair is determined to be the target audio segment pair; When all the first audio segment pairs corresponding to the candidate audio are determined to be the target audio segment pairs, the first preset duration audio segment is extracted from the audio segment to be identified in the first audio segment pair based on the alignment position of the local audio feature alignment to obtain the second audio segment. The second audio segment pair is obtained according to the second audio segment and the candidate audio segments in the first audio segment pair. Based on the third fingerprint information of the candidate audio segment and the fifth fingerprint information of the second audio segment in the second audio segment pair, the similarity between the candidate audio segment and the second audio segment in the second audio segment pair is determined. The similarity recognition result between the audio to be identified and the candidate audio is determined based on the similarity between the candidate audio segment and the second audio segment in each pair of second audio segments corresponding to the candidate audio.

9. The method according to claim 8, characterized in that, The determination of the similarity between the candidate audio segments and the second audio segment in the second audio segment pair based on the third fingerprint information of the candidate audio segments and the fifth fingerprint information of the second audio segment includes: Based on the overlap between the third fingerprint information and the fifth fingerprint information, the similarity between the candidate audio segment and the second audio segment in the second audio segment pair is determined.

10. The method according to claim 8, characterized in that, The step of determining the similarity recognition result between the audio to be identified and the candidate audio based on the similarity between the candidate audio segment and the second audio segment in each pair of second audio segments corresponding to the candidate audio includes: Calculate the average similarity between the candidate audio segment and the second audio segment in all pairs of second audio segments corresponding to the candidate audio; If the mean is greater than or equal to a third preset value, the similarity recognition result between the audio to be identified and the candidate audio is determined based on the similarity between the candidate audio segment and the second audio segment in each pair of second audio segments corresponding to the candidate audio.

11. The method according to claim 8 or 10, characterized in that, The similarity recognition result between the audio to be identified and the candidate audio is determined based on the similarity between the candidate audio segment and the second audio segment in each pair of second audio segments corresponding to the candidate audio, including: If the similarity between the candidate audio segment and the second audio segment in each pair of second audio segments corresponding to the candidate audio is greater than or equal to a fourth preset value, then the audio to be identified and the candidate audio are determined to be the same.

12. The method according to claim 11, characterized in that, The method further includes: If the similarity between the candidate audio segment and the second audio segment in any second audio segment pair corresponding to the candidate audio is less than the fourth preset value, it is determined that the audio to be identified and the candidate audio are not the same.

13. The method according to claim 11, characterized in that, The method further includes: Output the first prompt information for the candidate audio segment and the second audio segment whose similarity is less than the fourth preset value.

14. The method according to claim 10, characterized in that, The method further includes: If the mean value is less than the third preset value, it is determined that the candidate audio and the audio to be identified are different.

15. The method according to claim 7, characterized in that, The method further includes: If the second maximum occurrence count is less than the second preset value, it is determined that the candidate audio is different from the audio to be identified.

16. The method according to claim 4, characterized in that, The method further includes: If the first sum is less than the second start time, it is determined that the audio to be identified and the candidate audio are not the same.

17. The method according to claim 4, characterized in that, The method further includes: If the first sum is greater than the second start time and the energy of the audio segment to be identified is greater than the energy threshold, it is determined that the audio to be identified and the candidate audio are not the same.

18. The method according to claim 4, characterized in that, The method further includes: If the second end time of the audio to be identified is different from the predicted end time, it is determined that the audio to be identified and the candidate audio are not the same.

19. The method according to claim 18, characterized in that, The step of determining that the audio to be identified and the candidate audio are different when the second end time of the audio to be identified is different from the predicted end time includes: If the prediction end time is less than the second end time, or if the prediction end time is less than the second end time and the absolute value of the third difference is greater than the fifth preset value, it is determined that the audio to be identified and the candidate audio are not the same. If the prediction end time is greater than the second end time, it is determined that the audio to be identified and the candidate audio are not the same.

20. The method according to claim 3, characterized in that, The method further includes: If the first maximum occurrence count corresponding to the candidate audio is less than the first preset value, it is determined that the audio to be identified and the candidate audio are not the same.

21. An audio recognition device, characterized in that, include: The candidate audio determination module is configured to determine candidate audio from the existing audio based on the similarity between a first audio feature vector of the audio to be identified and a second audio feature vector of existing audio in the audio library; The audio feature alignment module is configured to perform audio feature alignment on the audio to be identified and the candidate audio for each candidate audio, based on the first fingerprint information of the audio to be identified and the second fingerprint information of the candidate audio. The same audio recognition module is configured to determine the sameness recognition result between the audio to be recognized and the candidate audio based on the result of audio feature alignment; The step of aligning the audio features of the audio to be identified and the candidate audio includes: aligning the audio features of the audio to be identified and the candidate audio as a whole; The method of determining the similarity recognition result between the audio to be identified and the candidate audio based on the audio feature alignment result includes: segmenting the candidate audio and the audio to be identified after overall alignment to obtain multiple first audio segment pairs, performing local audio feature alignment on each first audio segment pair, and determining the similarity recognition result between the audio to be identified and the candidate audio based on the local audio feature alignment result of each first audio segment pair.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 20.

23. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 20.

24. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 20.

Images