Audio recognition method, audio recognition device, medium, product and electronic equipment

Abstract

The invention relates to the technical field of computers, and provides an audio recognition method, an audio recognition device, a medium, a product and electronic equipment. The audio recognition method comprises the following steps: determining candidate audios from existing audios according to the similarity between a first audio feature vector of an audio to be recognized and a second audio feature vector of the existing audios in an audio library; for each candidate audio, according to the first fingerprint information of the to-be-identified audio and the second fingerprint information of the candidate audio, performing audio feature alignment on the to-be-identified audio and the candidate audio; and based on an audio feature alignment result, determining an identity identification result of the to-be-identified audio and the candidate audio. According to the scheme, the identification accuracy of the same audio can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing technology, and in particular, to an audio recognition method, an audio recognition device, a computer-readable storage medium, a computer program product, and an electronic device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims, and no description herein is admitted to be prior art by inclusion in this section.

[0003] Identical audio recognition can be applied to fields such as voice unlocking and identical song recognition. In related technologies, two audios are encoded into feature vectors through a neural network model, and the similarity between the two audios is measured based on the similarity between the feature vectors, thereby obtaining the recognition result of similar audios. Summary of the invention

[0004] However, the audio features extracted by related technologies are relatively sparse, and when the audio duration is long, the recognition accuracy of the same audio is low.

[0005] To this end, an audio recognition method is highly desired to improve the accuracy of the same audio recognition.

[0006] In this context, embodiments of the present invention are intended to provide an audio recognition method, an audio recognition device, a computer-readable storage medium, a computer program product, and an electronic device.

[0007] According to a first aspect of an embodiment of the present disclosure, an audio recognition method is provided, including: determining candidate audio from the existing audio according to the similarity between a first audio feature vector of the audio to be recognized and a second audio feature vector of the existing audio in an audio library; for each candidate audio, aligning audio features of the audio to be recognized and the candidate audio according to first fingerprint information of the audio to be recognized and second fingerprint information of the candidate audio; and determining a recognition result of the sameness of the audio to be recognized and the candidate audio based on the result of the audio feature alignment.

[0008] Optionally, the first fingerprint information includes a first set of spectral local peak features of the audio to be identified, and the second fingerprint information includes a second set of spectral local peak features of the candidate audio, and any spectral local peak feature is determined according to the time and frequency of the spectral local peak; the audio feature alignment of the audio to be identified and the candidate audio according to the first fingerprint information of the audio to be identified and the second fingerprint information of the candidate audio includes: generating a first inverted index library according to the time information corresponding to the same frequency in the first spectral local peak feature set; for each second spectral local peak feature in the second spectral local peak feature set, searching the first inverted index library for the time information corresponding to the frequency in the second spectral local peak feature, and calculating the first difference between the found time information and the time of the second spectral local peak feature; and aligning the audio features of the audio to be identified and the candidate audio as a whole according to the first difference.

[0009] Optionally, the overall alignment of audio features of the audio to be identified and the candidate audio based on the first difference includes: counting the first occurrence times of different first difference values ​​in the first difference values, and determining a first maximum occurrence time according to the maximum value of the first occurrence times; when the first maximum occurrence time is greater than or equal to a first preset value, using the first difference value corresponding to the first maximum occurrence time as a first alignment time difference; and based on the first alignment time difference, the overall alignment of audio features of the audio to be identified and the candidate audio.

[0010] Optionally, the result of determining the identity recognition result of the audio to be recognized and the candidate audio based on the audio feature alignment includes: calculating a first sum of the first start time of the candidate audio and the first alignment time difference; when the relationship between the first sum and the second start time of the audio to be recognized meets a first preset condition, obtaining the predicted end time of the audio to be recognized according to the first end time of the candidate audio and the second sum of the first alignment time difference; when the relationship between the second end time of the audio to be recognized and the predicted end time meets a second preset condition, segmenting the overall aligned candidate audio and the audio to be recognized to obtain multiple first audio segment pairs, and locally performing segmentation on each first audio segment pair. Audio feature alignment; determining the sameness recognition result of the audio to be identified and the candidate audio according to the result of the local audio feature alignment of each first audio segment pair; wherein, the first preset condition includes that the first sum is the same as the second start time of the audio to be identified, or the first sum is greater than the second start time and the energy of the audio segment to be confirmed is less than or equal to the energy threshold, and the audio segment to be confirmed includes an audio segment in the audio to be identified that takes the second start time as the starting point and the first alignment time difference as the length; the second preset condition includes that the second end time of the audio to be identified is the same as the predicted end time, or the predicted end time is less than the second end time and the absolute value of the third difference between the two is less than or equal to the fifth preset value.

[0011] Optionally, the determination of the sameness recognition result of the audio to be recognized and the candidate audio based on the result of audio feature alignment includes: segmenting the overall aligned candidate audio and the audio to be recognized to obtain multiple first audio segment pairs, and performing local audio feature alignment on each first audio segment pair; and determining the sameness recognition result of the audio to be recognized and the candidate audio based on the result of local audio feature alignment of each first audio segment pair.

[0012] Optionally, the candidate audio and the audio to be identified after overall alignment are segmented to obtain multiple first audio segment pairs, and the local audio feature alignment for each first audio segment pair includes: segmenting the candidate audio into multiple non-overlapping candidate audio segments according to a first preset duration; cutting out the audio segment to be identified corresponding to the candidate audio segment from the audio to be identified after overall alignment to obtain a first audio segment pair, wherein the third start time of the audio segment to be identified is less than the fourth start time of the candidate audio segment, and the third end time of the audio segment to be identified is greater than the fourth end time of the candidate audio segment; for each first audio segment pair, local audio feature alignment is performed on the candidate audio segment and the audio segment to be identified in the first audio segment pair according to the third fingerprint information of the candidate audio segment in the first audio segment pair and the fourth fingerprint information of the audio segment to be identified.

[0013] Optionally, the third fingerprint information includes a third spectral local feature set of the candidate audio segment, and the second fingerprint information includes a fourth spectral local peak feature set of the audio segment to be identified; the local audio feature alignment of the candidate audio segment and the audio segment to be identified in the first audio segment pair according to the third fingerprint information of the candidate audio segment and the fourth fingerprint information of the audio segment to be identified includes: generating a second inverted index library according to the time information corresponding to the same frequency in the fourth spectral local peak feature set; for each third spectral local peak feature in the third spectral local peak feature set, searching the second inverted index library for the time information corresponding to the frequency in the third spectral local peak feature, and calculating the second difference between the found time information and the time of the second spectral local peak feature; and according to the second difference, performing local audio feature alignment on the candidate audio segment and the audio segment to be identified in the first audio segment pair.

[0014] Optionally, performing local audio feature alignment on the candidate audio segment in the first audio segment pair and the audio segment to be identified based on the second difference includes: counting the second occurrence times of different second difference values ​​in the second difference values, and determining the second maximum occurrence time according to the maximum value of the second occurrence times; when the second maximum occurrence time is greater than or equal to a second preset value, using the second difference corresponding to the second maximum occurrence time as the second alignment time difference; and performing local audio feature alignment on the candidate audio segment in the first audio segment pair and the audio segment to be identified based on the second alignment time difference.

[0015] Optionally, determining the result of the sameness recognition between the audio to be recognized and the candidate audio according to the result of the local audio feature alignment of each first audio segment pair includes: determining the first audio segment pair as a target audio segment pair when the second maximum number of occurrences is greater than or equal to a second preset value; when all the first audio segment pairs corresponding to the candidate audio are determined as target audio segment pairs, cutting out an audio segment of the first preset duration from the audio segment to be recognized of the first audio segment pair based on the alignment position of the local audio feature alignment to obtain a second audio segment, and obtaining a second audio segment pair according to the second audio segment and the candidate audio segment in the first audio segment pair; determining the degree of sameness between the candidate audio segment and the second audio segment in the second audio segment pair based on the third fingerprint information of the candidate audio segment in the second audio segment pair and the fifth fingerprint information of the second audio segment; determining the result of the sameness recognition between the audio to be recognized and the candidate audio according to the degree of sameness between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio.

[0016] Optionally, determining the degree of identity between the candidate audio segment and the second audio segment in the second audio segment pair based on the third fingerprint information of the candidate audio segment and the fifth fingerprint information of the second audio segment in the second audio segment pair includes: determining the degree of identity between the candidate audio segment and the second audio segment in the second audio segment pair based on the degree of overlap between the third fingerprint information and the fifth fingerprint information.

[0017] Optionally, determining the identical recognition result of the audio to be identified and the candidate audio according to the identical degree between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio includes: calculating the mean of the identical degree between the candidate audio segment and the second audio segment in all second audio segment pairs corresponding to the candidate audio; and determining the identical recognition result of the audio to be identified and the candidate audio according to the identical degree between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio when the mean is greater than or equal to a third preset value.

[0018] Optionally, based on the degree of similarity between the candidate audio segment and the second audio segment in each pair of second audio segments corresponding to the candidate audio, determining the result of the similarity recognition between the audio to be identified and the candidate audio includes: when the degree of similarity between the candidate audio segment and the second audio segment in each pair of second audio segments corresponding to the candidate audio is greater than or equal to a fourth preset value, determining that the audio to be identified and the candidate audio are the same.

[0019] Optionally, the method further includes: determining that the audio to be identified is not identical to the candidate audio when the degree of similarity between the candidate audio segment and the second audio segment in any second audio segment pair corresponding to the candidate audio is less than the fourth preset value.

[0020] Optionally, the method further includes: outputting first prompt information of the candidate audio segment and the second audio segment whose similarity is less than the fourth preset value.

[0021] Optionally, the method further includes: when the mean is less than the third preset value, determining that the candidate audio is different from the audio to be identified.

[0022] Optionally, the method further includes: when the second maximum number of occurrences is less than the second preset value, determining that the candidate audio is different from the audio to be identified.

[0023] Optionally, the method further includes: when the first sum value is less than the second start time, determining that the audio to be recognized and the candidate audio are not the same.

[0024] Optionally, the method further includes: when the first sum value is greater than the second start time and the energy of the audio segment to be confirmed is greater than the energy threshold, determining that the audio to be identified and the candidate audio are not the same.

[0025] Optionally, the method further includes: when the second end time of the audio to be recognized and the predicted end time are different, determining that the audio to be recognized and the candidate audio are different.

[0026] Optionally, when the second end time of the audio to be identified and the predicted end time are different, determining that the audio to be identified and the candidate audio are not the same includes: when the predicted end time is less than the second end time, or the predicted end time is less than the second end time and the absolute value of the third difference is greater than the fifth preset value, determining that the audio to be identified and the candidate audio are not the same; when the predicted end time is greater than the second end time, determining that the audio to be identified and the candidate audio are not the same.

[0027] Optionally, the method further includes: when a first maximum number of occurrences corresponding to the candidate audio is less than the first preset value, determining that the audio to be identified is different from the candidate audio.

[0028] According to a second aspect of an embodiment of the present disclosure, an audio recognition device is provided, comprising: a candidate audio determination module, configured to determine candidate audio from the existing audio according to the similarity between a first audio feature vector of the audio to be recognized and a second audio feature vector of the existing audio in an audio library; an audio feature alignment module, configured to, for each candidate audio, perform audio feature alignment on the audio to be recognized and the candidate audio according to first fingerprint information of the audio to be recognized and second fingerprint information of the candidate audio; and an identical audio recognition module, configured to determine an identical recognition result of the audio to be recognized and the candidate audio based on the result of the audio feature alignment.

[0029] According to a third aspect of the present disclosure, a computer program product comprising instructions is provided. When the computer program product is run on a computer, the computer is enabled to perform the steps of the audio recognition method as described in the first aspect.

[0030] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the audio recognition method as described in the first aspect of the above embodiment is implemented.

[0031] According to the fifth aspect of the embodiments of the present disclosure, there is provided an electronic device, comprising: a processor; and a storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the audio recognition method as described in the first aspect of the above embodiment.

[0032] According to the audio recognition method, audio recognition device, computer-readable storage medium, computer program product and electronic device of the embodiment of the present disclosure, the audio similar to the audio to be recognized is determined from the audio library as the candidate audio through the similarity between the audio feature vectors, and then the audio to be recognized and the candidate audio are aligned according to the fingerprint information of the audio to be recognized and the candidate audio, and based on the audio feature alignment result, it is determined whether the candidate audio and the audio to be recognized are the same audio. On the one hand, on the basis of similarity recognition, the present disclosure further aligns the audio through the audio fingerprint information to recognize the same audio, which can improve the accuracy of the same audio recognition; on the other hand, the present disclosure first determines the candidate audio according to the similarity, and only needs to perform audio feature alignment processing on the candidate audio and the audio to be recognized based on the fingerprint information to obtain the recognition result, without the need to perform audio feature alignment processing on all the audio in the audio library and the audio to be recognized, thereby improving the efficiency of the same audio recognition; on the other hand, the present disclosure can perform deduplication processing on the same audio by recognizing the same audio, saving storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:

[0034] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0035] Figure 2 A schematic diagram showing a flow chart of an audio recognition method in an exemplary embodiment of the present disclosure;

[0036] Figure 3 A schematic structural diagram of a candidate audio determination system in an exemplary embodiment of the present disclosure is shown;

[0037] Figure 4 A schematic flow chart showing a method for overall alignment of audio features in an exemplary embodiment of the present disclosure;

[0038] Figure 5 A schematic diagram showing a result of overall alignment of audio in an exemplary embodiment of the present disclosure;

[0039] Figure 6 A schematic flow chart showing a method for determining an identity recognition result in an exemplary embodiment of the present disclosure;

[0040] Figure 7 A schematic flow chart showing a method for aligning local audio features in an exemplary embodiment of the present disclosure;

[0041] Figure 8 A schematic flow chart showing another method for aligning local audio features in an exemplary embodiment of the present disclosure;

[0042] Fig. 9 A flowchart showing another method for determining an identity recognition result in an exemplary embodiment of the present disclosure is shown;

[0043] Fig.10 A flowchart showing another method for determining an identity recognition result in an exemplary embodiment of the present disclosure is shown;

[0044] Fig.11 A schematic diagram showing the composition of an audio recognition device in an exemplary embodiment of the present disclosure;

[0045] Fig.12 A schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure is shown.

[0046] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0047] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0048] Those skilled in the art know that the embodiments of the present invention can be implemented as a system, device, equipment, medium, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0049] According to an embodiment of the present invention, an audio recognition method, an audio recognition device, a computer-readable storage medium, a computer program product, and an electronic device are provided.

[0050] In this document, any number of elements in the drawings is used for illustration rather than limitation, and any naming is used only for distinction and does not have any limiting meaning.

[0051] The principle and spirit of the present invention are described in detail below with reference to several representative embodiments of the present invention. SUMMARY OF THE INVENTION

[0053] The inventors of the present disclosure have discovered that the audio recognition method in the related art has the problem of low recognition accuracy and low recognition efficiency for the same audio.

[0054] In view of the above content, the basic idea of ​​the present disclosure is to provide an audio recognition method, device, computer-readable storage medium, computer program product and electronic device, determine the audio similar to the audio to be recognized from the audio library as the candidate audio through the similarity between the audio feature vectors, and then align the audio features of the audio to be recognized and the candidate audio according to the fingerprint information of the audio to be recognized and the candidate audio, and determine whether the candidate audio and the audio to be recognized are the same audio based on the audio feature alignment result. On the one hand, on the basis of similarity recognition, the present disclosure further aligns the audio through the audio fingerprint information to recognize the same audio, which can improve the accuracy of the same audio recognition; on the other hand, the present disclosure first determines the candidate audio according to the similarity, and only needs to perform audio feature alignment processing on the candidate audio and the audio to be recognized based on the fingerprint information to obtain the recognition result, without having to perform audio feature alignment processing on all the audio in the audio library and the audio to be recognized, thereby improving the efficiency of the same audio recognition; on the other hand, the present disclosure can perform deduplication processing on the same audio by recognizing the same audio, saving storage resources.

[0055] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are described in detail below.

[0056] Application Scenario Overview

[0057] It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principle of the present invention, and the embodiments of the present invention are not limited in this respect. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0058] In an exemplary application scenario, when a user needs to deduplicate audio in an audio library, any audio in the audio library can be used as audio to be recognized, thereby displaying the same audio as the audio to be recognized from other audio in the audio library according to the method in the present disclosure, so as to deduplicate the recognized identical audio.

[0059] In another exemplary application scenario, when a song infringement judgment is required, the method in the present disclosure can be used to determine whether the audio of the defendant's song is the same as that of the original song, thereby determining whether the defendant's song infringes the copyright of the original song.

[0060] In another exemplary music scenario, after a user inputs a song clip in a music client, songs in an audio library that are identical to the song clip input by the user can be identified according to the method of the present disclosure, thereby making song recommendations for the user.

[0061] Exemplary System Architecture

[0062] First, refer to Figure 1The system architecture of an exemplary application environment of the present disclosure is described.

[0063] like Figure 1 As shown, the system architecture 100 may include a terminal device 110 and a server 120. The terminal device 110 may be a terminal device such as a smart phone, a tablet computer, a desktop computer, a laptop computer, a smart wearable device, etc. The server 120 generally refers to a background system that provides services related to the audio recognition method in this exemplary embodiment, and may be a server or a cluster formed by multiple servers. The terminal device 110 and the server 120 may be connected via a wired or wireless communication link to perform data exchange.

[0064] In an exemplary embodiment, the above-mentioned audio recognition method can be performed by the server 120. Accordingly, the audio recognition device can be set in the server 120 to implement the corresponding module functions. For example, when it is necessary to confirm whether the audio to be recognized and the existing audio in the audio library are the same, the server 120 can determine the candidate audio from the existing audio according to the similarity between the first audio feature vector of the audio to be recognized and the second audio feature vector of the existing audio in the audio library. Then, for each candidate audio, the audio features of the audio to be recognized and the candidate audio are aligned according to the first fingerprint information of the audio to be recognized and the second fingerprint information of the candidate audio, and then according to the result of the fingerprint feature alignment, the audio identical to the audio to be recognized is determined from the audio library to realize the recognition of the same audio.

[0065] It should be understood that Figure 1 The number of terminal devices and servers in the example is only illustrative. Any number of terminal devices and servers may be provided as required. For example, the server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0066] It is easy for those skilled in the art to understand that the above application scenarios are only used as examples and are not limited to the present exemplary embodiment.

[0067] Exemplary Methods

[0068] Figure 2 A flowchart of an audio recognition method in an exemplary embodiment of the present disclosure is shown, referring to Figure 2 , the method comprising:

[0069] Step S210, determining candidate audio from the existing audio according to the similarity between the first audio feature vector of the audio to be recognized and the second audio feature vector of the existing audio in the audio library;

[0070] Step S220: for each candidate audio, aligning audio features of the audio to be identified and the candidate audio according to the first fingerprint information of the audio to be identified and the second fingerprint information of the candidate audio;

[0071] Step S230: Determine the sameness recognition result of the audio to be recognized and the candidate audio based on the result of the audio feature alignment.

[0072] Below, first, the specific implementation method of "step S210, determining candidate audio from the existing audio according to the similarity between the first audio feature vector of the audio to be identified and the second audio feature vector of the existing audio in the audio library" is described in detail.

[0073] In an exemplary implementation, the audio to be recognized may include any audio that needs to be recognized, such as a song, pure music, etc.

[0074] In an exemplary implementation, a neural network model may be trained to identify similarities between audios, thereby determining candidate audios.

[0075] Taking the audio to be identified as a song as an example, a song cover model can be constructed by a neural network model, and the song cover model determines the same song as the song to be identified, thereby determining the candidate song. The training data of the song cover model is a cover song group, and the songs in each song cover group are mutually cover relations. First, spectrum features are extracted for all songs. The spectrum features can adopt features such as Melp, CQT (Constant-QTransform, constant Q transformation, which can capture the energy characteristics at different frequencies in the audio signal). The neural network model can adopt convolutional neural network or recurrent neural network, and the loss function of training can adopt triple loss function. The goal of neural network learning is to minimize the distance between the songs in the cover song group and maximize the distance between the songs in different cover groups. The output of the neural network model can be a vector of fixed dimension, for example, a 512-dimensional vector. The larger the dimension, the stronger the expression ability, but the calculation cost and storage cost are also correspondingly more, so it can be determined according to one's actual needs, and this exemplary embodiment does not make special restrictions on this.

[0076] Next, combine Figure 3 The specific implementation of step S210 is described. For example, Figure 3FIG. 2 is a schematic diagram showing a structure of a candidate audio determination system in an exemplary embodiment of the present disclosure. Figure 3 The system may include a database building module 31 and a retrieval module 32.

[0077] For example, after training the neural network model, you can Figure 3 The library building module 31 in the library builds an audio retrieval library. The purpose of library building is to build the audio library into an efficient audio retrieval library. Taking the audio as a song and the neural network model as the above-mentioned song cover model as an example, the songs in the song library can be extracted from feature vectors using the song cover model to obtain high-dimensional feature vectors, and a song index library is established using a vector index library tool, such as faiss (Facebook AI Similarity Search, an open source library for fast and efficient vector database construction and similarity search), milvus (a cloud-native vector database that supports multiple index types and similarity measurement methods), etc. Tools all provide efficient vector retrieval capabilities, and support multiple index types such as inverted index and HNSW (Hierarchical Navigable Small World, a graph-based indexing algorithm).

[0078] After the audio library is established, the candidate audio corresponding to the audio to be identified can be retrieved in the audio library through the retrieval module 32. Continuing with the example of the audio being a song, the song to be identified can be input into the song cover model, the audio feature vector of the song to be identified can be extracted, and the song vector library constructed by the library building module 31 can be searched. For example, the cosine distance can be used for similarity measurement. The smaller the distance, the more similar the songs are. A similarity threshold can be set. Songs with similarities greater than the threshold are selected as the first candidate songs. Then, the similarities of the first candidate songs are sorted in descending order. The first N songs in the sorting result are selected as candidate songs for the song to be identified, thereby obtaining a candidate song list.

[0079] In other words, by training the neural network model, the audio in the audio library can be used to extract feature vectors using the neural network model, which is equivalent to encoding the audio into an audio vector, and then building the audio vector into an index library, using the same neural network model to extract the feature vector of the audio to be identified, using the feature vector of the audio to be identified to search in the index library, and outputting the audio indicated by the audio vectors in the first N nearest neighbor index libraries as candidate audio for the audio to be identified.

[0080] Through step S210, a similar audio set of the audio to be recognized can be retrieved from the audio library, thereby reducing the amount of data processing in subsequent steps S220 and S230, and improving the recognition efficiency of the same audio.

[0081] Below, the specific implementation method of "step S220, for each candidate audio, aligning the audio features of the audio to be identified and the candidate audio according to the first fingerprint information of the audio to be identified and the second fingerprint information of the candidate audio" is described in detail.

[0082] In an exemplary embodiment, the first fingerprint information includes a first set of local peak features of the frequency spectrum of the audio to be identified, and the second fingerprint information includes a second set of local peak features of the frequency spectrum of the candidate audio. Any local peak feature of the frequency spectrum is determined according to the time and frequency of the local peak of the frequency spectrum. Figure 4 The specific implementation of step S220 is further described.

[0083] For example, Figure 4 A flowchart of a method for overall alignment of audio features in an exemplary embodiment of the present disclosure is shown. Figure 4 , the method may include steps S410 to S430. Wherein:

[0084] In step S410, a first inverted index library is generated according to time information corresponding to the same frequency in the first spectrum local peak feature set.

[0085] For example, the fingerprint information of audio can be understood as feature points extracted from the audio spectrum. The two-dimensional spatial distribution of these feature points in time and frequency characterizes the characteristics of the audio content from the signal level. Therefore, the fingerprint information constructed based on the feature points is unique.

[0086] In the present disclosure, the local peak value of the audio spectrum can be used as a feature point to obtain the fingerprint information of the audio, that is, the fingerprint information of any audio can be understood as a set of local peak features of the audio spectrum.

[0087] Exemplarily, for any audio, the process of extracting fingerprint information can be: first, the audio can be subjected to STFT (Scale-invariant feature transform) transformation to obtain an audio spectrum. Specifically, the audio can be processed in frames, for example, each frame has a duration of 20ms and a frame shift of 10ms, and each frame data is subjected to fast Fourier transform to extract the logarithmic domain spectrum features of each frame, and the entire spectrum is averaged (i.e., the entire spectrum is standardized based on the mean), and the influence of different volumes is eliminated to obtain a two-dimensional table of time and frequency, which can be understood as an audio spectrum. Then, a filter (such as a 3-by-3 filter) is used to filter out local maximum points, and then a fixed length and width window is used to act on these maximum points, and the extreme points in each window are found as a set of feature points of the audio. The information of each feature point includes (t, f), where t represents time, f represents frequency, and height represents the energy of the feature point. Under normal circumstances, dozens of feature points can be selected per second of audio, and the number of feature points depends on the size of the window, wherein the size of the filter can be determined based on experience and actual conditions, and this exemplary embodiment does not make any characteristic limitation on this. The set of these feature points is used as the fingerprint information of the audio.

[0088] For example, after obtaining the set of feature points of the audio to be identified according to the above method, a first inverted index library can be generated based on the set of feature points. For example, the frequency in the feature point can be used as the fingerprint hash value keyword key, and the set of all times that the frequency appears in the feature point can be used as the value corresponding to the keyword of the frequency to generate the first inverted index library. In other words, the feature points in the set of feature points can be grouped according to the same frequency, and the feature points with the same frequency can be divided into the same group. Then, for each group, the frequency of the group can be used as the keyword, and all the time information in the group can be used as the value to form a key-value pair, and the set consisting of the key-value pairs corresponding to all groups can be used as the first inverted index library. That is, in the first inverted index library, all the time information corresponding to the frequency can be retrieved based on the frequency information.

[0089] In step S420, for each second spectrum local peak feature in the second spectrum local peak feature set, the time information corresponding to the frequency in the second spectrum local peak feature is searched in the first inverted index library, and the first difference between the found time information and the time of the second spectrum local peak feature is calculated.

[0090] For example, for the frequency of each feature point in the local peak feature set of the spectrum of each candidate audio, the time information corresponding to the frequency can be found in the first inverted index library, and then the found time can be subtracted from the time in the feature to obtain the first time difference subset corresponding to each feature point. Taking a feature point (f1, t1) in the candidate audio as an example, the time information corresponding to f1 is searched in the first inverted index library, such as t2, t3 and t4. For this feature point, there are 3 values ​​in the first time difference subset, namely t2 minus t1, t3 minus t1 and t4 minus t1. The first time difference set is obtained according to the set composed of the first time difference subsets corresponding to each feature point.

[0091] In other words, after obtaining the first inverted index library, for each candidate audio, the fingerprint information of the candidate audio can be used as input to search in the first inverted index library, and the hash value f1 of each feature point in the fingerprint information of the candidate audio corresponding to the time t1 in the first inverted index library is retrieved, and each time t2 is subtracted from t1 to obtain a fingerprint time difference tDiff = (t2-t1). After retrieving and subtracting all feature points of the fingerprint information of the candidate audio, a time difference set is obtained, namely the first time difference set mentioned above. That is, the set composed of each first difference value is the first time difference set.

[0092] In step S430, the audio features of the audio to be recognized and the candidate audio are aligned as a whole according to the first difference.

[0093] Exemplarily, an exemplary implementation of step S430 may include: counting the first occurrence times of different first difference values ​​in the first difference values, and determining the first maximum occurrence time according to the maximum value among the first occurrence times; when the first maximum occurrence time is greater than or equal to a first preset value, using the first difference value corresponding to the first maximum occurrence time as the first alignment time difference; and performing overall alignment of the audio features of the audio to be identified and the candidate audio according to the first alignment time difference.

[0094] For example, the first difference values ​​can be grouped according to the same difference value, and then the first number of the first difference values ​​in each group is counted. When the first number is less than the first preset value, it is determined that the audio to be identified and the candidate audio cannot be aligned as a whole, which means that the audio to be identified and the candidate audio are not the same. When the first number is greater than or equal to the first preset value, it means that there may be an overall offset between the audio to be identified and the candidate audio, that is, they can be aligned as a whole. Then, the first difference value corresponding to the largest first number can be used as the first alignment time difference. According to the first alignment time difference, the audio features of the audio to be identified and the candidate audio are aligned as a whole. According to the situation after the overall alignment, it is determined whether the audio to be identified and the candidate audio are the same audio.

[0095] Among them, according to the first alignment time difference, the implementation method of overall alignment of audio features of the audio to be identified and the candidate audio can be: adding the first alignment time difference to the original start time of the candidate audio as the new start time of the candidate audio, thereby moving the candidate audio as a whole by the first alignment time difference, so as to align the candidate audio to the video to be identified, thereby achieving alignment of the candidate audio and the audio to be identified.

[0096] For example, Figure 5 A schematic diagram showing the result of overall alignment of audio in an exemplary embodiment of the present disclosure. Figure 5 , Figure 5 The upper part of the image is the candidate audio, and the lower part is the video to be recognized. Figure 5 The audio to be recognized has an additional audio header time than the candidate audio as a whole. The original start time of the candidate audio plus the first time difference is used as the new start time of the candidate audio, so that the audio to be recognized and the candidate audio can be aligned as a whole.

[0097] In other words, the time boundary of the audio to be identified and the candidate audio can be aligned by overall alignment of the audio features, that is, as mentioned above, the fingerprint information of the audio to be identified is used to establish a first inverted index library, and then the features in the fingerprint information of the candidate audio are used as input to perform a search in the first inverted index library, and the hash value (frequency) of each feature point of the candidate audio is found in the first inverted index library. The time value t2, thereby obtaining the fingerprint time difference tDiff max =(t2-t1), after all the fingerprint features of the candidate audio are retrieved, a set of time differences is obtained, and then the number of occurrences of each time difference is counted. Assuming that the maximum number is n, the corresponding time difference is tDiff max If n is greater than or equal to the first preset value, it means that the two audios can be aligned as a whole, and the aligned world boundary pos=tDiff max, which indicates that the candidate audio is aligned to the time position of the audio to be identified. If n is less than the first preset value, it means that the two cannot be aligned as a whole, that is, the fingerprint information of the two is different, which means that the two belong to different audios.

[0098] The specific implementation of "step S230, determining the sameness recognition result of the audio to be recognized and the candidate audio based on the result of audio feature alignment" is described in detail below.

[0099] In an exemplary embodiment, after obtaining the first maximum number of occurrences in step S220, a specific implementation of step S230 may include: when the first maximum number of occurrences is greater than or equal to the first preset value, it can be directly determined that the audio to be identified and the candidate audio belong to the same audio, otherwise, it is determined that the audio to be identified and the candidate audio belong to different audio. Although this method is more accurate than the method of determining the same audio based on similarity alone, it cannot identify subtle differences in audio signals, and when the two are different, it cannot give the location of the difference.

[0100] Based on this, below, combined with Figure 6 Another specific implementation of step S230 is described. For example, Figure 6 A flow chart showing a method for determining a sameness recognition result in an exemplary embodiment of the present disclosure is shown. Figure 6 , the method may include steps S610 to S620. Wherein:

[0101] In step S610, the overall aligned candidate audio and the to-be-recognized audio are segmented to obtain a plurality of first audio segment pairs, and local audio feature alignment is performed on each first audio segment pair.

[0102] Next, combine Figure 7 An exemplary implementation of step S610 is described. For example, Figure 7 A flowchart of a method for aligning local audio features in an exemplary embodiment of the present disclosure is shown. Figure 7 , the method may include steps S710 to S730. Wherein:

[0103] In step S710, the candidate audio is divided into a plurality of non-overlapping candidate audio segments according to a first preset duration.

[0104] Among them, the first preset time length can be customized according to needs or experience, such as the first preset time length can be 5 seconds, 6 seconds, etc. For example, when the candidate audio is longer, the first preset time length can be set slightly longer; when the candidate audio is shorter, the first preset time length can be set slightly smaller accordingly, etc. This exemplary embodiment does not make any special limitations on this.

[0105] Taking the first preset duration of 5 seconds as an example, for a candidate audio with a length greater than 5 seconds, the candidate audio can be equally divided into multiple overlapping 5-second segments to obtain multiple candidate audio segments.

[0106] In step S720, an audio segment to be recognized corresponding to the candidate audio segment is extracted from the overall aligned audio segment to be recognized, so as to obtain a first audio segment pair.

[0107] In an exemplary embodiment, the third start time of the audio segment to be identified is less than the fourth start time of the candidate audio segment, and the third end time of the audio segment to be identified is greater than the fourth end time of the candidate audio segment.

[0108] For example, the first sub-start time of each candidate audio segment can be determined according to the actual start time of the candidate audio segment and the first preset duration. Then, the aligned sub-time of the first sub-start time in the audio to be identified can be found according to each first sub-start time and the first aligned time difference. For example, the aligned sub-time of each first sub-start time in the audio to be identified can be obtained by adding the first aligned time difference to the first sub-start time. For any aligned sub-time in the audio to be identified, the video segment of X seconds before and after the aligned sub-time can be cut out with the aligned sub-time as the center, so as to obtain the audio segment to be identified corresponding to the candidate audio segment, and to obtain a first audio segment pair according to the segment pair composed of the candidate audio segment and the audio segment to be identified corresponding to the candidate audio segment. Among them, X can be determined according to needs and experience, such as X can be 3 seconds, 4 seconds, etc. The length of the cut audio segment to be identified can be the same as or different from the length of the candidate audio segment, that is, the length of the audio segment to be identified is greater than or equal to the length of the candidate audio segment corresponding to it.

[0109] For example, the candidate audio can be divided into multiple non-overlapping candidate audio segments at equal distances, and then for each candidate audio segment, a segment extending X seconds to the left and right of the alignment position of the candidate audio segment in the audio to be identified is found in the audio to be identified, and the audio segment to be identified corresponding to the candidate audio segment is obtained, thereby obtaining the first audio segment pair.

[0110] When a slight speed change occurs, the overall alignment boundary may be deviated, which in turn leads to deviations in the calculation of the similarity of some segments. Therefore, more accurate local alignment can be achieved through the first audio segment pair to correct the deviation caused by the speed change. In other words, by expanding the alignment position to the left and right to obtain the first audio segment pair, the local alignment position can be further accurately determined, thereby improving the accuracy of identical audio recognition.

[0111] In step S730, for each first audio segment pair, local audio feature alignment is performed on the candidate audio segment in the first audio segment pair and the audio segment to be identified according to the third fingerprint information of the candidate audio segment in the first audio segment pair and the fourth fingerprint information of the audio segment to be identified.

[0112] In an exemplary embodiment, the third fingerprint information includes a third set of local frequency spectrum features of the candidate audio segment, and the second fingerprint information includes a fourth set of local frequency spectrum peak features of the audio segment to be identified.

[0113] Based on this, below, combined with Figure 8 An exemplary implementation of step S730 is described. Figure 8 A flowchart of another method for aligning local audio features in an exemplary embodiment of the present disclosure is shown. Figure 8 , the method may include steps S810 to S830. Wherein:

[0114] In step S810, a second inverted index library is generated according to time information corresponding to the same frequency in the fourth spectrum local peak feature set.

[0115] For example, with respect to the fourth fingerprint information of each audio segment to be identified, a second inverted index library of each audio segment to be identified may be generated.

[0116] The method for generating the second inverted index library may refer to the method for generating the first inverted index library described above, and will not be described in detail here.

[0117] In step S820, for each third spectrum local peak feature in the third spectrum local peak feature set, the time information corresponding to the frequency in the third spectrum local peak feature is searched in the second inverted index library, and the second difference between the found time information and the time of the second spectrum local peak feature is calculated.

[0118] The specific method for determining the second difference may refer to the specific method for determining the first difference described above, and will not be described in detail here.

[0119] In step S830, local audio features of the candidate audio segment and the audio segment to be identified in the first audio segment pair are aligned according to the second difference.

[0120] Exemplarily, a specific implementation of step S830 may include: counting the second occurrence numbers of different second difference values ​​in the second difference values, and determining the second maximum occurrence number based on the maximum value among the second occurrence numbers; when the second maximum occurrence number is greater than or equal to a second preset value, using the second difference value corresponding to the second maximum occurrence number as the second alignment time difference; and based on the second alignment time difference, performing local audio feature alignment on the candidate audio segment in the first audio segment pair and the audio segment to be identified.

[0121] In other words, for each pair of first audio segments, local audio feature alignment processing may be performed according to the method of step S810 to step S830.

[0122] Exemplarily, the specific implementation of step S830 may refer to the relevant content of the specific implementation of step S430 mentioned above, which will not be repeated here.

[0123] For example, the local audio feature alignment method is the same as the overall alignment method of the audio features in the above steps S410 to S430, except that the segments are different. The local audio feature alignment is performed on the two audio segments in the first audio segment pair, while the overall alignment is performed on the complete audio of the candidate audio and the audio to be recognized.

[0124] In step S620, the sameness recognition result of the audio to be recognized and the candidate audio is determined according to the result of the alignment of the local audio features of each first audio segment pair.

[0125] Next, combine Fig. 9 A specific implementation of step S620 is described. For example, Fig. 9 A flowchart showing another method for determining the sameness recognition result in an exemplary embodiment of the present disclosure is shown. Fig. 9 The method may include steps S910 to S940. Wherein:

[0126] In step S910, when the second maximum number of occurrences is greater than or equal to a second preset value, the first audio segment pair is determined to be a target audio segment pair.

[0127] For example, for each first audio segment pair, when the second maximum number of occurrences corresponding to the first audio segment pair is greater than or equal to the second preset value, it is determined that the two audio segments in the first audio segment pair can be aligned, and the first audio segment pair is determined as the target audio segment pair, otherwise, it is determined that the two audios in the first audio segment pair cannot be locally aligned. When any first audio segment pair cannot be locally aligned, it can be determined that the candidate audio and the audio to be recognized are not the same.

[0128] In step S920, when all the first audio segment pairs corresponding to the candidate audio are determined as the target audio segment pairs, the audio segment of the first preset length is cut out from the audio segment to be identified of the first audio segment pair based on the alignment position of the local audio feature alignment to obtain a second audio segment, and a second audio segment pair is obtained based on the second audio segment and the candidate audio segments in the first audio segment pair.

[0129] For example, when each first audio segment pair corresponding to the candidate audio can be partially aligned, the third start time of the candidate audio segment in each first audio segment pair can be determined based on the start time of the candidate audio segment in each first audio segment pair and the second alignment time difference of each first audio segment pair, and then the fourth start time is used as the starting point in the audio segment to be identified to cut out an audio segment with a length of the first preset time to obtain the second audio segment. A second audio segment pair is formed based on the second audio segment and the candidate audio segment in the first audio segment pair.

[0130] In step S930, based on the third fingerprint information of the candidate audio segment in the second audio segment pair and the fifth fingerprint information of the second audio segment, the degree of similarity between the candidate audio segment and the second audio segment in the second audio segment pair is determined.

[0131] Exemplarily, a specific implementation of step S930 may include: determining the degree of similarity between the candidate audio segment in the second audio segment pair and the second audio segment based on the degree of coincidence between the third fingerprint information and the fifth fingerprint information.

[0132] In an exemplary embodiment, the overlap degree of fingerprint information may be determined according to the number of identical fingerprint features.

[0133] For example, the similarity between the two audio segments in the second audio segment pair may be calculated using the following formula (1):

[0134]

[0135] In formula (1), fconsistency represents the degree of similarity, hit_count represents the number of overlaps between the fingerprints of the two audio segments, that is, the number of the same fingerprint features mentioned above, ref_seg_count represents the number of fingerprint features of the candidate audio segment, and query_seg_count represents the number of fingerprint features in the second audio segment.

[0136] In other words, for any first audio segment pair, it can be assumed that the total number of fingerprints of the candidate audio segments in the first audio segment pair is ref_seg_coun, and based on the second alignment time difference of the aforementioned local audio features, the total number of fingerprints query_seg_count of the audio segment to be identified in the first audio segment pair within the first preset time length (i.e., the second audio segment mentioned above) is calculated, and the number of overlapping fingerprints of the two audio segments hit_coun is determined, and then the degree of similarity of the two audio segments can be determined according to the above formula (1).

[0137] By calculating the similarity between two audio clips, we can directly measure whether the two audio clips sound consistent, thereby improving the accuracy of identical audio recognition.

[0138] In step S940, a similarity recognition result between the audio to be recognized and the candidate audio is determined according to the similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio.

[0139] Exemplarily, an exemplary implementation of step S940 may include: calculating the mean of the degree of similarity between the candidate audio segment and the second audio segment in all the second audio segment pairs corresponding to the candidate audio; when the mean is greater than or equal to a third preset value, determining the identity recognition result of the audio to be identified and the candidate audio according to the degree of similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio.

[0140] Exemplarily, the method of determining the identification result of the identity between the audio to be identified and the candidate audio based on the degree of identity between the candidate audio segment and the second audio segment in each pair of second audio segments corresponding to the candidate audio includes: when the degree of identity between the candidate audio segment and the second audio segment in each pair of second audio segments corresponding to the candidate audio is greater than or equal to a fourth preset value, determining that the audio to be identified and the candidate audio are the same.

[0141] For example, for each candidate audio, the average of the similarities between the two audio segments in all the corresponding second audio segment pairs can be calculated. For example, if a candidate audio is divided into 10 candidate audio segments, 10 first audio segment pairs can be determined. If the 10 first audio segment pairs can be locally aligned, 10 second audio segment pairs can be determined according to the alignment conditions, and then the average of the similarities between the 10 second audio segment pairs can be calculated. When the mean is greater than or equal to the third preset value, it is considered that the overall similarity between the candidate audio and the audio to be identified meets the requirement, and then it can be further determined whether the two are the same audio based on the similarity of each second audio segment pair. Specifically, if the similarity of the two audio segments in each second audio segment pair is greater than or equal to the fourth preset value, it indicates that the similarity of the two audio segments in each second audio segment pair is relatively high, and it can be determined that the audio to be identified is the same as the candidate audio. If the similarity of the two audio segments in a second audio segment pair is less than the fourth preset value, it indicates that the similarity of the two audio segments is low, which means that although the candidate audio and the audio to be identified have a high overall similarity, there are large differences in their local areas, so the two are still different audios.

[0142] Exemplarily, when the mean of the degree of identity between the candidate audio segment and the second audio segment in all the second audio segment pairs corresponding to the candidate audio is less than the third preset value, it can be directly determined that the candidate audio and the audio to be recognized are not the same. Of course, when the mean is greater than or equal to the third preset value, it can also be directly determined that the candidate audio and the audio to be recognized are the same, without further obtaining the sameness recognition result based on the degree of identity of each second audio segment pair, and this exemplary embodiment does not specifically limit this.

[0143] Exemplarily, another exemplary implementation of step S940 may include: determining that the audio to be identified is the same as the candidate audio when the degree of similarity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio is greater than or equal to a fourth preset value.

[0144] For example, after calculating the degree of identity of the two audio segments in each second audio segment pair, it is not necessary to first calculate the average of the degrees of identity and then determine whether the audio to be identified and the candidate audio are the same based on whether the degree of identity of the two audio segments in each second audio segment pair is greater than or equal to the fourth threshold. Instead, it is directly determined whether the audio to be identified and the candidate audio are the same based on whether the degree of identity of the two audio segments in each second audio segment pair is greater than or equal to the fourth preset value. As described above, when the degree of identity of the two audio segments in each second audio segment pair corresponding to the candidate audio is greater than or equal to the fourth preset value, it is determined that the audio to be identified and the candidate audio are the same.

[0145] Exemplarily, as described above, when the degree of similarity between the candidate audio segment and the second audio segment in any second audio segment pair corresponding to the candidate audio is less than the fourth preset value, it is determined that the audio to be identified is not the same as the candidate audio.

[0146] In an exemplary implementation, the audio recognition method in the present disclosure may further include: outputting first prompt information of the candidate audio segment and the second audio segment whose similarity is less than the fourth preset value.

[0147] For example, the candidate audio segment and the second audio segment whose similarity is less than a fourth preset value can be fed back to the user of the client through the first prompt information, so as to locate the reason and the position of the difference between the candidate audio and the audio to be identified.

[0148] In an exemplary embodiment, the audio recognition method in the present disclosure may further include: when the second maximum number of occurrences is less than the second preset value, determining that the candidate audio is different from the audio to be recognized.

[0149] For example, when the second maximum number of occurrences is less than the second preset value, it means that the two audio segments cannot be aligned in local audio features, that is, the two audio segments have large local differences and cannot be aligned, that is, there are segments in the candidate audio and the audio to be recognized that cannot be aligned in local audio features, which means that the candidate audio and the audio to be recognized are different. The second prompt information of the two audio segments whose second maximum number of occurrences is less than the second preset value can also be output.

[0150] Through the above steps S610 to S620, if it is determined that the audio to be identified and the candidate audio can be aligned as a whole, the candidate audio and the audio to be identified after the overall alignment can be further segmented, so that according to the segmentation results, the audio to be identified and the candidate audio continue to align local audio features on the basis of overall alignment, so as to determine whether the audio to be identified and the candidate audio are the same according to the results of the local audio feature alignment, and the segments with differences between the two can be obtained according to the situation of local feature alignment. In this way, not only can the subtle differences between the audios be identified and the accuracy of the same audio recognition be improved, but also the differences between the audios can be located, thereby helping to improve the efficiency of subsequent manual review.

[0151] In another exemplary embodiment, after overall alignment, the audio to be identified can be subjected to one or more of the following checks: head missing check, head insertion check, tail insertion check, and tail missing check. After all of the one or more checks are passed, local feature alignment can be performed to determine whether the audio to be identified and the candidate audio are the same based on the result of local feature alignment. In this way, not only can a more detailed comparison of the audio be performed to improve the accuracy of identifying the same audio, but also local feature alignment can be performed when the head missing check, head insertion check, tail missing check, and tail insertion check are passed to improve the recognition efficiency of the same audio.

[0152] Next, combine Fig.10 Another specific implementation of step S230 is described. For example, Fig.10 A flowchart showing another method for determining the sameness recognition result in an exemplary embodiment of the present disclosure is shown. Fig.10 , the method may include steps S1010 to S1040. Wherein:

[0153] In step S1010, a first sum of a first start time of the candidate audio and the first alignment time difference is calculated.

[0154] In an exemplary embodiment, a first sum of the first start time of the candidate audio and the first alignment time difference can be calculated to obtain a new start time of the candidate audio after overall alignment. The first start time can be understood as the actual start time of the candidate audio, that is, the start time when overall alignment is not performed.

[0155] In step S1020, when the relationship between the first sum value and the second start time of the audio to be identified meets a first preset condition, the predicted end time of the audio to be identified is obtained according to the first end time of the candidate audio and the second sum value of the first alignment time difference.

[0156] In an exemplary embodiment, the first preset condition includes that the first sum is the same as the second start time of the audio to be recognized, or the first sum is greater than the second start time and the energy of the audio segment to be confirmed is less than or equal to the energy threshold, and the audio segment to be confirmed includes an audio segment in the audio to be recognized that starts at the second start time and has the first alignment time difference as the length. The second start time can be understood as the actual start time of the audio to be recognized.

[0157] In an exemplary embodiment, the actual end time of the candidate audio segment can be detected according to the audio energy, and the predicted end time of the audio to be identified can be obtained based on the sum of the overall alignment time difference and the actual end time. The actual end time can also be obtained directly according to the end time of the candidate audio without detecting the actual end time of the candidate audio according to the energy.

[0158] For example, in the case where the overall alignment is possible, the audio to be identified can be further checked for missing and inserted headers. Specifically, when the first sum value is different from the start time of the audio to be identified, and the first sum value is less than the second start time of the audio to be identified, it means that the audio to be identified lacks the audio segment from the first sum value to the second start time in the candidate audio, that is, the audio to be identified lacks the header segment of the candidate audio, then it can be determined that the audio to be identified and the candidate audio are not the same. When the first sum value is greater than the second start time, it means that the audio to be identified has an additional audio segment from the second start time to the time period corresponding to the first sum value compared to the header of the candidate audio, that is, the audio to be identified after overall alignment has an audio segment inserted into the header compared to the candidate audio. At this time, the audio segment inserted into the header can be used as the audio segment to be confirmed. When the energy of the audio segment to be confirmed is greater than or equal to the energy threshold, it means that the segment to be confirmed is not a silent segment, then it means that the audio to be identified has an audio segment that does not exist in the candidate audio inserted into the header compared to the candidate audio, then it means that the audio to be identified and the candidate audio are not the same, and the difference includes the audio segment inserted into the header of the audio to be identified. When the energy of the segment to be confirmed is less than the energy threshold, it means that the segment to be confirmed is a silent segment. It can be determined that there is no head insertion in the audio to be identified compared with the candidate audio, which means that the head insertion check has passed.

[0159] Similarly, when the first sum is the same as the start time of the audio to be identified, it is determined that there is neither header missing nor header insertion in the audio to be identified, that is, the header missing and header insertion checks are passed.

[0160] In an exemplary embodiment, when the header missing check and / or the header insertion check pass after the overall alignment, it can be directly determined that the audio to be recognized is the same as the candidate audio.

[0161] In another exemplary embodiment, when the head insertion and head deletion checks are passed, tail deletion and / or tail insertion checks can be further performed in step S1030, and based on the tail check result, it is determined whether the audio to be identified and the candidate audio are the same.

[0162] Next, in step S1030, when the relationship between the second end time of the audio to be identified and the predicted end time meets the second preset condition, the overall aligned candidate audio and the audio to be identified are segmented to obtain multiple first audio segment pairs, and local audio feature alignment is performed on each first audio segment pair.

[0163] In an exemplary embodiment, the second preset condition includes that the second end time of the audio to be recognized is the same as the predicted end time, or the predicted end time is less than the second end time and the absolute value of the third difference between the two is less than or equal to a fifth preset value. The second end time can be understood as the actual end time of the audio to be recognized.

[0164] For example, when the actual end time of the audio to be recognized is the same as the predicted end time obtained based on the overall alignment, it means that the audio to be recognized has no tail insertion or tail deletion compared with the candidate audio, that is, the tail deletion check and tail insertion check of the audio to be recognized are both passed.

[0165] When the predicted end time and the second end time are different, if the predicted end time is greater than the actual end time of the audio to be identified, it means that the audio to be identified is missing the tail segment of the candidate audio, and the tail segment is the audio segment within the time period from the actual end time of the audio to be identified to the predicted end time, that is, the tail missing check fails, and it can be directly determined that the candidate audio and the audio to be identified are different.

[0166] When the predicted end time and the second end time are different, if the predicted end time is much smaller than the actual end time of the audio to be identified, it means that there is redundant content at the tail of the audio to be identified relative to the candidate audio, and it can be determined that the tail insertion check fails.

[0167] If the predicted end time is less than the audio to be identified, but the absolute value of the difference between the predicted end time and the actual end time of the audio to be identified is less than or equal to the fifth preset value, it means that the predicted end time is not much less than the actual end time of the audio to be identified, otherwise it means that the predicted end time is much less than the actual end time of the audio to be identified. If the predicted end time is less than the actual end time of the audio to be identified but does not reach the degree of being much less than, it is also determined that the tail insertion check has passed. This is to take into account the situation that the audio may fade out gradually, that is, the energy slowly decreases until the end, and the fade-out effect of each audio may be different, so a small time difference is allowed between the predicted end time and the actual end time. When the time difference between the two is small, the tail insertion check of the audio to be identified is still considered to have passed. Similarly, the above-mentioned tail missing check can also be that the predicted end time is much greater than the actual end time of the audio to be identified, and the tail missing check is determined to have failed, otherwise, the tail missing check is determined to have passed.

[0168] In an exemplary embodiment, when the tail deletion check and the tail insertion check fail, it is determined that the audio to be recognized and the candidate audio are not the same.

[0169] In an exemplary embodiment, when the tail deletion check and the tail insertion check pass, it can be directly determined that the audio to be recognized is the same as the candidate audio.

[0170] In another exemplary embodiment, when the tail missing check and the tail insertion check pass, the overall aligned candidate audio and the audio to be identified can also be segmented to obtain multiple first audio segment pairs, and local audio feature alignment is performed on each first audio segment pair, so that in step S1040, based on the result of the local audio feature alignment, it is determined whether the audio to be identified and the candidate audio are the same. Among them, the specific implementation of "segmenting the overall aligned candidate audio and the audio to be identified to obtain multiple first audio segment pairs, and local audio feature alignment for each first audio segment pair" in step S1030 can refer to the specific implementation of the relevant content in the above-mentioned step S610, which will not be repeated here.

[0171] In step S1040, the sameness recognition result of the audio to be recognized and the candidate audio is determined according to the result of the alignment of the local audio features of each first audio segment pair.

[0172] Exemplarily, the specific implementation of step S1040 may refer to the above-mentioned step S620, which will not be described in detail here.

[0173] Through the above-mentioned steps S1010 to S1040, the candidate audio and the audio to be recognized can be verified for identity from multiple dimensions to improve the accuracy of identical audio recognition.

[0174] Exemplarily, as mentioned above, the audio recognition method in the present disclosure may further include: when the above-mentioned first sum value is less than the second start time, determining that the audio to be recognized and the candidate audio are not the same.

[0175] For example, when the first sum value is less than the second start time, it indicates that the audio to be identified is missing a head compared to the candidate audio, that is, the audio segment in the time period corresponding to the first sum value to the second start time in the candidate audio is missing in the audio to be identified, so it can be directly determined that the audio to be identified and the candidate audio are different.

[0176] Exemplarily, as described above, the audio recognition method in the present disclosure may further include: when the above-mentioned first sum value is greater than the second start time and the energy of the audio segment to be confirmed is greater than the energy threshold, determining that the audio to be recognized and the candidate audio are not the same.

[0177] For example, when the first sum is greater than the second start time, it means that the audio to be identified has the above-mentioned audio segment to be confirmed inserted in the head compared to the candidate audio. When the energy of the audio segment to be confirmed is greater than the energy threshold, it means that the inserted audio segment to be confirmed is not a completely silent segment, but a segment with actual audio content. Therefore, it can be determined that the audio to be identified and the candidate audio are not the same.

[0178] Exemplarily, the audio recognition method in the present disclosure may further include: when the second end time of the audio to be recognized is different from the predicted end time, determining that the audio to be recognized and the candidate audio are different.

[0179] Exemplarily, in the above-mentioned case where the second end time of the audio to be identified and the predicted end time are different, determining that the audio to be identified and the candidate audio are not the same includes: when the predicted end time is less than the second end time, or the predicted end time is less than the second end time and the absolute value of the third difference is greater than the fifth preset value, determining that the audio to be identified and the candidate audio are not the same; when the predicted end time is greater than the second end time, determining that the audio to be identified and the candidate audio are not the same.

[0180] Exemplarily, the audio recognition method in the present disclosure may further include: when a first maximum number of occurrences corresponding to the candidate audio is less than the first preset value, determining that the audio to be recognized is different from the candidate audio.

[0181] For example, when the first maximum number of occurrences corresponding to the candidate audio is less than the first preset value, it means that the candidate audio and the audio to be identified cannot be aligned as a whole, that is, there are large differences between the candidate audio and the audio to be identified, so it can be determined that the video to be identified and the candidate audio are different.

[0182] In an exemplary scenario of the present disclosure, the recognition accuracy and efficiency of the same audio can be improved through two-stage detection. In the first stage, candidate audio is selected from the audio to be compared based on the similarity of the audio feature vector. In the second stage, for each candidate audio, the audio feature is aligned with the audio to be recognized, so that each audio pair consisting of the candidate audio and the audio to be recognized is verified in multiple dimensions, and based on the verification results of multiple dimensions, it is determined whether the two are the same, and then the audio that is the same as the audio to be recognized is determined from the candidate audio.

[0183] The above-mentioned multiple-dimensional verifications may include alignment confidence verification, header missing verification, header insertion verification, intermediate data verification, tail missing verification, and tail insertion verification.

[0184] Among them, the alignment confidence check can be understood as: if the overall fingerprint alignment fails, or the average of the calculated fragment similarity after alignment is lower than a certain threshold, it can be considered that the overall similarity of the two audios is insufficient, and the alignment confidence check fails. It is directly determined that the two are not the same and the check ends, otherwise it proceeds to the next step of header missing check.

[0185] The missing header check can be understood as follows: if the fingerprint can be aligned as a whole, but the alignment start time of the candidate audio to the audio to be identified is less than the actual start time of the audio to be identified, it means that the header content of the candidate audio cannot be found in the audio to be identified, that is, the audio to be identified is missing header data compared to the candidate audio, and the header check fails. It is directly determined that the two are different and the check ends. Otherwise, the next step of header insertion check is entered.

[0186] The header insertion check can be understood as: if after the overall alignment, the alignment start time of the candidate audio to the audio to be identified is greater than the actual start time of the audio to be identified, it means that compared with the candidate audio, the content of the audio to be identified in the time period from 0 to the alignment start time is redundant, then it can be detected whether this section of content is all silent segments. If it is not a silent segment, it means that the header of the audio to be identified has been inserted with content that is not in the candidate audio, then the header insertion check fails, and it is directly determined that the two are different and the check ends, otherwise it proceeds to the next step of intermediate data verification.

[0187] The intermediate data check can be understood as: checking the similarity of the two audio segments in the above-mentioned second audio segment pair corresponding to the candidate audio, so as to determine whether the content of each local segment pair is consistent. If there is a second audio segment pair whose similarity is lower than a certain threshold, all the second audio segment pairs lower than the threshold are saved and output. If the similarity of all the second audio segment pairs is greater than or equal to a certain threshold, the intermediate data check passes and enters the next step of tail missing check. Otherwise, it is determined that the intermediate data check fails and the check ends.

[0188] The tail missing check can be understood as: according to the end position of the candidate audio voice segment detected by energy, based on the overall alignment time difference, that is, the above-mentioned first alignment time difference and the end position of the candidate audio voice segment, the estimated end position of the audio voice segment to be identified is obtained. If the estimated end position exceeds the actual end position of the audio to be identified, it can be judged that the audio to be identified does not contain the tail content of the candidate audio, that is, the tail check fails and the check is ended, otherwise it proceeds to the next step of tail insertion check.

[0189] The tail insertion check can be understood as: if the estimated end position of the voiced segment of the audio to be identified does not exceed the actual end position of the audio to be identified, the actual end position of the voiced segment of the audio to be identified can be further detected. If the actual end position is much larger than the estimated end position, it means that there is redundant content at the end of the audio to be identified. Then it can be judged that the tail insertion check of the audio to be identified has failed, and it is determined that the two are not the same. Otherwise, it is determined that the two are the same.

[0190] In other words, in the above-mentioned multiple dimensional verifications, if the verification of any dimension fails or the verification fails, it can be determined that the audio to be identified and the candidate audio are different. After the verification of each dimension passes, it is determined that the audio to be identified and the candidate audio are the same.

[0191] It should be noted that the verification of the above-mentioned multiple dimensions can be selected according to needs, which does not mean that every dimension must be verified. In addition, the verification order of multiple dimensions can also be determined according to needs or experience, and this exemplary embodiment does not make any special limitations on this.

[0192] In the present disclosure, the recognition of the same audio is achieved by combining a machine learning model with fingerprint technology. While improving the recognition efficiency, the fingerprint technology can accurately compare the subtle differences in the audio signal, thereby improving the accuracy of the same audio recognition. Secondly, in the present disclosure, the recognition of the same audio is performed by aligning local audio features, which can identify different audios and accurately locate the differences between different audios. In the subsequent manual review, the located difference positions can improve the efficiency of the manual review. In addition, in the present disclosure, the same audio can be accurately identified. Taking songs as an example, by identifying the same song, the number of repeated songs in the song library can be reduced, thereby saving storage resources. Furthermore, the two-stage audio recognition method in the present disclosure can flexibly adjust the detection dimension according to demand, so as to adapt to the needs of different application scenarios, and has strong versatility and extensibility.

[0193] In addition, it should be noted that the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0194] Exemplary Devices

[0195] The exemplary embodiment of the present disclosure also provides an audio recognition device. Fig.11 As shown, the audio recognition device 1100 may include the following program modules: a candidate audio determination module 1110, configured to determine candidate audio from the existing audio according to the similarity between the first audio feature vector of the audio to be recognized and the second audio feature vector of the existing audio in the audio library; an audio feature alignment module 1120, configured to, for each candidate audio, perform audio feature alignment on the audio to be recognized and the candidate audio according to the first fingerprint information of the audio to be recognized and the second fingerprint information of the candidate audio; an identical audio recognition module 1130, configured to determine the identical recognition result of the audio to be recognized and the candidate audio based on the result of the audio feature alignment.

[0196] In an exemplary embodiment, the first fingerprint information includes a first set of spectral local peak features of the audio to be identified, and the second fingerprint information includes a second set of spectral local peak features of the candidate audio, and any spectral local peak feature is determined according to the time and frequency of the spectral local peak; the audio feature alignment module 1120 can be specifically configured as follows: generating a first inverted index library according to the time information corresponding to the same frequency in the first spectral local peak feature set; for each second spectral local peak feature in the second spectral local peak feature set, searching the first inverted index library for the time information corresponding to the frequency in the second spectral local peak feature, and calculating the first difference between the found time information and the time of the second spectral local peak feature; and performing overall audio feature alignment of the audio to be identified and the candidate audio according to the first difference.

[0197] In an exemplary implementation, the overall alignment of audio features of the audio to be recognized and the candidate audio according to the first difference includes:

[0198] Counting first occurrence times of different first difference values ​​in the first difference values, and determining a first maximum occurrence time according to a maximum value among the first occurrence times;

[0199] When the first maximum number of occurrences is greater than or equal to a first preset value, taking a first difference value corresponding to the first maximum number of occurrences as a first alignment time difference;

[0200] According to the first alignment time difference, overall audio feature alignment is performed on the audio to be recognized and the candidate audio.

[0201] In an exemplary embodiment, based on the aforementioned embodiment, the same audio recognition module 1130 can be specifically configured to: calculate a first sum of the first start time of the candidate audio and the first alignment time difference; when the relationship between the first sum and the second start time of the audio to be recognized satisfies a first preset condition, obtain the predicted end time of the audio to be recognized according to the first end time of the candidate audio and the second sum of the first alignment time difference; when the relationship between the second end time of the audio to be recognized and the predicted end time satisfies a second preset condition, segment the overall aligned candidate audio and the audio to be recognized to obtain multiple first audio segment pairs, and locally perform segmentation on each first audio segment pair. Audio feature alignment; determining the sameness recognition result of the audio to be identified and the candidate audio according to the result of the local audio feature alignment of each first audio segment pair; wherein, the first preset condition includes that the first sum is the same as the second start time of the audio to be identified, or the first sum is greater than the second start time and the energy of the audio segment to be confirmed is less than or equal to the energy threshold, and the audio segment to be confirmed includes an audio segment in the audio to be identified that takes the second start time as the starting point and the first alignment time difference as the length; the second preset condition includes that the second end time of the audio to be identified is the same as the predicted end time, or the predicted end time is less than the second end time and the absolute value of the third difference between the two is less than or equal to the fifth preset value.

[0202] In an exemplary embodiment, based on the aforementioned embodiment, the identical audio recognition module 1130 may be specifically configured as follows: segmenting the overall aligned candidate audio and the audio to be recognized to obtain a plurality of first audio segment pairs, and performing local audio feature alignment on each first audio segment pair; and determining the identical recognition result of the audio to be recognized and the candidate audio based on the result of the local audio feature alignment of each first audio segment pair.

[0203] In an exemplary embodiment, based on the aforementioned embodiment, the candidate audio and the audio to be identified after overall alignment are segmented to obtain multiple first audio segment pairs, and the local audio feature alignment for each first audio segment pair includes: according to a first preset duration, the candidate audio is segmented into multiple non-overlapping candidate audio segments; from the audio to be identified after overall alignment, the audio segment to be identified corresponding to the candidate audio segment is cut out to obtain a first audio segment pair, wherein the third start time of the audio segment to be identified is less than the fourth start time of the candidate audio segment, and the third end time of the audio segment to be identified is greater than the fourth end time of the candidate audio segment; for each first audio segment pair, according to the third fingerprint information of the candidate audio segment in the first audio segment pair and the fourth fingerprint information of the audio segment to be identified, the candidate audio segment and the audio segment to be identified in the first audio segment pair are locally aligned with each other.

[0204] In an exemplary embodiment, based on the aforementioned embodiment, the third fingerprint information includes a third spectral local feature set of the candidate audio segment, and the second fingerprint information includes a fourth spectral local peak feature set of the audio segment to be identified; the local audio feature alignment of the candidate audio segment and the audio segment to be identified in the first audio segment pair according to the third fingerprint information of the candidate audio segment and the fourth fingerprint information of the audio segment to be identified includes: generating a second inverted index library according to the time information corresponding to the same frequency in the fourth spectral local peak feature set; for each third spectral local peak feature in the third spectral local peak feature set, searching the second inverted index library for the time information corresponding to the frequency in the third spectral local peak feature, and calculating the second difference between the found time information and the time of the second spectral local peak feature; and according to the second difference, performing local audio feature alignment on the candidate audio segment and the audio segment to be identified in the first audio segment pair.

[0205] In an exemplary embodiment, based on the aforementioned embodiment, the local audio feature alignment of the candidate audio segment in the first audio segment pair and the audio segment to be identified according to the second difference includes: counting the second occurrence times of different second difference values ​​in the second difference values, and determining the second maximum occurrence times according to the maximum value of the second occurrence times; when the second maximum occurrence times is greater than or equal to a second preset value, using the second difference corresponding to the second maximum occurrence times as the second alignment time difference; and based on the second alignment time difference, the local audio feature alignment of the candidate audio segment in the first audio segment pair and the audio segment to be identified.

[0206] In an exemplary embodiment, based on the aforementioned embodiment, determining the sameness recognition result of the audio to be recognized and the candidate audio according to the result of the local audio feature alignment of each first audio segment pair includes: determining the first audio segment pair as a target audio segment pair when the second maximum number of occurrences is greater than or equal to a second preset value; when all first audio segment pairs corresponding to the candidate audio are determined as target audio segment pairs, cutting out an audio segment of the first preset duration from the audio segment to be recognized of the first audio segment pair based on the alignment position of the local audio feature alignment to obtain a second audio segment, and obtaining a second audio segment pair according to the second audio segment and the candidate audio segment in the first audio segment pair; determining the sameness between the candidate audio segment and the second audio segment in the second audio segment pair based on the third fingerprint information of the candidate audio segment in the second audio segment pair and the fifth fingerprint information of the second audio segment; determining the sameness recognition result of the audio to be recognized and the candidate audio according to the sameness between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio.

[0207] In an exemplary implementation, based on the foregoing embodiment, determining the degree of similarity between the candidate audio segment and the second audio segment in the second audio segment pair based on the third fingerprint information of the candidate audio segment and the fifth fingerprint information of the second audio segment in the second audio segment pair includes:

[0208] ==Based on the overlap between the third fingerprint information and the fifth fingerprint information, determine the similarity between the candidate audio segment and the second audio segment in the second audio segment pair.

[0209] In an exemplary embodiment, based on the aforementioned embodiment, determining the identity recognition result of the audio to be recognized and the candidate audio according to the degree of identity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio includes: calculating the mean of the degree of identity between the candidate audio segment and the second audio segment in all second audio segment pairs corresponding to the candidate audio; and when the mean is greater than or equal to a third preset value, determining the identity recognition result of the audio to be recognized and the candidate audio according to the degree of identity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio.

[0210] In an exemplary embodiment, based on the aforementioned embodiment, determining the identity recognition result of the audio to be identified and the candidate audio according to the degree of identity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio includes: when the degree of identity between the candidate audio segment and the second audio segment in each second audio segment pair corresponding to the candidate audio is greater than or equal to a fourth preset value, determining that the audio to be identified and the candidate audio are the same.

[0211] In an exemplary embodiment, based on the aforementioned embodiment, the identical audio recognition module 1130 may be specifically configured to determine that the audio to be recognized and the candidate audio are not identical when the degree of identity between the candidate audio segment and the second audio segment in any second audio segment pair corresponding to the candidate audio is less than the fourth preset value.

[0212] In an exemplary embodiment, the device may further include a first prompt information output module, which may be specifically configured to output first prompt information of the candidate audio segment and the second audio segment whose similarity is less than the fourth preset value.

[0213] In an exemplary implementation, based on the foregoing embodiment, the same audio recognition module 1130 may be specifically configured to: when the mean value is less than the third preset value, determine that the candidate audio and the audio to be recognized are different.

[0214] In an exemplary implementation, based on the foregoing embodiment, the same audio recognition module 1130 may be specifically configured to: determine that the candidate audio is different from the audio to be recognized when the second maximum number of occurrences is less than the second preset value.

[0215] In an exemplary implementation, based on the foregoing embodiment, the same audio recognition module 1130 may be specifically configured to: when the first sum value is less than the second start time, determine that the audio to be recognized and the candidate audio are not the same.

[0216] In an exemplary embodiment, based on the aforementioned embodiment, the same audio recognition module 1130 can be specifically configured to: when the first sum value is greater than the second start time and the energy of the audio segment to be confirmed is greater than the energy threshold, determine that the audio to be recognized and the candidate audio are not the same.

[0217] In an exemplary implementation, based on the aforementioned embodiment, the same audio recognition module 1130 may be specifically configured to: determine that the audio to be recognized and the candidate audio are not the same when the second end time of the audio to be recognized and the predicted end time are different.

[0218] In an exemplary embodiment, when the second end time of the audio to be identified and the predicted end time are different, determining that the audio to be identified and the candidate audio are not the same includes: when the predicted end time is less than the second end time, or the predicted end time is less than the second end time and the absolute value of the third difference is greater than the fifth preset value, determining that the audio to be identified and the candidate audio are not the same; when the predicted end time is greater than the second end time, determining that the audio to be identified and the candidate audio are not the same.

[0219] In an exemplary implementation, based on the aforementioned embodiment, the same audio recognition module 1130 may be specifically configured to: determine that the audio to be recognized and the candidate audio are not the same when the first maximum number of occurrences corresponding to the candidate audio is less than the first preset value.

[0220] The specific details of each part of the above-mentioned device have been described in detail in the implementation of the corresponding method part above. The undisclosed details can be found in the implementation of the method part above, and will not be repeated here.

[0221] Exemplary Storage Media

[0222] The storage medium according to the exemplary embodiment of the present invention is described below.

[0223] In this exemplary embodiment, the above method can be implemented by a program product, such as a portable compact disk read-only memory (CD-ROM) and including program code, and can be run on a device, such as a personal computer. However, the program product of the present invention is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device.

[0224] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0225] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0226] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RE, etc., or any suitable combination of the foregoing.

[0227] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0228] Exemplary computer program products

[0229] The exemplary embodiments of the present disclosure also provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the above-mentioned audio recognition method is implemented.

[0230] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0231] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0232] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, Python, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0233] Computer programs can be carried or transmitted through electrical, magnetic, optical, electromagnetic, infrared and other signals. Electronic devices can convert signals carrying computer programs into digital signals, and then run computer programs. When a computer program runs on an electronic device, its code is used to enable the electronic device to execute (more specifically, it can enable the processor of the electronic device to execute) the method steps of various exemplary embodiments of the present disclosure, such as the above-mentioned audio recognition method, which includes the following steps: according to the similarity between the first audio feature vector of the audio to be recognized and the second audio feature vector of the existing audio in the audio library, determine the candidate audio from the existing audio; for each candidate audio, according to the first fingerprint information of the audio to be recognized and the second fingerprint information of the candidate audio, align the audio features of the audio to be recognized and the candidate audio; based on the result of the audio feature alignment, determine the sameness recognition result of the audio to be recognized and the candidate audio.

[0234] The above method steps are executed by a computer program, and the above method is realized by a computer program. On the one hand, based on the similarity recognition, the audio is further aligned through the audio fingerprint information to identify the same audio, which can improve the accuracy of the same audio recognition; on the other hand, the candidate audio is first determined according to the similarity, and it is only necessary to perform audio feature alignment processing on the candidate audio and the audio to be identified based on the fingerprint information to obtain the recognition result. There is no need to perform audio feature alignment processing on all audios in the audio library and the audio to be identified, which improves the efficiency of the same audio recognition; on the other hand, by identifying the same audio, the same audio can be deduplicated, saving storage resources.

[0235] Exemplary Electronic Devices

[0236] refer to Fig.12 An electronic device of an exemplary embodiment of the present disclosure is described. The electronic device is the terminal device 110 or the server 120 mentioned above. The electronic device may include a processor and a memory. The memory stores executable instructions of the processor, such as a computer program. The processor executes the method steps of various exemplary embodiments of the present disclosure by executing the executable instructions. In addition, the electronic device may also include a display for displaying a graphical user interface.

[0237] Reference below Fig.12 , the electronic device is exemplarily described in the form of a general-purpose computing device. It should be understood that Fig.12 The electronic device 1200 shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0238] like Fig.12 As shown, the electronic device 1200 may include: a processor 1210 , a memory 1220 , a bus 1230 , an I / O (input / output) interface 1240 , a network adapter 1250 , and a display 1260 .

[0239] The memory 1220 may include a volatile memory, such as a RAM 1221, a cache unit 1222, and may also include a non-volatile memory, such as a ROM 1223. The memory 1220 may also include one or more program modules 1224, such program modules 1224 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 1224 may include each module in the above-mentioned device.

[0240] The processor 1210 may include one or more processing units. For example, the processor 1210 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0241] The processor 1210 can be used to execute executable instructions stored in the memory 1220, such as the above-mentioned audio recognition method, which includes the following steps: determining candidate audio from the existing audio based on the similarity between the first audio feature vector of the audio to be recognized and the second audio feature vector of the existing audio in the audio library; for each candidate audio, aligning the audio features of the audio to be recognized and the candidate audio based on the first fingerprint information of the audio to be recognized and the second fingerprint information of the candidate audio; and determining the sameness recognition result of the audio to be recognized and the candidate audio based on the result of the audio feature alignment.

[0242] The above method is implemented through a computer program. On the one hand, based on similarity recognition, the audio is further aligned through the audio fingerprint information to identify the same audio, which can improve the accuracy of identifying the same audio; on the other hand, the candidate audio is first determined according to the similarity, and only the audio feature alignment processing is performed on the candidate audio and the audio to be identified based on the fingerprint information to obtain the identification result. There is no need to align the audio features of all audios in the audio library with the audio to be identified, which improves the efficiency of identifying the same audio; on the other hand, by identifying the same audio, the same audio can be deduplicated, saving storage resources.

[0243] The bus 1230 is used to realize the connection between different components of the electronic device 1200, and may include a data bus, an address bus, and a control bus.

[0244] The electronic device 1200 can communicate with one or more external devices 1300 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 1240 .

[0245] The electronic device 1200 can communicate with one or more networks through the network adapter 1250. For example, the network adapter 1250 can provide mobile communication solutions such as 3G / 4G / 5G, or provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 1250 can communicate with other modules of the electronic device 1200 through the bus 1230.

[0246] The electronic device 1200 may display a graphical user interface through the display 1260 , such as a graphical user interface that displays the audio identity recognition result.

[0247] although Fig.12 Not shown, other hardware and / or software modules may also be provided in the electronic device 1200, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0248] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0249] It should be understood that the present disclosure is not limited to the specific method steps or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. Those skilled in the art will easily think of other embodiments based on the specific embodiments provided by the present disclosure. Therefore, the specific embodiments provided by the present disclosure are only exemplary, and the scope and spirit of the present disclosure are indicated by the claims, and any variations, uses or adaptive changes of the present disclosure should be covered, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the technical field that are not disclosed in the present disclosure.

Claims

1. An audio recognition method, characterized in that: include: Determine candidate audio from the existing audio according to the similarity between the first audio feature vector of the audio to be identified and the second audio feature vector of the existing audio in the audio library; For each candidate audio, align audio features of the audio to be identified and the candidate audio according to the first fingerprint information of the audio to be identified and the second fingerprint information of the candidate audio; Based on the result of the audio feature alignment, a recognition result of the identity between the audio to be recognized and the candidate audio is determined.

2. The method according to claim 1, characterized in that The first fingerprint information includes a first set of local peak features of the frequency spectrum of the audio to be identified, and the second fingerprint information includes a second set of local peak features of the frequency spectrum of the candidate audio, and any local peak feature of the frequency spectrum is determined according to the time and frequency of the local peak of the frequency spectrum; The aligning audio features of the audio to be identified and the candidate audio according to the first fingerprint information of the audio to be identified and the second fingerprint information of the candidate audio comprises: Generate a first inverted index library according to time information corresponding to the same frequency in the first spectrum local peak feature set; For each second spectrum local peak feature in the second spectrum local peak feature set, searching the first inverted index library for time information corresponding to the frequency in the second spectrum local peak feature, and calculating a first difference between the found time information and the time of the second spectrum local peak feature; According to the first difference, overall audio feature alignment is performed on the audio to be recognized and the candidate audio.

3. The method according to claim 2, characterized in that The step of performing overall audio feature alignment on the audio to be recognized and the candidate audio according to the first difference includes: Counting first occurrence times of different first difference values ​​in the first difference values, and determining a first maximum occurrence time according to a maximum value among the first occurrence times; When the first maximum number of occurrences is greater than or equal to a first preset value, taking a first difference value corresponding to the first maximum number of occurrences as a first alignment time difference; According to the first alignment time difference, overall audio feature alignment is performed on the audio to be recognized and the candidate audio.

4. The method according to claim 3, characterized in that The determining of the sameness recognition result of the audio to be recognized and the candidate audio based on the result of the audio feature alignment includes: Calculate a first sum of a first start time of the candidate audio and the first alignment time difference; When the relationship between the first sum and the second start time of the audio to be recognized satisfies a first preset condition, obtaining a predicted end time of the audio to be recognized according to the first end time of the candidate audio and the second sum of the first alignment time difference; When the relationship between the second end time of the audio to be recognized and the predicted end time meets a second preset condition, segmenting the overall aligned candidate audio and the audio to be recognized to obtain a plurality of first audio segment pairs, and performing local audio feature alignment on each first audio segment pair; Determine, according to the result of the alignment of the local audio features of each first audio segment pair, a result of the sameness recognition of the audio to be recognized and the candidate audio; The first preset condition includes that the first sum is the same as the second start time of the audio to be identified, or the first sum is greater than the second start time and the energy of the audio segment to be confirmed is less than or equal to an energy threshold, and the audio segment to be confirmed includes an audio segment in the audio to be identified that takes the second start time as the starting point and the first alignment time difference as the length; The second preset condition includes that the second end time of the audio to be identified is the same as the predicted end time, or the predicted end time is less than the second end time and the absolute value of the third difference between the two is less than or equal to a fifth preset value.

5. The method according to claim 3, characterized in that: The determining of the sameness recognition result of the audio to be recognized and the candidate audio based on the result of the audio feature alignment includes: Segmenting the overall aligned candidate audio and the to-be-recognized audio to obtain a plurality of first audio segment pairs, and aligning local audio features of each first audio segment pair; According to the result of the alignment of the local audio features of each first audio segment pair, the sameness recognition result of the audio to be recognized and the candidate audio is determined.

6. The method according to claim 4 or 5, characterized in that: The step of segmenting the overall aligned candidate audio and the to-be-recognized audio to obtain a plurality of first audio segment pairs, and performing local audio feature alignment on each first audio segment pair comprises: According to a first preset duration, dividing the candidate audio into a plurality of non-overlapping candidate audio segments; Extracting an audio segment to be identified that corresponds to the candidate audio segment from the overall aligned audio segment to obtain a first audio segment pair, wherein a third start time of the audio segment to be identified is less than a fourth start time of the candidate audio segment, and a third end time of the audio segment to be identified is greater than a fourth end time of the candidate audio segment; For each first audio segment pair, local audio feature alignment is performed on the candidate audio segment in the first audio segment pair and the audio segment to be identified according to the third fingerprint information of the candidate audio segment in the first audio segment pair and the fourth fingerprint information of the audio segment to be identified.

7. An audio recognition device, characterized in that: include: A candidate audio determination module is configured to determine a candidate audio from the existing audio according to a similarity between a first audio feature vector of the audio to be identified and a second audio feature vector of the existing audio in the audio library; An audio feature alignment module is configured to perform audio feature alignment on the audio to be identified and the candidate audio for each candidate audio according to the first fingerprint information of the audio to be identified and the second fingerprint information of the candidate audio; The same audio recognition module is configured to determine the sameness recognition result of the audio to be recognized and the candidate audio based on the result of audio feature alignment.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 6.

Images