Audio recognition method, apparatus, device, and storage medium

Abstract

This disclosure relates to an audio recognition method, apparatus, device, and storage medium. The method involves: acquiring note control command information of an audio to be recognized; filtering the audio based on the note control command information according to the instrument component dimension to obtain a first audio; processing the first audio based on its spectral characteristics to obtain a first candidate set, the first candidate set including multiple first candidate audio segments; determining the audio quality index of each second candidate audio segment in a second candidate set based on the musical characteristics of each first candidate audio segment; the second candidate set being the entire set or a subset of the first candidate set; and selecting a target audio segment based on the audio quality index of each second candidate audio segment. This achieves automatic selection of musical highlight segments without additional manpower and improves the accuracy of the selected musical highlight segments.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an audio recognition method, apparatus, device, and storage medium. Background Technology

[0002] Musical highlights typically refer to the most captivating and engaging parts of a piece of music, such as the chorus of a song. In short video background music, due to time constraints, highlights are often chosen as background music to quickly grab the user's attention.

[0003] In related technologies, a few bars of the chorus of a song are manually selected as the highlighted segments of each song to provide background music for short videos. However, this method requires a lot of manpower and cannot guarantee the accuracy of the selected music highlights. Summary of the Invention

[0004] This disclosure provides an audio recognition method, apparatus, device, and storage medium to at least solve at least one problem in the related art, such as low computational efficiency. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, an audio recognition method is provided, comprising:

[0006] Obtain the note control command information of the audio to be recognized;

[0007] Based on the note control instruction information, the audio to be identified is filtered based on the instrument component dimension to obtain the first audio.

[0008] The first audio is processed based on its spectral characteristics to obtain a first candidate set, which includes multiple first candidate audio segments.

[0009] Based on the musical features of each first candidate audio segment, determine the audio quality index of each second candidate audio segment in the second candidate set; the second candidate set is the entire set or a subset of the first candidate set;

[0010] Based on the audio quality index of each of the second candidate audio segments, target audio segments are selected; the target audio segments are used to characterize the musical highlight segments in the audio to be identified.

[0011] In an optional implementation, processing the first audio based on its spectral features to obtain a first candidate set includes:

[0012] Obtain the spectral characteristics of the first audio;

[0013] Based on the spectral characteristics of the first audio, the first audio is divided into audio segments to obtain multiple audio sub-segments;

[0014] Each audio sub-segment is determined to have a spectral sub-feature in the spectral features. Based on the spectral sub-feature corresponding to each audio sub-segment, multiple audio sub-segments are clustered to obtain multiple first candidate audio segments. The first candidate audio segments are then used as the first candidate set.

[0015] In an optional implementation, based on the musical features of each first candidate audio segment, an audio quality index is determined for each second candidate audio segment in the second candidate set, including:

[0016] Obtain the musical features of each of the first candidate audio segments;

[0017] Based on the lyrics control instruction information of the audio to be identified and the musical characteristics of each of the first candidate audio segments, the audio quality index of each second candidate audio segment in the second candidate set is determined.

[0018] In an optional implementation, determining the audio quality index of each second candidate audio segment in the second candidate set based on the lyrics control instruction information of the audio to be identified and the musical features of each first candidate audio segment includes:

[0019] Based on the musical features of each first candidate audio segment, a first audio quality index is determined for each first candidate audio segment.

[0020] Based on the lyrics control instruction information of the audio to be identified, multiple first audio quality indicators are filtered to obtain a second audio quality indicator for each second candidate audio segment in the second candidate set.

[0021] In an optional implementation, determining the first audio quality index for each first candidate audio segment based on the musical features of each first candidate audio segment includes:

[0022] Obtain the initial loudness feature, initial spectral brightness feature, and initial drum beat intensity feature for each of the first candidate audio segments;

[0023] The initial loudness feature, initial spectral brightness feature, and initial drum intensity feature of each first candidate audio segment are normalized to obtain the loudness feature, spectral brightness feature, and drum intensity feature of each first candidate audio segment, respectively.

[0024] Obtain the weight information corresponding to loudness features, spectral brightness features, and drumbeat intensity features, respectively;

[0025] Based on the weight information, the loudness feature, spectral brightness feature and drum beat intensity feature of each first candidate audio segment are weighted to obtain the first audio quality index of each first candidate audio segment.

[0026] In one optional implementation, the second audio quality indicator includes an audio quality score; the step of filtering target audio segments based on the audio quality indicator of each second candidate audio segment includes:

[0027] Based on the number of repetitions of each of the second candidate audio segments, the multiple second candidate audio segments are sorted sequentially, and the third and fourth candidate audio segments corresponding to the top two repetition counts are selected respectively.

[0028] If the difference between the number of repetitions of the third candidate audio segment and the number of repetitions of the fourth candidate audio segment is greater than a difference threshold, the candidate audio segment with the larger number of repetitions between the third and fourth candidate audio segments is selected as the target audio segment.

[0029] In an optional implementation, after selecting the third and fourth candidate audio segments corresponding to the two highest repetition counts, the method further includes:

[0030] If the difference between the number of repetitions of the third candidate audio segment and the number of repetitions of the fourth candidate audio segment is less than or equal to the difference threshold, the candidate audio segment with the larger audio quality score between the third and fourth candidate audio segments is selected as the target audio segment.

[0031] According to a second aspect of the present disclosure, an audio recognition device is provided, comprising:

[0032] The acquisition module is configured to acquire note control command information for the audio to be recognized;

[0033] The first processing module is configured to perform instrument component-based filtering processing on the audio to be identified based on the note control instruction information to obtain the first audio.

[0034] The second processing module is configured to perform processing on the first audio based on the spectral characteristics of the first audio to obtain a first candidate set, the first candidate set including a plurality of first candidate audio segments;

[0035] The third processing module is configured to perform an analysis based on the musical features of each of the first candidate audio segments to determine the audio quality index of each of the second candidate audio segments in the second candidate set; the second candidate set is the entire set or a subset of the first candidate set.

[0036] The filtering module is configured to perform filtering based on the audio quality index of each second candidate audio segment to obtain target audio segments; the target audio segments are used to characterize the musical highlight segments in the audio to be identified.

[0037] In one optional implementation, the second processing module is specifically used for:

[0038] Obtain the spectral characteristics of the first audio;

[0039] Based on the spectral characteristics of the first audio, the first audio is divided into audio segments to obtain multiple audio sub-segments;

[0040] Each audio sub-segment is determined to have a spectral sub-feature in the spectral features. Based on the spectral sub-feature corresponding to each audio sub-segment, multiple audio sub-segments are clustered to obtain multiple first candidate audio segments. The first candidate audio segments are then used as the first candidate set.

[0041] In an optional implementation, the third processing module includes:

[0042] The first acquisition submodule is configured to acquire the musical features of each of the first candidate audio segments;

[0043] The processing submodule is configured to execute, based on the lyrics control instruction information of the audio to be identified and the musical characteristics of each of the first candidate audio segments, to determine the audio quality index of each of the second candidate audio segments in the second candidate set.

[0044] In an optional implementation, the processing submodule includes:

[0045] The first processing unit is configured to perform a first audio quality index for each of the first candidate audio segments based on the musical features of each first candidate audio segment.

[0046] The second processing unit is configured to perform filtering on multiple first audio quality indicators based on lyrics control instruction information of the audio to be identified, so as to obtain a second audio quality indicator for each second candidate audio segment in the second candidate set.

[0047] In an optional implementation, the first processing unit is specifically used for:

[0048] Obtain the initial loudness feature, initial spectral brightness feature, and initial drum beat intensity feature for each of the first candidate audio segments;

[0049] The initial loudness feature, initial spectral brightness feature, and initial drum intensity feature of each first candidate audio segment are normalized to obtain the loudness feature, spectral brightness feature, and drum intensity feature of each first candidate audio segment, respectively.

[0050] Obtain the weight information corresponding to loudness features, spectral brightness features, and drumbeat intensity features, respectively;

[0051] Based on the weight information, the loudness feature, spectral brightness feature and drum beat intensity feature of each first candidate audio segment are weighted to obtain the first audio quality index of each first candidate audio segment.

[0052] In an optional implementation, where the second audio quality indicator includes an audio quality score, the filtering module includes:

[0053] The sorting submodule is configured to sort the multiple second candidate audio segments sequentially based on the number of repetitions of each of the second candidate audio segments, and select the third candidate audio segments and the fourth candidate audio segments corresponding to the two highest repetition counts in the sorting.

[0054] The first filtering submodule is configured to, when the difference between the number of repetitions of the third candidate audio segment and the number of repetitions of the fourth candidate audio segment is greater than a difference threshold, select the candidate audio segment with the larger number of repetitions between the third and fourth candidate audio segments as the target audio segment.

[0055] In an optional implementation, the first filtering module further includes:

[0056] The second filtering submodule is configured to, when the difference between the number of repetitions of the third candidate audio segment and the number of repetitions of the fourth candidate audio segment is less than or equal to a difference threshold, select the candidate audio segment with the larger audio quality score between the third and fourth candidate audio segments as the target audio segment.

[0057] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0058] processor;

[0059] Memory used to store the processor's executable instructions;

[0060] The processor is configured to execute the instructions to implement the audio recognition method as described in any of the above embodiments.

[0061] According to a fourth aspect of the present disclosure, a computer storage medium is provided, wherein when instructions in the computer storage medium are executed by a processor of an electronic device, the electronic device performs the audio recognition method described in any of the above embodiments.

[0062] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements the audio recognition method described in any of the above embodiments.

[0063] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0064] This embodiment of the disclosure obtains note control command information of the audio to be identified; filters the audio based on the instrument component dimension based on the note control command information to obtain a first audio; processes the first audio based on the spectral characteristics of the first audio to obtain a first candidate set, the first candidate set including multiple first candidate audio segments; determines the audio quality index of each second candidate audio segment in a second candidate set based on the musical characteristics of each first candidate audio segment; the second candidate set is the entire set or a subset of the first candidate set; and selects target audio segments based on the audio quality index of each second candidate audio segment; the target audio segments are used to characterize musical highlight segments in the audio to be identified. By combining note control command information to filter the audio to be identified based on the instrument component dimension, invalid components in the musical audio are removed, achieving automatic selection of musical highlight segments, reducing reliance on manual selection, eliminating the need for additional manpower, and improving the accuracy of selected musical highlight segments.

[0065] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0067] Figure 1 This is a flowchart illustrating an audio recognition method according to an exemplary embodiment.

[0068] Figure 2 This is a partial flowchart illustrating an audio recognition method according to an exemplary embodiment.

[0069] Figure 3 This is a partial flowchart illustrating another audio recognition method according to an exemplary embodiment.

[0070] Figure 4 This is a schematic diagram illustrating the process of an audio recognition method according to an exemplary embodiment.

[0071] Figure 5 This is a block diagram of an audio recognition device according to an exemplary embodiment.

[0072] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0073] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0074] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0075] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0076] Figure 1 This is a flowchart illustrating an audio recognition method according to an exemplary embodiment. The audio recognition method can be applied to electronic devices. Taking this electronic device as an example as a server, the server can be, but is not limited to, a standalone server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms, among others. Figure 1 As shown, the method includes the following steps.

[0077] In step S101, the note control instruction information of the audio to be recognized is obtained.

[0078] Note control command information refers to MIDI (Musical Instrument Digital Interface). It transmits not sound signals, but rather instruction information such as notes and control parameters. The note control command information itself does not contain musical waveform data; instead, it records each note as a number, describing the music through bytes. For example, it specifies which instrument is used at a given moment, what note it begins with, what tone it ends with, and what accompaniment is provided. The audio to be identified refers to the musical audio to be identified, such as songs or light music.

[0079] Optionally, note control instruction information can be obtained from the MIDI file of the audio to be recognized. Alternatively, the note control instruction information can be obtained by converting the audio data of the audio to be recognized.

[0080] In step S103, the audio to be identified is filtered based on the instrument component dimension based on the note control instruction information to obtain the first audio.

[0081] Generally, a song typically consists of the following elements: intro, verse, pre-chorus, chorus, bridge, and outro. The intro is the opening section, laying the foundation for the song's rhythm and melody. The verse is the main part of the song, guiding the listener to the song's main information and driving the music forward. The chorus is the most exciting and memorable part of the song, creating a stark contrast with the verse and usually repeating multiple times. The bridge connects different sections, typically consisting of only instruments and no vocals; this section may serve as a transition point between modes. The final outro is the end of the song.

[0082] Optionally, the note control instruction information can provide a wealth of information, such as various instrument information contained in the audio, specifically, when which instrument (e.g., piano, drums, etc.) begins. By acquiring the audio track used to characterize the instrument component dimension provided in the note control instruction information of the music signal, the audio to be identified is filtered based on the instrument component dimension. This removes disordered components from the music audio features, retaining effective components such as Vocal, bass, and drums, resulting in the first audio. Here, disordered components refer to other irregular parts of the audio.

[0083] In step S105, the first audio is processed based on the spectral characteristics of the first audio to obtain a first candidate set, which includes multiple first candidate audio segments.

[0084] Specifically, the first audio is segmented based on its spectral characteristics to obtain a first candidate set. This first candidate set includes multiple first candidate audio segments, each consisting of audio sub-segments belonging to the same music category. Each music category may correspond to one or more first candidate audio segments. In an optional embodiment, processing the first audio based on its spectral characteristics to obtain the first candidate set includes:

[0085] Obtain the spectral characteristics of the first audio;

[0086] Based on the spectral characteristics of the first audio, the first audio is divided into audio segments to obtain multiple audio sub-segments;

[0087] Each audio sub-segment is determined to have a spectral sub-feature in the spectral features. Based on the spectral sub-feature corresponding to each audio sub-segment, multiple audio sub-segments are clustered to obtain multiple first candidate audio segments. The first candidate audio segments are then used as the first candidate set.

[0088] Among them, the spectral features can be MFCC (Mel-Frequency Cepstral Coefficients) features, or CQT (Constant Q spectral Transform) local features, which are nonlinear frequency domain features obtained by filtering time-domain audio with a set of constant Q filters. These features are more in line with music theory, etc.

[0089] Optionally, taking MFCC features as an example, obtaining the spectral features of the first audio signal may include: converting the audio signal of the first audio signal to Mel frequencies, then performing cepstral analysis to extract the MFCC feature spectrum of the audio signal, thus obtaining the spectral features of the first audio signal. The low-order signal features of the spectral features include timbre and loudness. The spectral features (MFCC feature spectrum) X of the first audio signal can be represented as:

[0090] X = MFCC(x)

[0091] Where x is the audio time-domain signal.

[0092] Next, the self-similarity matrix is ​​obtained using the formula for calculating the self-similarity matrix. Then, an enhanced self-similarity matrix (SSM matrix) can be obtained through a series of feature smoothing methods, path smoothing, and thresholding. Then, segments are defined by capturing the local and global relationships within the matrix. In one implementation, the song's structure features can be obtained by calculating the cyclic time-lag matrix. The calculation process for the cyclic time-lag matrix is ​​as follows:

[0093] First, time-lag representation can be defined as:

[0094] L(k,n)=S((n+k),n)

[0095] Where k,n represent the number of time frames, n∈[0,N-1], k∈[-n,N-1-n], N represents the maximum number of time frames, and S represents the self-similarity matrix.

[0096] In practical applications, the number of time frames must be reasonably selected so that the sum of k and n, i.e., n+k, is within the range of [0, N-1]. Therefore, by simplifying the above formula, the formula for calculating the cyclic delay matrix L(k,n) can be obtained as follows:

[0097] L(k,n)=S((n+k)mod N,n)

[0098] Here, mod represents the modulo function.

[0099] Next, by smoothing the cyclic delay matrix and calculating the L2 norm of its first difference, the novelty curve of the song's structural features is obtained. The specific expression of the novelty curve is as follows:

[0100] Δ Structure (n)=||L(n+1)-L(n)||

[0101] Where n∈[0,N-2].

[0102] Next, by detecting the peak value of the novelty curve, the segment division time point (SegTime) of each musical segment is obtained. Specifically, SegTime = peakDetection(Δ Structure (n)), that is, to divide the first audio into audio segments based on the spectral characteristics of the first audio, and obtain multiple audio sub-segments.

[0103] Finally, the spectral sub-features (i.e., MFCC features) of each audio sub-segment in the spectral features are determined. Based on the spectral sub-features corresponding to each audio sub-segment, k-means clustering is performed on the multiple music sub-segments obtained from the division to obtain the clustered music tags. Here, k is a positive integer, for example, k can be set to 4. Next, by merging adjacent music tag segments of the same class, that is, the two segments have similar content, they can be regarded as the music content of the same segment, thus obtaining multiple first candidate audio segments, and the first candidate audio segments are used as the first candidate set.

[0104] In step S107, based on the musical features of each of the first candidate audio segments, the audio quality index of each of the second candidate audio segments in the second candidate set is determined; the second candidate set is the entire set or a subset of the first candidate set.

[0105] In an alternative implementation, such as Figure 2 As shown, based on the musical features of each first candidate audio segment, the audio quality index of each second candidate audio segment in the second candidate set is determined, including:

[0106] In step S201, the musical features of each of the first candidate audio segments are obtained.

[0107] Optionally, considering that the highlight segments of a song are usually the most emotionally intense parts, these segments typically have the richest musical color, the brightest overall color, the highest loudness, the heaviest drumbeats, and the fullest low frequencies, a comprehensive score is given to each extracted segment by calculating musical features such as average loudness, spectral brightness, and drumbeat intensity. Here, the musical features of each first candidate audio segment may include at least one of the following: initial loudness feature, initial spectral brightness feature, and initial drumbeat intensity feature. The RMS expression for the initial loudness feature is:

[0108]

[0109] Where x is the input audio signal, N is the window length, the length of a frame is N, m is the frame index, and n represents a point in the current frame, n∈[-N / 2,N / 2].

[0110] The initial spectral luminance feature expression for Spectral Centroid is:

[0111]

[0112] Among them, f kLet X be the center frequency of the k-th frequency band of the short-time Fourier transform, X be the spectrum of the short-time Fourier transform of the music signal, m be the frame index, and N be the total number of frequency bands, which is the length of the Fourier transform.

[0113] The initial drumbeat intensity characteristic spectral flux expression is:

[0114]

[0115] Where X is the spectrum of the short-time Fourier transform of the music signal, m is the frame index, K represents the total number of frequency bands, k represents the k-th frequency band, and Δspectral(m) is the intensity of the spectrum change in each frame, which is characterized by the intensity of the drum beats.

[0116] It should be noted that the specific details of the initial loudness characteristics, initial spectral brightness characteristics, and initial drumbeat intensity characteristics mentioned above can be found in relevant technical documents, and will not be repeated here.

[0117] In step S203, based on the lyrics control instruction information of the audio to be identified and the musical characteristics of each of the first candidate audio segments, the audio quality index of each second candidate audio segment in the second candidate set is determined.

[0118] In an alternative implementation, such as Figure 3 As shown, the step of determining the audio quality index of each second candidate audio segment in the second candidate set based on the lyrics control instruction information of the audio to be identified and the musical features of each first candidate audio segment includes:

[0119] In step S301, a first audio quality index is determined for each of the first candidate audio segments based on the musical features of each first candidate audio segment.

[0120] In an optional implementation, determining the first audio quality index for each first candidate audio segment based on the musical features of each first candidate audio segment includes:

[0121] The initial loudness feature, initial spectral brightness feature, and initial drum intensity feature of each first candidate audio segment are normalized to obtain the loudness feature, spectral brightness feature, and drum intensity feature of each first candidate audio segment, respectively.

[0122] Obtain the weight information corresponding to loudness features, spectral brightness features, and drumbeat intensity features, respectively;

[0123] Based on the weight information, the loudness feature, spectral brightness feature and drum beat intensity feature of each first candidate audio segment are weighted to obtain the first audio quality index of each first candidate audio segment.

[0124] Optionally, the initial loudness feature, initial spectral brightness feature, and initial drum beat intensity feature are smoothed to obtain and averaged, then normalized to a value between 0 and 1. Weights are assigned to each feature, and candidate highlight segments of the music are weighted based on these weights to calculate a score (Score1) for each segment, thus obtaining the first audio quality index. For example, the first audio quality index can be expressed as:

[0125] Score1 = k1 × P RMS +k2×P SC +k3×P sp

[0126] Among them, P RMS For loudness characteristics, P SC For spectral brightness characteristics, P sp The drumbeat intensity feature is represented by k1, k2, and k3, which are their respective weight information, and the sum of their weights is 1.

[0127] In step S303, multiple first audio quality indicators are filtered based on the lyrics control instruction information of the audio to be identified to obtain a second audio quality indicator for each second candidate audio segment in the second candidate set.

[0128] Optionally, the note control command information includes not only instrument information but also lyrics information. Here, the provided lyrics MIDI information is used to extract the audio VOCAL information of each segment, and segments without vocals are removed from the selected audio segments to prevent the bridge from being mistakenly detected as the chorus. After this secondary filtering process, the second audio quality index can be expressed as:

[0129] Score2=Delete(Score1, MIDI-Vocal)

[0130] Delete() is a filtering function used to filter out segments in Score1 that do not contain MIDI-Vocal, that is, to filter out segments in the first audio quality index that do not contain vocals.

[0131] In step S109, target audio segments are selected based on the audio quality index of each second candidate audio segment; the target audio segments are used to characterize the musical highlight segments in the audio to be identified.

[0132] In an optional implementation, where the second audio quality index includes an audio quality score, the step of filtering target audio segments based on the audio quality index of each second candidate audio segment includes:

[0133] Based on the number of repetitions of each of the second candidate audio segments, the multiple second candidate audio segments are sorted sequentially, and the third and fourth candidate audio segments corresponding to the top two repetition counts are selected respectively.

[0134] If the difference between the number of repetitions of the third candidate audio segment and the number of repetitions of the fourth candidate audio segment is greater than a difference threshold, the candidate audio segment with the larger number of repetitions between the third and fourth candidate audio segments is selected as the target audio segment.

[0135] In an optional implementation, after selecting the third and fourth candidate audio segments corresponding to the two highest repetition counts, the method further includes:

[0136] If the difference between the number of repetitions of the third candidate audio segment and the number of repetitions of the fourth candidate audio segment is less than or equal to the difference threshold, the candidate audio segment with the larger audio quality score between the third and fourth candidate audio segments is selected as the target audio segment.

[0137] Optionally, considering that the verses and choruses in a song will be repeated multiple times, the number of times each second candidate audio segment is repeated in the song (M) can be calculated using the lyrics. Combined with the previously calculated Score2, the final highlight segment is selected. For example, as one implementation, the top N (e.g., the top two) repetition counts can be selected from the existing selected segments, i.e., the third and fourth candidate audio segments corresponding to the top two repetition counts are selected. If the difference between the repetition counts of the third and fourth candidate audio segments is less than or equal to a threshold, i.e., if the difference in repetition counts between these two candidate audio segments is not significant, then the audio quality score (e.g., Score2) corresponding to the second audio quality index is combined to select the segment with the highest audio quality score from these two candidate audio segments. If the difference between the number of repetitions of the third candidate audio segment and the number of repetitions of the fourth candidate audio segment is greater than the difference threshold, that is, if the number of repetitions of these two segments differs greatly, the candidate audio segment with the larger number of repetitions between the third and fourth candidate audio segments is selected as the target audio segment, that is, the segment with the most repetitions is selected first.

[0138] In the above embodiment, note control command information of the audio to be identified is obtained; based on the note control command information, the audio to be identified is filtered according to the instrument component dimension to obtain a first audio; the first audio is processed based on the spectral characteristics of the first audio to obtain a first candidate set, which includes multiple first candidate audio segments; based on the musical characteristics of each first candidate audio segment, the audio quality index of each second candidate audio segment in the second candidate set is determined; the second candidate set is the entire set or a subset of the first candidate set; based on the audio quality index of each second candidate audio segment, a target audio segment is selected; the target audio segment is used to characterize the musical highlight segments in the audio to be identified. Thus, by combining note control command information to perform instrument component-based filtering on the audio to be identified, invalid components in the musical audio are removed, achieving automatic selection of musical highlight segments, reducing reliance on manual selection, and improving the accuracy of the selected musical highlight segments.

[0139] Furthermore, by filtering multiple first audio quality indicators using the lyrics control instructions of the audio to be identified, a second audio quality indicator is obtained for each second candidate audio segment in the second candidate set. This eliminates non-human vocal segments, reduces interference from transitional parts on chorus detection, and further improves the accuracy of selected musical highlight segments. Additionally, by combining loudness, brightness, and drum beat intensity with an understanding of the audio lyrics, a multi-dimensional comprehensive quality evaluation of the musical candidates is performed, further improving the accuracy of selected musical highlight segments.

[0140] For ease of understanding, Figure 4 This is a schematic diagram illustrating the process of an audio recognition method according to an exemplary embodiment, such as... Figure 4As shown, the electronic device can acquire the audio to be identified and its note control command information. On one hand, it extracts audio features based on the note control command information and uses a music structure clustering algorithm to divide the music segments, obtaining a first candidate set, which includes multiple first candidate audio segments. Then, through audio content understanding of the first candidate audio segments, such as audio analysis of loudness, spectral brightness, and drum beat intensity, a first audio quality index corresponding to a second candidate set is obtained. This second candidate set is the entirety or a subset of the first candidate set, and includes multiple second candidate audio segments. On the other hand, it extracts vocal segments from the audio to be identified based on the note control command information. Based on these vocal segments, the first audio quality index is filtered, removing segments without vocals, resulting in a second audio quality index. Furthermore, the electronic device acquires the lyrics of the audio to be identified and performs content understanding based on the lyrics, calculating the number of times each second candidate audio segment is repeated in the song (M). Combining this with the second audio quality index, it selects audio highlight segments to obtain target audio segments, which are used to characterize the musical highlight segments in the audio to be identified.

[0141] Figure 5 This is a block diagram illustrating an audio recognition device according to an exemplary embodiment. (Refer to...) Figure 5 The device includes:

[0142] The acquisition module 510 is configured to acquire note control instruction information of the audio to be recognized;

[0143] The first processing module 520 is configured to perform a filtering process on the audio to be identified based on the instrument component dimension, based on the note control instruction information, to obtain the first audio.

[0144] The second processing module 530 is configured to perform processing on the first audio based on the spectral features of the first audio to obtain a first candidate set, the first candidate set including a plurality of first candidate audio segments;

[0145] The third processing module 540 is configured to perform an analysis based on the musical features of each of the first candidate audio segments to determine the audio quality index of each of the second candidate audio segments in the second candidate set; the second candidate set is the entire set or a subset of the first candidate set.

[0146] The filtering module 550 is configured to perform filtering based on the audio quality index of each second candidate audio segment to obtain target audio segments; the target audio segments are used to characterize the musical highlight segments in the audio to be identified.

[0147] In one optional implementation, the second processing module is specifically used for:

[0148] Obtain the spectral characteristics of the first audio;

[0149] Based on the spectral characteristics of the first audio, the first audio is divided into audio segments to obtain multiple audio sub-segments;

[0150] Each audio sub-segment is determined to have a spectral sub-feature in the spectral features. Based on the spectral sub-feature corresponding to each audio sub-segment, multiple audio sub-segments are clustered to obtain multiple first candidate audio segments. The first candidate audio segments are then used as the first candidate set.

[0151] In an optional implementation, the third processing module includes:

[0152] The first acquisition submodule is configured to acquire the musical features of each of the first candidate audio segments;

[0153] The processing submodule is configured to execute, based on the lyrics control instruction information of the audio to be identified and the musical characteristics of each of the first candidate audio segments, to determine the audio quality index of each of the second candidate audio segments in the second candidate set.

[0154] In an optional implementation, the processing submodule includes:

[0155] The first processing unit is configured to perform a first audio quality index for each of the first candidate audio segments based on the musical features of each first candidate audio segment.

[0156] The second processing unit is configured to perform filtering on multiple first audio quality indicators based on lyrics control instruction information of the audio to be identified, so as to obtain a second audio quality indicator for each second candidate audio segment in the second candidate set.

[0157] In an optional implementation, the first processing unit is specifically used for:

[0158] Obtain the initial loudness feature, initial spectral brightness feature, and initial drum beat intensity feature for each of the first candidate audio segments;

[0159] The initial loudness feature, initial spectral brightness feature, and initial drum intensity feature of each first candidate audio segment are normalized to obtain the loudness feature, spectral brightness feature, and drum intensity feature of each first candidate audio segment, respectively.

[0160] Obtain the weight information corresponding to loudness features, spectral brightness features, and drumbeat intensity features, respectively;

[0161] Based on the weight information, the loudness feature, spectral brightness feature and drum beat intensity feature of each first candidate audio segment are weighted to obtain the first audio quality index of each first candidate audio segment.

[0162] In an optional implementation, where the second audio quality indicator includes an audio quality score, the filtering module includes:

[0163] The sorting submodule is configured to sort the multiple second candidate audio segments sequentially based on the number of repetitions of each of the second candidate audio segments, and select the third candidate audio segments and the fourth candidate audio segments corresponding to the two highest repetition counts in the sorting.

[0164] The first filtering submodule is configured to, when the difference between the number of repetitions of the third candidate audio segment and the number of repetitions of the fourth candidate audio segment is greater than a difference threshold, select the candidate audio segment with the larger number of repetitions between the third and fourth candidate audio segments as the target audio segment.

[0165] In an optional implementation, the first filtering module further includes:

[0166] The second filtering submodule is configured to, when the difference between the number of repetitions of the third candidate audio segment and the number of repetitions of the fourth candidate audio segment is less than or equal to a difference threshold, select the candidate audio segment with the larger audio quality score between the third and fourth candidate audio segments as the target audio segment.

[0167] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. (Refer to...) Figure 6 The electronic device includes a processor; a memory for storing processor-executable instructions; wherein, when the processor is configured to execute the instructions stored in the memory, it implements the steps of any of the audio recognition methods described above.

[0168] The electronic device can be a terminal, a server, or a similar computing device. Taking a server as an example... Figure 6This is a block diagram illustrating an electronic device 1000 for audio recognition according to an exemplary embodiment. The electronic device 1000 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 1010 (CPUs 1010 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 1030 for storing data, and one or more storage media 1020 (e.g., one or more mass storage devices) for storing application programs 1023 or data 1022. The memory 1030 and storage media 1020 may be temporary or persistent storage. The program stored in the storage media 1020 may include one or more modules, each module including a series of instruction operations on the electronic device. Furthermore, the CPU 1010 may be configured to communicate with the storage media 1020 and execute the series of instruction operations in the storage media 1020 on the electronic device 1000.

[0169] Electronic device 1000 may also include one or more power supplies 1060, one or more wired or wireless network interfaces 1050, one or more input / output interfaces 1040, and / or one or more operating systems 1021, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0170] The input / output interface 1040 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 1000. In one example, the input / output interface 1040 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In an exemplary embodiment, the input / output interface 1040 can be a radio frequency (RF) module for wireless communication with the Internet.

[0171] Those skilled in the art will understand that Figure 6 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, the electronic device 1000 may also include components that are more... Figure 6 The more or fewer components shown, or having the same Figure 6 The different configurations shown.

[0172] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of the electronic device 1000 to perform the above-described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0173] In an exemplary embodiment, a computer storage medium is also provided, which, when the instructions in the computer storage medium are executed by the processor of an electronic device, enables the electronic device to perform the method provided in any of the above embodiments.

[0174] In an exemplary embodiment, a computer program product is also provided, comprising a computer program / instructions that, when executed by a processor, implement the methods provided in any of the above embodiments. Optionally, the computer program is stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the methods provided in any of the above embodiments.

[0175] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0176] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0177] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio recognition method, characterized by, include: Obtain the note control command information of the audio to be recognized; Based on the note control instruction information, the audio to be identified is filtered based on the instrument component dimension to obtain the first audio. Obtain the spectral characteristics of the first audio; Based on the spectral characteristics of the first audio, the first audio is divided into audio segments to obtain multiple audio sub-segments; Determine the spectral sub-features of each audio sub-segment in the spectral features, cluster the multiple audio sub-segments based on the spectral sub-features corresponding to each audio sub-segment, obtain multiple first candidate audio segments, and take the first candidate audio segments as a first candidate set, the first candidate set including multiple first candidate audio segments; Obtain the musical features of each of the first candidate audio segments; Based on the musical features of each first candidate audio segment, a first audio quality index is determined for each first candidate audio segment. Based on the lyrics control instruction information of the audio to be identified, multiple first audio quality indicators are filtered to obtain a second audio quality indicator for each second candidate audio segment in the second candidate set; the second candidate set is the entire set or a subset of the first candidate set; Based on the audio quality index of each of the second candidate audio segments, target audio segments are selected; the target audio segments are used to characterize the musical highlight segments in the audio to be identified.

2. The method of claim 1, wherein, The step of determining the first audio quality index for each first candidate audio segment based on the musical features of each first candidate audio segment includes: Obtain the initial loudness feature, initial spectral brightness feature, and initial drum beat intensity feature for each of the first candidate audio segments; The initial loudness feature, initial spectral brightness feature, and initial drum intensity feature of each first candidate audio segment are normalized to obtain the loudness feature, spectral brightness feature, and drum intensity feature of each first candidate audio segment, respectively. Obtain the weight information corresponding to the loudness feature, spectral brightness feature, and drumbeat intensity feature, respectively; Based on the weight information, the loudness feature, spectral brightness feature, and drum beat intensity feature of each first candidate audio segment are weighted to obtain the first audio quality index of each first candidate audio segment.

3. The method of claim 1, wherein, The second audio quality indicator includes an audio quality score; the process of selecting target audio segments based on the audio quality indicator of each second candidate audio segment includes: Based on the number of repetitions of each second candidate audio segment, the multiple second candidate audio segments are sorted sequentially, and the third and fourth candidate audio segments corresponding to the top two repetition counts are selected respectively. If the difference between the number of repetitions of the third candidate audio segment and the number of repetitions of the fourth candidate audio segment is greater than a difference threshold, the candidate audio segment with the larger number of repetitions between the third and fourth candidate audio segments is selected as the target audio segment.

4. The method of claim 3, wherein, After selecting the third and fourth candidate audio segments corresponding to the top two repetition counts in the sorting, the method further includes: If the difference between the number of repetitions of the third candidate audio segment and the number of repetitions of the fourth candidate audio segment is less than or equal to the difference threshold, the candidate audio segment with the larger audio quality score between the third and fourth candidate audio segments is selected as the target audio segment.

5. An audio recognition device, characterized in that, include: The acquisition module is configured to acquire note control command information for the audio to be recognized; The first processing module is configured to perform instrument component-based filtering processing on the audio to be identified based on the note control instruction information to obtain the first audio. The second processing module is configured to perform the following actions: acquiring the spectral features of the first audio; dividing the first audio into audio segments based on the spectral features of the first audio to obtain multiple audio sub-segments; determining the spectral sub-features of each audio sub-segment in the spectral features; clustering the multiple audio sub-segments based on the spectral sub-features corresponding to each audio sub-segment to obtain multiple first candidate audio segments; and using the first candidate audio segments as a first candidate set, wherein the first candidate set includes multiple first candidate audio segments. The third processing module includes: a first acquisition submodule configured to acquire the musical features of each first candidate audio segment; a first processing unit configured to determine a first audio quality index of each first candidate audio segment based on the musical features of each first candidate audio segment; and a second processing unit configured to filter multiple first audio quality indices based on the lyrics control instruction information of the audio to be identified, to obtain a second audio quality index of each second candidate audio segment in the second candidate set. The filtering module is configured to perform filtering based on the audio quality index of each second candidate audio segment to obtain target audio segments; the target audio segments are used to characterize the musical highlight segments in the audio to be identified.

6. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the audio recognition method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device performs the audio recognition method as described in any one of claims 1 to 4.

Images