Audio feature extraction method and device, computer equipment and storage medium

By performing mapping transformation and Mel filtering on the initial audio features, combining nonlinear and multi-stage discrete cosine transformation, and fusing spectrum representation and frequency coefficients, the problem of insufficient features in traditional vehicle-mounted music feature extraction methods is solved, and different frequencies are achieved Accurate capture and analysis of features improves user experience.

CN119068906APending Publication Date: 2024-12-03CHONGQING JINKANG NEW ENERGY VEHICLE CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411196387.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The traditional in-vehicle music feature extraction method extracts insufficient features and cannot meet the growing user experience needs. Moreover, the feature extraction method based on deep learning is long and is not suitable for real-time feature extraction of music.

Method used

By mapping and transformation processing of the initial audio features, the audio features to be processed at different frequencies are obtained, and multiple Mel filter banks with different frequencies and resolutions are filtered, and nonlinear transformation and multi-stage discrete cosine transformation are performed. Finally, the representations of each spectrum and frequency coefficient are fused to obtain the target audio characteristics.

Benefits of technology

It improves the accuracy and completeness of audio feature extraction, meets user experience needs, avoids insufficient feature extraction, and realizes accurate capture and analysis of different frequency features, thereby improving user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068906A_ABST
    Figure CN119068906A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of vehicle music systems, and discloses an audio feature extraction method and device, computer equipment and a storage medium. According to the technical scheme, the Mel filter banks with different frequencies can be correspondingly adopted for filtering the features with different frequencies, then different obtained spectrum representations are processed respectively, and finally fusion is carried out, so that the features with different frequencies can be accurately captured and analyzed, and the accuracy of the features with different frequencies is improved. And the accuracy of audio feature extraction is improved. Meanwhile, the features of different frequencies correspond to different audio details, and the audio features are extracted and fused after being processed in parallel by distinguishing the frequencies, so that the integrity of audio feature extraction is ensured, the condition of insufficient feature extraction in the prior art is avoided, the user experience requirement is met, and the user experience degree is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of vehicle music systems, and in particular to a method, device, computer equipment and storage medium for extracting audio features. Background Art

[0002] With the widespread use of in-car music systems in modern cars, the demand for improving the in-car music experience is gradually increasing. Traditional in-car music feature extraction methods cannot meet the growing experience needs of users due to the technical problem of insufficient extracted features. Summary of the invention

[0003] In view of the above problems, the present application provides an audio feature extraction method, device, computer equipment and storage medium, which are used to solve the technical problem of insufficient feature extraction in the prior art. After the initial audio feature is mapped and transformed, the audio features to be processed including different frequencies are obtained, and then a plurality of Mel filter groups with different frequencies and resolutions are used to filter the audio features to be processed, so as to obtain multiple spectrum representations corresponding to the audio features to be processed; each spectrum representation is subjected to nonlinear transformation and multi-level discrete cosine transformation to obtain multiple frequency coefficients, and finally each spectrum representation and each frequency coefficient are fused to obtain the target audio feature. It is realized that the Mel filter groups with different frequencies are used to filter the features corresponding to different frequencies, and then the different spectrum representations obtained are processed separately, and finally fused, so that the features of different frequencies can be accurately captured and analyzed, thereby improving the accuracy of audio feature extraction. At the same time, the features of different frequencies correspond to different audio details. By distinguishing the frequencies, extracting them separately and processing them in parallel, and then merging them, the integrity of the audio feature extraction is guaranteed, and the situation of insufficient extracted features existing in the prior art is avoided, thereby meeting the user experience requirements and further improving the user experience.

[0004] According to one aspect of an embodiment of the present application, a method for extracting audio features is provided, the method comprising: performing mapping transformation processing on initial audio features to obtain audio features to be processed after mapping transformation; wherein the audio features to be processed include multiple audio features of different frequencies; using multiple preset Mel filter groups to perform filtering processing on the audio features to be processed to obtain multiple spectrum representations corresponding to the audio features to be processed; wherein the frequencies or resolutions of the Mel filter groups are different, and one spectrum representation corresponds to one Mel filter group; performing nonlinear transformation on each spectrum representation to obtain multiple transformed spectrum representations; performing multi-level discrete cosine transform on each transformed spectrum representation to obtain multiple frequency coefficients; and fusing each spectrum representation and each frequency coefficient to obtain the target audio feature.

[0005] In an optional manner, before the step of mapping and transforming the initial audio features to obtain the audio features to be processed, the method further includes: preprocessing the original music signal to obtain the initial audio features.

[0006] In an optional manner, the preprocessing of the original music signal to obtain the initial audio features further includes: encoding and decoding the original music signal to obtain music data in a unified format; and downsampling the music data to obtain the initial audio features.

[0007] In an optional manner, the step of performing mapping transformation processing on the initial audio features to obtain the audio features to be processed after the mapping transformation further includes: performing mapping transformation processing on the initial audio features based on a preset transformation to obtain a complex plane image including the frequency and phase of the initial audio features; wherein the complex plane image is the audio features to be processed after the mapping transformation, and the points in the complex plane image represent the frequency of the audio features to be processed.

[0008] In an optional manner, the step of performing a nonlinear transformation on each of the spectral representations to obtain a plurality of transformed spectral representations further includes: performing a nonlinear compression transformation on each of the spectral representations based on a preset function to obtain a plurality of transformed spectral representations; wherein the preset function includes a logarithmic function or an exponential function.

[0009] In an optional manner, the step of performing a multi-level discrete cosine transform on each of the transformed spectral representations to obtain a plurality of frequency coefficients further includes: traversing each of the transformed spectral representations, and using the traversed transformed spectral representation as the target spectral representation; performing a discrete cosine transform of the current level on the target spectral representation, and using the output of the current level as the input of the discrete cosine transform of the next level, repeating this step until the number of current levels reaches a preset number or the output of the current level meets a preset compression rate; using the output of the current level corresponding to the stop as the frequency coefficient of the target spectral representation, obtaining the frequency coefficients corresponding to each of the transformed spectral representations, and further obtaining a plurality of frequency coefficients.

[0010] In an optional manner, the step of fusing each of the spectral representations and each of the frequency coefficients to obtain the target audio feature further includes: splicing and fusing each of the frequency coefficients based on a weighted average algorithm to obtain a spliced ​​audio feature; fusing the spliced ​​audio feature and a preset feature based on a deep learning algorithm to obtain a target audio feature; wherein the preset feature includes a preset chroma feature and a preset spectral contrast feature.

[0011] According to another aspect of an embodiment of the present application, an audio feature extraction device is provided, comprising: a feature processing module, used to perform mapping transformation processing on the initial audio features to obtain the audio features to be processed after the mapping transformation; wherein the audio features to be processed include multiple audio features of different frequencies; the feature processing module is also used to use a preset multiple Mel filter groups to perform filtering processing on the audio features to be processed to obtain multiple spectrum representations corresponding to the audio features to be processed; wherein the frequencies or resolutions of the Mel filter groups are different, and one spectrum representation corresponds to one Mel filter group; the feature processing module is also used to perform nonlinear transformation on each of the spectrum representations to obtain multiple transformed spectrum representations; the feature processing module is also used to perform multi-level discrete cosine transform on each of the transformed spectrum representations to obtain multiple frequency coefficients; a feature fusion module is used to fuse each of the spectrum representations and each of the frequency coefficients to obtain the target audio feature.

[0012] According to another aspect of an embodiment of the present application, a computer device is provided, comprising: a controller; a memory for storing one or more programs, wherein when the one or more programs are executed by the controller, the controller implements the audio feature extraction method described in any one of the above claims.

[0013] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein the storage medium stores at least one executable instruction, and when the executable instruction is executed on a computer device / equipment, the computer device / equipment executes the operation of the audio feature extraction method described in any one of the above claims.

[0014] The embodiment of the present application maps and transforms the initial audio features to obtain the audio features to be processed including different frequencies, and then uses a plurality of Mel filter groups with different frequencies and resolutions to filter the audio features to be processed, so as to obtain a plurality of spectral representations corresponding to the audio features to be processed; each spectral representation is subjected to nonlinear transformation and multi-level discrete cosine transformation to obtain a plurality of frequency coefficients, and finally each spectral representation and each frequency coefficient are fused to obtain the target audio features. It is realized that the features of different frequencies are filtered by Mel filter groups of different frequencies, and then the obtained different spectral representations are processed separately and finally fused, so that the features of different frequencies can be accurately captured and analyzed, thereby improving the accuracy of audio feature extraction. At the same time, the features of different frequencies correspond to different audio details. By distinguishing the frequencies, extracting them separately and processing them in parallel, and then merging them, the integrity of the audio feature extraction is guaranteed, and the situation of insufficient extracted features existing in the prior art is avoided, thereby meeting the user experience requirements and further improving the user experience.

[0015] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0017] Figure 1 A schematic diagram showing a flow chart of an embodiment of a method for extracting audio features provided by the present application;

[0018] Figure 2 A schematic diagram showing a flow chart of another embodiment of the method for extracting audio features provided by the present application;

[0019] Figure 3 A schematic flow chart of another embodiment of the method for extracting audio features provided by the present application is shown;

[0020] Figure 4 A schematic flow chart of another embodiment of the method for extracting audio features provided by the present application is shown;

[0021] Figure 5 A schematic diagram showing the structure of an embodiment of an audio feature extraction device provided by the present application;

[0022] Figure 6 A schematic structural diagram of an embodiment of a computer device provided by the present application is shown. DETAILED DESCRIPTION

[0023] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0024] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0025] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0026] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0027] With the widespread use of in-car music systems in modern cars, the demand for improving the in-car music experience is gradually increasing. Traditional in-car music feature extraction methods mainly focus on basic sound amplification and equalization. The feature complexity based on neural networks is relatively large, and there are certain limitations in extracting key features of in-car music in real time.

[0028] Through analysis of the prior art, the applicant found that the existing music feature extraction methods have two main problems: 1. The traditional in-car music feature extraction method cannot meet the users' growing experience needs; 2. The feature extraction method based on deep learning takes a long time and is not suitable for real-time feature extraction of music.

[0029] The reasons are: 1. The traditional car music feature extraction method does not extract enough features; 2. The feature extraction method is too complicated and time-consuming.

[0030] In order to solve the problems existing in the prior art, the present application proposes an audio feature extraction method, device, computer equipment and storage medium for solving the above problems, which are described in detail with the following embodiments.

[0031] Figure 1 A flowchart of an embodiment of a method for extracting music features of the present application is shown, and the method is executed by a computer device. Figure 1 As shown, the method comprises the following steps:

[0032] Step S110: performing mapping transformation processing on the initial audio features to obtain audio features to be processed after mapping transformation.

[0033] The audio features to be processed include multiple audio features of different frequencies.

[0034] Specifically, the Z transform is applied to the input discrete music signal (i.e., the initial audio feature) x[n] to obtain its Z-domain representation X(z) (i.e., the audio feature to be processed). Using the inverse transform of the Z transform, the processed signal is returned to the time domain. The Z transform provides a more flexible signal analysis method by mapping discrete signals to the complex plane. Through the Z transform, the discrete time signal is mapped to the complex plane, which allows the frequency and phase information of the signal to be intuitively represented on the complex plane. This mapping allows us to analyze the characteristics of the signal by observing the poles and zeros on the z-plane. Each point on the complex plane can represent a specific frequency and attenuation characteristic, which can analyze the spectrum and attenuation behavior of the signal.

[0035] Step S120: using a plurality of preset Mel filter banks to perform filtering processing on the audio features to be processed, and obtaining a plurality of frequency spectrum representations corresponding to the audio features to be processed.

[0036] The frequencies or resolutions of the Mel filter banks are different, and one spectrum representation corresponds to one Mel filter bank.

[0037] Specifically, a multi-resolution Mel filter bank (i.e., Mel filter bank) is applied in the frequency domain to better capture the spectral details of the music signal. Multiple Mel filter banks with different frequency resolutions are designed so that each Mel filter bank with a frequency resolution can capture the characteristic information of the corresponding frequency. For example, a low-resolution filter bank is used to capture the overall spectral structure, and a high-resolution filter bank is used to capture the detailed features. The music signal is filtered through these filter banks to obtain multiple spectral representations. This method significantly improves the ability to capture the spectral details of the music signal and enhances the richness and refinement of feature extraction.

[0038] Step S130: performing nonlinear transformation on each frequency spectrum representation to obtain a plurality of transformed frequency spectrum representations.

[0039] Specifically, nonlinear compression is performed on the spectrum representation obtained by the filter bank to simulate the human ear's perception characteristics of changes in sound with different intensities.

[0040] Step S140: performing a multi-level discrete cosine transform on each transformed spectrum representation to obtain a plurality of frequency coefficients.

[0041] Specifically, a multi-level discrete cosine transform (DCT) is performed on the compressed Mel frequency energy spectrum to remove redundant information and extract representative frequency coefficients.

[0042] DCT is a linear transformation, which aims to convert discrete time signals into the frequency domain so that the energy of the signal can be concentrated on fewer frequency components, which is convenient for compression and analysis. The core idea of ​​DCT transformation is to use the orthogonal characteristics of the cosine function to expand the signal and express it as a weighted sum in the frequency domain.

[0043] For a discrete signal x[n] of length N, its DCT is defined as follows:

[0044]

[0045] Among them, X k is the frequency coefficient, N is the number of points of the audio feature to be processed, n=0,1,...,N-1, α(k) is a compensation coefficient, which can make the DCT transformation matrix an orthogonal matrix, k=0,1,...,N-1.

[0046] Step S150: fusing each spectrum representation and each frequency coefficient to obtain a target audio feature.

[0047] Specifically, the spectrum representation obtained from filter banks with different frequency resolutions and the frequency coefficients obtained from multi-level DCT are fused to form the final feature vector.

[0048] Beneficial effect: After mapping and transforming the initial audio features, the processed audio features including different frequencies are obtained, and then a plurality of Mel filter groups with different frequencies and resolutions are used to filter the processed audio features to obtain a plurality of spectral representations corresponding to the processed audio features; each spectral representation is subjected to nonlinear transformation and multi-level discrete cosine transformation to obtain a plurality of frequency coefficients, and finally the spectral representations and the frequency coefficients are fused to obtain the target audio features. It is realized that the features of different frequencies are filtered by Mel filter groups of different frequencies, and then the obtained different spectral representations are processed separately and finally fused, so that the features of different frequencies can be accurately captured and analyzed, thereby improving the accuracy of audio feature extraction. At the same time, the features of different frequencies correspond to different audio details. By distinguishing the frequencies, extracting them separately and processing them in parallel and then merging them, the integrity of the audio feature extraction is guaranteed, and the problem of insufficient extracted features existing in the prior art is avoided, thereby meeting the user experience requirements and further improving the user experience.

[0049] In some embodiments, before step S110, the process further includes: preprocessing the original music signal to obtain initial audio features.

[0050] The original music signal is the real-time music streaming data acquired by the acquisition device on the vehicle.

[0051] Specifically, the music stream is encoded and decoded to ensure the consistency of the data format, and the music data is downsampled to meet the requirements of subsequent processing.

[0052] Intentional effect: By preprocessing the original music signal so that the obtained initial audio features are adapted to the subsequent feature extraction operation, the feasibility of audio feature extraction is guaranteed.

[0053] In some embodiments, Figure 2 As shown, the original music signal is preprocessed to obtain the initial audio feature step, further comprising:

[0054] Step S210: Encode and decode the original music signal to obtain music data in a unified format.

[0055] Specifically, audio coding and decoding is an important part of the field of digital signal processing, which involves the process of converting analog audio signals into digital form and restoring digital audio data into analog signals.

[0056] Step S220: down-sample the music data to obtain initial audio features.

[0057] Specifically, downsampling audio mainly involves reducing the sampling rate of the audio, which is a common operation in the field of digital signal processing, usually used to reduce the data transmission rate or data size. Downsampling can be achieved in a variety of ways, including using specialized software libraries such as ffmpeg, NumPy or soundfile libraries, as well as implementing custom algorithms through programming.

[0058] Beneficial effects: The original music signal is processed by encoding, decoding and downsampling to obtain initial audio features with a unified format and easy operation, so that the obtained initial audio features are suitable for subsequent feature extraction operations, ensuring the feasibility of audio feature extraction.

[0059] In some embodiments, step S110 further includes: performing a mapping transformation process on the initial audio feature based on a preset transformation to obtain a complex plane image including the frequency and phase of the initial audio feature.

[0060] The complex plane image is the audio feature to be processed after mapping transformation, and the points in the complex plane image represent the frequency of the audio feature to be processed.

[0061] Specifically, the preset transformation is Z-transformation, and the Z-transformation is applied to the input discrete music signal (ie, the initial audio feature) x[n] to obtain its Z-domain representation X(z). The inverse transformation of the Z-transformation is used to return the processed signal to the time domain.

[0062] The Z transform is a power series representation of the discrete music signal (i.e., the initial audio feature) x[n], defined as:

[0063]

[0064] Where X(z) is the result of the Z transform, z is a complex number, usually expressed as z = re jθ , where r is the magnitude, θ is the angle, and j is the imaginary unit.

[0065] Through the Z transform, the discrete music signal (i.e. the original audio feature) is mapped to the complex plane, which allows the frequency and phase information of the signal to be intuitively represented on the complex plane. This mapping allows the actor to analyze the characteristics of the signal by observing the poles and zeros on the z-plane.

[0066] In the complex plane, the complex variable z is expressed as z = re jω (in polar form), or z = x + jy (in rectangular form). This representation allows detailed analysis of the signal in the frequency domain, covering the processing of different frequency components. Each point on the complex plane can represent a specific frequency and attenuation characteristic, enabling analysis of the signal's spectrum and attenuation behavior.

[0067] Beneficial effects: The steps of mapping and transforming the initial audio features are further refined, and each frequency of the audio features to be processed after mapping and transformation is mapped to the complex plane for display based on the preset transformation, so that the frequency can be directly analyzed based on the complex plane. The preset transformation provides a more flexible signal analysis method for audio features, which can more accurately capture the spectral characteristics of music signals and improve the accuracy of spectral analysis.

[0068] In some embodiments, step S130 further includes: performing a nonlinear compression transformation on each spectrum representation based on a preset function to obtain a plurality of transformed spectrum representations.

[0069] The preset function includes a logarithmic function or an exponential function.

[0070] Specifically, the spectrum representation obtained by the filter bank is nonlinearly compressed to simulate the human ear's perception of changes in sound intensity. The Mel frequency energy spectrum is nonlinearly transformed, such as using a logarithmic function or an exponential function for compression.

[0071] The nonlinear compression function can be selected as follows: Y = log (1 + αX) or Y = X β

[0072] Where X is the input energy spectrum, Y is the compressed output, and α and β are adjustment parameters.

[0073] Beneficial effects: By performing nonlinear transformation on the Mel frequency energy spectrum, such as logarithmic function or exponential function transformation, the human ear's perception characteristics of changes in sound intensities are simulated, the dynamic range of the music signal is compressed, and the output energy distribution is made more consistent with the actual auditory perception law, thereby improving the accuracy of dynamic range compression.

[0074] In some embodiments, Figure 3 As shown, step S140 further includes:

[0075] Step S141: traverse each transformed spectrum representation, and use the traversed transformed spectrum representation as the target spectrum representation.

[0076] Specifically, since each transformed spectrum representation needs to be subjected to multi-level discrete cosine transform processing, each transformed spectrum representation is traversed, and the traversed transformed spectrum representation is used as the target spectrum representation to avoid missing individual transformed spectrum representations.

[0077] Step S142: Perform discrete cosine transform of the current level on the target spectrum representation, and use the output of the current level as the input of discrete cosine transform of the next level. Repeat this step until the number of current levels reaches a preset number or the output of the current level meets the preset compression rate.

[0078] Specifically, a multi-level discrete cosine transform is performed on the compressed Mel frequency energy spectrum to remove redundant information and extract representative frequency coefficients. Discrete cosine transform is referred to as DCT.

[0079] First-level DCT: Perform a DCT on the compressed energy spectrum to extract preliminary frequency features. Multi-level DCT: Perform multiple DCT processes on the initially extracted frequency features, and each process further extracts high-level features. Finally, a set of frequency coefficients with high compactness and discriminability is obtained.

[0080] DCT is a linear transformation, the purpose of which is to convert discrete time signals into the frequency domain so that the energy of the signal can be concentrated on fewer frequency components, thereby facilitating compression and analysis. The core idea of ​​DCT transformation is to use the orthogonal characteristics of the cosine function to expand the signal and express it as a weighted sum in the frequency domain. The specific definition formula of DCT has been detailed in the above embodiment and will not be repeated here.

[0081] Multi-level DCT: In MDCT (Multi-level Discrete Cosine Transform), DCT is applied multiple times and the output of each transform is used as the input of the next transform. This nested transform structure can provide finer frequency resolution.

[0082] The initial signal is transformed by the first layer of DCT to obtain frequency domain coefficients. The high frequency coefficients are further decomposed and processed by the next layer of DCT until the predetermined number of layers or compression rate is reached.

[0083] Common range of levels: In audio analysis and compression, the number of levels of multi-level DCT is usually between 2 and 5.

[0084] Cut-off conditions: The number of levels of the multi-level DCT can be determined according to the following conditions:

[0085] 1. Signal length: Each level of DCT transform will halve the signal length, so the signal length is a natural cutoff condition.

[0086] 2. Frequency resolution requirement: Determine the number of levels based on the frequency resolution required by the application.

[0087] 3. Computational complexity: More levels mean higher computational complexity and time cost, so computing resources are a limiting factor.

[0088] For signals with rich frequency components and details, multi-level DCT can capture and process signal features more effectively. Multi-level DCT is necessary.

[0089] Step S143: using the output of the current level corresponding to the stop as the frequency coefficient of the target spectrum representation, obtaining the frequency coefficient corresponding to each transformed spectrum representation, and further obtaining a plurality of frequency coefficients.

[0090] Specifically, when the repetition of step S142 stops, the output of the current level corresponding to the stop is used as the frequency coefficient of the target spectrum representation, and the frequency coefficient corresponding to each transformed spectrum representation is obtained, thereby obtaining multiple frequency coefficients.

[0091] Beneficial effects: The frequency coefficients corresponding to each transformed spectrum representation are obtained by traversal, which can avoid omissions. At the same time, the transformed spectrum representation is processed by multi-level discrete cosine transform, which can remove redundant information and extract representative frequency coefficients. And the frequency features are extracted by preliminary discrete cosine transform, and the high-level features are further extracted by multi-level discrete cosine transform, and finally a set of frequency coefficients with high compactness and discrimination is obtained, which significantly improves the feature discrimination ability.

[0092] In some embodiments, after step S142, it also includes: using the output of the current level corresponding to the stop of repetition as the first frequency coefficient of the target spectrum representation, and using the output of the first level as the second frequency coefficient of the target spectrum representation; obtaining the frequency coefficient of the target spectrum representation according to the first frequency coefficient and the second frequency coefficient, and then obtaining the frequency coefficient corresponding to each transformed spectrum representation. Specifically, firstly, the frequency feature is extracted by preliminary discrete cosine transform, that is, the output of the first level, that is, the second frequency coefficient, and the first level is the first time to perform discrete cosine transform processing. When the repetition is stopped, it means that the current level corresponding to the stop of repetition of step S142 is the last time to perform discrete cosine transform processing, and the output of the last discrete cosine transform processing is the first frequency coefficient, and the corresponding extracted high-level features are. Therefore, according to the first frequency coefficient and the second frequency coefficient, the frequency features and high-level features of the target spectrum representation can be extracted at the same time, and finally a set of frequency coefficients with high compactness and discrimination are obtained, which significantly improves the feature discrimination ability.

[0093] In some embodiments, Figure 4 As shown, step S150 further includes:

[0094] Step S151: Based on a weighted average algorithm, each spectrum representation and each frequency coefficient are concatenated and fused to obtain a concatenated audio feature.

[0095] Specifically, the spectral representation is weighted averaged according to the frequency coefficients and concatenated to form a comprehensive feature vector, namely, the concatenated audio feature.

[0096] Step S152: Based on the deep learning algorithm, the spliced ​​audio features and the preset features are fused to obtain the target audio features.

[0097] The preset features include preset chromaticity features and preset spectrum contrast features.

[0098] Specifically, the chromaticity feature (also known as the Chroma feature) is an audio feature that describes the tonal content of music. It is based on the concept of representing notes as colors on a two-dimensional plane, where the vertical axis represents pitch and the horizontal axis represents time. The calculation method of the Chroma feature first divides the audio signal into short time frames, usually 20-30 milliseconds, in order to capture the chromaticity information of the audio, that is, the tone and harmony. This feature maps the audio signal to the musical mode space, representing the distribution of different pitches in each mode, thereby providing information about the degree and intensity of audio activity. The extraction and use of Chroma features are of great significance in audio processing and music information retrieval. It can help analyze the harmonic properties of music and then be applied to tasks such as music classification, similarity measurement, and music information retrieval.

[0099] Spectral contrast feature is a feature used for music style classification. It distinguishes different music styles by comparing their contrast on the spectrum. Spectral contrast feature expresses the style of music by analyzing the spectral characteristics of music, especially the distribution of relative spectrum rather than the average spectral envelope. This method can better reflect the timbre and pitch distribution of music, and thus shows good results in music style classification. Experiments show that the method based on spectral contrast feature can more effectively distinguish different music styles in music style classification than the traditional MFCC feature. The application of this feature improves the management efficiency of digital music library, makes automatic music style classification possible, and further optimizes the organization and retrieval of music content. MFCC: Mel Frequency Cepstrum Coefficient. Mel frequency is proposed based on the auditory characteristics of the human ear, and it has a nonlinear corresponding relationship with Hz frequency. Mel frequency cepstrum coefficient (MFCC) is the Hz spectrum feature calculated by using this relationship between them. The application of spectral contrast feature is not limited to music style classification, it can also be applied to other scenarios that need to distinguish different signals or sound types, such as speech emotion recognition. By analyzing the spectral characteristics of the signal, the spectral contrast feature can provide important information about the frequency distribution and intensity of the signal, thereby helping to identify different types of signals or sounds. In general, the spectral contrast feature is an effective tool that distinguishes different types of signals or sounds by analyzing the spectral contrast of the signal, especially in the fields of music style classification and speech emotion recognition.

[0100] Therefore, on the basis of splicing audio features, the preset chromaticity features and the preset spectral contrast features are integrated, and the deep learning algorithm is used for feature integration to obtain a comprehensive and more comprehensive target audio feature. This method improves the richness and expression ability of the target audio feature, so that the target audio feature can more accurately represent the multi-dimensional characteristics of the music signal.

[0101] Beneficial effects: Based on the weighted average algorithm, each spectrum representation and each frequency coefficient are spliced ​​and fused to obtain a spliced ​​audio feature composed of each spectrum representation and each frequency coefficient, which ensures the integrity of the extracted audio features. Then, based on the deep learning algorithm, the spliced ​​audio features and the preset features including the preset chromaticity features and the preset spectrum contrast features are fused to obtain a comprehensive target audio feature. By adding the preset features, the richness and expression ability of the target audio features are greatly improved, so that the target audio features can more accurately represent the multi-dimensional features of the music signal.

[0102] Figure 5 FIG. 1 is a schematic diagram showing a structure of an embodiment of an audio feature extraction device of the present application. Figure 5 As shown, the device 300 includes: a feature processing module 310 and a feature fusion module 320.

[0103] The feature processing module 310 is used to perform mapping transformation processing on the initial audio features to obtain the audio features to be processed after mapping transformation; wherein the audio features to be processed include audio features of multiple different frequencies;

[0104] The feature processing module 310 is further used to use a plurality of preset Mel filter banks to filter the audio features to be processed, so as to obtain a plurality of frequency spectrum representations corresponding to the audio features to be processed; wherein the frequencies or resolutions of the Mel filter banks are different, and one frequency spectrum representation corresponds to one Mel filter bank;

[0105] The feature processing module 310 is further used to perform nonlinear transformation on each spectrum representation to obtain a plurality of transformed spectrum representations;

[0106] The feature processing module 310 is further used to perform a multi-level discrete cosine transform on each transformed spectrum representation to obtain a plurality of frequency coefficients;

[0107] The feature fusion module 320 is used to fuse each spectrum representation and each frequency coefficient to obtain the target audio feature.

[0108] Beneficial effect: This embodiment maps and transforms the initial audio features to obtain audio features to be processed including different frequencies, and then uses multiple Mel filter groups with different frequencies and resolutions to filter the audio features to be processed, so as to obtain multiple spectrum representations corresponding to the audio features to be processed; each spectrum representation is subjected to nonlinear transformation and multi-level discrete cosine transformation to obtain multiple frequency coefficients, and finally each spectrum representation and each frequency coefficient are fused to obtain the target audio feature. It is realized that the features of different frequencies are filtered by Mel filter groups of different frequencies, and then the different spectrum representations are processed separately and finally fused, so that the features of different frequencies can be accurately captured and analyzed, thereby improving the accuracy of audio feature extraction. At the same time, the features of different frequencies correspond to different audio details. By distinguishing the frequencies, extracting them separately and processing them in parallel, and then merging them, the integrity of the audio feature extraction is guaranteed, avoiding the problem of insufficient extracted features in the prior art, thereby meeting the user experience requirements and further improving the user experience.

[0109] It should be noted that the audio feature extraction device provided in the above embodiment and the audio feature extraction method provided in the above embodiment belong to the same concept, and the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here.

[0110] Figure 6 A structural schematic diagram of an embodiment of a computer device of the present application is shown, which shows a structural schematic diagram of a computer system of a computer device suitable for implementing an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the computer device.

[0111] See also Figure 6 As shown, the computer device includes: a controller; a memory for storing one or more programs, and when the one or more programs are executed by the controller, the above-mentioned audio feature extraction method is executed.

[0112] Please continue reading Figure 6 As shown, the computer system 400 of the computer device includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 to the random access memory (RAM) 403, such as executing the method in the above embodiment. In RAM 403, various programs and data required for system operation are also stored. CPU 401, ROM 402 and RAM 403 are connected to each other through bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0113] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

[0114] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 409, and / or installed from a removable medium 411. When the computer program is executed by a central processing unit (CPU) 401, various functions defined in the system of the present application are executed.

[0115] Another aspect of the present application further provides a computer-readable storage medium, which stores at least one executable instruction. When the executable instruction is executed on a computer device / equipment, the computer device / equipment executes the operation of the audio feature extraction method as described above.

[0116] Beneficial effect: This embodiment maps and transforms the initial audio features to obtain audio features to be processed including different frequencies, and then uses multiple Mel filter groups with different frequencies and resolutions to filter the audio features to be processed, so as to obtain multiple spectrum representations corresponding to the audio features to be processed; each spectrum representation is subjected to nonlinear transformation and multi-level discrete cosine transformation to obtain multiple frequency coefficients, and finally each spectrum representation and each frequency coefficient are fused to obtain the target audio feature. It is realized that the features of different frequencies are filtered by Mel filter groups of different frequencies, and then the different spectrum representations are processed separately and finally fused, so that the features of different frequencies can be accurately captured and analyzed, thereby improving the accuracy of audio feature extraction. At the same time, the features of different frequencies correspond to different audio details. By distinguishing the frequencies, extracting them separately and processing them in parallel, and then merging them, the integrity of the audio feature extraction is guaranteed, avoiding the problem of insufficient extracted features in the prior art, thereby meeting the user experience requirements and further improving the user experience.

[0117] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0118] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0120] According to one aspect of an embodiment of the present application, a computer system is also provided, including a central processing unit (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM), such as executing the method in the above embodiment. In RAM, various programs and data required for system operation are also stored. CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0121] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.

[0122] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A method for extracting audio features, It is characterized in that The method comprises: Performing mapping transformation processing on the initial audio features to obtain audio features to be processed after mapping transformation; wherein the audio features to be processed include audio features of multiple different frequencies; Using a plurality of preset Mel filter banks to filter the audio features to be processed, and obtaining a plurality of frequency spectrum representations corresponding to the audio features to be processed; wherein the frequencies or resolutions of the Mel filter banks are different, and one frequency spectrum representation corresponds to one Mel filter bank; Performing a nonlinear transformation on each of the frequency spectrum representations to obtain a plurality of transformed frequency spectrum representations; Performing a multi-level discrete cosine transform on each of the transformed spectrum representations to obtain a plurality of frequency coefficients; Each of the spectral representations and each of the frequency coefficients are fused to obtain a target audio feature.

2. The method according to claim 1, It is characterized in that Before the step of mapping and transforming the initial audio features to obtain the audio features to be processed, the method further includes: The original music signal is preprocessed to obtain initial audio features.

3. The method according to claim 2, It is characterized in that The step of preprocessing the original music signal to obtain the initial audio feature further includes: Encode and decode the original music signal to obtain music data in a unified format; The music data is downsampled to obtain initial audio features.

4. The method according to claim 1, It is characterized in that The step of performing mapping transformation processing on the initial audio features to obtain the audio features to be processed after mapping transformation further includes: The initial audio feature is mapped and transformed based on a preset transformation to obtain a complex plane image including the frequency and phase of the initial audio feature; wherein the complex plane image is the audio feature to be processed after the mapping transformation, and the points in the complex plane image represent the frequency of the audio feature to be processed.

5. The method according to claim 1, It is characterized in that The step of performing a nonlinear transformation on each of the spectral representations to obtain a plurality of transformed spectral representations further comprises: A nonlinear compression transformation is performed on each of the frequency spectrum representations based on a preset function to obtain a plurality of transformed frequency spectrum representations; wherein the preset function includes a logarithmic function or an exponential function.

6. The method according to claim 1, It is characterized in that The step of performing a multi-level discrete cosine transform on each of the transformed spectrum representations to obtain a plurality of frequency coefficients further comprises: Traversing each of the transformed spectrum representations, and taking the traversed transformed spectrum representation as the target spectrum representation; Performing a discrete cosine transform of the current level on the target spectrum representation, and using the output of the current level as the input of the discrete cosine transform of the next level, repeating this step until the number of the current level reaches a preset number or the output of the current level meets the preset compression rate; The output of the current level corresponding to the stop is used as the frequency coefficient of the target spectrum representation, and the frequency coefficient corresponding to each of the transformed spectrum representations is obtained, thereby obtaining a plurality of frequency coefficients.

7. The method according to any one of claims 1 to 6, It is characterized in that The step of fusing each of the spectral representations and the frequency coefficients to obtain the target audio feature further includes: Based on a weighted average algorithm, each of the spectrum representations and each of the frequency coefficients are spliced ​​and fused to obtain a spliced ​​audio feature; Based on a deep learning algorithm, the spliced ​​audio features and preset features are fused to obtain target audio features; wherein the preset features include preset chromaticity features and preset spectrum contrast features.

8. An audio feature extraction device, It is characterized in that The device comprises: A feature processing module, used for performing mapping transformation processing on the initial audio features to obtain the audio features to be processed after mapping transformation; wherein the audio features to be processed include audio features of multiple different frequencies; The feature processing module is further used to use a plurality of preset Mel filter groups to filter the audio features to be processed, so as to obtain a plurality of frequency spectrum representations corresponding to the audio features to be processed; wherein the frequencies or resolutions of the Mel filter groups are different, and one frequency spectrum representation corresponds to one Mel filter group; The feature processing module is further used to perform nonlinear transformation on each of the frequency spectrum representations to obtain a plurality of transformed frequency spectrum representations; The feature processing module is further used to perform a multi-level discrete cosine transform on each of the transformed spectrum representations to obtain a plurality of frequency coefficients; The feature fusion module is used to fuse each of the spectrum representations and each of the frequency coefficients to obtain a target audio feature.

9. A computer device, It is characterized in that include: Controller; A memory for storing one or more programs, which, when executed by the controller, enables the controller to implement the audio feature extraction method described in any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The storage medium stores at least one executable instruction. When the executable instruction is executed on a computer device / equipment, the computer device / equipment executes the operation of the audio feature extraction method as described in any one of claims 1 to 7.

Citation Information

Cited By

  • Design method of visual piano key rhythm lamp

    CN120224527A