Audio recognition-based classification playback methods, audio equipment, and storage media

Abstract

This invention belongs to the field of audio detection technology and provides a classification and playback method, speaker, device, and storage medium based on audio recognition. The method includes: acquiring audio information; classifying the audio information using a detection model composed of a CNN network, a long short-term memory network, multimodal compact bilinear pooling, and a classifier; matching the audio information to a corresponding playback mode based on the classification result, and switching the speaker to the playback mode when playing the audio information. The playback mode includes one of music, movie, vocals, sports, games, and festivals; and when the playback mode is music, selecting the corresponding equalizer playback effect based on the music genre determined by the classification result. This invention can solve the problems encountered in audio playback in the prior art.

Description

Technical Field

[0001] This invention relates to the field of audio detection technology, and in particular to a classification playback method, audio system, device, and storage medium based on audio recognition. Background Technology

[0002] In existing technologies, when mobile phones, computers, or other smart devices play audio in different scenarios, different playback modes can be selected to achieve a better listening experience. For example, movie mode can be selected when watching a movie, game mode when playing a game, and music mode when listening to music. These playback modes require manual switching, which is quite inconvenient.

[0003] Furthermore, even after the playback mode is locked to music, different genres and styles of music allow users to select corresponding playback effects. For example, pop music can be set to a pop equalizer and room loudness setting, while rock music can be set to a rock equalizer and live loudness setting. This allows the speakers to play the music with the corresponding effects, resulting in more accurate vocals and pitch, and a more pleasing sound. However, this still requires manual selection, which is quite inconvenient. Summary of the Invention

[0004] In view of the above technical problems, the present invention provides a classification playback method, audio speaker, device and storage medium based on audio recognition to solve the problems encountered in audio playback in the prior art.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0006] According to one aspect of the present invention, a classification playback method based on audio recognition is disclosed, the method operating in an audio device, a computer, or a mobile device, the method comprising:

[0007] Obtain audio information;

[0008] A detection model based on a CNN network, a long short-term memory network, multimodal compact bilinear pooling, and a classifier is used to classify the audio information after training. During classification, only the Mel spectrum of the audio information is used as input. Audio features are extracted through stacked convolutional pooling layers of the CNN network. An attention mechanism is combined with the long short-term memory network. When lyrics are present in the audio information, important words in the lyrics are extracted to form lyrics features. When no lyrics are present, preset filler features are formed. The multimodal compact bilinear pooling is used to connect the lyrics features with the audio features, or the filler features with the audio features, to obtain combined features. These combined features are then input into the classifier, which consists of fully connected layers and a softmax function, to obtain the classification result of the audio information.

[0009] Based on the classification results, a corresponding playback mode is matched for the audio information, and the speaker switches to the playback mode when playing the audio information. The playback mode includes one of music, movie, vocals, sports, games, and festivals. When the playback mode is music, the corresponding equalizer playback effect is selected according to the music genre determined by the classification results.

[0010] Furthermore, before extracting the audio features, an n-second intermediate segment of the audio information is selected to generate the Mel spectrum of the intermediate segment, and the Mel spectrum is divided into multiple windows with 50% overlap between each pair of adjacent windows.

[0011] Furthermore, the extraction of the lyrics features specifically includes:

[0012] The lyrics are segmented into words to form a set of words represented by heat vectors;

[0013] Each word in the word set is embedded into a vector space;

[0014] Based on the Long Short-Term Memory network, high-level features of each word in the vector space are extracted, and attention mechanism is used to assign different weights to each word according to the influence of each word in the lyrics on the music genre classification.

[0015] The lyrics features are obtained by using weighted summation to calculate the high-level features of each weighted word.

[0016] Furthermore, the method of concatenating the lyrics features with the audio features using the multimodal compact bilinear pooling includes:

[0017] Based on the counting sketch projection function, the lyrics features and the audio features are projected into a low-dimensional representation respectively;

[0018] The convolution of the projected lyrics features and the audio features is calculated to obtain the combined features representing different modal interactions.

[0019] Furthermore, the classification based on the combined features specifically includes:

[0020] The combined features are processed by the fully connected layer, and the output of the fully connected layer is processed by the softmax function to convert the output of the fully connected layer into a probability distribution, so as to obtain the probability of the audio information in each playback mode and the probability of the music genre in the music mode, so as to obtain the classification result.

[0021] According to a second aspect of this disclosure, a classification-based audio playback system is provided. The audio system includes a speaker and further includes: a data acquisition module for acquiring audio information; and a classification module for classifying the audio information after training using a detection model composed of a CNN network, a long short-term memory network, multimodal compact bilinear pooling, and a classifier. During classification, only the Mel-spectrum of the audio information is used as input. Audio features are extracted through stacked convolutional pooling layers of the CNN network. An attention mechanism is combined with the long short-term memory network. When lyrics are present in the audio information, important words in the lyrics are extracted to form lyric features. When there are no lyrics in the audio information, preset fill features are formed. The multimodal compact bilinear pooling is used to connect the lyrics features with the audio features, or the padding features with the audio features, to obtain combined features. These combined features are then input into the classifier, which consists of a fully connected layer and a softmax function, to obtain the classification result of the audio information. The playback switching module is used to match the corresponding playback mode for the audio information according to the classification result, and to switch the speaker to the playback mode when playing the audio information. The playback mode includes one of music, movie, vocal, sports, game, and festival. When the playback mode is music, the module selects the corresponding equalizer playback effect according to the music genre determined by the classification result.

[0022] According to a third aspect of this disclosure, an audio recognition-based classification playback device is provided, comprising: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to implement the audio recognition-based classification playback method as described above.

[0023] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the audio recognition-based classification playback method as described above.

[0024] The technical solution disclosed herein has the following beneficial effects:

[0025] This invention can be executed on a computer or mobile phone, automatically identifying the type of audio and then switching the playback mode of the audio in the current playback task via software, thus eliminating the need for manual selection of the playback mode; alternatively, it can be executed directly on a speaker, identifying the audio currently being played and changing the speaker's playback mode based on the identification result, achieving high compatibility, ease of use, and improved product experience.

[0026] Combining audio and lyrics modalities can improve recognition results; bilinear pooling enables interaction of all elements of audio and lyrics features, while avoiding overfitting and high computational cost. Attached Figure Description

[0027] Figure 1 This is a flowchart of a classification playback method based on audio recognition, as described in the embodiments of this specification.

[0028] Figure 2 This is a structural block diagram of the audio recognition-based classification and playback system in the embodiments of this specification;

[0029] Figure 3 The terminal device that executes the audio recognition-based classification playback method in the embodiments of this specification;

[0030] Figure 4 This refers to a computer-readable storage medium storing a classification playback method based on audio recognition, as described in the embodiments of this specification. Detailed Implementation

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure may be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., may be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0032] Furthermore, the accompanying drawings are merely illustrative of this disclosure. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0033] like Figure 1 As shown in the embodiments of this specification, a classification playback method based on audio recognition is provided. The execution subject of this method can be a personal computer, a mobile smart device, a playback device, a server, etc. The playback device can be a speaker or headphones, and the mobile smart device can be a mobile phone, tablet, MP4 player, MP3 player, etc. The method specifically includes the following steps S101 to S103:

[0034] In step S101, audio information is acquired.

[0035] The audio information can be a real-time audio stream, such as an audio signal transmitted from a computer to a speaker, or an audio file, such as an audio file read by a computer or mobile phone. Additionally, if the audio stream and audio file are music, lyrics will be included. For example, MP3 audio files embed lyrics in their file tags, and audio streams played through music streaming platforms will provide lyrics along with the audio stream.

[0036] In step S102, a detection model based on a CNN network, a long short-term memory network, multimodal compact bilinear pooling, and a classifier is used to classify the audio information after training. During classification, only the Mel spectrum of the audio information is used as input. Audio features are extracted through stacked convolutional pooling layers of the CNN network. An attention mechanism and the long short-term memory network are combined. When lyrics are present in the audio information, important words in the lyrics are extracted to form lyrics features. When there are no lyrics, preset filler features are formed. The multimodal compact bilinear pooling is used to connect the lyrics features with the audio features, or the filler features with the audio features, to obtain combined features. These combined features are then input into the classifier, which consists of fully connected layers and a softmax function, to obtain the classification result of the audio information.

[0037] The detection model mainly consists of four parts: a CNN network and a Long Short-Term Memory (LSTM) network learn the most discriminative features from the audio and lyrics, respectively; an attention-based LSTM neural network highlights words in the lyrics used for genre classification; and a CNN network with several stacked convolutional pooling layers extracts audio features from the input audio. Instead of simply concatenating the generated audio and lyric features, the two types of features are combined using multi-peak compact bilinear pooling (MCB) before being fed into a classifier to determine the final classification decision. Even when there are no lyrics, the audio cannot be directly ruled out as not being music; in this case, only the audio features are classified, and the filler features are used as labels.

[0038] In step S103, according to the classification result, a corresponding playback mode is matched for the audio information, and the speaker switches to the playback mode when playing the audio information. The playback mode includes one of music, movie, vocals, sports, games, and festivals. When the playback mode is music, the corresponding equalizer playback effect is selected according to the music genre determined by the classification result.

[0039] In this context, playback mode refers to the specific configuration or settings that the speaker or audio output system will use during playback, such as scene settings. When the executing entity is a sound system, it switches to the playback mode most suitable for the characteristics of the audio to provide the best listening experience. Domestically and internationally, most films use a specific opening sequence, i.e., fixed audio. This can serve as a basis for classifying films and music. Furthermore, most film shots are dialogue-based, with minimal melody variation in the audio, represented by Mel spectrum, which can also be used as a criterion for film classification. If the audio is music, the melody varies greatly. Therefore, after training, the detection module can easily distinguish audio information and, in the case of lyrics, can perform more refined classification of music patterns.

[0040] In one embodiment, before extracting the audio features, an n-second intermediate segment of the audio information is selected, and the Mel spectrum of the intermediate segment is generated. The Mel spectrum is divided into multiple windows, with 50% overlap between each pair of adjacent windows.

[0041] The rhythm and melody of audio are usually closely related to genre, primarily determined by the distribution and variance of signal energy in the time and frequency domains. The Mel spectrum, a visual representation of the signal spectrum over time, is calculated by a nonlinear transformation of the frequency axis of the Short-Time Fourier Transform (STFT), and therefore can be used for audio signal recognition. CNN networks use only the Mel spectrum of the audio information as input and extract audio features through a small number of stacked convolutional pooling layers. The convolution calculation is as follows:

[0042]

[0043] C l It is the output feature map of the l-th layer, w l f k These are the parameters of the k-th filter in the l-th layer. φ is the bias of the l-th layer, and φ is the activation function, such as the ReLU function or the Tanh function.

[0044] After a convolutional layer, a max-pooling layer is then applied to parse the spectrogram, as follows:

[0045] V l =Maxpooling(C l );

[0046] V l This represents the audio features after the flattening operation.

[0047] In one embodiment, extracting the lyric features specifically includes: segmenting the lyrics into words to form a set of words represented by hot vectors; embedding each word in the word set into a vector space; extracting the high-level features of each word in the vector space based on a long short-term memory network, and assigning different weights to each word using an attention mechanism according to the influence of each word in the lyrics on music genre classification; and calculating the attention features of the high-level features of each weighted word using a weighted sum to obtain the lyric features.

[0048] In a song's lyrics, some words often better reflect its musical genre than others. Here, an attention-based LSTM (Long Short-Term Memory) neural network is used to highlight important words. Let W... i Given a set of lyrics for an audio message, embed them into a vector space. In, it can be represented as: S i =W i *P e S i ∈R L×E P e Let L be the parameter matrix, and E represent the dimension and length of the word embeddings in the lyrics, respectively. Then, from the word embedding vectors... Extracting higher-level features Advanced Feature F i The calculation is as follows:

[0049] F i =f s (Z i θ s ), F i ∈RL×B ;

[0050] θ s B is the parameter of the Long Short-Term Memory (LSTM) network, and B is the size of the LSM network.

[0051] Then, based on the influence of different words on the classification of music genres, an attention mechanism is used to analyze the high-level features F of each word. i Assign different weights The weights are calculated as follows;

[0052]

[0053] exp() is the exponential function.

[0054] Attention-based weights reflect the correlation between words and music genres. Parameters K and b are the learned weight matrix and bias term, respectively, and φ is the activation function Tanh.

[0055] Then, a weighted sum is used to calculate the features of interest:

[0056]

[0057] Therefore, the lyric features of the audio are represented as follows:

[0058]

[0059] parameter It is represented as a weight parameter.

[0060] In one embodiment, the step of concatenating the lyrics features and the audio features using the multimodal compact bilinear pooling includes: projecting the lyrics features and the audio features into a low-dimensional representation based on a counting sketch projection function; and calculating the convolution of the projected lyrics features and the audio features to obtain the combined features representing different modal interactions.

[0061] Bilinear pooling, which is the outer product between two vectors, allows all elements of the two vectors to interact compared to element-wise multiplication or simple concatenation. However, its high dimensionality and infeasible number of parameters lead to high computational cost and overfitting. To reduce the number of parameters and avoid explicitly calculating the outer product, multimodal compact bilinear pooling (MCB) is employed. This projects the joint outer product to a lower-dimensional space and avoids direct computation of the outer product, combining the extracted audio and lyrics features. Specifically, the features can be projected to a lower dimension using a counting sketch projection function. For example, the lyrics feature Z of audio i... i ∈R B Projected onto a representation Instead of directly calculating the outer product of the lyrics features and audio features, this embodiment calculates the outer product of the lyrics features and audio features as a convolution of two technical sketches, and the combined features are represented as:

[0062]

[0063] Ψ is the counting sketch projection function.

[0064] In one embodiment, the classification based on the combined features specifically includes: processing the combined features through the fully connected layer, processing the output of the fully connected layer through the softmax function, converting the output of the fully connected layer into a probability distribution, obtaining the probability of the audio information in each playback mode and the probability of the music genre in the music mode, so as to obtain the classification result.

[0065] The classification formula can be expressed as:

[0066]

[0067] The output vector o has several values, namely the probability that it belongs to different playback modes and the music genre in the music mode. These are the parameters of the fully connected layer, Ψ ZV It is a combination feature.

[0068] Based on the same line of thought, such as Figure 2As shown, an exemplary embodiment of this disclosure also provides a classification playback speaker based on audio recognition. The speaker includes a speaker 204 and further includes: a data acquisition module 201 for acquiring audio information; and a classification module 202 for classifying the audio information after training using a detection model composed of a CNN network, a long short-term memory network, multimodal compact bilinear pooling, and a classifier. During classification, only the Mel-spectrum of the audio information is used as input. Audio features are extracted through stacked convolutional pooling layers of the CNN network. An attention mechanism and the long short-term memory network are combined. When lyrics are present in the audio information, important words in the lyrics are extracted to form lyric features. When there are no lyrics in the audio information, a preset lyric feature is formed. The fill feature is obtained by concatenating the lyrics feature with the audio feature, or by concatenating the fill feature with the audio feature, using the multimodal compact bilinear pooling. The combined feature is then input into the classifier consisting of a fully connected layer and a softmax function to obtain the classification result of the audio information. The playback switching module 203 is used to match the corresponding playback mode for the audio information according to the classification result, and to switch the speaker 204 to the playback mode when playing the audio information. The playback mode includes one of music, movie, vocal, sports, game, and festival. When the playback mode is music, the corresponding equalizer playback effect is selected according to the music genre determined by the classification result.

[0069] This speaker system identifies the currently playing audio and adjusts its playback mode accordingly, achieving high compatibility, ease of use, and an improved user experience. Combining audio and lyrics modalities enhances recognition results; bilinear pooling enables interaction among all elements of audio and lyrics features, while avoiding overfitting and high computational cost.

[0070] The specific details of each module in the aforementioned audio system have been described in detail in the implementation method section. Any undisclosed details can be found in the implementation method section and will not be repeated here.

[0071] Following the same line of thought, this specification also provides an audio recognition-based classification playback device, such as... Figure 3 As shown.

[0072] The audio recognition-based classification playback device can be the terminal device or server provided in the above embodiments.

[0073] Audio recognition-based classification playback devices can vary significantly in configuration and performance, and may include one or more processors 301 and memory 302. Memory 302 may store one or more application programs or data. Memory 302 may include readable media in the form of volatile memory cells, such as random access memory (RAM) and / or cache memory, and may further include read-only memory cells. The application programs stored in memory 302 may include one or more program modules (not shown in the figures), including but not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Furthermore, processor 301 may be configured to communicate with memory 302 to execute a series of computer-executable instructions stored in memory 302 on the video privacy protection device. The video privacy protection device may also include one or more power supplies 303, one or more wired or wireless network interfaces 304, one or more I / O interfaces (input / output interfaces) 305, one or more external devices 306 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and may also communicate with one or more devices that enable users to interact with the device, and / or with any device that enables the device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed through I / O interfaces 305. Furthermore, the device can also communicate with one or more networks (e.g., local area networks (LANs)) through wired or wireless interfaces 304.

[0074] Specifically, in this embodiment, the video privacy protection device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the video privacy protection device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0075] Obtain audio information;

[0076] A detection model based on a CNN network, a long short-term memory network, multimodal compact bilinear pooling, and a classifier is used to classify the audio information after training. During classification, only the Mel spectrum of the audio information is used as input. Audio features are extracted through stacked convolutional pooling layers of the CNN network. An attention mechanism is combined with the long short-term memory network. When lyrics are present in the audio information, important words in the lyrics are extracted to form lyrics features. When no lyrics are present, preset filler features are formed. The multimodal compact bilinear pooling is used to connect the lyrics features with the audio features, or the filler features with the audio features, to obtain combined features. These combined features are then input into the classifier, which consists of fully connected layers and a softmax function, to obtain the classification result of the audio information.

[0077] Based on the classification results, a corresponding playback mode is matched for the audio information, and the speaker switches to the playback mode when playing the audio information. The playback mode includes one of music, movie, vocals, sports, games, and festivals. When the playback mode is music, the corresponding equalizer playback effect is selected according to the music genre determined by the classification results.

[0078] Based on the same idea, exemplary embodiments of this disclosure also provide a computer-readable storage medium storing a program product capable of implementing the methods described above. In some possible embodiments, various aspects of this disclosure can also be implemented as a program product including program code, which, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to the various exemplary embodiments of this disclosure.

[0079] refer to Figure 4 As shown, a program product 400 for implementing the above-described method according to an exemplary embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0080] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0081] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0082] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0083] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0084] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to the exemplary embodiments of this disclosure.

[0085] Furthermore, the above figures are merely illustrative representations of the processes included in the methods according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0086] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0087] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A classification playback method based on audio recognition, characterized in that, The method operates on an audio device, a computer, or a mobile device, and the method includes: Obtain audio information; A detection model based on a CNN network, a long short-term memory network, multimodal compact bilinear pooling, and a classifier is used to classify the audio information after training. During classification, only the Mel spectrum of the audio information is used as input. Audio features are extracted through stacked convolutional pooling layers of the CNN network. An attention mechanism is combined with the long short-term memory network. When lyrics are present in the audio information, important words in the lyrics are extracted to form lyrics features. When no lyrics are present, preset filler features are formed. The multimodal compact bilinear pooling is used to connect the lyrics features with the audio features, or the filler features with the audio features, to obtain combined features. These combined features are then input into the classifier, which consists of fully connected layers and a softmax function, to obtain the classification result of the audio information. The extraction of the lyrics features specifically includes: segmenting the lyrics into words to form a set of words represented by hot vectors; and embedding each word in the set of words into a vector space. In, among them, , , For the set of words, For parameter matrices, and These represent the dimension and length of the word embeddings in the lyrics, respectively. Based on a Long Short-Term Memory (LSTM) network, high-level features are extracted for each word in the vector space. These high-level features are... , High-level features, during computation , These are parameters of the Long Short-Term Memory (LSTM) network. The size of the Long Short-Term Memory network is used to assign different weights to each word based on its influence on music genre classification using an attention mechanism. When calculating weights, , It is an exponential function. Parameters K and b are the learned weight matrix and bias term, respectively. The activation function is Tanh; a weighted sum is used to calculate the high-level features of attention for each weighted word, and the calculation formula is as follows: The lyrics features are obtained and represented as follows: ,parameter Represented as weight parameters, For the first One audio file; Based on the classification results, a corresponding playback mode is matched for the audio information, and the speaker switches to the playback mode when playing the audio information. The playback mode includes one of music, movie, vocals, sports, games, and festivals. When the playback mode is music, the corresponding equalizer playback effect is selected according to the music genre determined by the classification results.

2. The classification and playback method based on audio recognition according to claim 1, characterized in that, Before extracting the audio features, an n-second intermediate segment of the audio information is selected, and the Mel spectrum of the intermediate segment is generated. The Mel spectrum is divided into multiple windows, with 50% overlap between each pair of adjacent windows.

3. The classification and playback method based on audio recognition according to claim 1, characterized in that, The method of concatenating the lyrics features with the audio features using the multimodal compact bilinear pooling includes: Based on the counting sketch projection function, the lyrics features and the audio features are projected into a low-dimensional representation respectively; The convolution of the projected lyrics features and the audio features is calculated to obtain the combined features representing different modal interactions.

4. The classification and playback method based on audio recognition according to claim 1, characterized in that, When classifying based on the combined features, the specific steps include: The combined features are processed by the fully connected layer, and the output of the fully connected layer is processed by the softmax function to convert the output of the fully connected layer into a probability distribution, so as to obtain the probability of the audio information in each playback mode and the probability of the music genre in the music mode, so as to obtain the classification result.

5. A classification-based audio playback system based on audio recognition, characterized in that, The audio system includes a speaker and also includes: The acquisition module is used to acquire audio information; The classification module is used to classify the audio information after training a detection model based on a CNN network, a long short-term memory network, multimodal compact bilinear pooling, and a classifier. During classification, only the Mel spectrum of the audio information is used as input. Audio features are extracted through stacked convolutional pooling layers of the CNN network. An attention mechanism is combined with the long short-term memory network. When lyrics are present in the audio information, important words in the lyrics are extracted to form lyrics features. When no lyrics are present, preset filler features are formed. The multimodal compact bilinear pooling is used to concatenate the lyrics features with the audio features, or the filler features with the audio features, to obtain combined features. These combined features are then input into the classifier, which consists of fully connected layers and a softmax function, to obtain the classification result of the audio information. Specifically, extracting the lyrics features includes: segmenting the lyrics into words to form a set of words represented by hot vectors; and embedding each word in the word set into a vector space. In, among them, , , For the set of words, For parameter matrices, and These represent the dimension and length of the word embeddings in the lyrics, respectively. Based on a Long Short-Term Memory (LSTM) network, high-level features are extracted for each word in the vector space. These high-level features are... , High-level features, during computation , These are parameters of the Long Short-Term Memory (LSTM) network. The size of the Long Short-Term Memory network is used to assign different weights to each word based on its influence on music genre classification using an attention mechanism. When calculating weights, , It is an exponential function. Parameters K and b are the learned weight matrix and bias term, respectively. The activation function is Tanh; a weighted sum is used to calculate the high-level features of attention for each weighted word, and the calculation formula is as follows: The lyrics features are obtained and represented as follows: ,parameter Represented as weight parameters, For the first One audio file; The playback switching module is used to match the corresponding playback mode for the audio information according to the classification result, and to switch the speaker to the playback mode when playing the audio information. The playback mode includes one of music, movie, vocal, sports, game, and festival. When the playback mode is music, the module selects the corresponding equalizer playback effect according to the music genre determined by the classification result.

6. A classification playback device based on audio recognition, characterized in that, include: processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to: Obtain audio information; A detection model based on a CNN network, a long short-term memory network, multimodal compact bilinear pooling, and a classifier is used to classify the audio information after training. During classification, only the Mel spectrum of the audio information is used as input. Audio features are extracted through stacked convolutional pooling layers of the CNN network. An attention mechanism is combined with the long short-term memory network. When lyrics are present in the audio information, important words in the lyrics are extracted to form lyrics features. When no lyrics are present, preset filler features are formed. The multimodal compact bilinear pooling is used to connect the lyrics features with the audio features, or the filler features with the audio features, to obtain combined features. These combined features are then input into the classifier, which consists of fully connected layers and a softmax function, to obtain the classification result of the audio information. The extraction of the lyrics features specifically includes: segmenting the lyrics into words to form a set of words represented by hot vectors; and embedding each word in the set of words into a vector space. In, among them, , , For the set of words, For parameter matrices, and These represent the dimension and length of the word embeddings in the lyrics, respectively. Based on a Long Short-Term Memory (LSTM) network, high-level features are extracted for each word in the vector space. These high-level features are... , High-level features, during computation , These are parameters of the Long Short-Term Memory (LSTM) network. The size of the Long Short-Term Memory network is used to assign different weights to each word based on its influence on music genre classification using an attention mechanism. When calculating weights, , It is an exponential function. Parameters K and b are the learned weight matrix and bias term, respectively. The activation function is Tanh; a weighted sum is used to calculate the high-level features of attention for each weighted word, and the calculation formula is as follows: The lyrics features are obtained and represented as follows: ,parameter Represented as weight parameters, For the first One audio file; Based on the classification results, a corresponding playback mode is matched for the audio information, and the speaker switches to the playback mode when playing the audio information. The playback mode includes one of music, movie, vocals, sports, games, and festivals. When the playback mode is music, the corresponding equalizer playback effect is selected according to the music genre determined by the classification results.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the audio recognition-based classification playback method as described in any one of claims 1 to 4.

Images