Classified playing method based on audio recognition, sound equipment, equipment and storable medium

Through audio recognition technology based on CNN network and long and short-term memory network, the audio playback mode and equalizer are automatically identified and switched, and the problem of manual operation of audio playback mode and equalizer selection in the prior art is solved, and the audio recognition is improved Accuracy and ease of use.

CN118972752AActive Publication Date: 2024-11-15HUIZHOU ZHONGZHE SHANGLANBO TECH CO LTD

Patent Information

Application Number
CN202411033880.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-15
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

In the prior art, the audio playback mode and equalizer selection require manual switching, and the audio playback effects of different music styles are difficult to match automatically, resulting in inconvenience.

Method used

Using detection models based on CNN network, long and short-term memory network and multimodal compact bilinear pooling, the audio type is automatically identified and the playback mode and equalizer are switched to achieve automated The audio classification and playback effect match.

Benefits of technology

It realizes audio playback mode switching and sound effect matching manually selected by unmanned users, improves the convenience of use and the accuracy of audio recognition, and avoids overfitting and high computational volume.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972752A_ABST
    Figure CN118972752A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of audio detection, and provides a classified playing method based on audio recognition, sound equipment, equipment and a storable medium. The method comprises the steps of obtaining audio information; classifying the audio information based on a detection model composed of a CNN network, a long short-term memory network, multi-modal compact bilinear pooling and a classifier; according to the classification result, a corresponding playing mode is matched for the audio information, the loudspeaker is switched to the playing mode when playing the audio information, the playing mode comprises one of music, movies, human voices, sports, games and festivals, and when the playing mode is music, the music genre is determined according to the classification result, and a corresponding equalizer playing effect is selected; according to the invention, problems encountered in audio playing in the prior art can be solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of audio detection technology, and in particular to a classification playback method, a speaker, a device and a storable medium based on audio recognition. Background Art

[0002] In the prior art, when a mobile phone, computer or other smart device plays audio in different scenarios, different playback modes can be selected to achieve better auditory effects, such as selecting movie mode when watching a movie, selecting game mode when playing a game, and selecting music mode when listening to a song. These playback modes require people to switch manually, which is very inconvenient.

[0003] Furthermore, even after the playback mode is locked to music, since different music has different genres and styles, people can select the corresponding playback sound effects when playing music, such as pop music corresponding to the equalizer and indoor loudness of pop music, rock corresponding to the rock equalizer and live loudness, and then make the speakers play music with the corresponding playback sound effects, so that the human voice and pitch in the music will be more accurate, and the played music will be more pleasant. This also requires manual selection, which is very inconvenient. Summary of the invention

[0004] In view of the above technical problems, the present invention provides a classification playback method, a speaker, a device and a storage medium based on audio recognition to solve the problems encountered in audio playback in the prior art.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present invention, a classification playback method based on audio recognition is disclosed, the method is run in an audio device, a computer or a mobile device, and the method comprises:

[0007] Get audio information;

[0008] A detection model based on a CNN network, a long short-term memory network, a multimodal compact bilinear pooling and a classifier is used to classify the audio information after training, wherein only the Mel spectrum of the audio information is used as input during classification, audio features are extracted through the stacked convolutional pooling layer of the CNN network, and the attention mechanism is combined with the long short-term memory network. When the audio information contains lyrics, important words in the lyrics are extracted to form lyrics features. When the audio information does not contain lyrics, preset filling features are formed. The lyrics features are connected to the audio features, or the filling features are connected to the audio features using the multimodal compact bilinear pooling to obtain combined features, and the combined features are input into the classifier composed of a fully connected layer and a softmax function to obtain the classification result of the audio information.

[0009] According to the classification result, a corresponding playback mode is matched for the audio information, and the speaker switches to the playback mode when playing the audio information. The playback mode includes one of music, movies, vocals, sports, games, and festivals. When the playback mode is music, a corresponding equalizer playback effect is selected according to the music genre determined by the classification result.

[0010] Furthermore, before extracting the audio features, an n-second middle segment of the audio information is selected to generate the Mel spectrum of the middle segment, and the Mel spectrum is divided into a plurality of windows, with 50% overlap between each two adjacent windows.

[0011] Furthermore, when extracting the lyrics features, it specifically includes:

[0012] Segmenting the lyrics to form a word set represented by a hot vector;

[0013] Embed each word in the word set into a vector space;

[0014] Extracting high-level features of each word in the vector space based on a long short-term memory network, and assigning different weights to each word in the lyrics using an attention mechanism according to the influence of each word on the music genre classification;

[0015] The lyrics feature is obtained by using the weighted sum to calculate the focus feature of the high-level features of each weighted word.

[0016] Furthermore, the method of using the multimodal compact bilinear pooling to connect the lyrics feature with the audio feature includes:

[0017] Based on the count sketch projection function, projecting the lyrics features and the audio features into a low-dimensional representation;

[0018] The convolution of the projected lyrics features and the audio features is calculated to obtain the combined features representing the interaction of different modalities.

[0019] Furthermore, when classifying according to the combined features, it specifically includes:

[0020] The combined features are processed by the fully connected layer, and the output of the fully connected layer is processed by the softmax function, and the output of the fully connected layer is converted into a probability distribution to obtain the probability of the audio information in each of the playback modes and the probability of the music genre in the music mode, so as to obtain the classification result.

[0021] According to a second aspect of the present disclosure, a classified playback audio system based on audio recognition is provided, the audio system includes a speaker, and also includes: an acquisition module for acquiring audio information; a classification module for classifying the audio information after training based on a detection model composed of a CNN network, a long short-term memory network, a multimodal compact bilinear pooling and a classifier, wherein during classification, only the Mel spectrum of the audio information is used as input, audio features are extracted through the stacked convolutional pooling layer of the CNN network, an attention mechanism is combined with the long short-term memory network, and when lyrics exist in the audio information, important words in the lyrics are extracted to form lyrics features, and when there are no lyrics in the audio information, preset filler features are formed. , using the multimodal compact bilinear pooling, connecting the lyrics feature with the audio feature, or connecting the filling feature with the audio feature to obtain a combined feature, and inputting the combined feature into the classifier composed of a fully connected layer and a softmax function to obtain a classification result of the audio information; a playback switching module, for matching a corresponding playback mode for the audio information according to the classification result, and enabling the speaker to switch to the playback mode when playing the audio information, wherein the playback mode includes one of music, movies, vocals, sports, games, and festivals, and when the playback mode is music, selecting a corresponding equalizer playback effect according to the music genre determined by the classification result.

[0022] According to a third aspect of the present disclosure, there is provided a classified playback device based on audio recognition, comprising: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor implements the classified playback method based on audio recognition as described above;

[0023] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, storing a computer program, wherein when the computer program is executed by a processor, the classification playback method based on audio recognition as described above is implemented.

[0024] The technical solution disclosed in this disclosure has the following beneficial effects:

[0025] The present invention can be executed on a computer or a mobile phone to automatically identify the type of audio, and then switch the playback mode of the audio in the current playback task through software, so that no one manually selects the playback mode; it can also be directly on the speaker to identify the audio currently playing, and change the playback mode of the speaker according to the recognition result, so as to achieve high compatibility, convenient use, and improved product experience.

[0026] Combining the two modalities of audio and lyrics can improve the recognition results; based on bilinear pooling, the interaction of all elements of audio and lyrics features can be achieved, while avoiding overfitting and high computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of a classification playback method based on audio recognition in an embodiment of this specification;

[0028] Figure 2 This is a structural block diagram of the audio classification playback based on audio recognition in the embodiment of this specification;

[0029] Figure 3 A terminal device that executes the classification playback method based on audio recognition in the embodiments of this specification;

[0030] Figure 4 A computer-readable storage medium storing a classification playback method based on audio recognition in an embodiment of this specification. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0032] In addition, the accompanying drawings are only schematic illustrations of the present disclosure. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0033] like Figure 1 As shown, the embodiment of this specification provides a classification playback method based on audio recognition. The execution subject of the method can be a personal computer, a mobile smart device, a playback device, a server, etc. The playback device can be a speaker, a headset, and the mobile smart device can be a mobile phone, a tablet, an MP4, an MP3, etc. The method can specifically include the following steps S101 to S103:

[0034] In step S101, audio information is acquired.

[0035] The audio information may be a real-time audio stream, such as an audio signal transmitted from a computer to a speaker, or an audio file, such as an audio file read by a computer, a mobile phone, etc. In addition, if the audio stream and the audio file are music, lyrics will be attached, such as MP3 format audio files will have lyrics embedded in the file tag, and the lyrics information of the audio stream played through the music streaming platform will be provided together with the audio stream.

[0036] In step S102, based on the detection model composed of CNN network, long short-term memory network, multimodal compact bilinear pooling and classifier, the audio information is classified after training, wherein only the Mel spectrum of the audio information is used as input during classification, and the audio features are extracted through the stacked convolutional pooling layer of the CNN network, and the attention mechanism is combined with the long short-term memory network. When the audio information has lyrics, the important words in the lyrics are extracted to form lyrics features. When the audio information has no lyrics, the preset filling features are formed. The multimodal compact bilinear pooling is used to connect the lyrics features with the audio features, or the filling features with the audio features to obtain combined features, and the combined features are input into the classifier composed of a fully connected layer and a softmax function to obtain the classification result of the audio information.

[0037] Among them, the detection model is mainly composed of four parts. The CNN network and the long short-term memory network learn the most discriminative features from the audio and lyrics respectively. That is, the LSTM (long short-term memory) neural network based on the attention mechanism is used to highlight the words in the lyrics for genre classification, and the CNN (convolutional neural network) network with several stacked convolutional pooling layers extracts audio features from the input audio. After that, instead of simply connecting the generated audio features and lyrics features, the two types of features are combined through multi-peak compact bilinear pooling (MCB), and then input into the classifier to determine the final classification decision. When there are no lyrics, it cannot be directly ruled out that the audio type is not music. Only the audio features are classified and identified, and the fill-in features are used as markers.

[0038] In step S103, according to the classification result, a corresponding playback mode is matched for the audio information, and the speaker switches to the playback mode when playing the audio information. The playback mode includes one of music, movies, vocals, sports, games, and festivals. When the playback mode is music, the corresponding equalizer playback effect is selected according to the music genre determined by the classification result.

[0039] Among them, the playback mode refers to the specific configuration or setting that the speaker or audio output system will adopt when playing, such as scene setting, etc. When the executor is the audio, the audio switches to the playback mode that best suits the audio characteristics when playing the audio to provide the best auditory experience. At home and abroad, most movies use a specific opening title, that is, a fixed audio, which can be used as a basis for the classification of movies and music. In addition, most of the shots in the movie are mainly dialogues, and the melody of the audio changes little, which is represented by the Mel spectrum, which can also be used as a basis for judging the movie. If the audio is music, the melody of the audio changes greatly. Therefore, after training, the detection module can easily distinguish the audio information, and in the case of lyrics, the music mode can be classified more finely.

[0040] In one embodiment, before extracting the audio features, an n-second middle segment of the audio information is selected to generate the Mel spectrum of the middle segment, and the Mel spectrum is divided into multiple windows, with 50% overlap between every two adjacent windows.

[0041] Among them, the rhythm and melody of the audio are usually closely related to the type, which is mainly determined by the distribution and variance of the signal energy in the time and frequency domains. The Mel spectrum is a visual representation of the change of the signal spectrum over time, which is calculated by the nonlinear transformation of the frequency axis of the short-time Fourier transform (STFT), so it can be used for audio signal recognition. The CNN network only uses the Mel spectrum of audio information as input and extracts audio features through a small number of stacked convolutional pooling layers. The convolution calculation is as follows:

[0042]

[0043] C l is the output feature map of layer l, w l f k is the parameter of the kth filter in the lth layer, is the bias of the lth layer, and φ is the activation function, such as the Relu function or the Tanh function.

[0044] After a convolutional layer, a maximum pooling layer is then performed to parse the spectrogram, such as:

[0045] V l =Maxpooling(C l );

[0046] V l represents the audio feature after the flattening operation.

[0047] In one embodiment, when extracting the lyrics features, it specifically includes: segmenting the lyrics to form a word set represented by a hot vector; embedding each word in the word set into a vector space; based on a long short-term memory network, extracting high-level features of each word in the vector space, and using an attention mechanism to assign different weights to each word in the lyrics according to the influence of each word on the music genre classification; using a weighted sum to calculate the attention feature of the high-level features of each weighted word to obtain the lyrics features.

[0048] Among them, in the lyrics of a song, some words usually reflect its musical genre better than other words. Here, an LSTM (Long Short-Term Memory) neural network based on the attention mechanism is used to highlight important words. Set W i is a set of lyrics for audio information, embedding it into the vector space It can be expressed as: S i =W i *P e , S i ∈R L×E , P e is a parameter matrix, L and E represent the dimension and length of the lyrics word embedding respectively. Then from the word embedding vector Extract higher-level features from Advanced Features F i The calculation of is as follows:

[0049] F i =f s (Z i ,θ s ), F i ∈RL×B ;

[0050] θ s is the parameter of the LSTM network, and B is the size of the LSTM network.

[0051] Then, based on the impact of different words on the classification of music genres, the attention mechanism is used to generate high-level features F for each word. i Assign different weights The weights are calculated as follows;

[0052]

[0053] exp() is the exponential function,

[0054] The attention-based weights reflect the relevance of words to music genres, the parameters K and b are the learned weight matrix and bias term, respectively, and φ is the activation function Tanh.

[0055] The weighted sum is then used to calculate the feature of interest:

[0056]

[0057] Therefore, the lyrics feature of the audio is represented as:

[0058]

[0059] parameter is represented as a weight parameter.

[0060] In one embodiment, the multimodal compact bilinear pooling is used to connect the lyric features with the audio features, including: projecting the lyric features and the audio features into a low-dimensional representation based on a count sketch projection function; calculating the convolution of the projected lyric features and the audio features to obtain the combined features representing the interaction of different modalities.

[0061] Among them, bilinear pooling is the outer product between two vectors. Compared with the element-by-element product or simple concatenation method, bilinear pooling allows all elements of the two vectors to interact, but the high dimensionality and infeasible number of parameters will lead to high computational cost and overfitting problems. In order to reduce the number of parameters and avoid explicit calculation of outer products, multimodal compact bilinear pooling (MCB) is adopted here to project the joint outer product to a lower dimensional space and avoid directly calculating the outer product to combine the extracted audio and lyrics features. Specifically, the features can be projected to a low dimension through a count sketch projection function. For example, the lyrics feature Z of audio i i ∈R B is projected into a representation Instead of directly calculating the outer product of the lyrics feature and the audio feature, the lower dimension (d<B) of the lyric feature is used. In this embodiment, the outer product of the lyrics feature and the audio feature is calculated as the convolution of two technical sketches, and the combined feature is expressed as:

[0062]

[0063] Ψ is the counting sketch projection function.

[0064] In one embodiment, when classification is performed based on the combined features, it specifically includes: processing the combined features through the fully connected layer, processing the output of the fully connected layer through the softmax function, converting the output of the fully connected layer into a probability distribution, obtaining the probability of the audio information in each of the playback modes and the probability of the music genre in the music mode, so as to obtain the classification result.

[0065] Among them, the classification formula can be expressed as:

[0066]

[0067] The output vector o has several values, namely the probability that it belongs to different playing modes and the music genre on the music mode, is the parameter of the fully connected layer, Ψ ZV It is a combined feature.

[0068] Based on the same idea, Figure 2As shown, the exemplary embodiment of the present disclosure also provides a classified playback audio system based on audio recognition, the audio system includes a speaker 204, and also includes: an acquisition module 201, used to obtain audio information; a classification module 202, used to classify the audio information after training based on a detection model composed of a CNN network, a long short-term memory network, a multimodal compact bilinear pooling and a classifier, wherein during classification, only the Mel spectrum of the audio information is used as input, audio features are extracted through the stacked convolutional pooling layer of the CNN network, the attention mechanism is combined with the long short-term memory network, when the audio information has lyrics, the important words in the lyrics are extracted to form lyrics features, and when the audio information has no lyrics, a preset Filling feature, using the multimodal compact bilinear pooling, connecting the lyrics feature with the audio feature, or connecting the filling feature with the audio feature to obtain a combined feature, and inputting the combined feature into the classifier composed of a fully connected layer and a softmax function to obtain a classification result of the audio information; a playback switching module 203, used to match the corresponding playback mode for the audio information according to the classification result, and enable the speaker 204 to switch to the playback mode when playing the audio information, the playback mode includes one of music, movies, vocals, sports, games, and festivals, and when the playback mode is music, select a corresponding equalizer playback effect according to the music genre determined by the classification result.

[0069] In this speaker, the audio currently being played is recognized and the playback mode of the speaker is changed according to the recognition result, achieving high compatibility, ease of use, and improved product experience. Combining the two modalities of audio and lyrics can improve the recognition results; based on bilinear pooling, all elements of audio and lyrics features can be interacted, while avoiding overfitting and high computational complexity.

[0070] The specific details of each module in the above-mentioned audio system have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0071] Based on the same idea, the embodiment of this specification also provides a classification playback device based on audio recognition, such as Figure 3 shown.

[0072] The classified playback device based on audio recognition may be the terminal device or the server provided in the above embodiment.

[0073] The classified playback device based on audio recognition may have relatively large differences due to different configurations or performances, and may include one or more processors 301 and memory 302, and the memory 302 may store one or more storage applications or data. Among them, the memory 502 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) and / or a cache storage unit, and may further include a read-only storage unit. The application stored in the memory 302 may include one or more program modules (not shown in the figure), such program modules include but are not limited to: an operating system, one or more applications, other program modules and program data, each of these examples or some combination may include the implementation of a network environment. Furthermore, the processor 301 can be configured to communicate with the memory 302 to execute a series of computer executable instructions in the memory 302 on the video privacy protection device. The video privacy protection device may also include one or more power supplies 303, one or more wired or wireless network interfaces 304, one or more I / O interfaces (input and output interfaces) 305, one or more external devices 306 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and may also communicate with one or more devices that enable a user to interact with the device, and / or communicate with any device that enables the device to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through the I / O interface 305. Furthermore, the device may also communicate with one or more networks (e.g., a local area network (LAN)) through the wired or wireless interface 304.

[0074] Specifically, in this embodiment, the video privacy protection device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions in the video privacy protection device, and the one or more programs are configured to be executed by one or more processors, including computer executable instructions for performing the following:

[0075] Get audio information;

[0076] A detection model based on a CNN network, a long short-term memory network, a multimodal compact bilinear pooling and a classifier is used to classify the audio information after training, wherein only the Mel spectrum of the audio information is used as input during classification, audio features are extracted through the stacked convolutional pooling layer of the CNN network, and the attention mechanism is combined with the long short-term memory network. When the audio information contains lyrics, important words in the lyrics are extracted to form lyrics features. When the audio information does not contain lyrics, preset filling features are formed. The lyrics features are connected to the audio features, or the filling features are connected to the audio features using the multimodal compact bilinear pooling to obtain combined features, and the combined features are input into the classifier composed of a fully connected layer and a softmax function to obtain the classification result of the audio information.

[0077] According to the classification result, a corresponding playback mode is matched for the audio information, and the speaker switches to the playback mode when playing the audio information. The playback mode includes one of music, movies, vocals, sports, games, and festivals. When the playback mode is music, a corresponding equalizer playback effect is selected according to the music genre determined by the classification result.

[0078] Based on the same idea, the exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0079] refer to Figure 4 As shown, a program product 400 for implementing the above method according to an exemplary embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0080] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0081] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0082] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0083] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0084] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary implementation of the present disclosure.

[0085] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiment of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0086] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0087] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A classification playback method based on audio recognition, It is characterized in that The method is run in an audio device, a computer or a mobile device, and the method comprises: Get audio information; A detection model based on a CNN network, a long short-term memory network, a multimodal compact bilinear pooling and a classifier is used to classify the audio information after training, wherein only the Mel spectrum of the audio information is used as input during classification, audio features are extracted through the stacked convolutional pooling layer of the CNN network, and the attention mechanism is combined with the long short-term memory network. When the audio information has lyrics, important words in the lyrics are extracted to form lyrics features. When the audio information has no lyrics, preset filling features are formed. The lyrics features are connected to the audio features, or the filling features are connected to the audio features using the multimodal compact bilinear pooling to obtain combined features, and the combined features are input into the classifier composed of a fully connected layer and a softmax function to obtain the classification result of the audio information. According to the classification result, a corresponding playback mode is matched for the audio information, and the speaker switches to the playback mode when playing the audio information. The playback mode includes one of music, movies, vocals, sports, games, and festivals. When the playback mode is music, a corresponding equalizer playback effect is selected according to the music genre determined by the classification result.

2. According to claim 1, a classification playback method based on audio recognition, It is characterized in that Before extracting the audio features, an n-second middle segment of the audio information is selected to generate the Mel spectrum of the middle segment, and the Mel spectrum is divided into a plurality of windows, with 50% overlap between each two adjacent windows.

3. The classification playback method based on audio recognition according to claim 1, It is characterized in that When extracting the lyrics features, specifically including: Segmenting the lyrics to form a word set represented by a hot vector; Embed each word in the word set into a vector space; Extracting high-level features of each word in the vector space based on a long short-term memory network, and assigning different weights to each word in the lyrics using an attention mechanism according to the influence of each word on the music genre classification; The lyrics feature is obtained by using the weighted sum to calculate the focus feature of the high-level features of each weighted word.

4. The classification playback method based on audio recognition according to claim 1, It is characterized in that The method of utilizing the multimodal compact bilinear pooling to connect the lyrics feature with the audio feature includes: Based on the count sketch projection function, projecting the lyrics features and the audio features into a low-dimensional representation; The convolution of the projected lyrics features and the audio features is calculated to obtain the combined features representing the interaction of different modalities.

5. The classification playback method based on audio recognition according to claim 1, It is characterized in that When classifying according to the combined features, it specifically includes: The combined features are processed by the fully connected layer, and the output of the fully connected layer is processed by the softmax function, and the output of the fully connected layer is converted into a probability distribution to obtain the probability of the audio information in each of the playback modes and the probability of the music genre in the music mode, so as to obtain the classification result.

6. A classified playback audio system based on audio recognition, It is characterized in that The audio system includes a loudspeaker and further includes: A collection module, used to obtain audio information; A classification module, for classifying the audio information after training based on a detection model composed of a CNN network, a long short-term memory network, a multimodal compact bilinear pooling and a classifier, wherein only the Mel spectrum of the audio information is used as input during classification, audio features are extracted through the stacked convolutional pooling layers of the CNN network, an attention mechanism is combined with the long short-term memory network, and when lyrics are present in the audio information, important words in the lyrics are extracted to form lyrics features, and when the audio information has no lyrics, a preset filling feature is formed, and the lyrics feature is connected to the audio feature or the filling feature is connected to the audio feature using the multimodal compact bilinear pooling to obtain a combined feature, and the combined feature is input into the classifier composed of a fully connected layer and a softmax function to obtain a classification result of the audio information; A playback switching module is used to match a corresponding playback mode for the audio information according to the classification result, and enable the speaker to switch to the playback mode when playing the audio information, wherein the playback mode includes one of music, movies, vocals, sports, games, and festivals, and when the playback mode is music, a corresponding equalizer playback effect is selected according to the music genre determined by the classification result.

7. A classification playback device based on audio recognition, It is characterized in that include: processor; and a memory arranged to store computer executable instructions which, when executed, cause the processor to: Get audio information; A detection model based on a CNN network, a long short-term memory network, a multimodal compact bilinear pooling and a classifier is used to classify the audio information after training, wherein only the Mel spectrum of the audio information is used as input during classification, audio features are extracted through the stacked convolutional pooling layer of the CNN network, and the attention mechanism is combined with the long short-term memory network. When the audio information has lyrics, important words in the lyrics are extracted to form lyrics features. When the audio information has no lyrics, preset filling features are formed. The lyrics features are connected to the audio features, or the filling features are connected to the audio features using the multimodal compact bilinear pooling to obtain combined features, and the combined features are input into the classifier composed of a fully connected layer and a softmax function to obtain the classification result of the audio information. According to the classification result, a corresponding playback mode is matched for the audio information, and the speaker switches to the playback mode when playing the audio information. The playback mode includes one of music, movies, vocals, sports, games, and festivals. When the playback mode is music, a corresponding equalizer playback effect is selected according to the music genre determined by the classification result.

8. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the classification and playback method based on audio recognition as described in any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Method and system for self-adaptive adjustment of audio effect

    CN104811864A

  • Method for image-text cross-mode sentiment classification based on compact bilinear fusion

    CN107066583A

  • A multi-view facial expression recognition method based on mobile terminal

    CN109409222A

  • Method and system for intelligently adjusting audio parameters

    CN110347864A

  • Music emotion classification method based on multi-modal learning

    CN111460213A

Cited By

  • Sound effect adjustment method, electronic equipment and storage medium

    CN119763516A