Music playing method, device, equipment and medium based on emotion recognition

Abstract

The embodiments of the present application provide a music playing method, device, equipment, and medium based on emotion recognition, which belongs to the field of artificial intelligence technology. The method determines the emotion recognition result based on the audio to be recognized, and then obtains the music to be played based on the emotion recognition result and the music generation model, and finally plays the music to be played. Among them, if the music generation model is a strong classifier, it is necessary to match the corresponding weak classifier according to the emotion recognition result, and generate music according to the weak classifier and the emotion recognition result, that is, the music to be played is obtained. The embodiments of the present application focus on the matching degree between the emotion recognition result and the music to be played, so the matching degree can be greatly increased by realizing music generation through a weak classifier. The generation accuracy of the music to be played can be further improved. It can be understood that due to the high matching degree between the music to be played and the emotion recognition result, and by playing the music to be played, the effect of relieving the emotions of the target object can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a music playing method and apparatus, device, and medium based on emotion recognition. Background Art

[0002] At present, there are many reasons for traffic accidents, generally speaking, they are human factors, vehicle factors, road and environmental factors, etc., among which human factors rank first. Among human factors, the driver's factor is the most important. The driver's bad mood is an important cause of traffic accidents. When stimulated, people generally express it through their faces, actions, eyes, etc. In the process of driving a vehicle, when the target object (driver) is in a good mood and satisfied, he can feel happy and comfortable, which has a positive role in promoting the observation and judgment of things. He often shows strong sensitivity, diligent observation, quick reaction, accurate judgment, and agile action, which is conducive to the safety of vehicle driving. On the contrary, if the target object has emotional fluctuations such as sadness, fear, distress, anger, etc., then the sensitivity decreases, the energy is scattered, and he is lazy to observe and think. Once encountering emergencies and emergencies, he will react slowly, make mistakes in judgment, and other bad behaviors. Therefore, how to provide a music playing method based on emotion recognition, which can determine music according to the emotion recognition results and relieve the emotions of the target object by playing music, has become a technical problem to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a music playing method and apparatus, device, and medium based on emotion recognition, which can obtain music according to the emotion recognition results and relieve the emotions of the target object by playing music.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a music playing method based on emotion recognition, the method comprising:

[0005] Acquire the audio to be recognized through a preset sound collection device;

[0006] Inputting the audio to be recognized into a preset emotion recognition model to obtain an emotion recognition result;

[0007] Inputting the emotion recognition result into a preset music generation model to obtain the music to be played through the music generation model;

[0008] Play the music to be played through a preset player;

[0009] Wherein, if the music generation model is a strong classifier, obtaining the music to be played through the music generation model includes:

[0010] Matching a corresponding weak classifier according to the emotion recognition result; the strong classifier includes a plurality of weak classifiers;

[0011] The music to be played for which the emotion recognition result is obtained by the weak classifier.

[0012] In some embodiments, the strong classifier is pre-trained by the following steps, including:

[0013] Get the audio data training set for sentiment classification;

[0014] Classifying the data of the audio data training set into at least two emotion categories by classification annotation;

[0015] At least two of the emotion categories are trained for sentiment classification through a preset algorithm to obtain the strong classifier; wherein a weak classifier is generated corresponding to each emotion category, and all the weak classifiers constitute a strong classifier.

[0016] In some embodiments, the music to be played for which the emotion recognition result is obtained by the weak classifier includes:

[0017] The emotion recognition result is judged by the weak classifier to obtain a music classification result;

[0018] Determining initial music according to the music classification result, and extracting music features of the initial music;

[0019] The music features are adjusted according to the music classification result to obtain the music to be played.

[0020] In some embodiments, the music to be played for which the emotion recognition result is obtained by the weak classifier includes:

[0021] The emotion recognition results are judged by the weak classifier to obtain the music classification results;

[0022] Determine the initial music feature according to the music classification result; wherein the initial music feature includes at least one of the following: a spectrum parameter feature, a rhythm sequence feature, and a pitch trajectory feature;

[0023] Predicting the initial music features through a preset prediction neural network to obtain prediction parameters;

[0024] The initial music features are adjusted according to the prediction parameters to obtain the target music features, and the music to be played is obtained according to the target music features.

[0025] In some embodiments, before classifying the data of the audio data training set into at least two emotion categories by classification labeling, the method further includes preprocessing the audio data training set, including:

[0026] Filtering the audio data training set through a preset filter to obtain a preliminary audio data training set; wherein the preliminary audio data training set includes a plurality of preliminary audio data;

[0027] converting the preliminary audio data into a spectrogram;

[0028] Performing frequency filtering on the frequency spectrum to obtain target audio data having a frequency less than a preset frequency threshold;

[0029] A preprocessed audio data training set is obtained according to the target audio data.

[0030] In some embodiments, before classifying the data of the audio data training set into at least two emotion categories by classification annotation, the method further includes expanding the audio data training set, including:

[0031] Performing cross-validation on the audio data of the audio data training set a preset number of times to obtain at least two parts of audio data;

[0032] Add Gaussian white noise to each part of the audio data to obtain enhanced audio data;

[0033] An expanded audio data training set is obtained based on at least two parts of the enhanced audio data.

[0034] In some embodiments, after the emotion recognition result is input into a preset music generation model, if the music generation model is a music automatic generation network, the method further includes pre-training the music automatic generation network, including:

[0035] Acquire a musical instrument audio data set, where the musical instrument audio data set includes at least two musical instrument audio data;

[0036] Performing cross-validation on the instrument audio data of the instrument audio data set for a preset number of times to obtain preliminary instrument audio data of at least two parts;

[0037] A test set and a training set are obtained according to at least two parts of the preliminary musical instrument audio data; wherein the test set includes one part of the preliminary musical instrument audio data, and the training set includes the preliminary musical instrument audio data of the other parts except the test set;

[0038] Adding Gaussian white noise to the preliminary instrument audio data of the training set to obtain enhanced instrument audio data;

[0039] Inputting the enhanced instrument audio data of the training set into a preset neural network model, and training the neural network model to obtain a preliminary music automatic generation network;

[0040] The preliminary instrument audio data of the test set is input into a preliminary music automatic generation network, and the preliminary music automatic generation network is tested to obtain a trained music automatic generation network.

[0041] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a music playing device based on emotion recognition, the device comprising:

[0042] An audio acquisition module is used to acquire the audio to be recognized through a preset sound acquisition device;

[0043] An emotion recognition module, used for inputting the audio to be recognized into a preset emotion recognition model to obtain an emotion recognition result;

[0044] A music determination module, used for inputting the emotion recognition result into a preset music generation model to obtain the music to be played through the music generation model; wherein, if the music generation model is a strong classifier, then obtaining the music to be played through the music generation model includes: matching a corresponding weak classifier according to the emotion recognition result; the strong classifier includes a plurality of weak classifiers; obtaining the music to be played of the emotion recognition result through the weak classifier;

[0045] The music playing module is used to play the music to be played through a preset player.

[0046] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program, when executed by the processor, implements the method described in the first aspect above.

[0047] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect above.

[0048] The music playing method based on emotion recognition, the music playing device based on emotion recognition, the computer equipment and the storage medium provided by the present application determine the emotion recognition result according to the audio, obtain the music to be played according to the emotion recognition result and the music generation model, and finally play the music to be played. In addition, the embodiment of the present application needs to match the corresponding weak classifier according to the emotion recognition result, generate music according to the weak classifier and the emotion recognition result, and obtain the music to be played. It should be noted that the embodiment of the present application focuses on the matching degree between the emotion recognition result and the music to be played, so the matching degree can be greatly increased by realizing the music generation through the weak classifier. The generation accuracy of the music to be played can be further improved. It can also be understood that due to the high matching degree between the music to be played and the emotion recognition result, and by playing the music to be played, the effect of relieving the emotions of the target object can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flowchart of a music playing method based on emotion recognition provided by an embodiment of the present application;

[0050] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0051] Figure 3 yes Figure 2 Flow chart of step S202 in FIG.

[0052] Figure 4 yes Figure 2 Flow chart of step S202 in FIG.

[0053] Figure 5 yes Figure 1 Flow chart of step S103 in FIG.

[0054] Figure 6 is a block diagram of the module structure of the music playing device based on emotion recognition provided by an embodiment of the present application;

[0055] Figure 7 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] First, some nouns involved in this application are analyzed:

[0060] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0061] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text comprehension. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0062] Information Extraction (NER): A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0063] Automatic Speech Recognition (ASR): Also known as automatic speech recognition, speech recognition is an interdisciplinary subfield of computational linguistics that develops methods and techniques that enable spoken language to be recognized and translated by computers. It integrates knowledge and research in the fields of linguistics, computer science, and electrical engineering.

[0064] AdaBoost algorithm: The basic principle is to reasonably combine multiple weak classifiers (weak classifiers generally use single-layer decision trees) to make them a strong classifier. Adaboost adopts the idea of ​​iteration. Only one weak classifier is trained in each iteration, and the trained weak classifier will be used in the next iteration. In other words, in the Nth iteration, there are a total of N weak classifiers, of which N-1 are previously trained, and their various parameters will no longer change. This time, the Nth classifier is trained. The relationship between the weak classifiers is that the Nth weak classifier is more likely to classify the data that the first N-1 weak classifiers did not classify correctly, and the final classification output depends on the comprehensive effect of these N classifiers.

[0065] At present, there are many reasons for traffic accidents, generally speaking, they are human factors, vehicle factors, road and environmental factors, etc., among which human factors rank first. Among human factors, the driver's factor is the most important. The driver's bad mood is an important cause of traffic accidents. For example, when the driver is driving, when the driver is in a bad mood, it may affect the driver's driving effect and may cause an accident. When stimulated, most people express it through their faces, actions, eyes, etc. In the process of driving a vehicle, when the target object (driver) is in a good mood and satisfied, he can feel happy and psychologically comfortable, and has a positive role in promoting the observation and judgment of things. He often shows strong sensitivity, diligent observation, quick reaction, accurate judgment, and agile action, which is conducive to the safety of vehicle driving. On the contrary, if the target object has emotional fluctuations such as sadness, fear, distress, anger, etc., then the sensitivity decreases, the energy is scattered, and he is lazy to observe and think. Once encountering emergencies and emergencies, he will react slowly, make mistakes in judgment, and other bad behaviors. If there is music to adjust the driver's emotions and make the driver happy, the driver will concentrate on driving the vehicle and reduce the occurrence of accidents.

[0066] In the related art, the corresponding music can be played according to the emotion type. However, generally, the music to be played is matched in the ready-made music database by determining the emotion type. Therefore, whether the music to be played is reasonable will be limited by the music reserve and music diversity of the music database. In addition, since the music database is fixed, similar emotion types are likely to match the same music to be played, which greatly reduces the matching degree between the emotion type and the music to be played, and further affects the effect of relieving the emotions of the target object.

[0067] Therefore, how to provide a music playing method based on emotion recognition, which can improve the matching rate between emotions and played music, and relieve the emotions of the target object by playing music, has become a technical problem that needs to be solved urgently.

[0068] Based on this, the main purpose of the embodiments of the present application is to propose a music playing method and apparatus, device, and medium based on emotion recognition, which aims to determine the emotion recognition result based on the audio, obtain the music to be played based on the emotion recognition result and the music generation model, and finally play the music to be played. It can be understood that since the music to be played is generated based on the emotion recognition result, the matching degree between the music to be played and the emotion recognition result is high. And by playing the music to be played, the effect of relieving the emotions of the target object can be improved.

[0069] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0070] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0071] The music playing method based on emotion recognition provided in the embodiment of the present application relates to the field of artificial intelligence technology. The music playing method based on emotion recognition provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a music playing method based on emotion recognition, etc., but is not limited to the above forms.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0073] The embodiments of the present application provide a music playback method and apparatus, device, and medium based on emotion recognition, which are specifically illustrated by the following embodiments. First, the music playback method based on emotion recognition in the embodiments of the present application is described.

[0074] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0075] Figure 1 is an optional flowchart of the music playing method based on emotion recognition provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0076] Step S101, obtaining the audio to be recognized through a preset sound collection device;

[0077] Step S102, inputting the audio to be recognized into a preset emotion recognition model to obtain an emotion recognition result;

[0078] Step S103, inputting the emotion recognition result into a preset music generation model to obtain the music to be played through the music generation model;

[0079] Step S104, playing the music to be played through a preset player.

[0080] In the steps S101 to S104 shown in the embodiment of the present application, the emotion recognition result is determined based on the audio, and then the music to be played is obtained based on the emotion recognition result and the music generation model, and finally the music to be played is played. It can be understood that since the music to be played is generated based on the emotion recognition result, the matching degree between the music to be played and the emotion recognition result is high. And by playing the music to be played, the effect of relieving the emotions of the target object can be improved.

[0081] In step S101 of some embodiments, the preset sound collection device refers to a voiceprint collection terminal device, such as a microphone, a voice recorder, or a mobile phone, which can collect the voice of the target object. The target object referred to in the embodiment of the present application may refer to the driver or to all persons in the vehicle, and the embodiment of the present application does not specifically limit this.

[0082] In one embodiment, when acquiring the current voice data of the driver, it can be acquired in real time without any reminder. The sound collection device is only started when the driver speaks, and the audio to be recognized can be obtained based on the voice data after the driver finishes speaking. In addition, the audio to be recognized can also be obtained based on the voice data collected in real time during the driver's speaking. In another embodiment, a prompt tone can also be played to the user to prompt the user to speak a voice, and then the voice data can be collected to obtain the audio to be recognized. Furthermore, in actual applications, if there are many passengers in the current vehicle and these passengers are talking, a prompt tone can be played to prompt the driver to speak, prompting the driver to speak, and collect the voice data to avoid collecting the voice data of other passengers except the driver.

[0083] In one embodiment, before step S102, the process further includes pre-processing the audio to be recognized, and the pre-processing may include but is not limited to:

[0084] Extract voiceprint features from the audio to be recognized to obtain preliminary voiceprint features;

[0085] Acquire driving information of the target vehicle; the driving information includes at least one of the following: driving speed information and throttle speed information;

[0086] Determine noise characteristics based on driving information;

[0087] Filter the preliminary voiceprint features according to the noise features to obtain the target voiceprint features;

[0088] The preprocessed audio to be recognized is obtained according to the target voiceprint features.

[0089] Specifically, the embodiments of the present application are mainly applied to vehicles, so the audio to be identified collected by the sound collection device will include some noise. The noise is related to the driving information of the vehicle. For example, if the driving speed is too fast, the vehicle noise will be greater. For another example, if the throttle speed is too high, the vehicle noise will also increase. Therefore, the embodiments of the present application determine the noise characteristics according to the driving information of the vehicle. The preliminary voiceprint features are de-noised according to the noise characteristics, and the preliminary voiceprint features are filtered to obtain the pre-processed audio to be identified. It can be understood that the accuracy of emotion recognition can be further improved by inputting the de-noised audio to be identified into the music generation model.

[0090] In one embodiment, step S102 includes:

[0091] Perform semantic analysis on the audio to be recognized to obtain text data;

[0092] Dividing the text data into at least two sentences according to a preset pause duration threshold, and determining language tags according to the sentences;

[0093] Extracting language features of the audio to be recognized, and obtaining the emotion probability result of each sentence according to the language label and the language features; wherein the emotion probability result includes at least two emotion types and the probability of each emotion type;

[0094] The emotion types are selected according to their probabilities to obtain the emotion recognition results.

[0095] Specifically, the extraction of speech features is to extract the content related to the main information from the audio, which is generally done from two different perspectives: visual domain and frequency domain. In practical applications, speech information in speech data can be identified through speech recognition, and then the text information in the speech information can be identified to obtain text features, that is, language features. Language labels mainly refer to the language to which each sentence belongs, such as Chinese or English. Determining the emotion probability result based on language labels and language features can improve the accuracy of emotion recognition.

[0096] Reference Figure 2 In one embodiment, if the music generation model in step S103 is a strong classifier, step S103 may include but is not limited to steps S201 to S202:

[0097] Step S201, matching a corresponding weak classifier according to the emotion recognition result; a strong classifier includes multiple weak classifiers;

[0098] Step S202, obtaining the music to be played with emotion recognition results through a weak classifier.

[0099] In steps S201 to S202 shown in the embodiment of the present application, a strong classifier generally includes multiple weak classifiers, and generally a strong classifier is obtained after the weak classifier is trained. Classification and recognition work is performed by the strong classifier, and a weak classifier is no longer needed. In the embodiment of the present application, it is necessary to match the corresponding weak classifier according to the emotion recognition result, and generate music according to the weak classifier and the emotion recognition result, that is, the music to be played is obtained. It should be noted that the embodiment of the present application focuses on the matching degree between the emotion recognition result and the music to be played, so the matching degree can be greatly increased by using a weak classifier to realize music generation. The generation accuracy of the music to be played can be further improved.

[0100] In one embodiment, if the music generation model in step S103 is a strong classifier, the strong classifier is pre-trained by the following steps, including:

[0101] Get the audio data training set for sentiment classification;

[0102] Classify the data in the audio data training set into at least two emotion categories through classification annotation;

[0103] At least two emotion categories are trained for sentiment classification through a preset algorithm to obtain a strong classifier; wherein a weak classifier is generated corresponding to each emotion category, and all weak classifiers constitute a strong classifier.

[0104] Specifically, the preset algorithm is the Adaboost algorithm. The Adaboost algorithm is used to implement the training of each weak classifier. After the training, if the classifier can determine the corresponding music classification result according to the emotion recognition result, the music to be played can be obtained according to the music classification result. In addition, in the subsequent music generation process, the discrimination effect of each weak classifier is retained to improve the matching degree between the music to be played and the emotion recognition result.

[0105] It should be noted that before classification and labeling, the audio data training set is also pre-processed to improve the accuracy of training. The pre-processing of the audio data training set specifically includes:

[0106] Filtering the audio data training set through a preset filter to obtain a preliminary audio data training set; wherein the preliminary audio data training set includes a plurality of preliminary audio data;

[0107] Convert preliminary audio data into spectrograms;

[0108] Performing frequency filtering on the spectrogram to obtain target audio data whose frequency is less than a preset frequency threshold;

[0109] A preprocessed audio data training set is obtained according to the target audio data.

[0110] Specifically, the preset filter is a FIP filter. In order to obtain a spectrogram, it is necessary to set the length of the Hamming windows to 5ms, overlap 4.4ms, and the number of DFT points = 512, so as to obtain a spectrogram of preliminary audio data through the Hamming window. In addition, in this embodiment, the part of the spectrogram that is greater than the preset frequency threshold is discarded. The preset frequency threshold can be 4kHz, because in normal communication, the frequency within 4kHz is sufficient to classify emotions. It does not affect the classification accuracy, but also reduces the classification workload and improves the classification efficiency.

[0111] In one embodiment, before classification and labeling, the audio data training set is also expanded. It should be noted that the expansion process can be performed on the unprocessed audio data training set, or on the preprocessed audio data training set. The expansion process specifically includes:

[0112] Performing cross-validation on the audio data of the audio data training set by a preset number of times to obtain at least two parts of audio data;

[0113] Add Gaussian white noise to each part of the audio data to obtain enhanced audio data;

[0114] An expanded audio data training set is obtained according to at least two parts of enhanced audio data.

[0115] Specifically, taking the preset number multiple as five times as an example, the audio data is cross-validated five times to obtain five parts of audio data. In order to prevent the adverse effects of overfitting and too small data volume, each part of the audio data is enhanced by adding Gaussian white noise with a signal-to-noise ratio of +15 (SNR) to each audio data, such as adding Gaussian white noise every 10 seconds or every 20 seconds to obtain enhanced audio data. Finally, the expanded audio data training set is obtained based on the five parts of enhanced audio data.

[0116] In one embodiment, step S202 includes:

[0117] The emotion recognition results are judged by the weak classifier to obtain the music classification results;

[0118] The music to be played is determined from a preset music database according to the music classification result.

[0119] Specifically, if the result of emotion recognition is anger type, then the result of music classification is joyful type, then joyful music to be played should be obtained from the music database. In another embodiment, the music database is determined according to the music classification result, and music in the music database is randomly extracted. For example, if the result of music classification is joyful type, the first music database can be determined, and music in the first music database can be randomly extracted as music to be played. If the result of music classification is anxious type, the second music database can be determined, and music in the second music database can be randomly extracted as music to be played. For another example, if the result of music classification is sad type, the third music database can be determined, and music in the third music database can be randomly extracted as music to be played. Among them, folk music, country music and slow-tempo music are stored in music library one, classical music, light music and joyful folk music are stored in music library two, and joyful modern music and joyful pop music are stored in music library three. It should be noted that the types of music stored in the music library and the corresponding relationship with emotions can be configured in advance.

[0120] Reference Figure 3 In another embodiment, step S202 includes steps S301 to S303:

[0121] Step S301, using a weak classifier to identify the emotion recognition result to obtain a music classification result;

[0122] Step S302, determining the initial music according to the music classification result, and extracting the music features of the initial music; wherein the music features include at least one of the following: a spectrum parameter feature, a rhythm sequence feature, and a pitch trajectory feature;

[0123] Step S303, adjusting the music features according to the music classification result to obtain the music to be played.

[0124] Specifically, considering that the purpose of playing music is to relieve the emotions of the target object, the music to be played is mainly cheerful music. Therefore, in the embodiment of the present application, initial music is introduced, and the initial music includes music features. Then, the music features are adjusted according to the music classification results to obtain target music, that is, the music to be played. In this way, different music to be played can be generated according to the emotion recognition results of the target object, thereby achieving the relief of the target object's emotions.

[0125] Reference Figure 4 In another embodiment, step S202 includes steps S401 to S404:

[0126] Step S401, using a weak classifier to identify the emotion recognition result, and obtaining a music classification result;

[0127] Step S402, determining initial music features according to the music classification result; wherein the initial music features include at least one of the following: a spectrum parameter feature, a rhythm sequence feature, and a pitch trajectory feature;

[0128] Step S403, predicting the initial music features through a preset prediction neural network to obtain prediction parameters;

[0129] Step S404, adjusting the initial music features according to the prediction parameters to obtain target music features, and obtaining the music to be played according to the target music features.

[0130] Referring to the above, considering that the purpose of playing music is to relieve the emotions of the target object, the music to be played is mainly cheerful music. Different from the above-mentioned embodiment in which the initial music is determined first, in the embodiment of the present application, the initial music features are first determined according to the music classification results, and then predicted by the prediction neural network. The initial music features are adjusted according to the obtained prediction parameters, and finally the music to be played is obtained. It can be understood that generating the music to be played by predicting through a prediction neural network can further improve the diversity of the generated music. It should be noted that the prediction neural network can be a convolutional neural network, such as a CNN model.

[0131] Reference Figure 5In another embodiment, if the music generation model of step S103 is a music automatic generation network, it also includes pre-training the music automatic generation network. Step S103 includes steps S501 to S506:

[0132] Step S501, obtaining a musical instrument audio data set, where the musical instrument audio data set includes at least two musical instrument audio data;

[0133] Step S502, cross-validating the instrument audio data of the instrument audio data set by a preset number of times to obtain at least two parts of preliminary instrument audio data;

[0134] Step S503, obtaining a test set and a training set according to at least two parts of preliminary instrument audio data; wherein the test set includes one part of preliminary instrument audio data, and the training set includes the other parts of preliminary instrument audio data except the test set;

[0135] Step S504, adding Gaussian white noise to the preliminary instrument audio data of the training set to obtain enhanced instrument audio data;

[0136] Step S505, inputting the enhanced instrument audio data of the training set into a preset neural network model, training the neural network model to obtain a preliminary music automatic generation network;

[0137] Step S506, inputting the preliminary instrument audio data of the test set into the preliminary music automatic generation network, testing the preliminary music automatic generation network to obtain a trained music automatic generation network.

[0138] Steps S501 to S506 shown in the embodiment of the present application, specifically, take the preset number multiple as five times as an example, the instrument audio data is cross-validated five times to obtain five parts of instrument audio data. The instrument audio data of the first part is used as a test set, and the instrument audio data of the other parts is used as a training set. That is, the training set and the test set are classified according to a 4 / 1 ratio. Afterwards, the instrument audio data of the second part is used as a test set, and the instrument audio data of the other parts is used as a training set. The training and testing of the model are cycled in sequence, and finally a music automatic generation network is obtained. The emotion recognition result is input into the music automatic generation network, and the music to be played can be output. In addition, in order to prevent the adverse effects of overfitting and too small a data volume, the instrument audio data of the training set is enhanced by adding Gaussian white noise with a signal-to-noise ratio of +15 (SNR) to each preliminary instrument audio data, such as adding Gaussian white noise every 10 seconds or every 20 seconds, to obtain enhanced instrument audio data. It should be noted that the preliminary music automatic generation network is a GPT-2 model.

[0139] See also Figure 6The embodiment of the present application also provides a music playing device based on emotion recognition, which can implement the above-mentioned music playing method based on emotion recognition. Figure 6 The module structure block diagram of the music playing device based on emotion recognition provided in the embodiment of the present application includes: an audio acquisition module 601, an emotion recognition module 602, a music determination module 603 and a music playing module 604. Among them, the audio acquisition module 601 is used to obtain the audio to be recognized through a preset sound collection device; the emotion recognition module 602 is used to input the audio to be recognized into a preset emotion recognition model to obtain the emotion recognition result; the music determination module 603 is used to input the emotion recognition result into a preset music generation model to obtain the music to be played through the music generation model; among them, if the music generation model is a strong classifier, the music to be played is obtained through the music generation model, including: matching the corresponding weak classifier according to the emotion recognition result; the strong classifier includes multiple weak classifiers; the music to be played with the emotion recognition result obtained through the weak classifier; the music playing module 604 is used to play the music to be played through a preset player.

[0140] It should be noted that the specific implementation of the music playing device based on emotion recognition is basically the same as the specific implementation of the music playing method based on emotion recognition mentioned above, and will not be repeated here.

[0141] The embodiment of the present application also provides a computer device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and when the program is executed by the processor, the above-mentioned music playing method based on emotion recognition is realized. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0142] See also Figure 7 , Figure 7 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0143] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0144] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the music playing method based on emotion recognition in the embodiment of this application;

[0145] Input / output interface 703, used to implement information input and output;

[0146] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0147] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0148] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0149] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned music playback method based on emotion recognition.

[0150] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0151] The music playing method based on emotion recognition, the music playing device based on emotion recognition, the computer equipment and the storage medium provided in the embodiment of the present application determine the emotion recognition result according to the audio, obtain the music to be played according to the emotion recognition result and the music generation model, and finally play the music to be played. In addition, the embodiment of the present application needs to match the corresponding weak classifier according to the emotion recognition result, generate music according to the weak classifier and the emotion recognition result, and obtain the music to be played. It should be noted that the embodiment of the present application focuses on the matching degree between the emotion recognition result and the music to be played, so the matching degree can be greatly increased by realizing the music generation through the weak classifier. The generation accuracy of the music to be played can be further improved. It can also be understood that due to the high matching degree between the music to be played and the emotion recognition result, and by playing the music to be played, the effect of relieving the emotions of the target object can be improved.

[0152] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0153] It can be understood by those skilled in the art that Figure 1-5 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0154] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0156] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0157] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0158] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0159] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0161] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0162] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A music playing method based on emotion recognition, characterized in that: The method comprises: Acquire the audio to be recognized through a preset sound collection device; Inputting the audio to be recognized into a preset emotion recognition model to obtain an emotion recognition result; Inputting the emotion recognition result into a preset music generation model to obtain the music to be played through the music generation model; Play the music to be played through a preset player; Wherein, if the music generation model is a strong classifier, obtaining the music to be played through the music generation model includes: Matching a corresponding weak classifier according to the emotion recognition result; the strong classifier includes a plurality of weak classifiers; The music to be played that obtains the emotion recognition result through the weak classifier; The strong classifier is pre-trained by the following steps, including: Get the audio data training set for sentiment classification; Filtering the audio data training set through a preset filter to obtain a preliminary audio data training set; wherein the preliminary audio data training set includes a plurality of preliminary audio data; converting the preliminary audio data into a spectrogram; Performing frequency filtering on the frequency spectrum to obtain target audio data having a frequency less than a preset frequency threshold; Obtaining a preprocessed audio data training set according to the target audio data; Classify the data in the audio data training set into at least two emotion categories by classification annotation; Performing sentiment classification training on at least two of the emotion categories through a preset algorithm to obtain the strong classifier; wherein a weak classifier is generated corresponding to each emotion category, and all the weak classifiers constitute a strong classifier; Wherein, after the emotion recognition result is input into a preset music generation model, if the music generation model is a music automatic generation network, the method further includes pre-training the music automatic generation network, including: Acquire a musical instrument audio data set, where the musical instrument audio data set includes at least two musical instrument audio data; Performing cross-validation on the instrument audio data of the instrument audio data set for a preset number of times to obtain preliminary instrument audio data of at least two parts; A test set and a training set are obtained according to at least two parts of the preliminary musical instrument audio data; wherein the test set includes one part of the preliminary musical instrument audio data, and the training set includes the preliminary musical instrument audio data of the other parts except the test set; Adding Gaussian white noise to the preliminary instrument audio data of the training set to obtain enhanced instrument audio data; Inputting the enhanced instrument audio data of the training set into a preset neural network model, and training the neural network model to obtain a preliminary music automatic generation network; The preliminary instrument audio data of the test set is input into a preliminary music automatic generation network, and the preliminary music automatic generation network is tested to obtain a trained music automatic generation network.

2. The method according to claim 1, characterized in that: The music to be played that obtains the emotion recognition result through the weak classifier includes: The emotion recognition result is judged by the weak classifier to obtain a music classification result; Determining initial music according to the music classification result, and extracting music features of the initial music; The music features are adjusted according to the music classification result to obtain the music to be played.

3. The method according to claim 1, characterized in that: The music to be played that obtains the emotion recognition result through the weak classifier includes: The emotion recognition results are judged by the weak classifier to obtain the music classification results; Determine the initial music feature according to the music classification result; wherein the initial music feature includes at least one of the following: a spectrum parameter feature, a rhythm sequence feature, and a pitch trajectory feature; Predicting the initial music features through a preset prediction neural network to obtain prediction parameters; The initial music features are adjusted according to the prediction parameters to obtain the target music features, and the music to be played is obtained according to the target music features.

4. The method according to claim 1, characterized in that: Before classifying the data in the audio data training set into at least two emotion categories by classification and labeling, the method further includes expanding the audio data training set, including: Performing cross-validation on the audio data of the audio data training set a preset number of times to obtain at least two parts of audio data; Add Gaussian white noise to each part of the audio data to obtain enhanced audio data; An expanded audio data training set is obtained based on at least two parts of the enhanced audio data.

5. A music playing device based on emotion recognition, characterized in that: The device comprises: An audio acquisition module is used to acquire the audio to be recognized through a preset sound acquisition device; An emotion recognition module, used for inputting the audio to be recognized into a preset emotion recognition model to obtain an emotion recognition result; A music determination module, used for inputting the emotion recognition result into a preset music generation model to obtain the music to be played through the music generation model; wherein, if the music generation model is a strong classifier, then obtaining the music to be played through the music generation model includes: matching a corresponding weak classifier according to the emotion recognition result; the strong classifier includes a plurality of weak classifiers; obtaining the music to be played of the emotion recognition result through the weak classifier; A music playing module, used for playing the music to be played through a preset player; The strong classifier is pre-trained by the following steps, including: Get the audio data training set for sentiment classification; Filtering the audio data training set through a preset filter to obtain a preliminary audio data training set; wherein the preliminary audio data training set includes a plurality of preliminary audio data; converting the preliminary audio data into a spectrogram; Performing frequency filtering on the frequency spectrum to obtain target audio data having a frequency less than a preset frequency threshold; Obtaining a preprocessed audio data training set according to the target audio data; Classify the data in the audio data training set into at least two emotion categories by classification annotation; Performing sentiment classification training on at least two of the emotion categories through a preset algorithm to obtain the strong classifier; wherein a weak classifier is generated corresponding to each emotion category, and all the weak classifiers constitute a strong classifier; Wherein, after the emotion recognition result is input into a preset music generation model, if the music generation model is a music automatic generation network, the device is further used to pre-train the music automatic generation network, including: Acquire a musical instrument audio data set, where the musical instrument audio data set includes at least two musical instrument audio data; Performing cross-validation on the instrument audio data of the instrument audio data set for a preset number of times to obtain preliminary instrument audio data of at least two parts; A test set and a training set are obtained according to at least two parts of the preliminary musical instrument audio data; wherein the test set includes one part of the preliminary musical instrument audio data, and the training set includes the preliminary musical instrument audio data of the other parts except the test set; Adding Gaussian white noise to the preliminary instrument audio data of the training set to obtain enhanced instrument audio data; Inputting the enhanced instrument audio data of the training set into a preset neural network model, and training the neural network model to obtain a preliminary music automatic generation network; The preliminary instrument audio data of the test set is input into a preliminary music automatic generation network, and the preliminary music automatic generation network is tested to obtain a trained music automatic generation network.

6. A computer device, characterized in that: The computer device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the method according to any one of claims 1 to 4 are realized.

7. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 4.

Images