Music sorting method and device, electronic device and storage medium

Abstract

The embodiments of the present application provide a music sorting method and device, an electronic device and a storage medium, which belong to the field of audio processing and digital medical technology. The method includes: obtaining a training sample set, constructing an emotion similarity matrix based on language audio emotion labels and music audio emotion labels; selecting the original language audio positive samples and the original language audio negative samples based on the emotion similarity matrix to determine the target language audio sample group; selecting the original music audio positive samples and the original music audio negative samples based on the emotion similarity matrix to determine the target music audio sample group; adjusting the model parameters based on the language audio emotion labels, the music audio emotion labels, the target language audio sample group and the target music audio sample group to obtain a music audio sorting model; inputting the target language audio and the preset music set into the music audio sorting model to perform music sorting. The embodiments of the present application can improve the accuracy of audio feature extraction, thereby improving the accuracy of music sorting.

Description

Technical Field

[0001] The present application relates to the fields of audio processing and digital medical technology, and in particular to a music sorting method and device, an electronic device, and a storage medium. Background Art

[0002] Music sequencing is the process of searching through a large library of music resources for music that matches a specific theme, style, or characteristics, and then sorting and outputting the matching results. In music therapy scenarios within digital healthcare, music sequencing can help better assist patients in their treatment by identifying music that matches their emotions.

[0003] Currently, cross-domain matching techniques for combining speech emotion recognition and music sorting typically use a large amount of paired data and learn a mapping function to minimize the distance between the paired data in the embedding space to train the sorting model. However, the music audio sorting model trained in this way does not accurately extract audio features, which affects the accuracy of music sorting. Therefore, how to improve the accuracy of audio feature extraction and, therefore, the accuracy of music sorting, has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a music sorting method and device, an electronic device and a storage medium, aiming to improve the accuracy of audio feature extraction, thereby improving the accuracy of music sorting.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a music sorting method, the method comprising:

[0006] Obtain a training sample set, the training sample set including a language audio subset and a music audio subset; wherein the language audio subset includes original language audio samples, original language audio positive samples, original language audio negative samples, and language audio emotion labels of the original language audio samples; the music audio subset includes original music audio samples, original music audio positive samples, original music audio negative samples, and music audio emotion labels of the original music audio samples; the original language audio samples are used to represent language audio segments emitted by sample subjects, and the original music audio samples are used to represent music audio segments to be sorted;

[0007] Constructing an emotion similarity matrix based on the language audio emotion label and the music audio emotion label to obtain the emotion similarity matrix;

[0008] Performing sample selection on the original language audio positive sample and the original language audio negative sample based on the emotion similarity matrix to determine a target language audio sample group, wherein the target language audio sample group includes a target language audio positive sample and a target language audio negative sample;

[0009] Performing sample selection on the original music audio positive sample and the original music audio negative sample based on the emotion similarity matrix to determine a target music audio sample group, wherein the target music audio sample group includes a target music audio positive sample and a target music audio negative sample;

[0010] Adjusting model parameters of an initial sorting model based on the language audio emotion label, the music audio emotion label, the target language audio sample group, and the target music audio sample group to obtain a music audio sorting model;

[0011] The target language audio is obtained, and the target language audio and a preset music set are input into the music audio sorting model to sort the music, thereby obtaining a music sorting list.

[0012] In some embodiments, the initial ranking model includes a feature extraction sub-model, and the sample selection of the original language audio positive samples and the original language audio negative samples based on the emotion similarity matrix to determine the target language audio sample group includes:

[0013] Performing audio feature extraction on the original language audio sample based on the feature extraction sub-model to obtain original language sample features;

[0014] Performing audio feature extraction on the original music audio sample based on the feature extraction sub-model to obtain original music sample features;

[0015] Constructing a feature similarity matrix based on the original language sample features and the original music sample features to obtain a feature similarity matrix;

[0016] performing sample selection on the original language audio positive samples based on the feature similarity matrix and the emotion similarity matrix to determine the target language audio positive samples of the target language audio sample group;

[0017] Sample selection is performed on the original language audio negative samples based on the feature similarity matrix and the emotion similarity matrix to determine the target language audio negative samples of the target language audio sample group.

[0018] In some embodiments, constructing an emotion similarity matrix based on the language audio emotion tag and the music audio emotion tag to obtain the emotion similarity matrix includes:

[0019] Performing emotion space mapping on the language audio emotion tag to obtain language emotion mapping data;

[0020] Performing emotional space mapping on the music audio emotion tag to obtain music emotion mapping data;

[0021] Vectorizing the language emotion mapping data to obtain a language emotion mapping vector;

[0022] Vectorizing the music emotion mapping data to obtain a music emotion mapping vector;

[0023] Performing vector similarity calculation on the language emotion mapping vector and the music emotion mapping vector to obtain a label mapping similarity value;

[0024] A sentiment similarity matrix is ​​constructed based on the tag mapping similarity values ​​to obtain the sentiment similarity matrix.

[0025] In some embodiments, adjusting the model parameters of the initial sorting model based on the language audio emotion tag, the music audio emotion tag, the target language audio sample group, and the target music audio sample group to obtain the music audio sorting model includes:

[0026] Perform speech emotion loss calculation based on the target language audio sample group, the language emotion mapping vector, and the music emotion mapping vector to obtain a language emotion loss function;

[0027] Performing music emotion loss calculation based on the target music audio sample group, the language emotion mapping vector, and the music emotion mapping vector to obtain a music emotion loss function;

[0028] Performing association loss calculation based on the original language sample features, the original music sample features, the target language audio sample group, and the target music audio sample group to obtain an association loss function;

[0029] Performing weighted calculation on the language emotion loss function, the music emotion loss function, and the association loss function to obtain a model loss function;

[0030] The model parameters of the feature extraction sub-model are adjusted based on the model loss function to obtain the music audio sorting model.

[0031] In some embodiments, the performing speech emotion loss calculation based on the target language audio sample group, the language emotion mapping vector, and the music emotion mapping vector to obtain a language emotion loss function includes:

[0032] Performing vector concatenation on the language emotion mapping vector and the music emotion mapping vector to obtain a target emotion label vector;

[0033] Performing audio feature extraction on the target language audio positive sample based on the feature extraction sub-model to obtain a first associated language audio feature;

[0034] Performing audio feature extraction on the target language audio negative sample based on the feature extraction sub-model to obtain a second associated language audio feature;

[0035] A triplet loss calculation is performed based on the target emotion label vector, the first associated language audio feature, and the second associated language audio feature to obtain the language emotion loss function.

[0036] In some embodiments, the feature extraction sub-model includes a feature extraction layer and a mapping layer, and extracting audio features from the original language audio sample based on the feature extraction sub-model to obtain the original language sample features includes:

[0037] Performing audio coding processing on the original language audio sample based on the feature extraction layer to obtain language audio coding features;

[0038] Feature mapping is performed on the language audio coding features based on the mapping layer to obtain the original language sample features.

[0039] In some embodiments, the music audio sorting model includes a feature extraction sub-model and a sorting sub-model, and inputting the target language audio and the preset music set into the music audio sorting model to sort the music to obtain a music sorting list includes:

[0040] Performing audio feature extraction on the target language audio based on the feature extraction sub-model to obtain target language audio features;

[0041] Performing audio feature extraction on the candidate music audio of the preset music set based on the feature extraction sub-model to obtain target music audio features;

[0042] Inputting the target language audio features and the target music audio features into the sorting sub-model to perform similarity calculation to obtain candidate similarity values;

[0043] The candidate music audios are sorted based on the candidate similarity values ​​to obtain the music sorting list.

[0044] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a music sorting device, the device comprising:

[0045] An acquisition module is configured to acquire a training sample set, wherein the training sample set includes a language audio subset and a music audio subset; wherein the language audio subset includes original language audio samples, original language audio positive samples, original language audio negative samples, and language audio emotion labels of the original language audio samples; the music audio subset includes original music audio samples, original music audio positive samples, original music audio negative samples, and music audio emotion labels of the original music audio samples; the original language audio samples are used to represent language audio segments emitted by sample subjects; and the original music audio samples are used to represent music audio segments to be sorted;

[0046] A matrix construction module is used to construct an emotion similarity matrix based on the language audio emotion label and the music audio emotion label to obtain the emotion similarity matrix;

[0047] A first sample selection module is configured to perform sample selection on the original language audio positive sample and the original language audio negative sample based on the emotion similarity matrix to determine a target language audio sample group, wherein the target language audio sample group includes a target language audio positive sample and a target language audio negative sample;

[0048] A second sample selection module is configured to perform sample selection on the original music audio positive sample and the original music audio negative sample based on the emotion similarity matrix to determine a target music audio sample group, wherein the target music audio sample group includes a target music audio positive sample and a target music audio negative sample;

[0049] a parameter adjustment module, configured to adjust model parameters of an initial sorting model based on the language audio emotion label, the music audio emotion label, the target language audio sample group, and the target music audio sample group to obtain a music audio sorting model;

[0050] The music sorting module is used to obtain target language audio, and input the target language audio and a preset music set into the music audio sorting model to perform music sorting to obtain a music sorting list.

[0051] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the music sorting method described in the first aspect above.

[0052] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the music sorting method described in the first aspect above.

[0053] The present application proposes a music sorting method and device, an electronic device and a storage medium, which trains an initial sorting model through a training sample set including a language audio subset and a music audio subset to obtain a music audio sorting model, which can deeply combine the emotional information of the language audio with the audio feature extraction. Specifically, a training sample set is obtained, which includes a language audio subset and a music audio subset; wherein the language audio subset includes the original language audio sample, the original language audio positive sample, the original language audio negative sample and the language audio emotion label of the original language audio sample, and the music audio subset includes the original music audio sample, the original music audio positive sample, the original music audio negative sample and the music audio emotion label of the original music audio sample, the original language audio sample is used to represent the language audio segment emitted by the sample object, and the original music audio sample is used to represent the music audio segment to be sorted; further, an emotional similarity matrix is ​​constructed based on the language audio emotion label and the music audio emotion label to obtain the emotional similarity matrix; the original language audio positive sample and the original music audio negative sample are sorted based on the emotional similarity matrix. The method comprises the following steps: selecting the original language audio negative sample to select the target language audio sample group, wherein the target language audio sample group includes the target language audio positive sample and the target language audio negative sample; and selecting the original music audio positive sample and the original music audio negative sample based on the emotion similarity matrix to determine the target music audio sample group, wherein the target music audio sample group includes the target music audio positive sample and the target music audio negative sample; further, adjusting the model parameters of the initial sorting model based on the language audio emotion label, the music audio emotion label, the target language audio sample group and the target music audio sample group to obtain a music audio sorting model; thereafter, obtaining the target language audio, and inputting the target language audio and the preset music set into the music audio sorting model for music sorting to obtain a music sorting list. Therefore, the embodiment of the present application can effectively improve the accuracy of audio feature extraction, thereby improving the accuracy of music sorting. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of the music sorting method provided in an embodiment of the present application;

[0055] Figure 2 yes Figure 1 A flowchart of step S120 in FIG.

[0056] Figure 3 It is a structural diagram of the sentiment similarity matrix provided by this application;

[0057] Figure 4 yes Figure 1 A flowchart of step S130 in FIG.

[0058] Figure 5This is a schematic diagram of the model structure of the initial sorting model provided by this application;

[0059] Figure 6 yes Figure 4 A flowchart of step S410 in FIG.

[0060] Figure 7 yes Figure 1 A flowchart of step S150 in FIG.

[0061] Figure 8 yes Figure 7 A flowchart of step S710 in FIG.

[0062] Figure 9 yes Figure 1 A flowchart of step S160 in FIG.

[0063] Figure 10 This is a structural diagram of a music sorting device provided in an embodiment of the present application;

[0064] Figure 11 This is a hardware structure diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0066] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0068] First, let’s analyze some of the terms used in this application:

[0069] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0070] Natural language processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese and English). A branch of artificial intelligence, NLP is an interdisciplinary field between computer science and linguistics, often referred to as computational linguistics. Natural language processing encompasses grammatical analysis, semantic analysis, and discourse comprehension. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It encompasses data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and linguistics research related to language computing.

[0071] Audio Emotion Recognition (AER) is a technology that identifies the emotional state of a speaker or listener by analyzing features in audio signals. This technology combines knowledge from signal processing, machine learning, deep learning, and artificial intelligence to enable computers to automatically detect and understand emotional information in audio.

[0072] Valence / Arousal (V / A mapping) is the process of mapping emotional states (such as happiness and sadness) into a two-dimensional space. V / A mapping converts emotional states into two dimensions: Valence and Arousal, with each dimension having a corresponding mapping value.

[0073] Triplet loss: A loss function that can be applied to deep learning models, enabling more refined modeling of details. The three elements in a triplet refer to the anchor sample, the positive sample, and the negative sample. By changing the specific value of the threshold (margin), the distance between positive and negative samples can be controlled to adapt to different model training requirements.

[0074] Music sorting is a process of searching for music audio that matches a specific theme, style or feature in a large number of music resources in a specific way, and sorting and outputting the matching results. The music sorting method of the present application can be better applied to scenarios such as intelligent diagnosis and treatment, remote consultation, etc. In the music therapy scenario in the field of digital medicine, music audio that matches the patient's emotions is determined by music sorting, which can better serve as an auxiliary treatment for the patient. Specifically, in the actual music-assisted therapy scenario, the emotion type expressed by the patient's voice clip can be used to search and sort the music. For example, the patient's current voice clip is "It's raining today, and I'm not in a good mood." At this time, the emotion type expressed by the patient's voice clip can be used to search and sort the music to obtain music that matches the emotion type contained in the voice clip. In this way, patients can be helped to strengthen their current emotions and better express their true thoughts, thereby better helping patients to vent their emotions and providing better intelligent auxiliary effects for music therapy scenarios.

[0075] However, the cross-domain matching method for combining speech emotion recognition and music sorting is very time-consuming and requires domain expertise. At present, the relevant technology usually converts data samples from different modalities into a joint embedding space, and the joint embedding space is optimized to align the data distribution of heterogeneous modes. However, speech (i.e., the language audio corresponding to the embodiment of the present application) and music (i.e., the music audio corresponding to the embodiment of the present application) cannot appear organically at the same time, that is, speech data and music data are two separate forms. For example, there is basically no speaking sound in music, except in some very rare cases, such as live concert music (because it may be mixed with the speaking sound of singers and fans greeting each other), or some rap music may mix speech and music together. Therefore, the relevant technology usually lacks an organic paired data set of speech and music. In this way, the relevant technology usually uses a large amount of paired data first and learns a mapping function to minimize the distance between the paired data in the embedding space to train the sorting model. However, expressing emotional happiness in speech and expressing emotional happiness in music will have a large difference in feature representation, which will lead to the insufficient accuracy of feature extraction of the relevant technology, thereby affecting the accuracy of music sorting.

[0076] Based on this, the embodiments of the present application provide a music sorting method and device, an electronic device and a storage medium, aiming to improve the accuracy of audio feature extraction, thereby improving the accuracy of music sorting.

[0077] The music sorting method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the music sorting method in the embodiments of the present application is described.

[0078] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0079] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0080] The music sorting method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The music sorting method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; the software can be an application that implements the music sorting method, etc., but is not limited to the above forms.

[0081] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0082] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as object information, object language audio data, and object location information, the permission or consent of the object will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the sensitive personal information of the object, the separate permission or consent of the object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or consent of the object, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.

[0083] See also Figure 1 , Figure 1 This is an optional flow chart of the music sorting method provided in the embodiment of the present application. In some embodiments, Figure 1 The method may include but is not limited to steps S110 to S160:

[0084] Step S110, obtaining a training sample set;

[0085] Step S120, constructing an emotion similarity matrix based on the language audio emotion label and the music audio emotion label to obtain an emotion similarity matrix;

[0086] Step S130, performing sample selection on the original language audio positive samples and the original language audio negative samples based on the emotion similarity matrix to determine a target language audio sample group;

[0087] Step S140, performing sample selection on the original music audio positive samples and the original music audio negative samples based on the emotion similarity matrix to determine a target music audio sample group;

[0088] Step S150, adjusting model parameters of the initial sorting model based on the language audio emotion label, the music audio emotion label, the target language audio sample group, and the target music audio sample group to obtain a music audio sorting model;

[0089] Step S160: Acquire target language audio, and input the target language audio and a preset music set into a music audio sorting model to sort the music, thereby obtaining a music sorting list.

[0090] In steps S110 to S160 shown in the embodiment of the present application, an emotion similarity matrix is ​​constructed using language audio emotion labels and music audio emotion labels, and a more representative target language audio sample group and target music audio sample group are determined from the training sample set using the similarity matrix, so as to train a music audio sorting model that can more accurately extract features related to emotion information from language audio and music audio, and accurately sort music audio clips based on the extracted features. Compared to the related art method of training the model based only on minimizing the distance of paired audio data, the present application can deeply explore the correlation information between emotion and audio features, improve the accuracy of audio feature extraction, and thus improve the accuracy of music sorting.

[0091] In step S110 of some embodiments, the present application may divide the original data set into multiple batches during training, and the data of each batch constitutes a training sample set, and the training sample set may include a language audio subset and a music audio subset. The language audio subset is used to record the language audio clips emitted by the sample object, and the audio clips recorded in the language audio subset may be derived from at least one sample object. In the music therapy scenario in the field of digital medicine, the audio clips recorded in the sample language audio subset may be audio clips in which the patient expresses his or her own state and thoughts to the doctor. The music audio subset is used to record multiple music audio clips stored in a preset music library, and the emotional type corresponding to each music audio clip has been determined. Among them, each music audio sample recorded in the music audio subset may be derived from audio clips of different candidate music in the preset music library.

[0092] It should be noted that the language audio subset may include multiple language audio samples, and may specifically include original language audio samples, original language audio positive samples, original language audio negative samples, and language audio emotion labels of original language audio samples. The original language audio sample refers to any language audio segment used for training in the current batch, and is used to characterize the language audio segment emitted by the sample object. The language audio emotion label is used to indicate the emotion type corresponding to the pre-labeled original language audio sample, and the preset emotion types may include happy, excited, neutral, sad, angry, depressed, etc. The emotion label is a numerical mark corresponding to each emotion type, for example, the emotion label corresponding to depression is 1, the emotion label corresponding to anger is 2, the emotion label corresponding to happiness is 6, etc., without specific limitation. The original language audio positive sample refers to a language audio sample with the same emotion label as the original language audio sample, and the original language audio negative sample refers to a language audio sample with a different emotion label from the original language audio sample. The language audio subset of this application can be derived from a constructed music dataset, such as the IEMOCAP dataset, the historical music therapy library, etc., without limitation.

[0093] It should be noted that the music audio subset may include multiple music audio samples, and may specifically include original music audio samples, original music audio positive samples, original music audio negative samples, and music audio emotion labels of original music audio samples. The original music audio sample refers to any music audio clip to be sorted in the current batch for training. The music audio emotion label is used to indicate the emotion type corresponding to the pre-labeled original music audio sample, and the preset emotion types may include happy, excited, funny, gentle, noisy, sad, angry, scary, etc. The emotion label is a numerical mark corresponding to each emotion type, for example, the emotion label corresponding to happy is 1, the emotion label corresponding to excited is 2, the emotion label corresponding to scary is 8, etc., without specific limitation. The original music audio positive sample refers to a music audio sample with the same emotion label as the original music audio sample, and the original music audio negative sample refers to a music audio sample with a different emotion label from the original music audio sample. The music audio subset of this application may be derived from a constructed music dataset, such as the AudioSet dataset, the historical music therapy library, etc., without limitation. It should be noted that the preset emotion type corresponding to the music audio emotion tag and the preset emotion type corresponding to the language audio emotion tag can be the same or different, without limitation.

[0094] In step S120 of some embodiments, the emotion similarity matrix is ​​a matrix used to quantify the similarity between two or more emotion expressions. Each element in the emotion similarity matrix represents the similarity score between a pair of emotion expressions. The embodiment of the present application constructs an emotion similarity matrix based on language audio emotion tags and music audio emotion tags, which can deeply learn the emotion representation in two different modalities (i.e., language audio and music audio) to enhance the accuracy and robustness of feature extraction.

[0095] It should be noted that the embodiment of the present application formalizes the speech-to-music matching task into a supervised cross-domain retrieval and ranking problem, better utilizes continuous audio emotion similarity information for model training, and effectively improves the accuracy of audio feature extraction associated with emotions.

[0096] See also Figure 2 In some embodiments, step S120 may specifically include but is not limited to steps S210 to S260:

[0097] Step S210, performing emotion space mapping on the language audio emotion tag to obtain language emotion mapping data;

[0098] Step S220, performing emotion space mapping on the music audio emotion tag to obtain music emotion mapping data;

[0099] Step S230, vectorizing the language emotion mapping data to obtain a language emotion mapping vector;

[0100] Step S240, vectorizing the music emotion mapping data to obtain a music emotion mapping vector;

[0101] Step S250: performing vector similarity calculation on the language emotion mapping vector and the music emotion mapping vector to obtain a label mapping similarity value;

[0102] Step S260 , constructing a sentiment similarity matrix based on the tag mapping similarity values ​​to obtain a sentiment similarity matrix.

[0103] In some embodiments, in step S210 and step S220, since data of different modal types (such as language and music) have differences in the specific categories of emotion classification, the language audio emotion tags can be first mapped to an emotion space to unify the emotion categories of data of different modalities into the same spatial dimension. Language emotion mapping data refers to mapping data that characterizes language audio emotion tags in the emotion space. For example, the present application can perform emotion space mapping on language audio emotion tags based on V / A mapping. The language emotion mapping data obtained at this time is a VA value, and the V / A mapping can be implemented using a pre-trained dimensional emotion model (Valence-Arousal-Dominance, VAD model, that is, using the two dimensions of emotional value Valence and arousal Arousal to represent emotion). In this model, Valence represents the degree of positivity or negativity of emotion, and Arousal represents the degree of arousal of emotion. The dimensional emotion model regards emotion analysis as a regression problem, and calculates the continuous values ​​of Valence and Arousal by analyzing the emotional words and contextual information in the text. Among them, the VA value of the present application can be a two-dimensional space coordinate. For example, the language audio emotion label is happy, and after V / A mapping, the language emotion mapping data can be obtained as (0.4, 0.2), 0.4 represents the emotion value of the language audio emotion label of the current original language audio sample, and 0.2 represents the arousal of the language audio emotion label of the current original language audio sample. Similarly, the music audio emotion label is emotionally mapped in space to obtain music emotion mapping data. For example, the music audio emotion label is sad, and after V / A mapping, the music emotion mapping data can be obtained as (0.3, 0.6), 0.3 represents the emotion value of the music audio emotion label of the current original music audio sample, and 0.6 represents the arousal of the music audio emotion label of the current original music audio sample.

[0104] In some embodiments of step S230 and step S240, further, the present application can vectorize the language emotion mapping data to obtain a language emotion mapping vector, and vectorize the music emotion mapping data to obtain a music emotion mapping vector. In this way, the performance of the code can be improved and the complexity of the code can be reduced, thereby reducing the calculation time of the algorithm and increasing the training speed. Therefore, the present application can convert the original emotion label from one form to another through V / A mapping, that is, unify the label types of different modalities into the same dimensional space, and further use vectorization to optimize code execution, so that data can be processed more effectively and computing efficiency can be improved. For example, the language emotion mapping data is (0.4, 0.2), which can be (1, 2) after vectorization, and the music emotion mapping data is (0.3, 0.6), which can be (3, 4) after vectorization.

[0105] It should be noted that the vectorization process of music emotion mapping data and language emotion mapping data in this application can use Word2vec (a pre-trained model for natural language processing) to perform vector conversion to obtain language emotion mapping vectors and music emotion mapping vectors. Other vector conversion methods, such as one-hot encoding, can also be used, which will not be elaborated on.

[0106] In some embodiments, in step S250, the tag mapping similarity value refers to the value obtained by calculating the vector similarity of the language emotion mapping vector and the music emotion mapping vector. The method for calculating the vector similarity can be a cosine similarity function, a Euclidean distance function, a Hamming distance function, etc., without specific limitation.

[0107] In step S260 of some embodiments, for a training sample set corresponding to a batch of data, an emotional similarity matrix can be obtained, and the label mapping similarity values ​​obtained by different combinations of language audio and music audio are recorded in this emotional similarity matrix.

[0108] Please note that, please refer to Figure 3 , Figure 3 A structural diagram of the emotion similarity matrix is ​​shown. The horizontal axis represents the emotion labels of the music audio subset, and the vertical axis represents the emotion labels of the language audio subset. The intersections of different horizontal and vertical axes represent the label similarity values ​​corresponding to different combinations of music emotion labels and language emotion labels. The emotion types corresponding to the language audio emotion labels can include happy, excited, neutral, sad, angry, and depressed, and the emotion types corresponding to the music audio emotion labels can include happy, excited, funny, gentle, noisy, sad, angry, and scary. Figure 3 Different color depths are used to represent the numerical value reflecting the similarity between two labels. The darker the color, the higher the similarity. In addition, the present application can be calculated through the training sample sets corresponding to multiple batches of data to continuously iterate the final generated emotion similarity matrix. Among them, the darker the diagonal color in the emotion similarity matrix, the more accurate the generated similarity matrix is, and the better the robustness of the corresponding feature representation. For example, the label mapping similarity value obtained by the combination of the language audio corresponding to the first emotion type (such as happy) and the music audio corresponding to the third emotion type (such as) represents the numerical value corresponding to the intersection of the first row and the third column in the emotion similarity matrix.

[0109] It should be noted that if Figure 3 As shown in FIG, the same pair of input speech audio and music audio has similar emotional VA values, that is, has a certain correspondence. For example, the happy type corresponds to the happy type, the excited type corresponds to the excited type, and the depressed type corresponds to a certain amount of anger and a certain amount of fear.

[0110] In the above embodiment, the present application connects the emotion tags of different domains (i.e., language audio and music audio) through V / A mapping, so as to facilitate the subsequent extraction of more emotion-related features, thereby improving the accuracy of audio feature extraction.

[0111] In step S130 of some embodiments, to better learn detailed features, the present application may select original language audio positive samples and original language audio negative samples based on the emotion similarity matrix during model training to determine a target language audio sample group, which includes the target language audio positive sample and the target language audio negative sample. The target language audio positive sample refers to the sample selected from the original language audio positive sample that has the best feature similarity with the original language audio sample. This ensures that the emotion label type of the two is the same and the detailed features are more similar, thereby improving the ability to extract audio features associated with emotion.

[0112] See also Figure 4 In some embodiments, step S130 may specifically include but is not limited to steps S410 to S450:

[0113] Step S410: extracting audio features from the original language audio sample based on the feature extraction sub-model to obtain features of the original language sample;

[0114] Step S420, extracting audio features from the original music audio sample based on the feature extraction sub-model to obtain features of the original music sample;

[0115] Step S430: constructing a feature similarity matrix based on the original language sample features and the original music sample features to obtain a feature similarity matrix;

[0116] Step S440 , performing sample selection on the original language audio positive samples based on the feature similarity matrix and the emotion similarity matrix to determine the target language audio positive samples of the target language audio sample group;

[0117] Step S450 : performing sample selection on the original language audio negative samples based on the feature similarity matrix and the emotion similarity matrix to determine the target language audio negative samples of the target language audio sample group.

[0118] Please note that Figure 5 , Figure 5A schematic diagram of the model structure of the initial sorting model is shown, which includes a feature extraction submodel 510, and the feature extraction submodel 510 includes a language encoding unit 511 and a music encoding unit 512. The language encoding unit 511 is used to encode the input language audio, and the music encoding unit 512 is used to encode the input music audio. Among them, the language encoding unit 511 of the present application can be based on Wav2vec (a self-supervised learning framework for learning speech representation from audio clips, which can average pool its phoneme-level features to aggregate these features into speech levels, such as Wav2vec1.0, Wav2vec2.0 and other network architectures), or other network architectures can be used without limitation. The music encoding unit 512 of the present application can be a music tagging model based on the Transformer architecture (Music Tagging Transformer, which can use the self-attention mechanism to process music data and realize effective classification and labeling of music content), or other network architectures can be used without limitation.

[0119] In steps S410 to S430 of some embodiments, the present application can perform audio feature extraction on the original language audio sample based on the language encoding unit 511 of the feature extraction sub-model 510 to obtain the original language sample features. And based on the music encoding unit 512 of the feature extraction sub-model, audio feature extraction is performed on the original music audio sample to obtain the original music sample features. Furthermore, a feature similarity matrix can be constructed based on the original language sample features and the original music sample features to obtain a feature similarity matrix, that is, the feature similarity of the values ​​at corresponding positions in the original language sample features and the original music sample features can be calculated, and the method for calculating the feature similarity can be a cosine similarity function, a Euclidean distance function, a Hamming distance function, etc., without specific limitation.

[0120] It should be noted that the matrix size of the feature similarity matrix obtained in this application is the same as the matrix size of the sentiment similarity matrix, and the label combinations corresponding to the corresponding positions of the two matrices are the same, which will not be elaborated.

[0121] In some embodiments, in steps S440 and S450, after obtaining the feature similarity matrix and the sentiment similarity matrix, the original language audio sample can be used as the anchor language sample, and based on the feature similarity matrix and the sentiment similarity matrix, the sample corresponding to the numerical value with the same label as the original language audio sample and the highest feature similarity is determined as the target language audio positive sample. Furthermore, based on the feature similarity matrix and the sentiment similarity matrix, the sample corresponding to the numerical value with a different label from the original language audio sample and the lowest feature similarity is determined as the target language audio negative sample.

[0122] Therefore, by combining the feature similarity matrix and the emotion similarity matrix, this application can deeply learn the emotion representation and feature representation in two different modalities (i.e., language audio and music audio), and select the most representative target language audio negative samples and target language audio positive samples in a triplet manner, thereby enhancing the accuracy and robustness of feature extraction in the subsequent training process.

[0123] See also Figure 6 In some embodiments, step S410 may specifically include but is not limited to steps S610 to S620:

[0124] Step S610, performing audio coding processing on the original language audio sample based on the feature extraction layer to obtain language audio coding features;

[0125] Step S620 , performing feature mapping on the language audio coding features based on the mapping layer to obtain original language sample features.

[0126] It should be noted that if Figure 5 As shown, the language encoding unit 511 and the music encoding unit 512 of the feature extraction sub-model 510 both include corresponding feature extraction layers and mapping layers for extracting features of different modalities.

[0127] In steps S610 and S620 of some embodiments, the language encoding unit 511 can perform audio encoding processing on the original language audio sample based on the corresponding feature extraction layer to obtain language audio encoding features. By performing audio encoding processing on the original language audio sample through the feature extraction layer, language audio encoding features can be obtained. These language audio encoding features can contain useful information in the original language audio sample, such as pitch, tone, sound intensity, etc., which is helpful for subsequent analysis and processing. Furthermore, feature mapping is performed on the language audio encoding features based on the mapping layer to obtain original language sample features. The purpose of this feature mapping is to convert the audio encoding features into a form more suitable for subsequent processing and analysis.

[0128] It should be noted that the mapping layer of the present application may be a multilayer perceptron (MLP) or a structure constructed by other mapping processing methods, which will not be described in detail.

[0129] In the above embodiment, the present application processes the input audio data through the feature extraction layer and the mapping layer, which helps the model to better adapt to different audio data, improve the generalization ability of the model, and effectively reduce the amount of data, thereby improving the efficiency of subsequent processing and analysis.

[0130] It should be noted that, in the music encoding unit 512, audio encoding processing can be performed on the original music audio sample based on the corresponding feature extraction layer to obtain music audio encoding features. Furthermore, feature mapping can be performed on the music audio encoding features based on the mapping layer to obtain original music sample features.

[0131] In step S140 of some embodiments, sample selection is performed on the original music audio positive samples and the original music audio negative samples based on the sentiment similarity matrix to determine a target music audio sample group, which includes the target music audio positive samples and the target music audio negative samples. The specific process of determining the target music audio sample group can be found in the process of determining the target language audio sample group in steps S410 to S450 above, that is, all language audio-related parameters are replaced with music audio-related parameters. To save space, this is not further described here.

[0132] In step S150 of some embodiments, the model parameters of the initial sorting model are adjusted based on the language audio emotion label, the music audio emotion label, the target language audio sample group, and the target music audio sample group to obtain a music audio sorting model. The music audio sorting model of the present application refers to a model that can sort the input music audio based on the emotion type of the input language audio, and the music audio sorting model includes a feature extraction submodel and a sorting submodel after parameter adjustment.

[0133] See also Figure 7 In some embodiments, step S150 may specifically include but is not limited to steps S710 to S750:

[0134] Step S710, performing speech emotion loss calculation based on the target language audio sample group, the language emotion mapping vector, and the music emotion mapping vector to obtain a language emotion loss function;

[0135] Step S720, performing music emotion loss calculation based on the target music audio sample group, the language emotion mapping vector, and the music emotion mapping vector to obtain a music emotion loss function;

[0136] Step S730, calculating an association loss based on the original language sample features, the original music sample features, the target language audio sample group, and the target music audio sample group to obtain an association loss function;

[0137] Step S740, performing weighted calculation on the language emotion loss function, the music emotion loss function, and the association loss function to obtain a model loss function;

[0138] Step S750: Adjust the model parameters of the feature extraction sub-model based on the model loss function to obtain a music audio sorting model.

[0139] In step S710 of some embodiments, the language sentiment loss function is a loss function used to characterize the influence relationship between the sentiment label and the language audio feature.

[0140] See also Figure 8 In some embodiments, step S710 may specifically include but is not limited to steps S810 to S840:

[0141] Step S810: performing vector concatenation on the language emotion mapping vector and the music emotion mapping vector to obtain a target emotion label vector;

[0142] Step S820: extracting audio features from the target language audio positive sample based on the feature extraction sub-model to obtain a first associated language audio feature;

[0143] Step S830: extracting audio features from the target language audio negative sample based on the feature extraction sub-model to obtain a second associated language audio feature;

[0144] Step S840 , performing triplet loss calculation based on the target emotion label vector, the first associated language audio feature, and the second associated language audio feature to obtain a language emotion loss function.

[0145] In steps S810 to S840 of some embodiments, the present application may construct the loss function of the present application based on the triple loss function, that is, optimize the loss function of the emotion label to the speech by embedding the emotion label. Specifically, the obtained language emotion mapping vector and the music emotion mapping vector may be vector spliced ​​to obtain the target emotion label vector. Furthermore, audio feature extraction may be performed on the target language audio positive sample based on the language coding unit of the feature extraction sub-model to obtain the first associated language audio feature, and audio feature extraction may be performed on the target language audio negative sample based on the language coding unit of the feature extraction sub-model to obtain the second associated language audio feature. Afterwards, a triple loss calculation is performed based on the target emotion label vector, the first associated language audio feature and the second associated language audio feature to obtain the language emotion loss function.

[0146] It should be noted that the triplet loss function is shown in the following formula 1. The specific process of calculating the language emotion loss function based on the target emotion label vector, the first associated language audio feature and the second associated language audio feature can be seen in the following formula 2:

[0147] L Triplet (z,z + ,z - )=max{0,D(z,z+ )-D(z,z - )+δ} (Formula 1)

[0148]

[0149] In Formula 1, z represents the anchor sample, z + represents a positive sample, z - represents negative samples, L Triplet (·) represents the triplet loss function, D(·) represents the set distance function, such as the cosine distance function, and δ represents a predefined margin value, such as 0.1, 0.5, etc., which can be a custom value without limitation. In formula 2, L SP-Speech represents the language sentiment loss function, z e represents the target sentiment label vector, represents the first associated language audio feature, In this way, by L Triplet The three parameters in (·) are replaced to obtain the language sentiment loss function.

[0150] In step S720 of some embodiments, the music emotion loss function is a loss function used to characterize the influence relationship between the emotion label and the music audio feature. Specifically, the present application can perform audio feature extraction on the target music audio positive sample based on the music encoding unit of the feature extraction sub-model to obtain the first associated music audio feature, and perform audio feature extraction on the target music audio negative sample based on the music encoding unit of the feature extraction sub-model to obtain the second associated music audio feature. Afterwards, a triplet loss calculation is performed based on the target emotion label vector, the first associated music audio feature and the second associated music audio feature to obtain the music emotion loss function. Therefore, the specific process of the present application for calculating the music emotion loss function based on the target music audio sample group, the language emotion mapping vector, and the music emotion mapping vector can be seen as shown in the following formula 3:

[0151]

[0152] In formula 3, L SP-Music represents the music emotion loss function, z e represents the target sentiment label vector, represents the first associated music audio feature, Represents the second associated music audio feature. Triplet The three parameters in (·) are replaced to obtain the music emotion loss function.

[0153] In the above embodiment, compared with the related art that calculates the loss function based on the characteristics of the language audio itself and the corresponding positive and negative samples, the present application can deeply learn the characteristic details of emotion to language and emotion to music by combining the characteristics of the embedded emotion tags, thereby improving the accuracy of feature extraction of audio clips.

[0154] In step S730 of some embodiments, the correlation between language audio and music audio features is further considered. The present application can also perform correlation loss calculation based on the original language sample features, the original music sample features, the target language audio sample group and the target music audio sample group to obtain a correlation loss function.

[0155] It should be noted that the association loss calculation based on the original language sample features, the original music sample features, the target language audio sample group, and the target music audio sample group is performed to obtain the association loss function, which can specifically include: performing a language-music association loss calculation based on the original language sample features, the target music audio positive sample, and the target music audio negative sample to obtain a first association loss sub-function, and the specific calculation process is shown in Formula 4 below. And performing a speech-music association loss calculation based on the original music sample features, the target language audio positive sample, and the target language audio negative sample to obtain a second association loss sub-function, and the specific calculation process is shown in Formula 5 below. Furthermore, performing a weighted calculation on the first association loss sub-function and the second association loss sub-function to obtain an association loss function, and the specific calculation process is shown in Formula 6 below.

[0156]

[0157] L cross0 =α1L cross1 +α2L cross2 (Formula 6)

[0158] Among them, L cross1 represents the first associated loss subfunction, z s Represents the original language sample features, L cross2 represents the second association loss subfunction, z m Represents the original music sample features, L cross0 Represents the associated loss function, α1 represents the weight of the first associated loss function, α2 represents the weight of the second associated loss function, and the sum of the weights of α1 and α2 is 1, such as α1 is 1, α2 is 0, or α1 and α2 are both 0.5, and can be flexibly adjusted according to actual needs, which is not described in detail.

[0159] In steps S740 and S750 of some embodiments, a weighted calculation is performed on the language emotion loss function, the music emotion loss function, and the association loss function to obtain a model loss function. Based on the model loss function, the model parameters of the feature extraction sub-model are adjusted until the model loss value meets the preset training end condition, and the feature extraction sub-model and the sorting sub-model after parameter adjustment are constructed to obtain a music audio sorting model. The preset training end condition may be when the model loss value is less than a preset loss value threshold, or when all batches of training sample sets are exhausted, without limitation.

[0160] In the above embodiment, the present application constructs the final model loss function of the present application by optimizing the triple loss function, that is, the triple loss of emotion to speech and emotion to music can be optimized respectively by combining the features of the embedded emotion tags, which can deeply learn more feature details and improve the accuracy of feature extraction of audio clips.

[0161] It should be noted that the calculation process of the model loss function can be seen in the following formula 7:

[0162] L 模 =ω1L SP-Speech +ω2L SP-Music +ω3L cross0 (Formula 7)

[0163] Among them, L 模 Represents the model loss function, ω1 represents the weight of the language emotion loss function, ω2 represents the weight of the music emotion loss function, ω3 represents the weight of the association loss function, and the sum of the weights of ω1, ω2 and ω3 is 1. For example, ω1 is 0.5, ω2 is 0.3, and ω3 is 0.2. ω1, ω2 and ω3 are hyperparameters that can be flexibly adjusted according to actual needs and will not be described in detail.

[0164] In the above embodiment, the present application can retrieve and match music audio that matches the speaker's emotion without language information (such as vocabulary-matched speech) by optimizing the model training process and optimizing the loss function.

[0165] In step S160 of some embodiments, after obtaining the music audio sorting model, the present application may input the target language audio and the preset music collection into the music audio sorting model to perform music sorting, thereby obtaining a music sorting list. The preset music collection may be automatically input into the model via a network, thereby achieving audio matching sorting for the target language audio.

[0166] It should be noted that this application can also be used in music software recommendations and information retrieval. That is, according to the target subject, a target language audio can be input to retrieve music audio that matches the same emotional type as the target language audio. The target language audio can be the target subject's own voiceprint audio or the audio of another subject's voiceprint input by the target subject.

[0167] See also Figure 9 In some embodiments, step S160 may specifically include but is not limited to steps S910 to S940:

[0168] Step S910: extracting audio features from the target language audio based on the feature extraction sub-model to obtain target language audio features;

[0169] Step S920: extracting audio features from candidate music audios in the preset music set based on the feature extraction sub-model to obtain target music audio features;

[0170] Step S930: Input the target language audio features and the target music audio features into the sorting sub-model to perform similarity calculation to obtain candidate similarity values;

[0171] Step S940: sort the candidate music audios based on the candidate similarity values ​​to obtain a music sorting list.

[0172] In steps S910 to S940 of some embodiments, audio feature extraction can be performed on the target language audio based on the feature extraction sub-model to obtain the target language audio features, and audio feature extraction can be performed on each candidate music audio of the preset music set based on the feature extraction sub-model to obtain the target music audio features. Furthermore, the target language audio features and each target music audio feature can be input into the sorting sub-model for similarity calculation to obtain multiple candidate similarity values. Thereafter, the candidate music audios can be sorted in descending audio order or ascending audio order based on the candidate similarity values ​​to obtain a music sorting list. For example, if the candidate music audios are sorted in descending audio order, the first music audio in the music sorting list is the music clip that best matches the emotional features of the target language audio. In addition, in music retrieval experiments in specific application scenarios, the present application can improve the retrieval recall hit rate by 13%.

[0173] A music sorting method provided by an embodiment of the present application can construct an emotion similarity matrix based on language audio emotion labels and music audio emotion labels, and can deeply learn the emotion representation in two different modalities (i.e., language audio and music audio). And by combining the feature similarity matrix and the emotion similarity matrix, the emotion representation and feature representation in two different modalities (i.e., language audio and music audio) can be deeply learned, and the most representative target language audio negative sample and target language audio positive sample are selected in combination with the triplet method, thereby enhancing the accuracy and robustness of feature extraction in the subsequent training process. Further, the present application can optimize the triplet loss of emotion to speech and emotion to music respectively by combining the features of the embedded emotion label. Compared with the method of related art that only trains the model based on minimizing the distance of paired audio data, the present application can deeply explore the correlation information between emotion and audio features, improve the accuracy of audio feature extraction, and thus improve the accuracy of music sorting.

[0174] See also Figure 10 The present application also provides a music sorting device that can implement the above-mentioned music sorting method. The device includes:

[0175] Acquisition module 1010 is used to acquire a training sample set, where the training sample set includes a language audio subset and a music audio subset; wherein the language audio subset includes original language audio samples, original language audio positive samples, original language audio negative samples, and language audio emotion labels of the original language audio samples; the music audio subset includes original music audio samples, original music audio positive samples, original music audio negative samples, and music audio emotion labels of the original music audio samples; the original language audio samples are used to represent language audio segments emitted by the sample subjects; and the original music audio samples are used to represent music audio segments to be sorted;

[0176] A matrix construction module 1020 is used to construct an emotion similarity matrix based on the language audio emotion label and the music audio emotion label to obtain an emotion similarity matrix;

[0177] A first sample selection module 1030 is configured to select the original language audio positive samples and the original language audio negative samples based on the sentiment similarity matrix to determine a target language audio sample group, where the target language audio sample group includes the target language audio positive samples and the target language audio negative samples;

[0178] A second sample selection module 1040 is configured to perform sample selection on the original music audio positive sample and the original music audio negative sample based on the emotion similarity matrix to determine a target music audio sample group, where the target music audio sample group includes the target music audio positive sample and the target music audio negative sample;

[0179] A parameter adjustment module 1050 is configured to adjust model parameters of the initial sorting model based on the language audio emotion label, the music audio emotion label, the target language audio sample group, and the target music audio sample group to obtain a music audio sorting model;

[0180] The music sorting module 1060 is used to obtain target language audio, and input the target language audio and a preset music set into a music audio sorting model to sort the music and obtain a music sorting list.

[0181] The specific implementation of the music sorting device is basically the same as the specific embodiment of the above-mentioned music sorting method, and will not be repeated here.

[0182] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the aforementioned music sorting method when executing the computer program. The electronic device may be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0183] See also Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0184] The processor 1110 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0185] The memory 1120 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1120 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1120 and is called by the processor 1110 to execute the music sorting method of the embodiments of this application.

[0186] Input / output interface 1130, used for information input and output;

[0187] Communication interface 1140, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0188] bus 1150 , which transmits information between various components of the device (e.g., processor 1110 , memory 1120 , input / output interface 1130 , and communication interface 1140 );

[0189] The processor 1110 , the memory 1120 , the input / output interface 1130 , and the communication interface 1140 are communicatively connected to each other within the device via a bus 1150 .

[0190] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned music sorting method is implemented.

[0191] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0192] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0193] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0194] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0195] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0196] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0197] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0199] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0201] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0202] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A music sorting method, characterized in that: The method comprises: Obtain a training sample set, the training sample set including a language audio subset and a music audio subset; wherein the language audio subset includes original language audio samples, original language audio positive samples, original language audio negative samples, and language audio emotion labels of the original language audio samples; the music audio subset includes original music audio samples, original music audio positive samples, original music audio negative samples, and music audio emotion labels of the original music audio samples; the original language audio samples are used to represent language audio segments emitted by sample subjects, and the original music audio samples are used to represent music audio segments to be sorted; Constructing an emotion similarity matrix based on the language audio emotion label and the music audio emotion label to obtain the emotion similarity matrix; Performing audio feature extraction on the original language audio sample based on the feature extraction sub-model to obtain original language sample features; performing audio feature extraction on the original music audio sample based on the feature extraction sub-model to obtain original music sample features; constructing a feature similarity matrix based on the original language sample features and the original music sample features to obtain a feature similarity matrix; performing sample selection on the original language audio positive sample based on the feature similarity matrix and the emotion similarity matrix to determine the target language audio positive sample of the target language audio sample group; performing sample selection on the original language audio negative sample based on the feature similarity matrix and the emotion similarity matrix to determine the target language audio negative sample of the target language audio sample group; Performing sample selection on the original music audio positive sample and the original music audio negative sample based on the emotion similarity matrix to determine a target music audio sample group, wherein the target music audio sample group includes a target music audio positive sample and a target music audio negative sample; Adjusting model parameters of an initial sorting model based on the language audio emotion label, the music audio emotion label, the target language audio sample group, and the target music audio sample group to obtain a music audio sorting model, wherein the initial sorting model includes the feature extraction sub-model; The target language audio is obtained, and the target language audio and a preset music set are input into the music audio sorting model to sort the music, thereby obtaining a music sorting list.

2. The method according to claim 1, characterized in that The constructing of an emotion similarity matrix based on the language audio emotion tag and the music audio emotion tag to obtain the emotion similarity matrix includes: Performing emotion space mapping on the language audio emotion tag to obtain language emotion mapping data; Performing emotional space mapping on the music audio emotion tag to obtain music emotion mapping data; Vectorizing the language emotion mapping data to obtain a language emotion mapping vector; Vectorizing the music emotion mapping data to obtain a music emotion mapping vector; Performing vector similarity calculation on the language emotion mapping vector and the music emotion mapping vector to obtain a label mapping similarity value; A sentiment similarity matrix is ​​constructed based on the tag mapping similarity values ​​to obtain the sentiment similarity matrix.

3. The method according to claim 2, characterized in that The step of adjusting the model parameters of the initial sorting model based on the language audio emotion label, the music audio emotion label, the target language audio sample group, and the target music audio sample group to obtain a music audio sorting model includes: Perform speech emotion loss calculation based on the target language audio sample group, the language emotion mapping vector, and the music emotion mapping vector to obtain a language emotion loss function; Performing music emotion loss calculation based on the target music audio sample group, the language emotion mapping vector, and the music emotion mapping vector to obtain a music emotion loss function; Performing association loss calculation based on the original language sample features, the original music sample features, the target language audio sample group, and the target music audio sample group to obtain an association loss function; Performing weighted calculation on the language emotion loss function, the music emotion loss function, and the association loss function to obtain a model loss function; The model parameters of the feature extraction sub-model are adjusted based on the model loss function to obtain the music audio sorting model.

4. The method according to claim 3, characterized in that The performing speech emotion loss calculation based on the target language audio sample group, the language emotion mapping vector, and the music emotion mapping vector to obtain a language emotion loss function includes: Performing vector concatenation on the language emotion mapping vector and the music emotion mapping vector to obtain a target emotion label vector; Performing audio feature extraction on the target language audio positive sample based on the feature extraction sub-model to obtain a first associated language audio feature; Performing audio feature extraction on the target language audio negative sample based on the feature extraction sub-model to obtain a second associated language audio feature; A triplet loss calculation is performed based on the target emotion label vector, the first associated language audio feature, and the second associated language audio feature to obtain the language emotion loss function.

5. The method according to any one of claims 1 to 4, characterized in that The feature extraction sub-model includes a feature extraction layer and a mapping layer. The feature extraction sub-model is used to extract audio features from the original language audio sample to obtain the original language sample features, including: Performing audio coding processing on the original language audio sample based on the feature extraction layer to obtain language audio coding features; Feature mapping is performed on the language audio coding features based on the mapping layer to obtain the original language sample features.

6. The method according to claim 1, characterized in that The music audio sorting model includes a feature extraction sub-model and a sorting sub-model. Inputting the target language audio and the preset music set into the music audio sorting model to sort the music and obtain a music sorting list includes: Performing audio feature extraction on the target language audio based on the feature extraction sub-model to obtain target language audio features; Performing audio feature extraction on the candidate music audio of the preset music set based on the feature extraction sub-model to obtain target music audio features; Inputting the target language audio features and the target music audio features into the sorting sub-model to perform similarity calculation to obtain candidate similarity values; The candidate music audios are sorted based on the candidate similarity values ​​to obtain the music sorting list.

7. A music sorting device, characterized in that: The device comprises: An acquisition module is configured to acquire a training sample set, wherein the training sample set includes a language audio subset and a music audio subset; wherein the language audio subset includes original language audio samples, original language audio positive samples, original language audio negative samples, and language audio emotion labels of the original language audio samples; the music audio subset includes original music audio samples, original music audio positive samples, original music audio negative samples, and music audio emotion labels of the original music audio samples; the original language audio samples are used to represent language audio segments emitted by sample subjects; and the original music audio samples are used to represent music audio segments to be sorted; A matrix construction module is used to construct an emotion similarity matrix based on the language audio emotion label and the music audio emotion label to obtain the emotion similarity matrix; A first sample selection module is configured to extract audio features from the original language audio sample based on the feature extraction submodel to obtain original language sample features; extract audio features from the original music audio sample based on the feature extraction submodel to obtain original music sample features; construct a feature similarity matrix based on the original language sample features and the original music sample features to obtain a feature similarity matrix; select the original language audio positive samples based on the feature similarity matrix and the emotion similarity matrix to determine the target language audio positive samples of the target language audio sample group; select the original language audio negative samples based on the feature similarity matrix and the emotion similarity matrix to determine the target language audio negative samples of the target language audio sample group; A second sample selection module is configured to perform sample selection on the original music audio positive sample and the original music audio negative sample based on the emotion similarity matrix to determine a target music audio sample group, wherein the target music audio sample group includes a target music audio positive sample and a target music audio negative sample; a parameter adjustment module, configured to adjust model parameters of an initial sorting model based on the language audio emotion label, the music audio emotion label, the target language audio sample group, and the target music audio sample group to obtain a music audio sorting model, wherein the initial sorting model includes the feature extraction sub-model; The music sorting module is used to obtain target language audio, and input the target language audio and a preset music set into the music audio sorting model to perform music sorting to obtain a music sorting list.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements a music sorting method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a music sorting method according to any one of claims 1 to 6 is implemented.

Images