Music sorting method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a music sorting method and device, electronic equipment and a storage medium, and belongs to the technical field of audio processing and digital medical treatment. The method comprises the following steps: acquiring a training sample set, and constructing an emotion similarity matrix based on language audio emotion tags and music audio emotion tags; selecting the original language audio positive sample and the original language audio negative sample based on the emotion similarity matrix to determine a target language audio sample group; selecting the original music audio positive sample and the original music audio negative sample based on the emotion similarity matrix to determine a target music audio sample group; adjusting model parameters based on the language audio emotion label, the music audio emotion label, the target language audio sample group and the target music audio sample group to obtain a music audio sorting model; and inputting the target language audio and the preset music set into a music audio sorting model for music sorting. According to the embodiment of the invention, the accuracy of audio feature extraction can be improved, so that the accuracy of music sorting is improved.

Description

Technical Field

[0001] This application relates to the fields of audio processing and digital medical technology, and in particular to a music sorting method and apparatus, electronic device and storage medium. Background Technology

[0002] Music sequencing is a process of searching through a large body of music resources for audio tracks that match specific themes, styles, or characteristics, and then sorting and outputting the results. In the context of music therapy in digital healthcare, music sequencing can help identify audio tracks that resonate with a patient's emotions, thus providing better support for treatment.

[0003] Currently, related technologies for cross-domain matching combining speech emotion recognition and music ranking typically involve first using a large amount of paired data and learning a mapping function that minimizes the distance between paired data in the embedding space to train the ranking model. However, the music audio ranking model trained in this way has poor accuracy in audio feature extraction, thus affecting the accuracy of music ranking. Therefore, how to improve the accuracy of audio feature extraction, thereby improving the accuracy of music ranking, has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main objective of this application is to provide a music sorting method, apparatus, electronic device, and storage medium, which aims to improve the accuracy of audio feature extraction, thereby improving the accuracy of music sorting.

[0005] To achieve the above objectives, a first aspect of this application proposes a music sorting method, the method comprising:

[0006] A training sample set is obtained, which includes a language audio subset and a music audio subset; wherein, the language audio subset includes original language audio samples, original language audio positive samples, original language audio negative samples, and language audio sentiment labels of the original language audio samples; the music audio subset includes original music audio samples, original music audio positive samples, original music audio negative samples, and music audio sentiment labels of the original music audio samples; the original language audio samples are used to represent language audio segments emitted by the sample objects, and the original music audio samples are used to represent music audio segments to be sorted;

[0007] An emotion similarity matrix is ​​constructed based on the language audio emotion tags and the music audio emotion tags to obtain the emotion similarity matrix;

[0008] Based on the sentiment similarity matrix, sample selection is performed on the original language audio positive samples and the original language audio negative samples to determine the target language audio sample group, which includes target language audio positive samples and target language audio negative samples;

[0009] Based on the emotional similarity matrix, sample selection is performed on the original positive music audio samples and the original negative music audio samples to determine the target music audio sample group, which includes target music audio positive samples and target music audio negative samples;

[0010] Based on the language audio emotion tags, the music audio emotion tags, the target language audio sample group, and the target music audio sample group, the model parameters of the initial ranking model are adjusted to obtain the music audio ranking model.

[0011] The target language audio is obtained, and the target language audio and a preset music set are input into the music audio sorting model to sort the music and obtain a music sorting list.

[0012] In some embodiments, the initial ranking model includes a feature extraction sub-model, and the step of selecting samples from the original language audio positive samples and the original language audio negative samples based on the sentiment similarity matrix to determine the target language audio sample group includes:

[0013] Based on the feature extraction sub-model, audio features are extracted from the original language audio samples to obtain the original language sample features;

[0014] Based on the feature extraction sub-model, audio features are extracted from the original music audio samples to obtain the features of the original music samples.

[0015] A feature similarity matrix is ​​constructed based on the features of the original language samples and the features of the original music samples to obtain the feature similarity matrix;

[0016] Based on the feature similarity matrix and the sentiment similarity matrix, sample selection is performed on the original language audio positive samples to determine the target language audio positive samples of the target language audio sample group;

[0017] Based on the feature similarity matrix and the sentiment similarity matrix, sample selection is performed on the original language audio negative samples to determine the target language audio negative samples of the target language audio sample group.

[0018] In some embodiments, the construction of the emotion similarity matrix based on the language audio emotion tags and the music audio emotion tags to obtain the emotion similarity matrix includes:

[0019] The emotional space is mapped to the emotional tags of the language audio to obtain language emotional mapping data;

[0020] The emotional space of the music audio emotional tags is mapped to obtain music emotional mapping data;

[0021] The language sentiment mapping data is vectorized to obtain language sentiment mapping vectors;

[0022] The music emotion mapping data is vectorized to obtain the music emotion mapping vector;

[0023] The similarity between the language emotion mapping vector and the music emotion mapping vector is calculated to obtain the tag mapping similarity value;

[0024] The sentiment similarity matrix is ​​constructed based on the label mapping similarity values.

[0025] In some embodiments, adjusting the model parameters of the initial ranking model based on the language audio sentiment tags, the music audio sentiment tags, the target language audio sample group, and the target music audio sample group to obtain a music audio ranking model includes:

[0026] Based on the target language audio sample group, the language emotion mapping vector, and the music emotion mapping vector, speech emotion loss is calculated to obtain the language emotion loss function;

[0027] Based on the target music audio sample group, the language emotion mapping vector, and the music emotion mapping vector, music emotion loss is calculated to obtain the music emotion loss function;

[0028] Based on the original language sample features, the original music sample features, the target language audio sample group, and the target music audio sample group, the association loss is calculated to obtain the association loss function;

[0029] The model loss function is obtained by weighting the language emotion loss function, the music emotion loss function, and the association loss function.

[0030] The model parameters of the feature extraction sub-model are adjusted based on the model loss function to obtain the music audio sorting model.

[0031] In some embodiments, the step of calculating speech emotion loss based on the target language audio sample group, the language emotion mapping vector, and the music emotion mapping vector to obtain a language emotion loss function includes:

[0032] The language emotion mapping vector and the music emotion mapping vector are concatenated to obtain the target emotion tag vector;

[0033] Based on the feature extraction sub-model, audio features are extracted from the positive samples of the target language audio to obtain the first associated language audio features;

[0034] Based on the feature extraction sub-model, audio features are extracted from the negative samples of the target language audio to obtain the second associated language audio features;

[0035] The language sentiment loss function is obtained by calculating triplet loss based on the target sentiment tag vector, the first associated language audio feature, and the second associated language audio feature.

[0036] In some embodiments, the feature extraction sub-model includes a feature extraction layer and a mapping layer. The step of extracting audio features from the original language audio samples based on the feature extraction sub-model to obtain the original language sample features includes:

[0037] Based on the feature extraction layer, the original language audio sample is subjected to audio encoding processing to obtain language audio encoding features;

[0038] Based on the mapping layer, feature mapping is performed on the language audio coding features to obtain the original language sample features.

[0039] In some embodiments, the music audio sorting model includes a feature extraction sub-model and a sorting sub-model. The step of inputting the target language audio and a preset music set into the music audio sorting model to sort the music and obtain a music sorting list includes:

[0040] Based on the feature extraction sub-model, audio features of the target language audio are extracted to obtain target language audio features;

[0041] Based on the feature extraction sub-model, audio features are extracted from the candidate music audios of the preset music set to obtain the target music audio features;

[0042] The target language audio features and the target music audio features are input into the ranking sub-model for similarity calculation to obtain candidate similarity values;

[0043] The candidate music audios are sorted based on the candidate similarity values ​​to obtain the music sorting list.

[0044] To achieve the above objectives, a second aspect of this application provides a music sorting apparatus, the apparatus comprising:

[0045] The acquisition module is used to acquire a training sample set, which includes a language audio subset and a music audio subset. The language audio subset includes original language audio samples, original positive language audio samples, original negative language audio samples, and language audio sentiment tags for the original language audio samples. The music audio subset includes original music audio samples, original positive music audio samples, original negative music audio samples, and music audio sentiment tags for the original music audio samples. The original language audio samples are used to represent language audio segments emitted by the sample objects, and the original music audio samples are used to represent music audio segments to be sorted.

[0046] A matrix construction module is used to construct an emotion similarity matrix based on the language audio emotion tags and the music audio emotion tags, thereby obtaining the emotion similarity matrix;

[0047] The first sample selection module is used to select samples from the original language audio positive samples and the original language audio negative samples based on the emotion similarity matrix, and determine the target language audio sample group, wherein the target language audio sample group includes target language audio positive samples and target language audio negative samples;

[0048] The second sample selection module is used to select samples from the original positive music audio samples and the original negative music audio samples based on the emotion similarity matrix, and determine the target music audio sample group, which includes target music audio positive samples and target music audio negative samples.

[0049] The parameter adjustment module is used to adjust the model parameters of the initial ranking model based on the language audio emotion tags, the music audio emotion tags, the target language audio sample group, and the target music audio sample group to obtain a music audio ranking model.

[0050] The music sorting module is used to acquire target language audio and input the target language audio and a preset music set into the music audio sorting model to sort the music and obtain a music sorting list.

[0051] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the music sorting method described in the first aspect.

[0052] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the music sorting method described in the first aspect.

[0053] This application proposes a music sorting method, apparatus, electronic device, and storage medium. It trains an initial sorting model using a training sample set including a subset of language audio and a subset of music audio to obtain a music audio sorting model. This model deeply integrates the emotional information of language audio with audio feature extraction. Specifically, a training sample set is obtained, comprising a subset of language audio and a subset of music audio. The language audio subset includes original language audio samples, original positive language audio samples, original negative language audio samples, and language audio emotional tags for the original language audio samples. The music audio subset includes original music audio samples, original positive music audio samples, original negative music audio samples, and music audio emotional tags for the original music audio samples. The original language audio samples represent language audio segments emitted by the sample object, and the original music audio samples represent music audio segments to be sorted. Further, an emotional similarity matrix is ​​constructed based on the language audio emotional tags and the music audio emotional tags to obtain an emotional similarity matrix. Based on the emotional similarity matrix, the original positive language audio samples and the original music audio samples are sorted. The process involves selecting samples from negative samples of the initial language audio to determine a target language audio sample group, which includes both positive and negative samples of the target language audio. Then, based on an emotion similarity matrix, it selects samples from the original positive and negative samples of the original music audio to determine a target music audio sample group, which also includes both positive and negative samples of the target music audio. Further, based on the emotion tags of the language audio, the emotion tags of the music audio, the target language audio sample group, and the target music audio sample group, the initial ranking model's parameters are adjusted to obtain a music audio ranking model. Subsequently, the target language audio is acquired, and the target language audio and a preset music set are input into the music audio ranking model for music ranking, resulting in a music ranking list. Therefore, this embodiment can effectively improve the accuracy of audio feature extraction, thereby improving the accuracy of music ranking. Attached Figure Description

[0054] Figure 1 This is a flowchart of a music sorting method provided in an embodiment of this application;

[0055] Figure 2 yes Figure 1 A flowchart of step S120 in the process;

[0056] Figure 3 This is a schematic diagram of the structure of the sentiment similarity matrix provided in this application;

[0057] Figure 4 yes Figure 1 A flowchart of step S130 in the process;

[0058] Figure 5This is a schematic diagram of the model structure of the initial sorting model provided in this application;

[0059] Figure 6 yes Figure 4 A flowchart of step S410 in the process;

[0060] Figure 7 yes Figure 1 A flowchart of step S150 in the process;

[0061] Figure 8 yes Figure 7 A flowchart of step S710 in the process;

[0062] Figure 9 yes Figure 1 A flowchart of step S160 in the process;

[0063] Figure 10 This is a schematic diagram of a music sorting device provided in an embodiment of this application;

[0064] Figure 11 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0068] First, let's analyze some of the terms used in this application:

[0069] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0070] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0071] Audio Emotion Recognition (AER) is a technology that identifies the emotional state of a speaker or listener by analyzing features in an audio signal. This technology combines knowledge from signal processing, machine learning, deep learning, and artificial intelligence to enable computers to automatically detect and understand emotional information in audio.

[0072] Valence / Arousal mapping (V / A mapping) refers to the process of mapping emotional states (such as happiness, sadness, etc.) to a two-dimensional space. V / A mapping can transform emotional states into two dimensions, Valence and Arousal, and each dimension has a corresponding mapping value.

[0073] Triplet loss is a loss function applicable to deep learning models that allows for more detailed modeling. The "three" in a triplet refers to the anchor sample, the positive sample, and the negative sample. The distance between the positive and negative samples can be controlled by changing the specific value of the threshold (margin), thus adapting to different model training needs.

[0074] Music sorting is a process of searching through a large amount of music resources for music audio that matches a specific theme, style, or characteristic, and then sorting the results. The music sorting method in this application can be better applied to scenarios such as intelligent diagnosis and treatment, and remote consultation. In the music therapy scenario of digital healthcare, music sorting can identify music audio that matches the patient's emotions, thus providing better support for treatment. Specifically, in actual music-assisted therapy scenarios, music retrieval and sorting can be performed using the emotional type expressed in the patient's speech fragments. For example, if the patient's current speech fragment is "It's raining today, and I'm not in a good mood," then music retrieval and sorting can be performed using the emotional type expressed in the patient's speech fragment to obtain music that matches the emotional type contained in that speech fragment. This can help patients strengthen their current emotions, better express their true thoughts, and thus better assist in emotional management, providing better intelligent support for music therapy scenarios.

[0075] However, cross-domain matching methods combining speech emotion recognition and music ranking are very time-consuming and require domain expertise. Currently, related technologies typically convert data samples from different modalities into a joint embedding space, which is then optimized to align the data distribution of heterogeneous modalities. However, speech (i.e., the audio of speech as described in the embodiments of this application) and music (i.e., the audio of music as described in the embodiments of this application) cannot organically coexist; that is, speech data and music data are two separate forms. For example, music generally does not contain spoken voices, except in very rare cases, such as live concert music (where there may be spoken voices from singers and fans greeting each other), or some rap music, which may mix speech and music. Therefore, related technologies typically lack organically paired datasets of speech and music. Thus, related technologies usually first use a large amount of paired data and learn a mapping function that minimizes the distance between paired data in the embedding space to train the ranking model. However, the expression of happiness in speech and the expression of happiness in music will have significant differences in feature representation, leading to insufficient accuracy in feature extraction and thus affecting the accuracy of music ranking.

[0076] Based on this, embodiments of this application provide a music sorting method and apparatus, an electronic device and a storage medium, which aim to improve the accuracy of audio feature extraction, thereby improving the accuracy of music sorting.

[0077] The music sorting method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the music sorting method in this application is described.

[0078] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0079] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0080] The music sorting method provided in this application relates to the field of artificial intelligence technology. The music sorting method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; the software can be an application implementing the music sorting method, but is not limited to the above forms.

[0081] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0082] It should be noted that in various specific embodiments of this application, when processing data related to the identity or characteristics of an object, such as object information, object language audio data, and object location information, the object's permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining sensitive personal information of an object, separate permission or consent from the object is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the object's separate permission or consent is the necessary object-related data required for the proper functioning of the embodiments of this application obtained.

[0083] Please see Figure 1 , Figure 1 This is an optional flowchart of the music sorting method provided in the embodiments of this application. In some embodiments, Figure 1 The method may include, but is not limited to, steps S110 to S160:

[0084] Step S110: Obtain the training sample set;

[0085] Step S120: Construct an emotion similarity matrix based on the emotional tags of language audio and music audio to obtain the emotion similarity matrix;

[0086] Step S130: Select samples from the original language audio positive samples and the original language audio negative samples based on the sentiment similarity matrix to determine the target language audio sample group;

[0087] Step S140: Select samples from the original positive and negative music audio samples based on the sentiment similarity matrix to determine the target music audio sample group;

[0088] Step S150: Adjust the model parameters of the initial ranking model based on the language audio emotion tags, music audio emotion tags, target language audio sample group, and target music audio sample group to obtain the music audio ranking model;

[0089] Step S160: Obtain the target language audio and input the target language audio and preset music set into the music audio sorting model to sort the music and obtain the music sorting list.

[0090] Steps S110 to S160 of this application embodiment involve constructing an emotion similarity matrix using language audio emotion tags and music audio emotion tags. This similarity matrix is ​​then used to determine more representative target language audio sample groups and target music audio sample groups from the training sample set. This process trains a music audio ranking model capable of more accurately extracting emotion-related features from language and music audio, and accurately ranking music audio segments based on the extracted features. Compared to related technologies that train models solely based on minimizing the distance between paired audio data, this application delves deeper into the correlation between emotion and audio features, improving the accuracy of audio feature extraction and thus enhancing the accuracy of music ranking.

[0091] In step S110 of some embodiments, during training, the original dataset can be divided into multiple batches, and each batch of data constitutes a training sample set. This training sample set may include a language audio subset and a music audio subset. The language audio subset records language audio segments emitted by sample objects, and the audio segments recorded in the language audio subset may originate from at least one sample object. In the music therapy scenario of digital healthcare, the audio segments recorded in the sample language audio subset can be audio segments in which patients express their state and thoughts to doctors. The music audio subset records multiple music audio segments stored in a preset music library, and the emotional type corresponding to each music audio segment has been determined. Each music audio sample recorded in the music audio subset may originate from audio segments of different candidate music in the preset music library.

[0092] It should be noted that the language audio subset can include multiple language audio samples, specifically including original language audio samples, original language audio positive samples, original language audio negative samples, and language audio sentiment labels for the original language audio samples. Original language audio samples refer to any language audio segment used for training in the current batch, representing the language audio segment emitted by the sample object. Language audio sentiment labels indicate the pre-labeled sentiment type corresponding to the original language audio sample. Preset sentiment types can include happy, excited, neutral, sad, angry, frustrated, etc., and the sentiment label is a numerical marker corresponding to each sentiment type. For example, the sentiment label for frustrated is 1, for angry it is 2, and for happy it is 6, etc., without specific limitations. Original language audio positive samples refer to language audio samples with the same sentiment labels as the original language audio samples, and original language audio negative samples refer to language audio samples with different sentiment labels than the original language audio samples. The language audio subset of this application can be derived from existing music datasets, such as the IEMOCAP dataset, historical music therapy databases, etc., without limitation.

[0093] It should be noted that the music audio subset can include multiple music audio samples, specifically including original music audio samples, original positive music audio samples, original negative music audio samples, and music audio sentiment labels for the original music audio samples. The original music audio sample refers to any unsorted music audio segment used for training in the current batch. The music audio sentiment label indicates the pre-labeled sentiment type corresponding to the original music audio sample, and the preset sentiment types can include happy, excited, funny, gentle, noisy, sad, angry, and scary, etc. The sentiment label is a numerical marker corresponding to each sentiment type; for example, the sentiment label for happy is 1, for excited it is 2, for scary it is 8, etc., without specific limitations. Original positive music audio samples are music audio samples with the same sentiment labels as the original music audio samples, and original negative music audio samples are music audio samples with different sentiment labels than the original music audio samples. The music audio subset in this application can come from an existing music dataset, such as the AudioSet dataset, a historical music therapy database, etc., without limitation. It should be noted that the preset emotion types corresponding to the emotional tags of music audio and the preset emotion types corresponding to the emotional tags of language audio can be the same or different, without limitation.

[0094] In step S120 of some embodiments, the sentiment similarity matrix is ​​a matrix used to quantify the degree of similarity between two or more sentiment expressions. Each element in the sentiment similarity matrix represents a similarity score between a pair of sentiment expressions. Embodiments of this application construct sentiment similarity matrices based on language audio sentiment tags and music audio sentiment tags, enabling in-depth learning of sentiment representations in two different modalities (i.e., language audio and music audio) to enhance the accuracy and robustness of feature extraction.

[0095] It should be noted that the embodiments of this application formalize the speech-to-music matching task into a supervised cross-domain retrieval and ranking problem, which makes better use of continuous audio emotion similarity information for model training and effectively improves the accuracy of audio feature extraction related to emotion.

[0096] Please see Figure 2 In some embodiments, step S120 may specifically include, but is not limited to, steps S210 to S260:

[0097] Step S210: Perform emotional space mapping on the language audio emotional tags to obtain language emotional mapping data;

[0098] Step S220: Perform emotional space mapping on the music audio emotional tags to obtain music emotional mapping data;

[0099] Step S230: Vectorize the language sentiment mapping data to obtain the language sentiment mapping vector;

[0100] Step S240: Vectorize the music emotion mapping data to obtain the music emotion mapping vector;

[0101] Step S250: Calculate the vector similarity between the language emotion mapping vector and the music emotion mapping vector to obtain the tag mapping similarity value;

[0102] Step S260: Construct the sentiment similarity matrix based on the label mapping similarity value to obtain the sentiment similarity matrix.

[0103] In steps S210 and S220 of some embodiments, since different modalities of data (such as language and music) have different specific categories for emotion classification, emotion space mapping can be performed on the language audio emotion tags first to unify the emotion categories of data from different modalities to the same spatial dimension. Language emotion mapping data refers to the mapping data representing language audio emotion tags in the emotion space. For example, this application can perform emotion space mapping on language audio emotion tags based on V / A mapping. The resulting language emotion mapping data is a VA value, and V / A mapping can be implemented using a pre-trained dimensional emotion model (Valence-Arousal-Dominance, VAD model, which uses two dimensions, Valence and Arousal, to represent emotion). In this model, Valence represents the positive or negative degree of emotion, while Arousal represents the arousal level of emotion. The dimensional emotion model treats emotion analysis as a regression problem, calculating continuous values ​​of Valence and Arousal by analyzing emotion words and contextual information in the text. In this application, the VA value can be a two-dimensional spatial coordinate. For example, if the emotional label of a language audio is "happy," and after V / A mapping, the language emotional mapping data can be (0.4, 0.2), where 0.4 represents the emotional value of the language audio emotional label of the current original language audio sample, and 0.2 represents the arousal level of the language audio emotional label of the current original language audio sample. Similarly, emotional spatial mapping is performed on the emotional label of music audio to obtain music emotional mapping data. For example, if the emotional label of a music audio is "sad," and after V / A mapping, the music emotional mapping data can be (0.3, 0.6), where 0.3 represents the emotional value of the music audio emotional label of the current original music audio sample, and 0.6 represents the arousal level of the music audio emotional label of the current original music audio sample.

[0104] In steps S230 and S240 of some embodiments, this application further vectorizes the language emotion mapping data to obtain language emotion mapping vectors, and vectorizes the music emotion mapping data to obtain music emotion mapping vectors. This improves code performance and reduces code complexity, thereby reducing algorithm computation time and increasing training speed. Therefore, this application can transform the original emotion labels from one form to another through V / A mapping, that is, unifying label types of different modalities into the same dimensional space, and further utilizes vectorization to optimize code execution, thus processing data more effectively and improving computational efficiency. For example, language emotion mapping data (0.4, 0.2) can be vectorized to (1, 2), and music emotion mapping data (0.3, 0.6) can be vectorized to (3, 4).

[0105] It should be noted that the vectorization process of music emotion mapping data and language emotion mapping data in this application can use Word2vec (a pre-trained model for natural language processing) to perform vector transformation to obtain language emotion mapping vectors and music emotion mapping vectors, or other vector transformation methods, such as one-hot encoding, can be used, which will not be elaborated here.

[0106] In step S250 of some embodiments, the label mapping similarity value refers to the numerical value obtained after calculating the vector similarity between the language emotion mapping vector and the music emotion mapping vector. The method for calculating vector similarity can be the cosine similarity function, Euclidean distance function, Hamming distance function, etc., and is not specifically limited.

[0107] In step S260 of some embodiments, for a training sample set corresponding to a batch of data, an emotion similarity matrix can be obtained, and the label mapping similarity value obtained by different combinations of language audio and music audio is recorded in this emotion similarity matrix.

[0108] It should be noted that, please refer to Figure 3 , Figure 3 This diagram illustrates the structure of an emotion similarity matrix. The horizontal axis represents the emotion tags of the music audio subset, and the vertical axis represents the emotion tags of the language audio subset. The intersections of different horizontal and vertical axes represent the tag similarity values ​​corresponding to different combinations of music and language emotion tags. The emotion types corresponding to language audio emotion tags can include happy, excited, neutral, sad, angry, and frustrated, while the emotion types corresponding to music audio emotion tags can include happy, excited, funny, gentle, noisy, sad, angry, and scary. Figure 3 This application uses different color depths to represent the numerical similarity between two labels, with darker colors indicating higher similarity. Furthermore, the application can calculate the final sentiment similarity matrix iteratively using training sample sets from multiple batches of data. Specifically, darker diagonal colors in the sentiment similarity matrix indicate a more accurate generated similarity matrix and better robustness of the corresponding feature representation. For example, the label mapping similarity value obtained by combining the language audio corresponding to the first sentiment type (e.g., happiness) and the music audio corresponding to the third sentiment type is represented by the value at the intersection of the first row and third column in the sentiment similarity matrix.

[0109] It should be noted that, as Figure 3 As shown, the same pair of input speech audio and music audio have similar emotional VA values, meaning they have a certain correspondence. For example, happy type corresponds to happy type, excited type corresponds to excited type, and depressed type corresponds to a certain degree of anger and a certain degree of fear.

[0110] In the above embodiments, this application connects emotion tags from different domains (i.e., speech audio and music audio) through V / A mapping, which facilitates the subsequent extraction of more emotion-related features, thereby improving the accuracy of audio feature extraction.

[0111] In step S130 of some embodiments, to better learn detailed features, this application can select samples from the original language audio positive samples and original language audio negative samples based on the sentiment similarity matrix during model training to determine a target language audio sample group. This target language audio sample group includes target language audio positive samples and target language audio negative samples. The target language audio positive samples refer to the samples selected from the original language audio positive samples that have the best feature similarity to the original language audio samples. This ensures that both samples have the same sentiment label type and that their detailed features are more similar, thereby improving the ability to extract audio features associated with sentiment.

[0112] Please see Figure 4 In some embodiments, step S130 may specifically include, but is not limited to, steps S410 to S450:

[0113] Step S410: Extract audio features from the original language audio samples based on the feature extraction sub-model to obtain the original language sample features;

[0114] Step S420: Extract audio features from the original music audio samples based on the feature extraction sub-model to obtain the features of the original music samples;

[0115] Step S430: Construct a feature similarity matrix based on the features of the original language samples and the features of the original music samples to obtain the feature similarity matrix;

[0116] Step S440: Select positive samples of the original language audio based on the feature similarity matrix and the sentiment similarity matrix to determine the positive samples of the target language audio sample group.

[0117] Step S450: Select negative samples of the original language audio based on the feature similarity matrix and the sentiment similarity matrix to determine the negative samples of the target language audio sample group.

[0118] It should be noted that you should refer to [link / reference]. Figure 5 , Figure 5A schematic diagram of the initial ranking model is shown. This initial ranking model includes a feature extraction sub-model 510, which comprises a language encoding unit 511 and a music encoding unit 512. The language encoding unit 511 is used to encode the input language audio, and the music encoding unit 512 is used to encode the input music audio. Specifically, the language encoding unit 511 can be based on Wav2vec (a self-supervised learning framework used to learn speech representations from audio segments, which can perform average pooling of phoneme-level features to aggregate these features into speech-level features, such as Wav2vec1.0, Wav2vec2.0, etc.), or other network architectures, without limitation. The music encoding unit 512 can be a music tagging model based on the Transformer architecture (Music Tagging Transformer, which can utilize a self-attention mechanism to process music data and achieve effective classification and tagging of music content), or other network architectures, without limitation.

[0119] In steps S410 to S430 of some embodiments, this application can extract audio features from the original language audio samples based on the language encoding unit 511 of the feature extraction sub-model 510 to obtain the original language sample features. Similarly, it can extract audio features from the original music audio samples based on the music encoding unit 512 of the feature extraction sub-model to obtain the original music sample features. Furthermore, a feature similarity matrix can be constructed based on the original language sample features and the original music sample features to obtain the feature similarity matrix. This means that the numerical values ​​at corresponding positions in the original language sample features and the original music sample features can be used to calculate the feature similarity. The method for calculating the feature similarity can be a cosine similarity function, a Euclidean distance function, a Hamming distance function, etc., without specific limitations.

[0120] It should be noted that the feature similarity matrix obtained in this application has the same matrix size as the sentiment similarity matrix, and the corresponding label combinations at corresponding positions in the two matrices are the same, which will not be elaborated further.

[0121] In steps S440 and S450 of some embodiments, after obtaining the feature similarity matrix and the sentiment similarity matrix, the original language audio sample can be used as the anchor language sample. Based on the feature similarity matrix and the sentiment similarity matrix, the sample with the same label as the original language audio sample and the highest feature similarity value is determined as the positive sample of the target language audio. Furthermore, based on the feature similarity matrix and the sentiment similarity matrix, the sample with the different label from the original language audio sample and the lowest feature similarity value is determined as the negative sample of the target language audio.

[0122] Therefore, by combining the feature similarity matrix and the sentiment similarity matrix, this application can deeply learn the sentiment representation and feature representation in two different modalities (i.e., language audio and music audio), and select the most representative negative and positive samples of the target language audio by combining triples, thereby enhancing the accuracy and robustness of feature extraction in subsequent training.

[0123] Please see Figure 6 In some embodiments, step S410 may specifically include, but is not limited to, steps S610 to S620:

[0124] Step S610: Perform audio encoding processing on the original language audio samples based on the feature extraction layer to obtain language audio encoding features;

[0125] Step S620: Based on the mapping layer, feature mapping is performed on the language audio coding features to obtain the original language sample features.

[0126] It should be noted that, as Figure 5 As shown, the language coding unit 511 and music coding unit 512 of the feature extraction sub-model 510 each include their respective feature extraction layer and mapping layer, which are used to extract features of different modalities.

[0127] In steps S610 and S620 of some embodiments, in the language encoding unit 511, audio encoding processing can be performed on the original language audio sample based on the corresponding feature extraction layer to obtain language audio encoding features. These language audio encoding features can contain useful information from the original language audio sample, such as pitch, tone, and sound intensity, which is helpful for subsequent analysis and processing. Furthermore, feature mapping can be performed on the language audio encoding features based on the mapping layer to obtain the original language sample features. The purpose of this feature mapping is to transform the audio encoding features into a form more suitable for subsequent processing and analysis.

[0128] It should be noted that the mapping layer in this application can be a multilayer perceptron (MLP) or a structure constructed by other mapping processing methods, which will not be elaborated further.

[0129] In the above embodiments, this application processes the input audio data through a feature extraction layer and a mapping layer, which helps the model better adapt to different audio data, improves the model's generalization ability, and effectively reduces the amount of data, thereby improving the efficiency of subsequent processing and analysis.

[0130] It should be noted that in the music encoding unit 512, the original music audio samples can be processed by audio encoding based on the corresponding feature extraction layer to obtain music audio encoding features. Furthermore, by performing feature mapping on the music audio encoding features based on the mapping layer, the original music sample features can be obtained.

[0131] In step S140 of some embodiments, sample selection is performed on the original positive and negative music audio samples based on the emotion similarity matrix to determine a target music audio sample group, which includes target music audio positive and target music audio negative samples. The specific process for determining the target music audio sample group can be found in steps S410 to S450 above, where the parameters related to language audio are replaced with parameters related to music audio. For brevity, this will not be elaborated further here.

[0132] In step S150 of some embodiments, the model parameters of the initial ranking model are adjusted based on the language audio sentiment tags, music audio sentiment tags, target language audio sample groups, and target music audio sample groups to obtain a music audio ranking model. The music audio ranking model of this application refers to a model that can rank input music audio based on the sentiment type of the input language audio, and this music audio ranking model includes a feature extraction sub-model and a ranking sub-model with adjusted parameters.

[0133] Please see Figure 7 In some embodiments, step S150 may specifically include, but is not limited to, steps S710 to S750:

[0134] Step S710: Based on the target language audio sample group, language emotion mapping vector, and music emotion mapping vector, calculate the speech emotion loss to obtain the language emotion loss function;

[0135] Step S720: Based on the target music audio sample group, language emotion mapping vector, and music emotion mapping vector, calculate the music emotion loss to obtain the music emotion loss function;

[0136] Step S730: Calculate the association loss based on the original language sample features, the original music sample features, the target language audio sample group, and the target music audio sample group to obtain the association loss function;

[0137] Step S740: The language emotion loss function, music emotion loss function, and association loss function are weighted and calculated to obtain the model loss function;

[0138] Step S750: Adjust the model parameters of the feature extraction sub-model based on the model loss function to obtain the music audio ranking model.

[0139] In step S710 of some embodiments, the language emotion loss function is used to characterize the influence relationship between emotion tags and language audio features.

[0140] Please see Figure 8 In some embodiments, step S710 may specifically include, but is not limited to, steps S810 to S840:

[0141] Step S810: Concatenate the language emotion mapping vector and the music emotion mapping vector to obtain the target emotion tag vector;

[0142] Step S820: Based on the feature extraction sub-model, audio features are extracted from the positive samples of the target language audio to obtain the first associated language audio features;

[0143] Step S830: Based on the feature extraction sub-model, audio features are extracted from the negative audio samples of the target language to obtain the second associated language audio features;

[0144] Step S840: Based on the target sentiment tag vector, the first associated language audio features, and the second associated language audio features, triplet loss is calculated to obtain the language sentiment loss function.

[0145] In steps S810 to S840 of some embodiments, this application can construct its own loss function based on a triplet loss function, that is, optimize the loss function from emotion tags to speech by embedding emotion tags. Specifically, the obtained language emotion mapping vector and music emotion mapping vector can be concatenated to obtain the target emotion tag vector. Further, audio features can be extracted from positive samples of target language audio based on the language encoding unit of the feature extraction sub-model to obtain the first associated language audio features, and audio features can be extracted from negative samples of target language audio based on the language encoding unit of the feature extraction sub-model to obtain the second associated language audio features. Then, triplet loss is calculated based on the target emotion tag vector, the first associated language audio features, and the second associated language audio features to obtain the language emotion loss function.

[0146] It should be noted that the triplet loss function is shown in Formula 1 below. The specific process of calculating the language sentiment loss function based on the target sentiment tag vector, the first associated language audio feature, and the second associated language audio feature can be seen in Formula 2 below:

[0147] L Triplet (z,z + ,z - )=max{0,D(z,z+ )-D(z,z - )+δ} (Formula 1)

[0148]

[0149] In Formula 1, z represents the anchored sample, z + Let z represent a positive sample. - L represents a negative sample. Triplet (·) represents the triplet loss function, D(·) represents the set distance function, such as the cosine distance function, and δ represents the predefined margin value, such as 0.1, 0.5, etc., which can be a user-defined value without limitation. In Formula 2, L SP-Speech Let z represent the language sentiment loss function. e Represents the target sentiment tag vector. Indicates the first associated language audio features, This represents the audio features of the second associated language. Thus, by using L... Triplet By replacing the three parameters in (·), we obtain the language emotion loss function.

[0150] In step S720 of some embodiments, the music emotion loss function is used to characterize the influence relationship between emotion tags and music audio features. Specifically, this application can extract audio features from positive samples of target music audio based on the music encoding unit of the feature extraction sub-model to obtain first associated music audio features, and extract audio features from negative samples of target music audio based on the music encoding unit of the feature extraction sub-model to obtain second associated music audio features. Then, a triplet loss is calculated based on the target emotion tag vector, the first associated music audio features, and the second associated music audio features to obtain the music emotion loss function. Therefore, the specific process by which this application calculates the music emotion loss function based on the target music audio sample group, the language emotion mapping vector, and the music emotion mapping vector can be seen in the following formula 3:

[0151]

[0152] In formula 3, L SP-Music Let z represent the music emotion loss function. e Represents the target sentiment tag vector. Indicates the first associated music audio feature, This represents the second associated musical audio feature. Thus, by using L... Triplet By replacing the three parameters in (·), we obtain the music emotion loss function.

[0153] In the above embodiments, compared with the related technologies that calculate the loss function based on the characteristics of the language audio itself and the corresponding positive and negative samples, this application can learn the feature details of emotion to language and emotion to music by combining the features of embedded emotion tags, thereby improving the accuracy of feature extraction of audio segments.

[0154] In step S730 of some embodiments, the correlation between language audio and music audio features is further considered. This application can also calculate the correlation loss based on the original language sample features, the original music sample features, the target language audio sample group and the target music audio sample group to obtain the correlation loss function.

[0155] It should be noted that the association loss is calculated based on the features of the original language samples, the features of the original music samples, the target language audio sample group, and the target music audio sample group to obtain the association loss function. Specifically, this may include: calculating the language-music association loss based on the features of the original language samples, positive samples of the target music audio, and negative samples of the target music audio to obtain the first association loss sub-function, and the specific calculation process is shown in Formula 4 below. Then, calculating the speech-music association loss based on the features of the original music samples, positive samples of the target language audio, and negative samples of the target language audio to obtain the second association loss sub-function, and the specific calculation process is shown in Formula 5 below. Further, the first association loss sub-function and the second association loss sub-function are weighted and calculated to obtain the association loss function, and the specific calculation process is shown in Formula 6 below.

[0156]

[0157] L cross0 =α1L cross1 +α2L cross2 (Formula 6)

[0158] Among them, L cross1 Let z represent the first association loss function. s L represents the features of the original language sample. cross2 Let z represent the second association loss function. m L represents the features of the original music sample. cross0 Let α1 represent the weight of the first association loss function and α2 represent the weight of the second association loss function. The sum of the weights of α1 and α2 is 1. For example, α1 is 1 and α2 is 0, or both α1 and α2 are 0.5. They can be flexibly adjusted according to actual needs, which will not be elaborated further.

[0159] In steps S740 and S750 of some embodiments, the language emotion loss function, music emotion loss function, and association loss function are weighted and calculated to obtain the model loss function. Based on the model loss function, the model parameters of the feature extraction sub-model are adjusted until the model loss value meets a preset training termination condition. The adjusted feature extraction sub-model and ranking sub-model are then used to construct a music audio ranking model. The preset training termination condition can be when the model loss value is less than a preset loss value threshold, or when all batches of training samples have been used; it is not limited to these conditions.

[0160] In the above embodiments, this application constructs the final model loss function by optimizing the triplet loss function. That is, by combining the features of the embedded emotion tags, the triplet loss from emotion to speech and from emotion to music can be optimized respectively, which can learn more feature details and improve the accuracy of feature extraction from audio segments.

[0161] It should be noted that the calculation process of the model loss function can be seen in Equation 7 below:

[0162] L 模 =ω1L SP-Speech +ω2L SP-Music +ω3L cross0 (Formula 7)

[0163] Among them, L 模 Let ω1 represent the weight of the language emotion loss function, ω2 represent the weight of the music emotion loss function, and ω3 represent the weight of the association loss function. The sum of the weights of ω1, ω2, and ω3 is 1, such as ω1 being 0.5, ω2 being 0.3, and ω3 being 0.2. Furthermore, ω1, ω2, and ω3 are hyperparameters that can be flexibly adjusted according to actual needs, and will not be elaborated further.

[0164] In the above embodiments, by optimizing the model training process and the loss function, this application can still retrieve music audio that matches the speaker's emotion even without linguistic information (such as speech with matching words).

[0165] In step S160 of some embodiments, after obtaining the music audio sorting model, this application can input the target language audio and a preset music set into the music audio sorting model to sort the music and obtain a music sorting list. The preset music set can be automatically input into the model via a network connection, thereby achieving audio matching and sorting of the target language audio.

[0166] It should be noted that this application can also be used in music software recommendations and information retrieval. Specifically, based on the input of target language audio by the target user, music audio matching the emotional type of that target language audio can be retrieved. The target language audio can be the target user's own voiceprint or the voiceprint of another user input by the target user.

[0167] Please see Figure 9 In some embodiments, step S160 may specifically include, but is not limited to, steps S910 to S940:

[0168] Step S910: Extract audio features from the target language audio based on the feature extraction sub-model to obtain the target language audio features;

[0169] Step S920: Based on the feature extraction sub-model, audio features are extracted from the candidate music audios of the preset music set to obtain the target music audio features;

[0170] Step S930: Input the target language audio features and the target music audio features into the ranking sub-model to calculate the similarity and obtain the candidate similarity values;

[0171] Step S940: Sort the candidate music audios based on the candidate similarity values ​​to obtain a music sorting list.

[0172] In steps S910 to S940 of some embodiments, audio features can be extracted from the target language audio based on a feature extraction sub-model to obtain target language audio features. Audio features can also be extracted from each candidate music audio in a preset music set based on the feature extraction sub-model to obtain target music audio features. Further, the target language audio features and each target music audio feature can be input into a sorting sub-model for similarity calculation to obtain multiple candidate similarity values. Then, the candidate music audios can be sorted in descending or ascending order based on the candidate similarity values ​​to obtain a music sorting list. For example, if the candidate music audios are sorted in descending order, the first music audio in the music sorting list is the music segment that best matches the emotional features of the target language audio. Furthermore, in music retrieval experiments in specific application scenarios, this application can improve the retrieval recall hit rate by 13%.

[0173] This application provides a music ranking method that constructs an emotion similarity matrix based on language audio emotion tags and music audio emotion tags, enabling deep learning of emotion representations in two different modalities (i.e., language audio and music audio). By combining the feature similarity matrix and the emotion similarity matrix, it learns both emotion and feature representations in the two modalities (i.e., language audio and music audio), and selects the most representative negative and positive samples of the target language audio using a triplet approach, thereby enhancing the accuracy and robustness of feature extraction during subsequent training. Furthermore, this application can optimize the emotion-to-speech and emotion-to-music triplet losses by combining the features of the embedded emotion tags. Compared to related techniques that train models solely based on minimizing the distance between paired audio data, this application can deeply mine the correlation information between emotion and audio features, improving the accuracy of audio feature extraction and thus enhancing the accuracy of music ranking.

[0174] Please see Figure 10 This application also provides a music sorting apparatus that can implement the above-described music sorting method. The apparatus includes:

[0175] The acquisition module 1010 is used to acquire a training sample set, which includes a language audio subset and a music audio subset. The language audio subset includes original language audio samples, original language audio positive samples, original language audio negative samples, and language audio sentiment labels of the original language audio samples. The music audio subset includes original music audio samples, original music audio positive samples, original music audio negative samples, and music audio sentiment labels of the original music audio samples. The original language audio samples are used to represent the language audio segments emitted by the sample objects, and the original music audio samples are used to represent the music audio segments to be sorted.

[0176] The matrix construction module 1020 is used to construct an emotion similarity matrix based on language audio emotion tags and music audio emotion tags to obtain the emotion similarity matrix;

[0177] The first sample selection module 1030 is used to select samples from the positive and negative samples of the original language audio based on the sentiment similarity matrix, and to determine the target language audio sample group, which includes the positive and negative samples of the target language audio.

[0178] The second sample selection module 1040 is used to select samples from the original music audio positive samples and the original music audio negative samples based on the emotion similarity matrix, and determine the target music audio sample group, which includes the target music audio positive samples and the target music audio negative samples.

[0179] The parameter adjustment module 1050 is used to adjust the model parameters of the initial ranking model based on the language audio emotion tags, music audio emotion tags, target language audio sample groups, and target music audio sample groups to obtain a music audio ranking model.

[0180] The music sorting module 1060 is used to acquire target language audio and input the target language audio and preset music set into the music audio sorting model to sort the music and obtain a music sorting list.

[0181] The specific implementation of this music sorting device is basically the same as the specific implementation of the music sorting method described above, and will not be repeated here.

[0182] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned music sorting method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0183] Please see Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0184] The processor 1110 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0185] The memory 1120 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1120 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1120 and called and executed by the processor 1110 using the music sorting method of the embodiments of this application.

[0186] The input / output interface 1130 is used to implement information input and output;

[0187] The communication interface 1140 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0188] Bus 1150 transmits information between various components of the device (e.g., processor 1110, memory 1120, input / output interface 1130, and communication interface 1140);

[0189] The processor 1110, memory 1120, input / output interface 1130 and communication interface 1140 are connected to each other within the device via bus 1150.

[0190] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described music sorting method.

[0191] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0192] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0193] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0194] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0195] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0196] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0197] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0199] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0200] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0201] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0202] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A music sorting method, characterized in that: The method comprises: Acquire a training sample set, the training sample set comprising a language audio subset and a music audio subset; wherein the language audio subset comprises an original language audio sample, an original language audio positive sample, an original language audio negative sample, and a language audio emotion label of the original language audio sample, the music audio subset comprises an original music audio sample, an original music audio positive sample, an original music audio negative sample, and a music audio emotion label of the original music audio sample, the original language audio sample is used to characterize a language audio segment emitted by a sample object, and the original music audio sample is used to characterize a music audio segment to be sorted; Constructing an emotion similarity matrix based on the language audio emotion label and the music audio emotion label to obtain the emotion similarity matrix; Based on the emotion similarity matrix, sample selection is performed on the original language audio positive sample and the original language audio negative sample to determine a target language audio sample group, wherein the target language audio sample group includes a target language audio positive sample and a target language audio negative sample; Based on the emotion similarity matrix, sample selection is performed on the original music audio positive sample and the original music audio negative sample to determine a target music audio sample group, wherein the target music audio sample group includes a target music audio positive sample and a target music audio negative sample; Adjusting the model parameters of the initial sorting model based on the language audio emotion label, the music audio emotion label, the target language audio sample group, and the target music audio sample group to obtain a music audio sorting model; The target language audio is obtained, and the target language audio and a preset music set are input into the music audio sorting model to sort the music, so as to obtain a music sorting list.

2. The method according to claim 1, characterized in that The initial sorting model includes a feature extraction sub-model, and the sample selection of the original language audio positive sample and the original language audio negative sample based on the emotion similarity matrix to determine the target language audio sample group includes: Extracting audio features from the original language audio sample based on the feature extraction sub-model to obtain features of the original language sample; Performing audio feature extraction on the original music audio sample based on the feature extraction sub-model to obtain features of the original music sample; Constructing a feature similarity matrix based on the original language sample features and the original music sample features to obtain a feature similarity matrix; Performing sample selection on the original language audio positive samples based on the feature similarity matrix and the emotion similarity matrix to determine the target language audio positive samples of the target language audio sample group; The original language audio negative samples are selected based on the feature similarity matrix and the emotion similarity matrix to determine the target language audio negative samples of the target language audio sample group.

3. The method according to claim 2, characterized in that The constructing of the emotion similarity matrix based on the language audio emotion tag and the music audio emotion tag to obtain the emotion similarity matrix includes: Performing emotion space mapping on the language audio emotion tag to obtain language emotion mapping data; Performing emotional space mapping on the music audio emotion tags to obtain music emotion mapping data; Vectorizing the language emotion mapping data to obtain a language emotion mapping vector; Vectorizing the music emotion mapping data to obtain a music emotion mapping vector; Performing vector similarity calculation on the language emotion mapping vector and the music emotion mapping vector to obtain a label mapping similarity value; A sentiment similarity matrix is ​​constructed based on the label mapping similarity values ​​to obtain the sentiment similarity matrix.

4. The method according to claim 3, characterized in that The method of adjusting the model parameters of the initial sorting model based on the language audio emotion label, the music audio emotion label, the target language audio sample group, and the target music audio sample group to obtain the music audio sorting model includes: Perform speech emotion loss calculation based on the target language audio sample group, the language emotion mapping vector, and the music emotion mapping vector to obtain a language emotion loss function; Performing music emotion loss calculation based on the target music audio sample group, the language emotion mapping vector, and the music emotion mapping vector to obtain a music emotion loss function; Calculate the association loss based on the original language sample features, the original music sample features, the target language audio sample group and the target music audio sample group to obtain an association loss function; Performing weighted calculation on the language emotion loss function, the music emotion loss function and the association loss function to obtain a model loss function; The model parameters of the feature extraction sub-model are adjusted based on the model loss function to obtain the music audio sorting model.

5. The method according to claim 4, characterized in that The performing speech emotion loss calculation based on the target language audio sample group, the language emotion mapping vector, and the music emotion mapping vector to obtain a language emotion loss function includes: Performing vector concatenation on the language emotion mapping vector and the music emotion mapping vector to obtain a target emotion label vector; Performing audio feature extraction on the target language audio positive sample based on the feature extraction sub-model to obtain a first associated language audio feature; Extracting audio features from the target language audio negative sample based on the feature extraction sub-model to obtain a second associated language audio feature; A triplet loss calculation is performed based on the target emotion label vector, the first associated language audio feature, and the second associated language audio feature to obtain the language emotion loss function.

6. The method according to any one of claims 2 to 5, characterized in that The feature extraction sub-model includes a feature extraction layer and a mapping layer. The audio feature extraction is performed on the original language audio sample based on the feature extraction sub-model to obtain the original language sample feature, including: Performing audio coding processing on the original language audio sample based on the feature extraction layer to obtain language audio coding features; Feature mapping is performed on the language audio coding features based on the mapping layer to obtain the original language sample features.

7. The method according to claim 1, characterized in that The music audio sorting model includes a feature extraction sub-model and a sorting sub-model. The target language audio and the preset music set are input into the music audio sorting model to sort the music to obtain a music sorting list, including: Performing audio feature extraction on the target language audio based on the feature extraction sub-model to obtain target language audio features; Extract audio features from candidate music audio of the preset music set based on the feature extraction sub-model to obtain target music audio features; Inputting the target language audio feature and the target music audio feature into the sorting sub-model to perform similarity calculation to obtain a candidate similarity value; The candidate music audios are sorted based on the candidate similarity values ​​to obtain the music sorting list.

8. A music sorting device, characterized in that: The device comprises: An acquisition module is used to acquire a training sample set, wherein the training sample set includes a language audio subset and a music audio subset; wherein the language audio subset includes an original language audio sample, an original language audio positive sample, an original language audio negative sample, and a language audio emotion label of the original language audio sample; the music audio subset includes an original music audio sample, an original music audio positive sample, an original music audio negative sample, and a music audio emotion label of the original music audio sample; the original language audio sample is used to characterize a language audio segment emitted by a sample object; and the original music audio sample is used to characterize a music audio segment to be sorted; A matrix construction module, used for constructing an emotion similarity matrix based on the language audio emotion label and the music audio emotion label to obtain the emotion similarity matrix; A first sample selection module is used to perform sample selection on the original language audio positive sample and the original language audio negative sample based on the emotion similarity matrix to determine a target language audio sample group, wherein the target language audio sample group includes a target language audio positive sample and a target language audio negative sample; A second sample selection module is used to perform sample selection on the original music audio positive sample and the original music audio negative sample based on the emotion similarity matrix to determine a target music audio sample group, wherein the target music audio sample group includes a target music audio positive sample and a target music audio negative sample; A parameter adjustment module, used to adjust the model parameters of the initial sorting model based on the language audio emotion label, the music audio emotion label, the target language audio sample group and the target music audio sample group to obtain a music audio sorting model; The music sorting module is used to obtain target language audio, and input the target language audio and a preset music set into the music audio sorting model to sort the music, so as to obtain a music sorting list.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements a music sorting method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a music sorting method as described in any one of claims 1 to 7 is implemented.

Images