Music search method, multi-modal fusion model training method, device, storage medium and program product

Abstract

The application relates to a music search method, a training method of a multi-modal fusion model, equipment, a storage medium and a program product. The method comprises the following steps: in response to a music search request, obtaining music query information corresponding to the music search request; identifying the information modal type of the music query information; converting the music query information into a corresponding music query vector through a feature extraction model corresponding to the information modal type; searching for a target song vector with a similarity to the music query vector meeting a preset condition in a song vector database; the song vector database comprises a song vector corresponding to each song in a preset song library; each song vector is obtained by fusing audio features, text features and visual features of a corresponding song by a pre-trained multi-modal fusion model; and generating a music search result corresponding to the music search request according to a song resource corresponding to the target song vector. The method can improve the accuracy of music search.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a music search method, a training method for a multimodal fusion model, an apparatus, a storage medium, and a program product. Background Technology

[0002] With the rapid development of internet and digital multimedia technologies, online music platforms have become the primary channel for accessing music resources. Massive amounts of music media assets (hereinafter referred to as media assets) are stored on cloud servers, including various forms of data such as audio files, lyrics, album art, and music videos. How to quickly and accurately retrieve target music from this massive amount of data is the core problem faced by music retrieval systems.

[0003] In traditional technology, music search mainly relies on single text tags such as song title, artist, and album for indexing and searching, which greatly limits the understanding of music content and results in low accuracy of music search. Summary of the Invention

[0004] Therefore, it is necessary to provide a music search method, a training method for a multimodal fusion model, a computer device, a computer-readable storage medium, and a computer program product that can improve the accuracy of music search, in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a music search method, including:

[0006] In response to a music search request, obtain the music query information corresponding to the music search request;

[0007] The information modality type of the music query information is identified, and the music query information is converted into a corresponding music query vector through a feature extraction model corresponding to the information modality type;

[0008] In the song vector database, target song vectors whose similarity to the music query vector meets preset conditions are searched; wherein, the song vector database includes song vectors corresponding to each song in a preset song library; each song vector is obtained by fusing the audio features, text features and visual features of the corresponding song by a pre-trained multimodal fusion model; the audio features, text features and visual features of the same song are mapped to neighboring positions in the same semantic space by the pre-trained multimodal fusion model;

[0009] Based on the song resources corresponding to the target song vector, generate the music search results corresponding to the music search request.

[0010] In one embodiment, the pre-trained multimodal fusion model includes an audio feature encoder, a text feature encoder, and a visual feature encoder. Prior to the step of searching for target song vectors in the song vector database that have a similarity to the music query vector that meets preset conditions, the method further includes:

[0011] Obtain the multimodal features of each song in the preset song library; the multimodal features include at least the song's audio features, text features, and visual features;

[0012] For each song, the audio features of the song are input into the audio feature encoder to obtain an audio encoding vector; the text features of the song are input into the text feature encoder to obtain a text encoding vector; and the visual features of the song are input into the visual feature encoder to obtain a visual encoding vector; the audio encoding vector, the text encoding vector, and the visual encoding vector of the song are located in adjacent positions in the same semantic space.

[0013] The song vector of the song is obtained by fusing the audio encoding vector, the text encoding vector, and the visual encoding vector.

[0014] In one embodiment, when the music query information corresponding to the music search request includes multiple different types of music query information, the information modality type identified from the music query information includes multiple different types of information modality types; the step of converting the music query information into a corresponding music query vector using a feature extraction model corresponding to the information modality type includes:

[0015] The music query information is converted into modal feature vectors corresponding to each of the aforementioned information modal types using feature extraction models.

[0016] The modal feature vectors corresponding to each of the aforementioned information modal types are input into the pre-trained multimodal fusion model to obtain the music query vector; the music query vector is obtained by the pre-trained multimodal fusion model fusing the modal feature vectors mapped to the same semantic space.

[0017] In one embodiment, generating music search results corresponding to the music search request based on the song resources corresponding to the target song vector includes:

[0018] Identify the search intent of the music query information and determine the music attributes corresponding to the search intent;

[0019] From the target song vectors, select the target song vectors whose song attributes match the music attributes;

[0020] Based on the song resources corresponding to the selected target song vector, generate the music search results corresponding to the music search request.

[0021] In one embodiment, searching a song vector database for a target song vector whose similarity to the music query vector meets preset conditions includes:

[0022] Calculate the similarity between each song vector in the song vector database and the music query vector;

[0023] The song vectors are sorted in descending order according to their similarity. The top N song vectors are selected as the target song vectors that meet the preset conditions, where N is a preset number.

[0024] Secondly, this application provides a training method for a multimodal fusion model, including:

[0025] Obtain a training sample set; the training sample set includes positive samples obtained by pairing at least two modal features from the multimodal features of the same song, and negative samples obtained by combining any of the modal features of different songs; the modal features are one of audio features, text features, and visual features;

[0026] The training sample set is input into the multimodal fusion model to be trained. Based on the difference between the positive and negative samples, the multimodal fusion model is trained to obtain a pre-trained multimodal fusion model. The pre-trained multimodal fusion model is used to map the audio features, text features, and visual features of each song in the preset song library to neighboring positions in the same semantic space to obtain a song vector for each song. The song vector is used to calculate the similarity between it and the music query vector to determine the target song vector that meets the preset conditions. The song resources corresponding to the target song vector are used to generate music search results corresponding to the music search request. The music query vector is obtained by identifying the information modality type of the music query information corresponding to the music search request and transforming the music query information using the feature extraction model corresponding to the information modality type.

[0027] In one embodiment, training the multimodal fusion model to be trained based on the difference between the positive samples and the negative samples includes:

[0028] Each modal feature in the positive sample is input into the encoder corresponding to the corresponding modality to obtain the positive sample coding feature of each modality; and each modal feature in the negative sample is input into the encoder corresponding to the corresponding modality to obtain the negative sample coding feature of each modality.

[0029] Calculate the similarity between the positive sample coding features of each modality to obtain the positive sample similarity; and calculate the similarity between the negative sample coding features of each modality to obtain the negative sample similarity.

[0030] The multimodal fusion model to be trained is trained based on the difference between the positive sample similarity and the negative sample similarity.

[0031] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0032] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0033] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0034] The aforementioned music search method, multimodal fusion model training method, computer equipment, computer-readable storage medium, and computer program product, respond to music search requests and identify the information modality type of the music query information corresponding to the music search request. Through a feature extraction model corresponding to the information modality type, the music query information is converted into a corresponding music query vector. This vector is then matched with the similarity of each song vector in a song vector database. The song vector is obtained by a pre-trained multimodal fusion model fusing heterogeneous features such as audio, text, and visual features of the corresponding song and mapping them to neighboring positions in the same semantic space. By constructing a unified semantic space, heterogeneous features of different modalities can be mapped to a unified semantic space for measurement and comparison. This allows the emotion of the melody, the artistic conception of the lyrics, and the style of the album cover to be associated and understood, improving the comprehensiveness of the understanding of music content. This enables the understanding of users' more abstract and emotional music search requests. Users can search for music based on a melody, an emotional description, or a visual style. By calculating vector similarity in a unified semantic space, music content that meets the user's deeper needs is accurately recalled, greatly improving the accuracy and personalized experience of music search. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a diagram illustrating the application environment of a music search method in one embodiment.

[0037] Figure 2 This is a flowchart illustrating a music search method in one embodiment;

[0038] Figure 3 This is a structural diagram of a multimodal fusion model in one embodiment;

[0039] Figure 4 This is a logic diagram of an online music search method in one embodiment;

[0040] Figure 5 This is a flowchart illustrating a training method for a multimodal fusion model in one embodiment;

[0041] Figure 6 This is a flowchart illustrating a music search method in another embodiment;

[0042] Figure 7 This is a structural block diagram of a music search device in one embodiment;

[0043] Figure 8 This is a structural block diagram of a training device for a multimodal fusion model in one embodiment;

[0044] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] In related technologies, most music search methods rely solely on single text tags such as song title, artist, and album for indexing and searching, ignoring the rich information contained in the music itself, such as melody, rhythm, emotional atmosphere, and visual elements such as album covers, resulting in a very limited understanding of the music content.

[0047] Furthermore, due to a limited understanding, current music search methods are limited to keyword matching. Users cannot search in a more emotional or abstract way (such as "a piano piece that sounds very sad" or "music similar to this album cover"), resulting in search results that often do not match the user's true intent and have low accuracy.

[0048] Furthermore, information in different modalities such as audio, text, and images is fragmented, and there is a lack of effective technical means to link them together. For example, current music search methods cannot semantically connect an exciting melody with powerful lyrics and a vividly colored album cover, creating data silos.

[0049] The music search method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0050] In practical applications, server 104 responds to music search requests by obtaining music query information corresponding to the music search request; server 104 identifies the information modality type of the music query information and converts the music query information into a corresponding music query vector through a feature extraction model corresponding to the information modality type; server 104 searches for target song vectors in the song vector database whose similarity to the music query vector meets preset conditions; wherein, the song vector database includes song vectors corresponding to each song in the preset song library; each song vector is obtained by fusing the audio features, text features, and visual features of the corresponding song by a pre-trained multimodal fusion model; the audio features, text features, and visual features of the same song are mapped to neighboring positions in the same semantic space by the pre-trained multimodal fusion model; server 104 generates music search results corresponding to the music search request based on the song resources corresponding to the target song vector.

[0051] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses.

[0052] Among them, server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0053] In one exemplary embodiment, such as Figure 2 As shown, a music search method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes:

[0054] Step S202: In response to the music search request, obtain the music query information corresponding to the music search request.

[0055] A music search request refers to a user-initiated request to search for music. In practice, users can enter music search information in the music application on their device, such as entering text, uploading images, or recording audio in the search box on the music search page. The device encapsulates the user's music search information into a music search request and sends it to the server over the network. The server responds to the music search request by retrieving the corresponding music search information.

[0056] Music query information refers to the data content submitted by users to describe their music search intent. This music query information can be a combination of one or more modalities, including text information, audio information, and visual information. Optionally, text information can include lyrics, song titles, artists, album descriptions, user comments, etc.; audio information can include background music, vocals, etc.; and visual information can include album covers, music video screenshots, artist photos, etc.

[0057] For example, when music query information includes text information, it can include text descriptions entered by the user, such as song titles, artist names, lyrics snippets, genre descriptions, and emotional descriptions, such as user-entered descriptions like "upbeat electronic music suitable for listening to at the beach" or "sad piano music." When music query information includes audio information, it can include audio snippets uploaded or recorded by the user, such as melodies hummed by the user or music clips being played. When music query information includes visual information, it can include images uploaded by the user, such as album cover images, music video screenshots, and art-style images.

[0058] In one embodiment, music query information can simultaneously include information from multiple modalities. For example, a user can simultaneously input text descriptions and audio clips, or simultaneously upload images and input text descriptions. By comprehensively processing music query information from multiple modalities, the accuracy of the search can be improved.

[0059] Step S204: Identify the information modality type of the music query information, and convert the music query information into a corresponding music query vector using a feature extraction model corresponding to the information modality type.

[0060] The information modality type refers to the data type category of the music query information, including but not limited to text modality, audio modality, and visual modality. In practice, it is necessary to identify which information modality type(s) the music query information belongs to in order to call the corresponding feature extraction model for processing.

[0061] Optionally, the information modality type can be identified based on the data format of the music query information. For example, when the music query information includes a text string, the information modality type can be identified as text; when the music query information includes an audio file, the information modality type can be identified as audio; and when the music query information includes an image file, the information modality type can be identified as visual.

[0062] In one embodiment, when the music query information includes information of multiple information modalities, the various different information modalities can be identified separately. For example, when a user submits both a text description and an audio clip, the information modalities of the music query information can be identified as including both text and audio modalities.

[0063] In this context, a feature extraction model refers to a model used to extract feature vectors from information of a specific modality. Different information modality types correspond to different feature extraction models. In specific implementations, based on the identified information modality type, the corresponding feature extraction model is invoked to convert the music query information into a music query vector.

[0064] In this context, the music query vector refers to a vector that represents the semantic meaning of music query information. The music query vector is extracted from the music query information using a feature extraction model, and this vector can be compared with the song vector in subsequent similarity calculations.

[0065] In one embodiment, when the music query information includes text information in a text modality, a text feature extraction model can be invoked to convert the text information into a text feature vector. The text feature extraction model can be a model based on natural language processing techniques, such as a pre-trained language model based on BERT or RoBERTa. The text information is input into the text feature extraction model, and the model outputs a text feature vector representing the semantic meaning of the text.

[0066] In one embodiment, when the music query information includes audio information of an audio modality, an audio feature extraction model can be invoked to convert the audio information into an audio feature vector. The audio feature extraction model can be a deep learning model for audio processing, such as a VGGish model, OpenL3 model, or MusiCNN model. The audio information is input into the audio feature extraction model, and the model outputs an audio feature vector representing the acoustic characteristics and semantic meaning of the audio.

[0067] In one embodiment, when the music query information includes visual information of a visual modality, a visual feature extraction model can be invoked to convert the image information into a visual feature vector. The visual feature extraction model can be a deep learning model for image processing, such as a ResNet-based model, EfficientNet model, CLIP model, or ViT model. Image information is input into the visual feature extraction model, and the model outputs a visual feature vector representing the visual features and semantic meaning of the image.

[0068] If the music query information includes text information in the text modality, audio information in the audio modality, and visual information in the visual modality, then feature extraction models of the corresponding information modality type can be used to extract features, resulting in text feature vectors, audio feature vectors, and visual feature vectors. Based on the text feature vectors, audio feature vectors, and visual feature vectors, the music query vector corresponding to the music query information can be constructed.

[0069] Step S206: In the song vector database, search for target song vectors whose similarity to music query vectors meets preset conditions.

[0070] The song vector database includes the song vector corresponding to each song in the preset song library; each song vector is obtained by fusing the audio features, text features and visual features of the corresponding song by a pre-trained multimodal fusion model; the audio features, text features and visual features of the same song are mapped to neighboring positions in the same semantic space by the pre-trained multimodal fusion model.

[0071] The song vector database refers to a database that stores song vectors. It stores the song vector corresponding to each song in a pre-built song library, along with the song's identifier information. The pre-built song library is a pre-constructed music library containing numerous songs, potentially including millions or even hundreds of millions of songs.

[0072] In this context, a song vector is a vector representing the semantic meaning of a song. Each song vector is obtained by fusing the audio, text, and visual features of the corresponding song through a pre-trained multimodal fusion model. Song vectors comprehensively reflect information about a song across multiple dimensions, including audio, text, and visual elements.

[0073] Among them, the pre-trained multimodal fusion model refers to a deep learning model that has been pre-trained and can fuse features from multiple modalities. This model can map the audio features, text features, and visual features of the same song to neighboring locations in the same semantic space, enabling features from different modalities to be compared and measured in a unified semantic space.

[0074] In this context, the same semantic space refers to the unified vector space constructed by the pre-trained multimodal fusion model. Within this semantic space, feature vectors from different modalities are comparable, meaning their semantic similarity can be measured by calculating the distance or similarity between vectors. Different modal features of the same song are mapped to neighboring locations within this semantic space, implying that these feature vectors are close in distance and have high similarity within the semantic space.

[0075] As an example, Approximate Nearest Neighbor (ANN) algorithms can be used to search for target song vectors in a song vector database that meet preset similarity criteria to the music query vector. Examples include hash-based methods, tree-based methods, and graph-based methods. These algorithms can significantly improve search speed while sacrificing a small amount of accuracy.

[0076] As another example, song vector databases can be database systems specifically designed for storing and retrieving high-dimensional vectors, such as Faiss, Milvus, and Weaviate. These database systems have built-in efficient vector indexing structures and search algorithms that can quickly calculate the similarity between vectors and return target song vectors that meet preset conditions.

[0077] As another example, the similarity between a music query vector and each song vector in a song vector database can be calculated, and target song vectors that meet preset conditions can be searched based on the similarity. The preset conditions could be that the similarity exceeds a preset threshold, or that the similarity ranks within the top N, where N is a preset number. Specifically, song vectors with similarity exceeding the preset threshold can be selected first, and then the top N similarity vectors can be chosen as target song vectors.

[0078] Similarity refers to the degree of similarity between two vectors. Various similarity calculation methods can be used, such as cosine similarity, Euclidean distance, and Manhattan distance. For example, cosine similarity can be used to calculate the similarity between a music query vector and the vectors of each song. The value of cosine similarity ranges from -1 to 1; the closer the value is to 1, the more similar the two vectors are.

[0079] The target song vector refers to a song vector whose similarity to the music query vector meets preset conditions. The song corresponding to the target song vector is the retrieved candidate song that matches the user's query intent.

[0080] Step S208: Generate music search results corresponding to the music search request based on the song resources corresponding to the target song vector.

[0081] The song resources may include relevant information and data of the song corresponding to the target song vector, including but not limited to song identifier, song name, artist name, album name, song audio file, lyrics, album cover image, music video, etc.

[0082] Music search results refer to the search results generated based on music search requests and returned to the user. Music search results may include a list of songs corresponding to each song resource. Optionally, the songs in the song list can be sorted by similarity; the higher the similarity between the song vector and the music query vector, the higher the ranking in the sorting.

[0083] In the aforementioned music search method, in response to a music search request, the information modality type of the music query information corresponding to the music search request is identified. A feature extraction model corresponding to the information modality type is used to convert the music query information into a corresponding music query vector. This vector is then matched with the similarity of each song vector in a song vector database. The song vector is obtained by a pre-trained multimodal fusion model that fuses heterogeneous features such as audio, text, and visual features of the corresponding song and maps them to neighboring positions in the same semantic space. By constructing a unified semantic space, heterogeneous features from different modalities can be mapped to a unified semantic space for measurement and comparison. This allows the emotion of the melody, the artistic conception of the lyrics, and the style of the album cover to be associated and understood, improving the comprehensiveness of the understanding of music content. This enables the understanding of more abstract and emotional music search requests from users. Users can search for music based on a melody, an emotional description, or a visual style. By calculating vector similarity in a unified semantic space, music content that meets the user's deeper needs is accurately recalled, greatly improving the accuracy and personalized experience of music search.

[0084] In another embodiment, the pre-trained multimodal fusion model includes an audio feature encoder, a text feature encoder, and a visual feature encoder. Before the step of searching for a target song vector in the song vector database whose similarity to a music query vector meets a preset condition, the model further includes: obtaining the multimodal features of each song in a preset song library; for each song, inputting the song's audio features into the audio feature encoder to obtain an audio encoded vector, inputting the song's text features into the text feature encoder to obtain a text encoded vector, and inputting the song's visual features into the visual feature encoder to obtain a visual encoded vector; the song's audio encoded vector, text encoded vector, and visual encoded vector are located in adjacent positions in the same semantic space; and fusing the audio encoded vector, text encoded vector, and visual encoded vector to obtain the song vector.

[0085] Multimodal features refer to the features of a song across different modal dimensions. These features include at least audio, textual, and visual features. By acquiring the multimodal features of each song in a pre-defined song library, it becomes easier to generate song vectors using a multimodal fusion model.

[0086] Audio features refer to the characteristics of a song's audio information. This audio information includes the song's audio file, audio waveform, etc. In practice, audio features can be extracted from the song's audio information. These features reflect the song's acoustic characteristics and semantic information, such as melody, rhythm, timbre, chords, genre, and mood.

[0087] For example, to obtain audio features, the audio information of a song can first be converted into a Mel spectrogram. A Mel spectrogram is a time-frequency representation method that can simulate the perceptual characteristics of sound by the human ear. Then, the Mel spectrogram can be input into an audio feature extraction model to obtain the audio features. Optionally, the audio feature extraction model can be a pre-trained audio domain-specific model, such as VGGish or OpenL3 models, which can be used to extract general features such as scene and ambient sound; or, the audio feature extraction model can be a deep convolutional neural network model in the audio domain, which can be used to extract high-level semantic features of music such as rhythm, timbre, melody, chords, genre, and emotion. The audio features output by the audio feature extraction model can be a high-dimensional vector, such as 128-dimensional, 256-dimensional, or 768-dimensional.

[0088] Textual features refer to the characteristics of a song's textual information. This textual information includes lyrics, song title, artist name, album name, song tags, song description, user comments, etc. Textual features extracted from a song's textual information can reflect its semantic content, theme, and emotional content.

[0089] For example, to obtain text features, the text information of a song can be input into a text feature extraction model to obtain text features. Optionally, the text feature extraction model can be a pre-trained language model based on natural language processing techniques, such as the BERT model, RoBERTa model, GPT model, etc. The text information is input into the text feature extraction model, and the model outputs text features representing the semantic meaning of the text. Text features can be vectors with special labels output by the text feature extraction model, such as the vector corresponding to the [CLS] label in the BERT model, or vectors obtained by pooling the vectors corresponding to all words. Text features can be high-dimensional vectors, such as 768-dimensional or 1024-dimensional.

[0090] Visual features refer to the characteristics of a song's visual information. This visual information includes album cover images, music video screenshots, and artist photos. Visual features extracted from a song's visual information can reflect information such as the image's style, color, scene, objects, and atmosphere.

[0091] For example, to obtain visual features, the visual information of the song can be input into a visual feature extraction model to obtain visual features. Optionally, the visual feature extraction model can be a pre-trained image model based on computer vision technology. This model can be a ResNet model, an EfficientNet model, etc., used to extract general visual features; alternatively, it can be a CLIP model, a ViT model, etc., which, because they incorporate textual information during training, extract visual features with stronger semantic relevance and are more suitable for retrieval tasks. The visual information is input into the visual feature extraction model, and the model outputs visual features representing the visual characteristics and semantic meaning of the image. Visual features can be high-dimensional vectors, such as 512-dimensional, 768-dimensional, 1024-dimensional, etc.

[0092] By processing textual, audio, and visual information separately through feature extraction models of corresponding modalities, and extracting corresponding textual, audio, and visual features, multimodal features are obtained. This multi-dimensional and deep feature extraction method constructs a comprehensive understanding of the music content. Then, the multimodal features can be input into a pre-trained multimodal fusion model, that is, the audio, textual, and visual features in the multimodal features are respectively input into the encoders corresponding to the respective modalities in the multimodal fusion model.

[0093] The pre-trained multimodal fusion model includes multiple encoders, each responsible for processing features from one modality. The multimodal fusion model includes an audio feature encoder, a text feature encoder, and a visual feature encoder.

[0094] For the convenience of those skilled in the art, Figure 3 An exemplary structural diagram of a multimodal fusion model is provided. This pre-trained multimodal fusion model can receive input features from different modalities, and can be primarily a two-tower or multi-tower model. Each tower in the multimodal fusion model is an independent encoder responsible for processing one modality. The multimodal fusion model includes an audio feature extraction tower (audio feature encoder), a text feature extraction tower (text feature encoder), and a visual feature extraction tower (visual feature encoder), each processing one of the three corresponding modalities. Each tower (encoder) in the multimodal fusion model maps different modal features of the same song to neighboring locations in the semantic space; then, a central fusion module fuses the features mapped to neighboring locations.

[0095] The audio encoding vector refers to the vector output by the audio feature encoder. In practice, the audio features of a song are input into the audio feature encoder, which encodes these features and outputs the audio encoding vector. The audio encoding vector represents the semantic information of the song in the audio modality dimension.

[0096] Here, the text encoding vector refers to the vector output by the text feature encoder. In the specific implementation, the text features of the song are input into the text feature encoder, which encodes the text features and outputs the text encoding vector. The text encoding vector represents the semantic information of the song in the text modality dimension.

[0097] Here, the visual encoding vector refers to the vector output by the visual feature encoder. In the specific implementation, the visual features of the song are input into the visual feature encoder, which encodes the visual features and outputs the visual encoding vector. The visual encoding vector represents the semantic information of the song in the visual modality dimension.

[0098] In a pre-trained multimodal fusion model, audio features, text features, and visual features are input into their respective encoders, mapping the audio, text, and visual encoding vectors of the same song to neighboring locations in the same semantic space. During the training process, the model learns to map different modal features of the same song to neighboring locations in the semantic space, enabling features from different modalities to be semantically correlated and aligned.

[0099] In practice, the audio encoding vector, text encoding vector, and visual encoding vector of a song are fused to obtain the song vector. This fusion can be an operation that combines multiple vectors into one. The song vector comprehensively reflects the semantic information of the song across multiple modal dimensions, including audio, text, and visual.

[0100] As an example, audio, text, and visual encoded vectors can be fused using a weighted average. Specifically, each encoded vector is assigned a weight that reflects the importance of that modality feature in the song vector. In practice, the product of each encoded vector and its corresponding weight is calculated, and these products are then summed to obtain the song vector. The weights can be preset fixed values ​​or learned through training.

[0101] As another example, audio encoding vectors, text encoding vectors, and visual encoding vectors can be merged by concatenation. Specifically, the three encoding vectors are concatenated in a certain order into a longer vector, which is the song vector. The dimension of the concatenated song vector is equal to the sum of the dimensions of the three encoding vectors.

[0102] As another example, audio encoding vectors, text encoding vectors, and visual encoding vectors can be fused using a fusion layer. The fusion layer can be one or more layers of a neural network, capable of learning how to effectively fuse encoding vectors from different modalities. The audio encoding vector, text encoding vector, and visual encoding vector are input into the fusion layer, which outputs the fused song vector.

[0103] The technical solution of this embodiment maps heterogeneous features extracted from different sources such as audio, text, and images into a unified semantic vector space. In this space, information from different modalities is assigned comparable and unified semantic coordinates, from which the fused song vector is obtained. This allows previously unrelated elements such as melody emotion, lyrical imagery, and album art style to be mutually measurable and understood, achieving a leap from data silos to semantic interoperability and constructing a three-dimensional, in-depth understanding of songs. Based on this unified semantic understanding, it supports intelligent music search based on abstract concepts such as emotion, style, and melody, helping to understand users' deeper music search needs and improving the accuracy, quality, and personalized experience of music searches.

[0104] In another embodiment, music query information is converted into corresponding music query vectors using a feature extraction model corresponding to each information modality type. This includes: converting music query information into modal feature vectors corresponding to each information modality type using a feature extraction model corresponding to each information modality type; inputting the modal feature vectors corresponding to each information modality type into a pre-trained multimodal fusion model to obtain music query vectors; the music query vectors are obtained by fusing the modal feature vectors mapped to the same semantic space using the pre-trained multimodal fusion model.

[0105] In cases where the music search request corresponds to multiple different types of music query information, the information modality types identified from the music query information include multiple different information modality types. For example, if the music query information includes text information, audio information, and visual information, the information modality types identified from the music query information include text modality types, audio modality types, and visual modality types.

[0106] In practice, when music query information includes information of multiple information modalities, it is necessary to process the information of each information modality separately and convert it into modal feature vectors corresponding to each information modality.

[0107] In one embodiment, when the music query information includes text information in a text modality, a text feature extraction model can be invoked to convert the text information into a text feature vector, which serves as the modality feature vector corresponding to the text modality type. When the music query information includes audio information in an audio modality, an audio feature extraction model can be invoked to convert the audio information into an audio feature vector, which serves as the modality feature vector corresponding to the audio modality type. When the music query information includes visual information in a visual modality, a visual feature extraction model can be invoked to convert the image information into a visual feature vector, which serves as the modality feature vector corresponding to the visual modality type. The descriptions of the text feature extraction model, audio feature extraction model, and visual feature extraction model can be found in the above embodiments and will not be repeated here.

[0108] In the specific implementation, the modal feature vectors corresponding to each information modality type are input into a pre-trained multimodal fusion model. This model maps each modal feature vector to the same semantic space and fuses the modal feature vectors to obtain the music query vector.

[0109] In one embodiment, the pre-trained multimodal fusion model includes encoders corresponding to each information modality type. Text feature vectors are input to the text feature encoder to obtain text encoded vectors; audio feature vectors are input to the audio feature encoder to obtain audio encoded vectors; and visual feature vectors are input to the visual feature encoder to obtain visual encoded vectors. These encoded vectors are mapped to the same semantic space; then, a fusion operation can be performed to combine the encoded vectors into a music query vector, i.e., fusing the audio encoded vector, text encoded vector, and visual encoded vector to obtain the music query vector. The fusion operation can be the aforementioned weighted averaging, concatenation, or processing through a fusion layer, etc.

[0110] It should be noted that when the music query information only includes information of one information modality type, it is only necessary to input the feature vector of that modality into the corresponding encoder to obtain the encoded vector, which can then be used as the music query vector.

[0111] The technical solution of this application embodiment can convert music query information of multiple modalities into a unified music query vector. The music query vector and the song vector are in the same semantic space, which improves the efficiency and accuracy of subsequent similarity calculation, thereby improving the efficiency and accuracy of music search.

[0112] In another embodiment, in the song vector database, searching for target song vectors whose similarity to music query vectors meets preset conditions includes: calculating the similarity between each song vector in the song vector database and the music query vector; sorting each song vector in descending order according to similarity; and selecting the top N song vectors from each song vector as target song vectors that meet preset conditions, where N is a preset number.

[0113] By calculating the similarity between each song vector in the song vector database and the music query vector, it is possible to determine which songs best match the user's query intent.

[0114] For example, cosine similarity can be used as a similarity metric, which measures the degree of similarity between two vectors by calculating the cosine of the angle between them. Alternatively, other similarity metrics such as Euclidean distance or Manhattan distance can be used.

[0115] In the specific implementation, based on the calculated similarity, the song vectors in the song vector database are sorted in descending order, that is, arranged in order of similarity from high to low. Furthermore, the top N song vectors are selected as target song vectors that meet preset conditions; N can be 5, 10, 20, etc. These target song vectors correspond to the songs that best match the user's query intent.

[0116] The technical solution of this embodiment can quickly and accurately retrieve the target song vector that is most similar to the music query vector from the song vector database, thereby improving the accuracy and efficiency of music search.

[0117] In one embodiment, the offline construction method of the song vector database may include: traversing each song in a preset song library, using a pre-trained multimodal fusion model to infer and calculate each modal feature of each song to obtain its representation vector in a unified semantic space, i.e., the song vector corresponding to each song; determining the vector database, which may be a database specifically designed for storing and retrieving high-dimensional vectors, capable of providing efficient vector similarity search functionality, such as an open-source library that can be integrated into the application, or a complete, independently deployable open-source database service that provides rich functions such as data management, dynamic indexing, and scalar filtering.

[0118] Then, the generated large number of song vectors, along with their unique song identifiers, are stored in a vector database. Next, the Approximate Nearest Neighbor (ANN) algorithm is used to build a vector index structure in the vector database, thus generating the song vector database. Building the vector index structure in the vector database using the ANN algorithm significantly improves retrieval speed while sacrificing a small amount of precision, achieving a balance between accuracy and efficiency.

[0119] In another embodiment, generating music search results corresponding to a music search request based on the song resources corresponding to the target song vector includes: identifying the search intent of the music query information and determining the music attribute corresponding to the search intent; filtering out target song vectors whose song attributes match the music attribute from the target song vectors; and generating music search results corresponding to the music search request based on the song resources corresponding to the filtered target song vectors.

[0120] Search intent refers to the purpose or expectation of a user initiating a music search request. A user may want to search for specific types of music, such as songs, instrumental music, music videos, or albums. Therefore, it is necessary to identify the user's search intent in order to return search results that better match the user's needs.

[0121] The music attributes corresponding to the search intent may include the type of music the user wants to search for, such as songs, instrumental music, music videos, albums, etc.; or the music style, emotion, scene, etc. that the user expects.

[0122] As an example, natural language processing techniques can be used to analyze the music query information corresponding to a music search request and extract keywords or phrases that indicate the search intent.

[0123] As another example, an intent recognition model can be used to identify search intent. An intent recognition model is a classification model that categorizes music queries into several predefined intent categories. For example, intent categories could include: searching for songs, searching for albums, searching for artists, searching for instrumental music, searching for music videos, etc. The music query information is input into the intent recognition model, which outputs the intent category corresponding to the query. Based on this intent category, the music attribute corresponding to the search intent can be determined.

[0124] As another example, more granular information about search intent can be identified, such as the user's desired music style, mood, and context. By analyzing adjectives, verbs, nouns, and other words in music query information, the music attributes corresponding to the search intent can be determined.

[0125] The song attributes associated with the target song vector may include song name, singer, instrumental track, video, album, etc., and may also include song style, emotion, scene, etc.

[0126] In practical applications, target song vectors that match the song attributes and music attributes are selected from the target song vectors; that is, the song attributes and music attributes are first determined.

[0127] For example, if a user's music attribute requirement is jazz, and the song attribute of a target song vector is pop, even if the target song vector is close to the query vector in the vector space (possibly due to similar melodies), the target song vector still needs to be removed, and the target song vectors with the song attribute of jazz should be retained.

[0128] For example, if a user's music attribute requires searching for a song A, and the name of song A is the same as the name of a singer B, meaning song A is a song with the same name as the singer, then if the song attributes of each target song vector include both songs performed by singer B and songs with the same name as singer B, then when matching the music attribute with the song attributes of each target song vector, only songs with the same name as singer B can be retained.

[0129] For example, when the search intent is for songs, target song vectors corresponding to the song type are filtered from the target song vectors, while target song vectors corresponding to other types such as albums, artists, and music videos are excluded.

[0130] For example, when the search intent includes a specific music style, target song vectors that match that music style are filtered from the target song vectors.

[0131] The process involves retrieving corresponding song resources based on the selected target song vectors and generating music search results. These search results include relevant information and data about the songs corresponding to the selected target song vectors, such as song title, artist name, album name, playback link, and album cover image.

[0132] Optionally, based on the song identifier corresponding to the target song vector, the corresponding song resources can be queried from the database, and the queried song resources can be organized into a list or other forms to generate music search results.

[0133] Furthermore, the selected target song vectors can be further sorted. Sorting can be based on factors such as similarity, song popularity, and user historical preferences. The song resources are then organized according to the sorted order, ensuring that the songs most relevant to the user's needs are listed first.

[0134] The technical solution of this application embodiment can filter out the songs that best meet the user's needs from the target song vector according to the user's search intent, and generate accurate music search results, thereby improving the user experience.

[0135] For the convenience of those skilled in the art, Figure 4 An exemplary logic diagram of an online music search method is provided. In practical applications, this online music search service can directly receive music search requests submitted by users and obtain the corresponding music query information. Regardless of whether the music query information is a descriptive text, a humming melody, or an ambient image, an intent recognition model can be invoked to determine the user's search intent, which includes the type of target result the user desires. Simultaneously, an encoder corresponding to the modality of the music query information can be used for real-time calculation to convert the music query information into a corresponding music query vector. For example, if a user inputs the text-modal music query information "upbeat electronic music suitable for listening to at the beach," this music query information can be converted into a query vector and intent content representing the search intent. Then, the generated query vector and intent content can be sent to a pre-indexed song vector database. The song vector database performs an approximate nearest neighbor algorithm search to calculate the similarity and similarity score between the query vector and each song vector in the database. Based on the similarity score, the song vector database quickly returns the N song vectors closest to the query vector, along with their corresponding song identifiers and similarity scores. Then, based on the intent content corresponding to the search intent, the top N song vectors are filtered and sorted to obtain the final music search results, which are then returned to the front end for the user.

[0136] The technical solution of this application embodiment can, for music media assets such as songs, music, music videos, and albums, utilize domain-specific models to deeply extract heterogeneous features in three dimensions: audio, text, and visual, constructing a comprehensive understanding of music content; by training a multimodal fusion model, the extracted heterogeneous features are mapped and aligned to a unified high-dimensional semantic space, achieving semantic interoperability; all music media assets in a preset song library are transformed into song vectors through the above multimodal fusion model, and an efficient song vector database is constructed based on each song vector to support fast online similarity retrieval; in the online music search service stage, the user's multimodal music query information (text, audio, image) is encoded into query vectors in real time, and efficient semantic similarity retrieval is performed in the song vector database to obtain music search results.

[0137] In some embodiments, such as Figure 5 As shown, a training method for a multimodal fusion model is provided, which is then applied to... Figure 1 Taking server 104 as an example, the method includes the following steps:

[0138] Step S502: Obtain the training sample set.

[0139] The training sample set includes positive samples obtained by pairing at least two modal features from the multimodal features of the same song, and negative samples obtained by combining any modal features of different songs; the modal features are one of audio features, text features, and visual features.

[0140] Step S504: Input the training sample set into the multimodal fusion model to be trained, and train the multimodal fusion model to be trained according to the difference between positive and negative samples to obtain a pre-trained multimodal fusion model.

[0141] The pre-trained multimodal fusion model maps the audio, text, and visual features of each song in the preset song library to neighboring locations in the same semantic space, resulting in a song vector for each song. The song vector is used to calculate the similarity with the music query vector to determine the target song vector that meets the preset conditions. The song resources corresponding to the target song vector are used to generate the music search results corresponding to the music search request. The music query vector is obtained by identifying the information modality type of the music query information corresponding to the music search request and transforming the music query information using the feature extraction model corresponding to the information modality type.

[0142] The training sample set is the dataset used to train the multimodal fusion model. The training sample set includes positive samples and negative samples. Positive samples are pairs of different modal features belonging to the same song, while negative samples are combinations of arbitrary modal features belonging to different songs.

[0143] In one embodiment, multiple songs can be selected from a music library as training data. For each song, its audio features, text features, and visual features are obtained. Different modal features of the same song are paired to generate positive samples. For example, when constructing samples, the audio features of song A can be paired with the text features of song A to obtain one positive sample; the audio features of song A can be paired with the visual features of song A to obtain another positive sample; and the text features of song A can be paired with the visual features of song A to obtain yet another positive sample. For example, the (audio, text), (audio, cover), and (text, cover) pairings of the same song can be used as positive samples.

[0144] In one embodiment, arbitrary modal features from different songs are combined to generate negative samples. For example, combining the audio features of song A with the text features of song B yields one negative sample; combining the visual features of song A with the audio features of song C yields another negative sample.

[0145] In one embodiment, the training sample set may also include trimodal samples, that is, combining audio features, text features and visual features of the same song together as positive samples, or combining three modal features from different songs together as negative samples.

[0146] The multimodal fusion model to be trained refers to a multimodal fusion model that has not yet been trained or is in the process of being trained. The multimodal fusion model to be trained includes an audio feature encoder, a text feature encoder, and a visual feature encoder.

[0147] In practice, contrastive learning can be used to train the multimodal fusion model. Contrastive learning is a self-supervised learning method that learns effective feature representations by narrowing the distance between positive sample pairs and widening the distance between negative sample pairs in the semantic space.

[0148] In practice, the loss function value is calculated based on the difference between positive and negative samples. The loss function measures the gap between the model's prediction and the expected result. By optimizing the loss function, the parameters of the multimodal fusion model to be trained are adjusted, enabling the model to map different modal features of the same song to nearby positions in the semantic space, while mapping features of different songs to distant positions in the semantic space. Optionally, a contrastive loss function can be used. Contrastive loss functions calculate the loss value based on the similarity difference between positive and negative samples, such as the InfoNCE loss function and the Triplet loss function. The basic idea of ​​the InfoNCE loss function is to encourage the model to accurately identify the unique positive sample from among many negative samples, given a positive sample and multiple negative samples.

[0149] Then, by calculating the contrastive loss function value and updating the parameters of each encoder of the multimodal fusion model to be trained through backpropagation, all encoders of the multimodal fusion model will eventually map the different modal features of the same song to the neighboring positions in the semantic space, thus obtaining the pre-trained multimodal fusion model.

[0150] In one embodiment, the multimodal fusion model to be trained is trained based on the difference between positive and negative samples, including: inputting the modal features of each positive sample into the encoder corresponding to the corresponding modality to obtain the positive sample encoding features of each modality; and inputting the modal features of each negative sample into the encoder corresponding to the corresponding modality to obtain the negative sample encoding features of each modality; calculating the similarity between the positive sample encoding features of each modality to obtain the positive sample similarity; and calculating the similarity between the negative sample encoding features of each modality to obtain the negative sample similarity; and training the multimodal fusion model to be trained based on the difference between the positive sample similarity and the negative sample similarity.

[0151] The multimodal fusion model to be trained includes encoders corresponding to each modality type, such as audio feature encoders, text feature encoders, and visual feature encoders.

[0152] In practice, training samples are input into the multimodal fusion model to be trained. For each sample, the different modal features in the sample are input into the encoder corresponding to the respective modality to obtain the encoding vector of each modality. For example, for a sample containing audio features and text features, the audio features are input into the audio feature encoder to obtain the audio encoding vector, and the text features are input into the text feature encoder to obtain the text encoding vector.

[0153] Among them, the positive sample coding features include the coding features obtained after the modal features in the positive samples are encoded by the corresponding encoder.

[0154] Among them, the negative sample coding features include the coding features obtained after the modal features in the negative samples are encoded by the corresponding encoder.

[0155] In practice, the similarity between modal encoding vectors in a sample is calculated. For positive samples, the goal is to maximize the similarity between their modal encoding vectors, i.e., to maximize the similarity of positive samples. For negative samples, the goal is to minimize the similarity between their modal encoding vectors, i.e., to minimize the similarity of negative samples. Specifically, a contrastive learning training method can be used to train the multimodal fusion model. Contrastive learning is a self-supervised learning method that learns effective feature representations by narrowing the distance between positive sample pairs and widening the distance between negative sample pairs.

[0156] Optionally, a contrastive loss function value can be calculated based on the difference between the similarity of positive samples and the similarity of negative samples, and the multimodal fusion model to be trained can be trained based on the contrastive loss function value. Optionally, the contrastive loss function may include the InfoNCE loss function, where the basic idea of ​​the InfoNCE loss function is that, given a positive sample and multiple negative samples, the model should be able to identify the positive sample from the many negative samples.

[0157] The technical solution of this application embodiment can obtain a pre-trained multimodal fusion model through training with rich positive and negative samples. This model can map different modal features of the same song to neighboring positions in the same semantic space, thereby improving the robustness and accuracy of the multimodal fusion model and thus improving the robustness and accuracy of subsequent music search.

[0158] In another embodiment, such as Figure 6 As shown, a music search method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0159] Step S602: In response to the music search request, obtain the music query information corresponding to the music search request.

[0160] Step S604: Identify the information modality type of the music query information, and convert the music query information into modality feature vectors corresponding to each information modality type through the feature extraction model corresponding to each information modality type.

[0161] Step S606: Input the modal feature vectors corresponding to each information modality type into the pre-trained multimodal fusion model to obtain the music query vector.

[0162] The music query vector is obtained by fusing the feature vectors of each modality mapped to the same semantic space using a pre-trained multimodal fusion model.

[0163] In cases where the music query information corresponding to a music search request includes multiple different types of music query information, the information modality types identified from the music query information include multiple different types of information modality types.

[0164] Step S608: Obtain the multimodal features of each song in the preset song library.

[0165] Among them, multimodal features include at least the song's audio features, text features, and visual features.

[0166] In step S610, for each song, the audio features of the song are input into the audio feature encoder to obtain the audio encoding vector, and the text features of the song are input into the text feature encoder to obtain the text encoding vector, and the visual features of the song are input into the visual feature encoder to obtain the visual encoding vector.

[0167] In this context, the song's audio encoding vector, text encoding vector, and visual encoding vector are located in adjacent positions within the same semantic space.

[0168] Step S612: Merge the audio encoding vector, text encoding vector, and visual encoding vector to obtain the song vector of the song; generate a song vector database based on the song vector corresponding to each song in the preset song library.

[0169] The song vector database includes the song vector corresponding to each song in the preset song library; each song vector is obtained by fusing the audio features, text features and visual features of the corresponding song by a pre-trained multimodal fusion model; the audio features, text features and visual features of the same song are mapped to neighboring positions in the same semantic space by the pre-trained multimodal fusion model.

[0170] The pre-trained multimodal fusion model includes an audio feature encoder, a text feature encoder, and a visual feature encoder.

[0171] In one embodiment, before the step of searching for a target song vector in the song vector database whose similarity to a music query vector meets a preset condition, the method further includes: obtaining a training sample set; the training sample set includes positive samples obtained by pairing at least two modal features from the multimodal features of the same song, and negative samples obtained by combining any modal features of different songs; the modal features are one of audio features, text features, and visual features; the training sample set is input into the multimodal fusion model to be trained, and the multimodal fusion model to be trained is trained according to the difference between the positive and negative samples to obtain a pre-trained multimodal fusion model; the pre-trained multimodal fusion model is used to map the audio features, text features, and visual features of each song in the preset song library to neighboring positions in the same semantic space to obtain a song vector for each song; the song vector is used to calculate the similarity with the music query vector to determine the target song vector that meets the preset condition; the song resources corresponding to the target song vector are used to generate music search results corresponding to the music search request; the music query vector is obtained by identifying the information modality type of the music query information corresponding to the music search request, and by using the feature extraction model corresponding to the information modality type to transform the music query information.

[0172] In one embodiment, the multimodal fusion model to be trained is trained based on the difference between positive and negative samples, including: inputting the modal features of each positive sample into the encoder corresponding to the corresponding modality to obtain the positive sample encoding features of each modality; and inputting the modal features of each negative sample into the encoder corresponding to the corresponding modality to obtain the negative sample encoding features of each modality; calculating the similarity between the positive sample encoding features of each modality to obtain the positive sample similarity; and calculating the similarity between the negative sample encoding features of each modality to obtain the negative sample similarity; and training the multimodal fusion model to be trained based on the difference between the positive sample similarity and the negative sample similarity.

[0173] Step S614: Calculate the similarity between each song vector in the song vector database and the music query vector; sort each song vector in descending order according to the similarity, and select the song vector in the top N positions as the target song vector that meets the preset conditions.

[0174] Where N is the preset quantity.

[0175] Step S616: Identify the search intent for the music query information and determine the music attributes corresponding to the search intent.

[0176] Step S618: Filter out target song vectors from the target song vectors whose song attributes match the music attributes.

[0177] Step S620: Generate music search results corresponding to the music search request based on the song resources corresponding to the selected target song vector.

[0178] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a music search method described above.

[0179] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0180] Based on the same inventive concept, this application also provides a music search device for implementing the music search method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more music search device embodiments provided below can be found in the limitations of the music search method described above, and will not be repeated here.

[0181] In one exemplary embodiment, such as Figure 7 As shown, a music search device is provided, comprising:

[0182] The acquisition module 710 is used to acquire music query information corresponding to the music search request in response to the music search request.

[0183] The conversion module 720 is used to identify the information modality type of the music query information and convert the music query information into a corresponding music query vector through a feature extraction model corresponding to the information modality type.

[0184] The search module 730 is used to search for target song vectors in the song vector database that have a similarity to the music query vector that meets preset conditions; wherein, the song vector database includes song vectors corresponding to each song in the preset song library; each song vector is obtained by fusing the audio features, text features and visual features of the corresponding song by a pre-trained multimodal fusion model; the audio features, text features and visual features of the same song are mapped to neighboring positions in the same semantic space by the pre-trained multimodal fusion model.

[0185] The generation module 740 is used to generate music search results corresponding to the music search request based on the song resources corresponding to the target song vector.

[0186] In one embodiment, the pre-trained multimodal fusion model includes an audio feature encoder, a text feature encoder, and a visual feature encoder; the acquisition module 710 is specifically used to acquire the multimodal features of each song in the preset song library; the multimodal features include at least the audio features, text features, and visual features of the song; for each song, the audio features of the song are input to the audio feature encoder to obtain an audio encoding vector, and the text features of the song are input to the text feature encoder to obtain a text encoding vector, and the visual features of the song are input to the visual feature encoder to obtain a visual encoding vector; the audio encoding vector, the text encoding vector, and the visual encoding vector of the song are in adjacent positions in the same semantic space; the audio encoding vector, the text encoding vector, and the visual encoding vector are fused to obtain the song vector of the song.

[0187] In one embodiment, when the music query information corresponding to the music search request includes multiple different types of music query information, the information modality types identified from the music query information include multiple different types of information modality types; the conversion module 720 is specifically used to convert the music query information into modality feature vectors corresponding to each of the information modality types through a feature extraction model corresponding to each of the information modality types; input the modality feature vectors corresponding to each of the information modality types into the pre-trained multimodal fusion model to obtain the music query vector; the music query vector is obtained by the pre-trained multimodal fusion model fusing the modality feature vectors mapped to the same semantic space.

[0188] In one embodiment, the generation module 740 is specifically used to identify the search intent of the music query information, determine the music attribute corresponding to the search intent; filter out target song vectors whose song attributes match the music attribute from the target song vectors; and generate music search results corresponding to the music search request based on the song resources corresponding to the filtered target song vectors.

[0189] In one embodiment, the search module 730 is specifically used to calculate the similarity between each song vector in the song vector database and the music query vector; sort each song vector in descending order according to the similarity; and select the song vector ranked in the top N positions from each song vector as the target song vector that meets the preset conditions, where N is a preset number.

[0190] In one exemplary embodiment, such as Figure 8 As shown, a training device for a multimodal fusion model is provided, comprising:

[0191] The sample acquisition module 810 is used to acquire a training sample set; the training sample set includes positive samples obtained by pairing at least two modal features from the multimodal features of the same song, and negative samples obtained by combining any of the modal features of different songs; the modal features are one of audio features, text features and visual features.

[0192] The model training module 820 is used to input the training sample set into the multimodal fusion model to be trained, and to train the multimodal fusion model to be trained based on the difference between the positive samples and the negative samples, thereby obtaining a pre-trained multimodal fusion model. The pre-trained multimodal fusion model is used to map the audio features, text features, and visual features of each song in the preset song library to neighboring positions in the same semantic space, thereby obtaining a song vector for each song. The song vector is used to calculate the similarity between it and the music query vector to determine the target song vector that meets the preset conditions. The song resources corresponding to the target song vector are used to generate music search results corresponding to the music search request. The music query vector is obtained by identifying the information modality type of the music query information corresponding to the music search request and converting the music query information using the feature extraction model corresponding to the information modality type.

[0193] In one embodiment, the model training module is specifically configured to input each modal feature in the positive samples into the encoder corresponding to the corresponding modality to obtain positive sample coding features of each modality, and input each modal feature in the negative samples into the encoder corresponding to the corresponding modality to obtain negative sample coding features of each modality; calculate the similarity between the positive sample coding features of each modality to obtain positive sample similarity, and calculate the similarity between the negative sample coding features of each modality to obtain negative sample similarity; and train the multimodal fusion model to be trained based on the difference between the positive sample similarity and the negative sample similarity.

[0194] Each module in the aforementioned music search device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0195] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores music data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a music search method or a training method for a multimodal fusion model.

[0196] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0197] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0198] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0199] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0200] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0201] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0202] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0203] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A music search method, characterized in that, The method includes: In response to a music search request, obtain the music query information corresponding to the music search request; The information modality type of the music query information is identified, and the music query information is converted into a corresponding music query vector through a feature extraction model corresponding to the information modality type; In the song vector database, target song vectors whose similarity to the music query vector meets preset conditions are searched; wherein, the song vector database includes song vectors corresponding to each song in a preset song library; each song vector is obtained by fusing the audio features, text features and visual features of the corresponding song by a pre-trained multimodal fusion model; the audio features, text features and visual features of the same song are mapped to neighboring positions in the same semantic space by the pre-trained multimodal fusion model; Based on the song resources corresponding to the target song vector, generate the music search results corresponding to the music search request.

2. The method of claim 1, wherein, The pre-trained multimodal fusion model includes an audio feature encoder, a text feature encoder, and a visual feature encoder. Before the step of searching for target song vectors in the song vector database that have a similarity to the music query vector that meets preset conditions, the method further includes: Obtain the multimodal features of each song in the preset song library; the multimodal features include at least the song's audio features, text features, and visual features; For each song, the audio features of the song are input into the audio feature encoder to obtain an audio encoding vector; the text features of the song are input into the text feature encoder to obtain a text encoding vector; and the visual features of the song are input into the visual feature encoder to obtain a visual encoding vector; the audio encoding vector, the text encoding vector, and the visual encoding vector of the song are located in adjacent positions in the same semantic space. The song vector of the song is obtained by fusing the audio encoding vector, the text encoding vector, and the visual encoding vector.

3. The method of claim 1, wherein, When the music query information corresponding to the music search request includes multiple different types of music query information, the information modality type identified from the music query information includes multiple different types of information modality types; The step of converting the music query information into a corresponding music query vector using a feature extraction model corresponding to the information modality type includes: The music query information is converted into modal feature vectors corresponding to each of the aforementioned information modal types using feature extraction models. The modal feature vectors corresponding to each of the aforementioned information modal types are input into the pre-trained multimodal fusion model to obtain the music query vector; the music query vector is obtained by the pre-trained multimodal fusion model fusing the modal feature vectors mapped to the same semantic space.

4. The method of claim 1, wherein, The step of generating music search results corresponding to the music search request based on the song resources corresponding to the target song vector includes: Identify the search intent of the music query information and determine the music attributes corresponding to the search intent; From the target song vectors, select the target song vectors whose song attributes match the music attributes; Based on the song resources corresponding to the selected target song vector, generate the music search results corresponding to the music search request.

5. The method of claim 1, wherein, The step of searching the song vector database for target song vectors whose similarity to the music query vector meets preset conditions includes: Calculate the similarity between each song vector in the song vector database and the music query vector; The song vectors are sorted in descending order according to their similarity. The top N song vectors are selected as the target song vectors that meet the preset conditions, where N is a preset number.

6. A method for training a multi-modal fusion model, comprising: The method includes: Obtain a training sample set; the training sample set includes positive samples obtained by pairing at least two modal features from the multimodal features of the same song, and negative samples obtained by combining any of the modal features of different songs; the modal features are one of audio features, text features, and visual features; The training sample set is input into the multimodal fusion model to be trained. Based on the difference between the positive and negative samples, the multimodal fusion model is trained to obtain a pre-trained multimodal fusion model. The pre-trained multimodal fusion model is used to map the audio features, text features, and visual features of each song in the preset song library to neighboring positions in the same semantic space to obtain a song vector for each song. The song vector is used to calculate the similarity between it and the music query vector to determine the target song vector that meets the preset conditions. The song resources corresponding to the target song vector are used to generate music search results corresponding to the music search request. The music query vector is obtained by identifying the information modality type of the music query information corresponding to the music search request and transforming the music query information using the feature extraction model corresponding to the information modality type.

7. The method of claim 6, wherein, The step of training the multimodal fusion model to be trained based on the difference between the positive samples and the negative samples includes: Each modal feature in the positive sample is input into the encoder corresponding to the corresponding modality to obtain the positive sample coding feature of each modality; and each modal feature in the negative sample is input into the encoder corresponding to the corresponding modality to obtain the negative sample coding feature of each modality. Calculate the similarity between the positive sample coding features of each modality to obtain the positive sample similarity; and calculate the similarity between the negative sample coding features of each modality to obtain the negative sample similarity. The multimodal fusion model to be trained is trained based on the difference between the positive sample similarity and the negative sample similarity. 8.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-7. When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, The computer program, which is executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images