Trade music recommendation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a score recommendation method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a target material, determining a first target score based on the target material through a first model, and adding a score to the target material based on the first target score; obtaining text information, and determining a material feature, a text feature and an intention type through a second model based on the target material and the text information; and according to the intention type, based on at least one target object in a target object set, determining a second target incidental music corresponding to the target material, and based on the second target incidental music, replacing the incidental music of the target material, the target object set comprising keywords in the text features, text vectors corresponding to the text features and material vectors corresponding to the material features. According to the embodiment of the invention, accurate recommendation of the matched music of the target material can be realized, and the matching degree of the matched music and the material content is improved, so that the matched music effect of the target material is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to data processing technologies, and in particular, to a background music recommendation method, apparatus, electronic device, and storage medium. Background Art

[0002] With the development of mobile communication technologies and Internet technologies, it has become possible to instantaneously edit and produce user-shot videos and quickly share them on social platforms.

[0003] During video editing, selecting appropriate background music for the video helps to optimize the presentation effect of the video and thus expand the spread of the video. Currently, the background music for a video can be obtained by keyword retrieval of a pre-constructed music library. However, the retrieval results of the above method depend on the determination of the retrieval terms. If the input retrieval terms are incorrect, inappropriate background music will be returned, or even no background music can be retrieved. In addition, the background music retrieved based on the retrieval terms may not match the video content either, resulting in poor background music effects. Summary of the Invention

[0004] The present disclosure provides a background music recommendation method, apparatus, electronic device, and storage medium, which can improve the accuracy of background music recommendation and the matching degree between the background music and the material content.

[0005] In a first aspect, embodiments of the present disclosure provide a background music recommendation method, including:

[0006] Obtain a target material, determine a first target background music based on the target material through a first model, and add the background music to the target material based on the first target background music, where the first model is trained based on target material samples determined by historical interaction operations and background music samples corresponding to the target material samples;

[0007] Obtain text information, and determine a material feature, a text feature, and an intention type based on the target material and the text information through a second model, where the text information is natural language representing the background music intention of the target material;

[0008] Determine a second target background music corresponding to the target material based on at least one target object in a target object set according to the intention type, and replace the background music of the target material based on the second target background music, where the target object set includes keywords in the text feature, text vectors corresponding to the text feature, and material vectors corresponding to the material feature.

[0009] In a second aspect, embodiments of the present disclosure further provide a background music recommendation apparatus, and the apparatus includes:

[0010] The first background music determination module is configured to obtain target materials, determine a first target background music based on the target materials through a first model, and add the background music to the target materials based on the first target background music, where the first model is trained based on target material samples determined by historical interaction operations and the background music samples corresponding to the target material samples;

[0011] The feature determination module is configured to obtain text information, and determine a material feature, a text feature, and an intention type based on the target materials and the text information through a second model, where the text information is natural language characterizing the background music intention of the target materials;

[0012] The second background music determination module is configured to determine a second target background music corresponding to the target materials based on at least one target object in a target object set according to the intention type, and replace the background music of the target materials based on the second target background music, where the target object set includes keywords in the text feature, the text vector corresponding to the text feature, and the material vector corresponding to the material feature.

[0013] In a third aspect, an embodiment of the present disclosure further provides an electronic device, where the electronic device includes:

[0014] One or more processors;

[0015] A storage device for storing one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the background music recommendation method as described in any embodiment of the present disclosure.

[0017] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium containing computer-executable instructions, where the computer-executable instructions are used to execute the background music recommendation method as described in any embodiment of the present disclosure when executed by a computer processor.

[0018] Embodiments of the present disclosure provide a method, apparatus, electronic device, and storage medium for soundtrack recommendation. The first model is used to determine the first target soundtrack corresponding to the target material, and the soundtrack is added to the target material based on the first target soundtrack. Text information is obtained, and the second model is used to deeply understand and judge the intention of the target material and the text information, and the second target soundtrack is determined based on different target objects according to different intentions. Then, the soundtrack of the target material is replaced with the second target soundtrack. The technical solution of the embodiments of the present disclosure first uses the first model to determine the first target soundtrack corresponding to the target material. If text information representing the soundtrack intention is obtained, the second target soundtrack corresponding to the target material is determined by combining the soundtrack intention and the material content, and the first target soundtrack is replaced with the second target soundtrack as the soundtrack of the target material, realizing the accurate recommendation of the target material soundtrack, improving the matching degree between the soundtrack and the material content, and thus enhancing the soundtrack effect of the target material. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In combination with the accompanying drawings and with reference to the following specific embodiments, the above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic, and the original and elements are not necessarily drawn to scale.

[0020] Figure 1 It is a flowchart of a method for soundtrack recommendation provided by an embodiment of the present disclosure;

[0021] Figure 2 It is a flowchart of a method for training a first model provided by an embodiment of the present disclosure;

[0022] Figure 3 It is a flowchart of another method for soundtrack recommendation provided by an embodiment of the present disclosure;

[0023] Figure 4 It is a schematic diagram of a soundtrack link provided by an embodiment of the present disclosure;

[0024] Figure 5 It is a schematic diagram of the structure of a soundtrack recommendation apparatus provided by an embodiment of the present disclosure;

[0025] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0027] It should be understood that the various steps recited in the method embodiments of the present disclosure can be executed in a different order and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.

[0028] As used herein, the term "including" and its variants are open-ended, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of the functions executed by these devices, modules or units or their interdependent relationships.

[0030] It should be noted that the modifications of "one" and "plural" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0033] For example, in response to receiving an active request from the user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that executes the operation of the technical solution of the present disclosure according to the prompt message.

[0034] As an optional but non-limiting implementation, in response to receiving an active request from a user, the way of sending a prompt message to the user can be, for example, in the form of a pop-up window, and the prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry selection controls for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0035] It can be understood that the above notification and the process of obtaining user authorization are only illustrative and do not limit the implementation of the present disclosure. Other ways that meet relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0036] It can be understood that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of corresponding laws, regulations and related regulations.

[0037] Figure 1 FIG. is a schematic flowchart of a method for recommending background music provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the situation of background music recommendation. The method can be executed by a background music recommendation device, and the device can be implemented in the form of software and / or hardware. Optionally, it is implemented through an electronic device, and the electronic device can be a mobile terminal, a PC terminal or a server, etc.

[0038] As Figure 1 shown, the method includes:

[0039] S110. Obtain a target material, determine a first target background music based on the target material through a first model, and add the background music to the target material based on the first target background music.

[0040] Among them, the first model is trained based on a target material sample determined by historical interaction operations and a background music sample corresponding to the target material sample. For example, the first model can be a neural network model trained based on a target material sample determined by historical interaction operations and a background music sample corresponding to the target material sample. Historical interaction operations can represent interaction information corresponding to historical materials within a sampling period. For example, historical interaction operations include operations such as background music search, collection, like or sharing of historical materials.

[0041] The background music represents a music segment that is presented in cooperation with the material. By matching the music with the content of the target material, the artistic effect of the target material is enhanced. In the embodiment of the present disclosure, a music segment database can be used to store music segments. Optionally, the music segment database can store music segments in the form of vectors. For example, music information such as the lyrics or song name of a music segment is mapped into the form of a music text vector, and the music text vector is saved. The music video of the music segment is mapped into the form of a music video vector, and the music text vector and the music video vector are stored through the music segment database.

[0042] Optionally, determining the target material sample according to historical interaction operations may include: sorting candidate material samples according to the frequency of historical interaction operations, and determining the target material sample according to the sorting result. Obtain the target material, obtain the music text features and music video features of the background music in the target material sample, and train a first model in combination with the target material sample, music text features, and music video features, so that the first model learns the correlation between the features of the target material sample and the background music features. Among them, the music text features may include music text vectors mapped from music information such as the lyrics or song names of music segments. The music video features may include music video vectors mapped from the music videos of music segments. For example, within a set inspection time period, determine the sorting of candidate materials according to the occurrence frequency of the same slot features when the user searches for background music, and use the candidate materials corresponding to the frequently occurring slot features as the target materials. The slot feature is the identification information representing the background music intention in the text information. For example, the slot features include music genre, author, and language, etc. Optionally, obtain the search request input by the user, and perform slot extraction on the search request using a preset template to obtain slot features. The preset template is a template containing pre-set slots.

[0043] The target material may be a video or picture to be background-music added, etc. For example, the target material may include a video or picture with a set content theme. Obtaining the target material may be to obtain a set of background-music to-be video or background-music to-be picture pre-stored locally, etc. Or, obtain a set of submission videos or submission pictures currently shot and uploaded by the user.

[0044] Exemplarily, obtain a video or picture to be background-music added as the target material. Input the target material into the first model, and determine the first target background music based on the material features of the target material through the first model, and add background music to the target material based on the first target background music.

[0045] Optionally, obtain the music information corresponding to the first target background music; if the first target background music meets the copyright verification condition, generate a background-music material according to the first target background music, music information, and target material, where the background-music material is the target material with background music added. The music information may include music information related to the background music. For example, the music information includes music segment identification, song name, author, and music segment duration, etc. Perform copyright verification on the first target background music. If the first target background music passes the copyright verification, after encapsulating the first target background music according to the music information, combine the encapsulated first target background music and the target material to obtain the background-music material.

[0046] Due to the limitations of the training samples of the first model, it may not be able to perceive some new features in the target material. Therefore, the first target background music may not match the user's background music intention. To better recommend background music, the embodiments of the present disclosure also need to obtain the user's background music intention.

[0047] S120. Obtain text information, and determine material features, text features, and intention types based on the target material and the text information through a second model.

[0048] Among them, the text information is natural language characterizing the background music intention of the target material. The background music intention of the target material is obtained by parsing the text information. For example, the text information may include information associated with the background music intention input by the user. For example, the text information is determined through the conversation content between the user and the intelligent robot. The conversation content may include a set scene, a set location, a set time, a set event, and a set music preference, etc.

[0049] The second model represents a deep learning model trained using a large amount of text data, and is used to generate natural language text, and generate image description information corresponding to the image content, etc.

[0050] The material features can characterize the description information corresponding to the target material. For example, the material features may include video features or picture features, etc. The video features may include long text (such as content description text) and target labels about the video description, etc. The picture features may include long text (such as content description text) and target labels about the picture description, etc. The text features may include long text (such as semantic description text) and target labels summarizing or describing the text information. The long text includes semantic description text or content description text, etc.

[0051] Optionally, keyword extraction is performed on the long text to obtain keywords. Optionally, slot extraction is performed on the long text using a preset template to obtain slot features.

[0052] The intention type can be the music scoring intention determined by the second model based on the text information. For example, the second model performs intention parsing on the text information to obtain the intention type. The intention type can include the first intention type, the second intention type, the third intention type, etc. The first intention type represents that the text information contains an explicit description of the music scoring intention for the target material. The second intention type represents that the text information contains a vague description of the music scoring intention for the target material and / or a description of the attributes of the target material. Among them, the attribute description can include the theme attribute of the target material, etc. The third intention type represents that the text information contains the music scoring requirement for the target material, but does not contain the description of the music scoring intention. For example, the first intention type can represent an explicit intention, the second intention type can represent a vague intention, and the third intention type can represent a completely vague intention. The explicit intention can be the music scoring intention corresponding to a text such as "I want a video using XX music". The vague intention can be the music scoring intention corresponding to a text such as "I want a warm video", where warm is the attribute description of the video. Or, the vague intention can also be the music scoring intention corresponding to a text such as "I want a video with a travel theme, and the music is by xx singer", where the travel theme is the attribute description of the video and xx singer is the vague description of the music. Or, the vague intention can also be the music scoring intention corresponding to a text such as "I want a lively song", where lively is the vague description of the music. The completely vague intention can be something like "I want to add music to the video".

[0053] Exemplarily, based on the target material and the text information, the second model determines the material feature, the text feature, and the intention type, including: inputting the target material and the text information into the second model; outputting the material feature by the second model based on the target material; outputting the text feature and the intention type by the second model based on the text information.

[0054] In the embodiments of the present disclosure, the target material and the text information are respectively input into the second model. The second model can understand the text information to obtain the semantic description text and the target label corresponding to the text information as the text feature. In addition, the second model can also perform intention parsing on the text information to obtain the intention type. The second model can also understand the target material to obtain the content description text and the target label corresponding to the target material as the material feature. For example, according to the pre-set correspondence between the video duration and the number of frames extracted, frames are extracted from the video to be scored to obtain a video frame sequence. The video frame sequence is input into the second model, and the second model understands the content of each video frame and outputs the content description text and the target label about the video content.

[0055] In some embodiments, keywords in the text features are obtained by performing keyword extraction processing on the semantic description text. Slot features corresponding to the text features are obtained by performing slot extraction processing on the semantic description text using a preset template. Keywords in the material features are obtained by performing keyword extraction processing on the content description text. Slot features corresponding to the material features are obtained by performing slot extraction processing on the content description text using a preset template.

[0056] S130. Based on the intention type, determine a second target background music corresponding to the target material based on at least one target object in the target object set, and replace the background music of the target material with the second target background music.

[0057] Among them, the target object set includes keywords in the text features, text vectors corresponding to the text features, and material vectors corresponding to the material features. For example, the target object set includes multiple target objects, and the target object can be a keyword in the text features, a text vector corresponding to the text features, or a material vector corresponding to the material features, etc.

[0058] The second target background music can represent a music segment that matches the content of the target material and conforms to the background music intention included in the text information. Optionally, the second target background music can also be a candidate set of music segments that match the content of the video to be background-music and conform to the background music intention, and the candidate set of music segments is displayed for the user to select. Thus, the background music of the target material is replaced with the selected background music.

[0059] Exemplarily, if the intention type represents a single intention, a second target background music corresponding to the target material is determined based on at least one target object in the target object set according to the single intention.

[0060] If the intention type represents at least two intentions, a candidate background music set corresponding to the target material is determined based on each intention based on at least one target object in the target object set, the candidate background music in the candidate background music set is sorted according to historical interaction operations, and a second target background music corresponding to the target material is determined according to the sorting result.

[0061] Among them, the candidate background music set includes at least two of the background music recall results corresponding to clear intentions, the background music recall results corresponding to fuzzy intentions, and the background music recall results corresponding to completely fuzzy intentions. The background music recall results corresponding to clear intentions include candidate background music obtained by retrieving a preset multimedia content library based on keywords. The background music recall results corresponding to fuzzy intentions include candidate background music obtained by retrieving a preset multimedia content library based on text vectors, and candidate background music determined by a first model based on text vectors and material vectors. The background music recall results corresponding to completely fuzzy intentions include candidate background music output by the first model based on material vectors and text vectors. The preset multimedia content library can include a music segment database, etc.

[0062] Further, if the intention type represents a single intention, determining the second target background music corresponding to the target material based on at least one target object in the target object set according to the single intention includes: for the first intention type, retrieving a preset multimedia content library according to keywords in the text features to obtain the second target background music corresponding to the target material, where the first intention type represents that the text information contains an explicit description of the background music intention for the target material.

[0063] Further, for the second intention type, retrieving a preset multimedia content library according to the text vector corresponding to the text features to obtain a first candidate background music set, where the second intention type represents that the text information contains a vague description of the background music intention for the target material and / or an attribute description of the target material. Inputting the material vector and the text vector into the first model, and determining a second candidate background music set through the first model based on the material vector and the text vector. Determining the second target background music corresponding to the target material according to the first candidate background music set and the second candidate background music set.

[0064] Optionally, determining the second target background music corresponding to the target material according to the first candidate background music set and the second candidate background music set includes:

[0065] Sorting the first candidate background music set and the second candidate background music set according to the historical interaction operations corresponding to the first candidate background music set and the historical interaction operations corresponding to the second candidate background music set, and determining the second target background music corresponding to the target material according to the sorting result.

[0066] Specifically, obtaining the historical interaction operation data of each candidate background music in the first candidate background music set; obtaining the historical interaction operation data of each candidate background music in the second candidate background music set; sorting the candidate background music according to the historical interaction operation data, and selecting the second target background music corresponding to the target material from the first candidate background music set and the second candidate background music set according to the sorting result. For example, the candidate background music with the top N sorting can be selected as the second target background music.

[0067] Further, for the third intention type, inputting the material vector into the first model, and determining the second target background music corresponding to the target material through the first model based on the material vector.

[0068] In the embodiments of the present disclosure, replacing the background music of the target material based on the second target background music includes: if the intention type is an explicit intention, directly using the second target background music to replace the first background music in the target material as the background music of the target material.

[0069] If the intention type is the second intention type, sort the music in the second target background music in descending order according to the historical interaction operations of each music segment in the second target background music. Select the background music ranked TOP1 in the sorting to replace the first background music in the target material. Alternatively, the background music ranked TOP M can be returned to the client for the user to select. According to the single background music selection operation input by the user for the background music ranked TOP M, use one selected music segment to replace the background music of the target material. Alternatively, according to the multi-background music selection operation input by the user for the background music ranked TOP M, splice at least two selected music segments and use the spliced background music to replace the background music of the target material.

[0070] If the intention type is the third intention type, obtain the second target background music output by the first model based on the material vector and the text vector, sort the music in the second target background music in descending order according to the historical interaction operations of each music segment in the second target background music, and determine the background music of the target material according to the sorting result. The specific implementation manner is similar to that of the above embodiments and will not be elaborated here.

[0071] The technical solution of the embodiment of the present disclosure determines the first target background music corresponding to the target material through the first model, and adds the background music to the target material based on the first target background music; obtains the text information, deeply understands and judges the intention of the target material and the text information through the second model, and determines the second target background music based on different targets according to different intentions; then, uses the second target background music to replace the background music of the target material. The technical solution of the embodiment of the present disclosure first uses the first model to determine the first target background music corresponding to the target material. If the text information representing the background music intention is obtained, the second target background music corresponding to the target material is determined by combining the background music intention and the material content, and the second target background music is used to replace the first target background music as the background music of the target material, realizing the accurate recommendation of the background music of the target material, improving the matching degree between the background music and the material content, and thus, enhancing the background music effect of the target material.

[0072] Figure 2 It is a schematic flowchart of a training method of a first model provided by an embodiment of the present disclosure. On the basis of the above embodiments, the training method of the first model is additionally limited to enhance the background music recommendation ability of the first model through incremental features.

[0073] S210. Obtain the target material, determine the first target background music based on the target material through the first model, and add the background music to the target material based on the first target background music.

[0074] S220. Obtain the text information, and determine the material feature, the text feature, and the intention type based on the target material and the text information through the second model.

[0075] S230. Determine the second target background music corresponding to the target material based on at least one target object in the target object set according to the intention type, and replace the background music of the target material based on the second target background music.

[0076] S240. Determine the incremental features corresponding to the training sample set of the first model according to the text features and material features, where the incremental features are used to update the first model.

[0077] Among them, the incremental features represent the features that do not exist in the material features corresponding to the target material samples in the training sample set of the first model. For example, the text features and material features include: feature A, feature B, feature C, feature D, feature E, feature F, and feature G, and the material features corresponding to the target material samples in the training sample set of the first model include: feature A, feature B, feature C, and feature D. Then, feature E, feature F, and feature G are the incremental features.

[0078] Exemplarily, compare the text features and material features with the material features corresponding to the target material samples in the training sample set of the first model to obtain the incremental features. Record the incremental features, and monitor the interaction operations for the targeted incremental features. Determine the features used to update the first model from the incremental features according to the interaction operations. For example, monitor the candidate material samples containing feature E, feature F, or feature G, and determine that feature F is the feature whose interaction frequency meets the preset conditions. Then, use the candidate material samples containing feature F as the target material samples, add the material features and background music corresponding to the target material samples to the training sample set, and retrain the first model.

[0079] Further, updating the first model based on the incremental features includes:

[0080] S241. Obtain the historical interaction operations of the candidate material samples corresponding to the incremental features.

[0081] Exemplarily, determine the candidate material samples containing a single incremental feature, and obtain the historical interaction operations of the candidate material samples through the log.

[0082] S242. Determine the target material samples according to the historical interaction operations of the candidate material samples, and update the training sample set according to the target material samples and their corresponding background music samples.

[0083] Exemplarily, if the historical interaction operations of the candidate material samples meet the preset conditions, then determine the candidate material samples as the target material samples. Among them, the preset conditions can be determined based on factors such as the interaction operation frequency, popularity, or preference degree.

[0084] For example, candidate material samples containing incremental features are sorted in descending order according to the frequency of interaction operations, and the TOP X candidate material samples in the sorting result are determined as target material samples. The material features corresponding to the target material samples, the music text features corresponding to the music score samples, and the music video features are used as new training samples, and the new training samples are added to the training sample set.

[0085] S243. Train the first model using the updated training sample set.

[0086] Exemplarily, each training sample in the updated training sample set is input into the first model so that the first model learns the associations between the material features and the music text features and the music video features respectively. Thus, the first model learns the relationship between the incremental features and the music score recommendation, achieving recommendation enhancement.

[0087] For example, the material features corresponding to the target material samples containing incremental features are mapped into material vectors according to a predefined instruction format and input into the first model. The music text features and the music video features corresponding to the music score samples are used to supervise the model output result, and the first model is trained through forward propagation and backpropagation.

[0088] The technical solution of the embodiments of the present disclosure obtains incremental features by comparing the text features and the material features with the training sample set of the first model respectively, determines the target material samples according to the historical interaction operations of the candidate material samples corresponding to the incremental features, and then updates the training sample set according to the target material samples and their corresponding music score samples. The first model is trained using the updated training sample set, realizing the automatic perception of the incremental features of the first model and enabling the first model to learn the association between the incremental features and the music score recommendation, achieving recommendation enhancement.

[0089] Figure 3 FIG. is a schematic flowchart of another music score recommendation method provided by the embodiments of the present disclosure. On the basis of the above embodiments, it is limited to determine the second target music score corresponding to the target material when the intention type represents at least two intentions. As Figure 3 shown, the method includes:

[0090] S310. Obtain the target material, determine the first target music score based on the target material through the first model, and add the music score to the target material based on the first target music score.

[0091] S320. Obtain the text information, and determine the material features, the text features, and the intention type based on the target material and the text information through the second model, where the intention type represents at least two intentions.

[0092] For example, the intention type may include a combination of at least two of clear intention, fuzzy intention, and completely fuzzy intention.

[0093] In some embodiments, a video to be scored with music uploaded by a user is obtained, and a first target music score is determined for the video to be scored with music through a first model. Since the first target music score is a music score determined based on video features, it may not meet the user's music scoring intention. At this time, the user can input text information through multiple rounds of conversations with the robot. For example, the text information input by the user includes "I shot a video during my travel and I want to add a music score to the video. In addition, I heard song C by singer A. I want a travel video with the music score being the song by singer A and preferably a music segment similar to song C". The second model is used to perform intention recognition on the text information to obtain an intention type. The intention types include clear intention, fuzzy intention, and completely fuzzy intention Figure 3 These three types of intentions. Among them, "song C" corresponds to a clear intention. And, "a travel video with the music score being the song by singer A" corresponds to a fuzzy intention. And, "I want to add a music score to the video" corresponds to a completely fuzzy intention.

[0094] S330. For a clear intention, at least one candidate music score is retrieved from a preset multimedia content library according to the keywords in the text features.

[0095] Exemplarily, for the case of a clear intention, the keywords in the text features included in the target object set are obtained according to the clear intention, and a music segment database is retrieved according to the keywords to obtain a clear intention recall result.

[0096] S340. For a fuzzy intention, at least one candidate music score is retrieved from a preset multimedia content library according to the text vector corresponding to the text features.

[0097] Exemplarily, at least one candidate music score corresponding to the target material is retrieved according to the text vector. The music segment data includes music text vectors corresponding to music segments. The music text vectors can be determined based on contents such as the lyrics or song names corresponding to the music segments. The similarity between the text vector and the music text vectors in the music segment database is determined, and the first music segment with the similarity exceeding a preset similarity threshold is determined, and the first music segment is used as at least one candidate music score corresponding to the target material, that is, a first candidate music score set is obtained.

[0098] S350. For a fuzzy intention, the material vector and the text vector are input into the first model, and at least one candidate music score is determined by the first model based on the material vector and the text vector.

[0099] Exemplarily, the material vector and the text vector are mapped into the input information of the first model according to a predefined instruction format. Among them, the instruction information is used to inform the first model of the specific task. The first model performs music score recall and rearrangement based on the material vector and the text vector to obtain at least one candidate music score, which is the second candidate music score set.

[0100] S360. For the ambiguous intention, the at least one candidate music score retrieved and the at least one candidate music score output by the first model are mixed and arranged to obtain the ambiguous intention recall result.

[0101] S370. For the completely ambiguous intention, obtain at least one candidate music score output by the first model based on the material vector and the text vector.

[0102] Exemplarily, for the completely ambiguous intention, obtain at least one candidate music score output by the first model based on the material vector and the text vector as the completely ambiguous intention recall result.

[0103] S380. Sort the clear intention recall result, the ambiguous intention recall result, and the completely ambiguous intention recall result according to the historical interaction operations, and determine the second target music score corresponding to the target material according to the sorting result.

[0104] Exemplarily, respectively obtain the historical interaction operations corresponding to each music score in the clear intention recall result, the ambiguous intention recall result, and the completely ambiguous intention recall result, arrange the music scores in descending order according to the historical interaction operations, and use the music scores ranked in TOPX as the second target music score corresponding to the target material.

[0105] S390. Replace the music score of the target material based on the second target music score.

[0106] Figure 4 This is a schematic diagram of a music score link provided by an embodiment of the present disclosure. As Figure 4As shown, the client sends the target material 401 to the server. After the server obtains the target material 401, it determines the first target background music 402 based on the target material 401 through the first model 418. The client sends the text information 403 corresponding to the target material 401 to the server, and the text information 403 is a natural language representing the background music intention of the target material 401. Through the second model 404, intention understanding and reasoning are performed based on the target material 401 and the text information 403 to obtain text features and intention types. Through the second model 404, content understanding and reasoning are performed on the target material 401 to obtain material features. Then, according to the intention type, the corresponding retrieval service 405 is called to perform the following steps: For the case where the intention type includes a clear intention 406, the preset multimedia content library is retrieved according to the keywords included in the text features to obtain the corresponding search result list 407, and the top 1 music segment is selected from the search result list 407 as the clear intention recall result 408. For the case where the intention type includes a fuzzy intention 409, the vector similarity between the text vector corresponding to the text features and the music text vectors of each music segment in the music segment vector database 410 is determined, and the music segments with vector similarity exceeding the set threshold are used as the first candidate background music set 411. The material vector and text vector corresponding to the material features are mapped into the instruction format corresponding to the first model 418 and then input into the first model 418, and the recommendation result is determined through the first model 418, which is the second candidate background music set 412. According to the historical interaction operations, the background music in the first candidate background music set 411 and the second candidate background music set 412 are mixed, and the fuzzy intention recall result 413 is determined based on the mixing result. For the intention type including a completely fuzzy intention 414, the second candidate background music set 412 determined by the first model 418 based on the material features and text features is obtained as the completely fuzzy intention recall result 415. The historical interaction operations corresponding to each target music segment in the clear intention recall result 408, the fuzzy intention recall result 413, and the completely fuzzy intention recall result 415 are respectively obtained, and the target music segments are sorted in descending order according to the historical interaction operations, and the second target background music 416 is determined according to the sorting result. Through the result output module 417, actions such as encapsulation of music information related to the second target background music 416 and copyright verification are implemented. If the second target background music 416 meets the copyright verification conditions, the first target background music 402 in the target material is replaced according to the encapsulated second target background music 416 to obtain a new background music material, and the new background music material is sent to the client. Among them, the result output module 417 can be the Natural Language Generation (NLG) module in the second model 404, etc. The NLG module is determined based on specific rules and neural network models.

[0107] The technical solution of the embodiment of the present disclosure performs a recall operation by combining multiple intents corresponding to intent types with keywords, text features, and material features, determines a second target background music according to the historical interaction operations of the recall results, and replaces the first target background music in the target material with the second target background music, optimizing the background music retrieval process, improving the retrieval accuracy, and enhancing the background music matching degree of the target material.

[0108] Figure 5 FIG. is a schematic structural diagram of a background music recommendation device provided by an embodiment of the present disclosure. The device can be implemented in the form of software and / or hardware. Optionally, it is implemented by an electronic device, which can be a mobile terminal, a PC, or a server, etc.

[0109] As Figure 5 shown, the device includes: a first background music determination module 510, a feature determination module 520, and a second background music determination module 530.

[0110] The first background music determination module 510 is configured to obtain a target material, determine a first target background music based on the target material through a first model, and add a background music to the target material based on the first target background music, where the first model is trained based on a target material sample determined by historical interaction operations and a background music sample corresponding to the target material sample;

[0111] The feature determination module 520 is configured to obtain text information, and determine a material feature, a text feature, and an intent type based on the target material and the text information through a second model, where the text information is a natural language representing the background music intent of the target material;

[0112] The second background music determination module 530 is configured to determine a second target background music corresponding to the target material based on at least one target object in a target object set according to the intent type, and replace the background music of the target material based on the second target background music, where the target object set includes keywords in the text feature, a text vector corresponding to the text feature, and a material vector corresponding to the material feature.

[0113] Optionally, the device further includes:

[0114] An incremental feature determination module, configured to determine an incremental feature corresponding to the training sample set of the first model according to the text feature and the material feature after determining the material feature, the text feature, and the intent type based on the target material and the text information through the second model, where the incremental feature is used to update the first model.

[0115] Further, the device further includes a model training module, configured to:

[0116] Obtain the historical interaction operations of the candidate material samples corresponding to the incremental features;

[0117] Determine a target material sample based on the historical interaction operations of the candidate material samples, and update the training sample set according to the target material sample and its corresponding background music sample;

[0118] Train the first model using the updated training sample set.

[0119] Optionally, the second background music determination module 530 includes:

[0120] A first background music determination unit, configured to, if the intent type represents a single intent, determine a second target background music corresponding to the target material based on at least one target object in the target object set according to the single intent;

[0121] A second background music determination unit, configured to, if the intent type represents at least two intents, determine a candidate background music set corresponding to the target material based on each intent and at least one target object in the target object set, sort the candidate background music in the candidate background music set according to historical interaction operations, and determine a second target background music corresponding to the target material according to the sorting result.

[0122] Optionally, the first background music determination unit is specifically configured to:

[0123] For the first intent type, retrieve a preset multimedia content library according to the keywords in the text features to obtain a second target background music corresponding to the target material, where the first intent type represents that the text information contains a clear description of the background music intent for the target material.

[0124] Optionally, the first background music determination unit is specifically configured to:

[0125] For the second intent type, retrieve a preset multimedia content library according to the text vector corresponding to the text features to obtain a first candidate background music set, where the second intent type represents that the text information contains a vague description of the background music intent for the target material and / or a description of the attributes of the target material;

[0126] Input the material vector and the text vector into the first model, and determine a second candidate background music set through the first model based on the material vector and the text vector;

[0127] Determine a second target background music corresponding to the target material according to the first candidate background music set and the second candidate background music set.

[0128] Optionally, the determining a second target background music corresponding to the target material according to the first candidate background music set and the second candidate background music set includes:

[0129] Sort the first candidate music score set and the second candidate music score set according to the historical interaction operations corresponding to the first candidate music score set and the historical interaction operations corresponding to the second candidate music score set, and determine the second target music score corresponding to the target material according to the sorting result.

[0130] The music score recommendation device provided by the embodiments of the present disclosure can execute the music score recommendation method provided by any embodiment of the present disclosure, and has corresponding functional modules and beneficial effects for executing the method.

[0131] It should be noted that the various units and modules included in the above device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiments of the present disclosure.

[0132] Figure 6 It is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. Next, refer to Figure 6 , which shows a schematic structural diagram of an electronic device 600 suitable for implementing the embodiments of the present disclosure (such as Figure 6 the terminal device or server in). The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present disclosure.

[0133] As Figure 6 shown, the electronic device 600 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 into the random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The editing / output (I / O) interface 605 is also connected to the bus 604.

[0134] Typically, the following devices can be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 can allow the electronic device 600 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 6 the electronic device 600 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices can be implemented or had.

[0135] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

[0136] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0137] The electronic device provided by the embodiment of the present disclosure and the music score recommendation method provided by the above embodiment belong to the same inventive concept. The technical details not described in detail in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0138] The embodiment of the present disclosure provides a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, the music score recommendation method provided by the above embodiment is implemented.

[0139] It should be noted that the computer-readable medium described above can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, and this computer-readable signal medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0140] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0141] The above computer-readable medium can be included in the above electronic device; or it can exist separately without being assembled into the electronic device.

[0142] The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device is caused to:

[0143] Obtain target material, determine a first target background music based on the target material through a first model, and add the background music to the target material based on the first target background music, where the first model is trained based on target material samples determined by historical interaction operations and the background music samples corresponding to the target material samples;

[0144] Obtain text information, and determine material features, text features, and intent types based on the target material and the text information through a second model, where the text information is natural language representing the background music intent of the target material;

[0145] Determine the second target background music corresponding to the target material based on at least one target object in the target object set according to the intent type, and replace the background music of the target material based on the second target background music, where the target object set includes keywords in the text features, text vectors corresponding to the text features, and material vectors corresponding to the material features.

[0146] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include, but are not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions.

[0148] The units involved in the embodiments described in the present disclosure can be implemented in software or in hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself.

[0149] The functions described above herein can be performed, at least in part, by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0150] In the context of the present disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) having similar functions disclosed in the present disclosure.

[0152] In addition, although the operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although a number of specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0153] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms for implementing the claims.

Claims

1. A method for recommending background music, characterized in that, Including: Obtain a target material, determine a first target background music based on the target material through a first model, and add the background music to the target material based on the first target background music, where the first model is trained based on a target material sample determined by a historical interaction operation and a background music sample corresponding to the target material sample; Obtain text information, and determine a material feature, a text feature, and an intention type based on the target material and the text information through a second model, where the text information is natural language characterizing the background music intention of the target material; Based on the intention type, determine a second target background music corresponding to the target material based on at least one target object in a target object set, and replace the background music of the target material based on the second target background music, where the target object set includes keywords in the text feature, a text vector corresponding to the text feature, and a material vector corresponding to the material feature.

2. The method according to claim 1, wherein After determining the material feature, the text feature, and the intention type based on the target material and the text information through the second model, it further includes: Determine an incremental feature corresponding to a training sample set of the first model according to the text feature and the material feature, where the incremental feature is used to update the first model.

3. The method according to claim 2, characterized in that, Updating the first model based on the incremental feature includes: Obtain a historical interaction operation of a candidate material sample corresponding to the incremental feature; Determine a target material sample according to the historical interaction operation of the candidate material sample, and update the training sample set according to the target material sample and its corresponding background music sample; Train the first model using the updated training sample set.

4. The method according to claim 1, wherein The determining the second target background music corresponding to the target material based on at least one target object in the target object set according to the intention type includes: If the intention type represents a single intention, determine the second target background music corresponding to the target material based on the single intention and at least one target object in the target object set; If the intention type represents at least two intentions, determine a candidate background music set corresponding to the target material according to each intention and at least one target object in the target object set, sort the candidate background music in the candidate background music set according to the historical interaction operation, and determine the second target background music corresponding to the target material according to the sorting result.

5. The method according to claim 4, wherein The if the intention type represents a single intention, determining the second target background music corresponding to the target material based on the single intention and at least one target object in the target object set includes: For a first intention type, retrieve a preset multimedia content library according to the keywords in the text feature to obtain the second target background music corresponding to the target material, where the first intention type represents that the text information contains a clear description of the background music intention of the target material.

6. The method according to claim 4, characterized in that The if the intention type represents a single intention, determining the second target background music corresponding to the target material based on the single intention and at least one target object in the target object set includes: For the second intention type, retrieve a preset multimedia content library according to the text vector corresponding to the text feature to obtain a first candidate background music set, where the second intention type represents that the text information contains a fuzzy description of the background music intention for the target material and / or a description of the attributes of the target material; Input the material vector and the text vector into the first model, and determine a second candidate background music set through the first model based on the material vector and the text vector; Determine the second target background music corresponding to the target material according to the first candidate background music set and the second candidate background music set.

7. The method according to claim 6, wherein The determining the second target background music corresponding to the target material according to the first candidate background music set and the second candidate background music set includes: Sort the first candidate background music set and the second candidate background music set according to the historical interaction operations corresponding to the first candidate background music set and the historical interaction operations corresponding to the second candidate background music set, and determine the second target background music corresponding to the target material according to the sorting result.

8. A music score recommendation device, characterized in that, including: A first background music determination module, configured to obtain a target material, determine a first target background music based on the target material through a first model, and add background music to the target material based on the first target background music, where the first model is trained based on a target material sample determined by historical interaction operations and a background music sample corresponding to the target material sample; A feature determination module, configured to obtain text information, and determine a material feature, a text feature, and an intention type based on the target material and the text information through a second model, where the text information is natural language representing the background music intention of the target material; A second background music determination module, configured to determine the second target background music corresponding to the target material based on the intention type based on at least one target object in a target object set, and replace the background music of the target material based on the second target background music, where the target object set includes keywords in the text feature, the text vector corresponding to the text feature, and the material vector corresponding to the material feature.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the background music recommendation method according to any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions are used to execute the background music recommendation method according to any one of claims 1-7 when executed by a computer processor.

Images