Model training method and device, video music generation method and device, equipment and medium
By extracting multimodal data from film, television, audio and video data and building a background music matching model, the problem of low matching between soundtrack and video content in the existing technology is solved, and the background music matching driven by video content is realized, and the video quality is improved. Match degree.
Patent Information
- Application Number
- CN202411087042.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-12-06
AI Technical Summary
The existing short video soundtracking methods rely on the producer's subjective preferences, resulting in low matching between the soundtrack and the video content. The existing deep learning models cannot automatically match the most suitable background music based on the video content.
By extracting sample audio and video clips from film, television, audio and video data, extracting multimodal data and background music data, building training samples, using multimodal models to train a background music matching model, and matching the theme, atmosphere and environment information of the video content based on the video content Background music.
It realizes automatic matching of the most suitable background music based on the video content, improves the matching degree and quality of the video soundtrack, and meets the needs of non-professional video producers.
Smart Images

Figure CN119094857A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video processing technology, and in particular to a model training method, a video music matching method, a device, equipment and a medium. Background Art
[0002] With the rapid rise of the short video industry, more and more Internet users are turning from consuming video content to making their own video content. However, most people are not professional video production experts and lack certain video production knowledge and experience reserves. Background music is a very important component of short videos and is crucial to the quality of the video and the attractiveness of the video to people. Therefore, it is necessary to use artificial intelligence technology to select appropriate background music for the video.
[0003] In the related art, existing short videos with background music are usually used as training samples to train deep learning models, and then the trained deep learning models are used to compose music for short videos. However, in the existing short video industry, the music of existing short videos is often determined by the subjective preferences of short video producers, and most of them are popular music when short videos are produced. Therefore, the music of existing short videos has a low match with the video itself. The deep learning model trained based on such training samples is also unable to determine the background music with the highest match for the video based on the characteristics of the video content (including: pictures, dialogues, etc.) itself. Summary of the invention
[0004] The purpose of the embodiments of the present invention is to provide a model training method, a video music matching method, an apparatus, a device and a medium to match the most suitable background music for the video content to be matched based on the characteristics of the video content itself. The specific technical solution is as follows:
[0005] In a first aspect of the present invention, a model training method is first provided, the method comprising:
[0006] Get film and television audio and video data;
[0007] Extracting sample audio and video clips from the film and television audio and video data;
[0008] Extracting sample multimodal data and sample background music data from the sample audio and video clips, wherein the sample multimodal data represents theme information, atmosphere information, and environment information of the sample audio and video;
[0009] Constructing a training sample based on the sample background music data and the sample multimodal data;
[0010] Inputting the sample multimodal data into a multimodal model to obtain predicted background music data determined and output by the multimodal model from a candidate background music library;
[0011] The model parameters of the multimodal model are iteratively updated based on the predicted background music data and the sample background music data to obtain a background music matching model.
[0012] Optionally, the sample multimodal data includes sample text data, sample voice data and sample video data, the sample text data is used to characterize the subject information of the sample audio and video, the sample voice data is used to characterize the atmosphere information of the sample audio and video, and the sample video data is used to characterize the environmental information of the sample audio and video.
[0013] Optionally, extracting sample multimodal data and sample background music data from the sample audio and video clips includes:
[0014] Determine sample voice data and sample video data corresponding to the sample audio and video clip based on the time point information;
[0015] The sample video data is subjected to text recognition to obtain sample text data, or the sample voice data is subjected to audio conversion to obtain sample text data.
[0016] Optionally, the method further comprises:
[0017] Determine the duration information of each sample audio and video clip;
[0018] According to the duration information and target duration information of each sample audio and video segment, each sample audio and video segment is split respectively.
[0019] Optionally, based on the time point information of the segment with background music, extracting a sample audio and video segment from the film and television audio and video data includes:
[0020] Determine the duration information of the segment with background music based on the time point information of the segment with background music;
[0021] According to the duration information and the target duration information, the segment with background music is divided to obtain a plurality of sample audio and video segments.
[0022] In a second aspect of the present invention, a video music matching method is also provided, the method comprising:
[0023] Acquire multimodal data of a target video, wherein the multimodal data represents subject information, atmosphere information, and environment information of the target video;
[0024] Inputting the multimodal data and the candidate background music library into a pre-trained background music matching model to obtain the target background music matched with the target video determined and output by the background music matching model from the candidate background music library;
[0025] The background music matching is obtained by training based on any of the model training methods described above.
[0026] Optionally, before inputting the multimodal data and the candidate background music library into a pre-trained background music matching model, the method further includes:
[0027] Get background music from one or more background music libraries;
[0028] Background music with a click rate greater than a click rate threshold and / or a forwarding rate greater than a forwarding rate threshold is added to the candidate background music library.
[0029] In a third aspect of the present invention, a model training device is provided, the device comprising:
[0030] The acquisition module is used to obtain film and television audio and video data;
[0031] A first extraction module, used to extract sample audio and video clips from the film and television audio and video data;
[0032] A second extraction module is used to extract sample multimodal data and sample background music data from the sample audio and video clips, wherein the sample multimodal data represents the theme information, atmosphere information, and environment information of the sample audio and video;
[0033] A construction module, used for constructing a training sample based on the sample background music data and the sample multimodal data;
[0034] A training module is used to input the sample multimodal data into a multimodal model to obtain predicted background music data determined and output by the multimodal model from a candidate background music library; iteratively update the model parameters of the multimodal model based on the predicted background music data and the sample background music data to obtain a background music matching model.
[0035] Optionally, the sample multimodal data includes sample text data, sample voice data and video data, the sample text data is used to characterize the subject information of the sample audio and video, the sample voice data is used to characterize the atmosphere information of the sample audio and video, and the sample video data is used to characterize the environmental information of the sample audio and video.
[0036] Optionally, the first extraction module includes:
[0037] A background music segment determination submodule, used to determine the background music segments in the film and television audio and video data;
[0038] The extraction submodule is used to extract sample audio and video clips from the film and television audio and video data based on the time point information of the clip with background music.
[0039] Optionally, the second extraction module includes:
[0040] A first determination module, configured to determine voice data and video data corresponding to the sample audio and video clip based on the time point information;
[0041] The second determination module is used to perform text recognition on the video data to obtain text data, or to perform audio conversion on the voice data to obtain text data.
[0042] Optionally, the extraction submodule is specifically used to:
[0043] Determine the duration information of the segment with background music based on the time point information of the segment with background music;
[0044] According to the duration information and the target duration information, the segment with background music is divided to obtain a plurality of sample audio and video segments.
[0045] In a fourth aspect of the present invention, a video music matching device is provided, the device comprising:
[0046] An acquisition module, used to acquire multimodal data of a target video, wherein the multimodal data represents subject information, atmosphere information, and environment information of the target video;
[0047] A matching module, used for inputting the multimodal data and the candidate background music library into a pre-trained background music matching model to obtain a target background music matched with the target video, which is determined and output by the background music matching model from the candidate background music library;
[0048] The background music matching model is trained based on any one of the model training devices described above.
[0049] Optionally, the device comprises:
[0050] A capture module, used for acquiring background music from one or more background music libraries;
[0051] The determination module is used to add background music with a click rate greater than a click rate threshold and / or a forwarding rate greater than a forwarding rate threshold to a candidate background music library.
[0052] In another aspect of the present invention, there is provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0053] Memory, used to store computer programs;
[0054] The processor is used to implement the steps of any of the above-mentioned model training methods or any of the above-mentioned video music matching methods when executing the program stored in the memory.
[0055] In another aspect of the implementation of the present invention, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes any of the above-mentioned model training methods or any of the above-mentioned video soundtrack methods.
[0056] In the model training method provided in the embodiment of the present invention, training samples are constructed based on film and television audio and video data. It can be understood that a large number of professionals are invested in polishing film and television audio and video, whether from shooting or post-production, and its background music is also matched by professionals according to the needs of video content. It can be seen that the background music in the film and television audio and video and the corresponding video content (for example: video theme, video atmosphere, video screen characters, video screen environment, etc.) must have a high degree of matching. In the embodiment of the present invention, sample audio and video clips are extracted from the film and television audio and video, and then sample multimodal data and sample background music data are extracted from the sample audio and video clips, wherein the sample multimodal data can characterize the theme information, atmosphere information, and environmental information of the sample audio and video. Since the background music in the film and television audio and video is matched with the theme information, atmosphere information, and environmental information of the corresponding video content, the sample multimodal data and the sample background music data are also matched. In the embodiment of the present invention, the multimodal model is trained based on the sample multimodal data and the sample background music data to obtain a background music matching model. Therefore, the background music matching model can also match suitable background music based on the theme information, atmosphere information, and environmental information of the target video. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.
[0058] Figure 1 A schematic diagram of the steps of a model training method provided in an embodiment of the present invention;
[0059] Figure 2 A schematic diagram of the sub-step flow of step S102 in a model training method provided in an embodiment of the present invention;
[0060] Figure 3 A schematic diagram of the sub-step flow of step S103 in a model training method provided in an embodiment of the present invention;
[0061] Figure 4 A schematic diagram of an exemplary flow chart of a model training method provided in an embodiment of the present invention;
[0062] Figure 5 A schematic diagram of a step flow of a video music matching method provided in an embodiment of the present invention;
[0063] Figure 6 A structural block diagram of a model training device provided in an embodiment of the present invention;
[0064] Figure 7 A structural block diagram of a video music matching device provided in an embodiment of the present invention;
[0065] Figure 8 The present invention is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0066] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0067] See also Figure 1 , which shows a schematic flow chart of the steps of a model training method provided in an embodiment of the present invention, the model training method comprises the following steps:
[0068] S101, obtaining film and television audio and video data.
[0069] In the embodiment of the present invention, film and television audio and video data refers to the long video data corresponding to a movie or a TV series. The long video data has not been edited and created by Internet users and is the complete audio and video data corresponding to a movie or an episode of a TV series.
[0070] In the embodiment of the present invention, some film and television audio and video data can be collected from the Internet platform.
[0071] In the embodiments of the present invention, a large number of professionals are involved in polishing the film and television audio and video both in shooting and post-production, and the background music is also matched by professionals according to the needs of the video content. It can be seen that the background music in the film and television audio and video and the corresponding video content (for example: video theme, video atmosphere, video screen characters, video screen environment, etc.) must have a high degree of matching.
[0072] S102, extracting sample audio and video clips from the film and television audio and video data.
[0073] In the embodiment of the present invention, the sample audio and video clips may be obtained by dividing the film and television audio and video data according to time units. The specific time units may be determined according to actual sample requirements, and the embodiment of the present invention does not limit this.
[0074] In the embodiment of the present invention, the sample audio and video clips may also be audio and video clips with background music parts divided according to whether background music exists.
[0075] In the embodiment of the present invention, the sample audio and video clips may also be classic clips extracted from film and television audio and video data. Generally speaking, the background music in the classic clips and the corresponding video content (e.g., video theme, video atmosphere, video screen characters, video screen environment, etc.) have a higher matching degree than general film and television audio and video clips. The background music matching model trained by using such sample audio and video clips can match more suitable background music for the target video. Specifically, manual analysis or big data analysis can be used to determine the classic clips, wherein the basis of big data analysis may include: the number of times used for secondary creation, the click rate of film and television audio and video clips, etc.
[0076] S103, extracting sample multimodal data and sample background music data from the sample audio and video clips, wherein the sample multimodal data represents theme information, atmosphere information, and environment information of the sample audio and video.
[0077] In an embodiment of the present invention, sample multimodal data may include: sample text data, sample voice data, sample video data, etc., wherein the sample text data may include: lyrics text of background music or lines text of video content, the sample voice data may be: dialogue voice in the video content, and the sample video data may be the picture in the video content, including: picture subject object, picture environment background, etc.
[0078] In the embodiment of the present invention, the sample text data (the lyrics text of the background music or the dialogue text of the video content) of the sample multimodal data usually contains some content related to the theme of the video, such as a conversation between a couple, two people quarreling, a warm family conversation, etc.; the sample voice data of the sample multimodal data can usually characterize the emotion and atmosphere of the video, such as quarreling, sadness, joy, etc.; the sample video data of the sample multimodal data can characterize the location environment where the plot of the video takes place, the main object of the picture, etc. Therefore, in the embodiment of the present invention, the sample multimodal data can characterize the theme information, atmosphere information, environmental information, etc. of the sample audio and video. Specifically, the sample text data is used to characterize the theme information of the sample audio and video, the sample voice data is used to characterize the atmosphere information of the sample audio and video, and the sample video data is used to characterize the environmental information of the sample audio and video.
[0079] In an embodiment of the present invention, audio and video segmentation software (for example, ffmpeg) can be used to separate sample audio data and sample video data of each sample audio and video clip, and then separate sample background music data from the sample audio data, and then extract corresponding sample multimodal data from the sample audio and video clip based on the sample background music data.
[0080] S104: constructing a training sample based on the sample background music data and the sample multimodal data.
[0081] In an embodiment of the present invention, the sample multimodal data extracted based on the above steps S101 to S103 and the corresponding sample background music data have a high degree of matching. The sample multimodal data and the corresponding sample background music data can be used as training samples so that the multimodal model can learn the correlation between the two.
[0082] S105, inputting the sample multimodal data into a multimodal model to obtain predicted background music data determined and output by the multimodal model from a candidate background music library.
[0083] The multimodal model has the ability to perform fusion processing, reasoning and analysis on multimodal data, and the multimodal data may specifically include text data, voice data and video data.
[0084] In an embodiment of the present invention, the multimodal model is a model that is pre-trained on large-scale cross-modal data to learn shared representations between different modalities. The multimodal model can understand the basic structure and semantic information of multimodal data. In an embodiment of the present invention, the multimodal model can integrate information from different modalities (including: text modality, speech modality and image modality) to realize the model's ability to understand complex data.
[0085] In an embodiment of the present invention, the multimodal model can fuse and infer the feature information of multimodal data (including text data, voice data and video data). For example, the multimodal model can be: Multimodal Recurrent Neural Networks, Multimodal Convolutional Neural Networks, Multimodal Generative Adversarial Networks, etc.
[0086] In an embodiment of the present invention, the multimodal model may be a pre-trained model having the function of matching background music for film and television audio and video data. The multimodal model may be retrained based on the training samples (including: sample background music data and sample multimodal data) constructed in an embodiment of the present invention.
[0087] S106, iteratively updating the model parameters of the multimodal model based on the predicted background music data and the sample background music data to obtain a background music matching model.
[0088] In the embodiment of the present invention, the above steps S101 to S104 can be used to construct a preset number of training samples, each of which includes: sample background music data and corresponding sample multimodal data. The multimodal model is trained based on the preset number of training samples to obtain a background music matching model.
[0089] In an embodiment of the present invention, during the process of training the multimodal model, the multimodal model can fuse and infer the theme information, atmosphere information, and environmental information of the sample audio and video represented by the sample multimodal data, determine and output predicted background music data from the candidate background music library, and then iteratively update the model parameters of the multimodal model based on the difference between the predicted background music data and the sample background music data corresponding to the sample multimodal data in the training sample.
[0090] Specifically, vector representations of the predicted background music data and the sample multimodal data can be extracted, and the difference between the predicted background music and the sample background music can be determined based on the difference between the two vector representations. Further, a loss function can be defined to quantify the difference between the predicted background music and the sample background music. The loss function can be a mean square error loss function, a cross entropy loss function, etc. In the iterative update process, gradient descent or other optimization algorithms can be used to iteratively update the model parameters according to the loss function to minimize the difference between the predicted background music data and the sample multimodal data.
[0091] In the embodiment of the present invention, since the background music in the film and television audio and video is matched with the theme information, atmosphere information, and environmental information of the corresponding video content, the sample multimodal data and the sample background music data are also matched. Based on this, the background music matching model trained by the training samples constructed in the embodiment of the present invention can also match the appropriate background music based on the theme information, atmosphere information, and environmental information of the target video.
[0092] In combination with the above embodiments, in an optional implementation manner, the embodiment of the present invention further provides another model training method. In the model training method, the above step 102 may specifically include steps S1021 to S1022, such as Figure 2As shown, it shows a schematic diagram of the sub-step flow of step S102 in a model training method provided in an embodiment of the present invention:
[0093] S1021, determining whether there is a background music segment in the film and television audio and video data.
[0094] S1022: extracting sample audio and video clips from the film and television audio and video data based on the time point information of the clip with background music.
[0095] Specifically, it can be understood that not all segments of film and television audio and video data have background music. Therefore, an embodiment of the present invention proposes that the film and television audio and video data can be analyzed to determine the segments with background music in the audio and video data, and determine the time point information of the segments with background music. Then, based on the time point information, the segments with background music are determined from the film and television audio and video data as sample audio and video segments, and then sample multimodal data is extracted based on the sample audio and video segments.
[0096] In the embodiment of the present invention, the audio and video data of the film and television can be specifically segmented to obtain the film and television audio data and the film and television video data, and then the background music and voice contained in the audio data can be separated by using audio separation software (for example: spleeter) to obtain background music data and voice data. It can be understood that since not the entire segment of the film and television audio data has background music, the separated background music data is composed of one or more background music clips, and then the time point information of each background music clip can be obtained to determine the sample audio and video clip.
[0097] In an embodiment of the present invention, background music data and corresponding multimodal data are determined based on time point information of film and television audio and video clips with background music, so that background music data and multimodal data can be extracted from the film and television audio and video data more accurately, and a sample data set can be constructed more conveniently. Moreover, when the film and television audio and video data contains multiple clips with background music, multiple background music data and corresponding multimodal data can be extracted at one time to construct multiple sample data.
[0098] In the embodiment of the present invention, sample audio and video segments can be accurately extracted from the film and television audio and video data based on the time point information. For example, if the segment with background music in the film and television audio and video data is determined to be 10 minutes and 10 seconds to 13 minutes and 15 seconds, the sample audio and video segment can be extracted from the film and television audio and video data based on 10 minutes and 10 seconds to 13 minutes and 15 seconds.
[0099] In the embodiment of the present invention, the first preset time length can be pushed forward based on the start time point in the time point information, and the second preset time length can be pushed backward based on the end time point to obtain the second time point information, and then based on the determined second time point information, sample audio and video clips can be extracted from the film and television audio and video data, so that the film and television audio and video clips containing background music can be determined, wherein the first preset time length and the second preset time length can be the same or different, and can be determined by technical personnel based on experience. For example: the determined film and television audio and video data contains background music clips of 10 minutes and 10 seconds to 13 minutes and 15 seconds, the first preset time length and the second preset time length are both 5 seconds, and the determined second time point information is: 10 minutes and 05 seconds to 13 minutes and 20 seconds, then the sample audio and video clips can be extracted from the film and television audio and video data based on 10 minutes and 05 seconds to 13 minutes and 20 seconds.
[0100] In combination with the above embodiments, in an optional implementation manner, the embodiment of the present invention further provides another model training method. In the model training method, step S1022 includes:
[0101] Based on the time point information of the segment with background music, the duration information of the segment with background music is determined; according to the duration information and the target duration information, the segment with background music is divided to obtain multiple sample audio and video segments.
[0102] Specifically, in an embodiment of the present invention, the target duration information can be determined based on the video duration requirement of the application scenario of the background music matching model. For example, when the application scenario of the background music matching model is a short video platform, the target duration information can be obtained based on statistics of the duration information of existing short videos on the current short video platform.
[0103] In an embodiment of the present invention, based on the target duration information, the segments with background music determined in the above embodiment can be divided to obtain more training samples, and the duration of the training samples fits the video duration requirement of the application scenario.
[0104] In an embodiment of the present invention, when the duration information of the segment with background music is split according to the duration information and target duration information of the segment with background music, the segment with background music can be evenly split according to the target duration, or the core part of the video can be determined from the segment with background music and then the core part of the video can be split.
[0105] For example, if the target duration is 30 seconds and the clip with background music is 2 minutes, the clip with background music can be evenly split into four sample audio and video clips according to time, thereby determining four groups of training samples (including sample background music data and corresponding sample multimodal data). Alternatively, the middle 1 minute of the clip with background music can be used as the core part of the video, and then split to obtain two sample audio and video clips, thereby determining two groups of training samples (including sample background music data and corresponding sample multimodal data).
[0106] It is understandable that the core part of the video can be determined based on the background music data, for example, it can be the climax of the background music, or it can be determined based on the corresponding video content or the corresponding voice data.
[0107] In the embodiment of the present invention, the background music data of the training samples determined based on the core part of the video clip with background music often have a higher matching degree with the corresponding video content, and thus a background music matching model with better matching results and more accuracy can be trained.
[0108] In combination with the above embodiments, in an optional implementation manner, the embodiment of the present invention further provides another model training method. In the model training method, the above step 103 may specifically include steps S1031 to S1032, such as Figure 3 As shown, it shows a schematic diagram of the sub-step flow of step S103 in a model training method provided in an embodiment of the present invention:
[0109] S1031, determining sample voice data and sample video data corresponding to the sample audio and video clip based on the time point information.
[0110] S1032, performing text recognition on the sample video data to obtain sample text data, or performing audio conversion on the sample voice data to obtain sample text data.
[0111] In the embodiment of the present invention, based on the above step S1021, it can be determined whether there is a segment of background music in the film and television audio and video data, and further the corresponding time point information can be determined.
[0112] In the embodiment of the present invention, the sample audio and video clips may be firstly segmented into audio and video to obtain sample audio clips and sample video clips, wherein the sample video clips may be determined as sample video class data as one of the sample multimodal data.
[0113] In the embodiment of the present invention, sample voice data may be further extracted from the sample audio segment and determined as sample voice data as one of the sample multimodal data.
[0114] In the embodiment of the present invention, generally speaking, there are subtitles in the film and television audio and video data, so the corresponding sample dialogue text can be determined based on the subtitles. Specifically, text recognition can be performed on the picture frames included in the above sample video clips to obtain the dialogue text.
[0115] In the embodiment of the present invention, considering that some film and television audio and video data do not have subtitles, audio conversion can also be performed based on the sample voice data to obtain corresponding sample dialogue texts.
[0116] In the embodiment of the present invention, the sample dialogue text is determined as sample text type data as one of the sample multimodal data.
[0117] In the embodiment of the present invention, there is no specific limitation on the specific text recognition method and audio conversion method. For example, the text recognition method may be: for example, OCR (Optical Character Recognition) technology is used to perform text recognition on the picture frame image included in the video; the audio conversion method may be: using an automatic speech recognition method to extract speech text from the audio.
[0118] It can be understood that the speech data extracted from the sample audio clip can usually express the emotion and atmosphere of the video, such as quarreling, sadness, joy, etc. The video data extracted from the sample audio clip can determine the location environment, the main object of the picture, etc. of the video plot. The text data (the lyrics text of the background music or the lines text of the video content) extracted from the sample audio clip usually contains some content related to the theme of the video, such as a conversation between a couple, two people quarreling, a warm family conversation, etc. Therefore, in an embodiment of the present invention, multimodal data can characterize the theme information, atmosphere information, environmental information, etc. of the sample audio and video. Therefore, the background music matching model trained based on multimodal data and background music data can match appropriate background music based on the information of the video content itself, such as the theme information, atmosphere information, and environmental information of the target video.
[0119] For ease of understanding, in combination with the above embodiments, the present invention also provides an exemplary embodiment of a model training method, such as Figure 4 As shown, it shows an exemplary flow chart of a model training method provided in an embodiment of the present invention. Specifically, the method includes:
[0120] S1, obtain film and television audio and video data.
[0121] S2, split the film and television audio and video data to obtain video data P and audio data V.
[0122] Specifically, audio and video segmentation software (eg FFmpeg) may be used to segment the audio and video data of a film and television. In the embodiment of the present invention, the specific audio and video segmentation method is not limited.
[0123] S3, segment the audio data V to obtain background music M1 and voice data S1.
[0124] Specifically, audio separation software (eg Spleeter) may be used to separate the background music M1 and the voice data S1 contained in the audio data V. In the embodiment of the present invention, the specific audio separation method is not limited.
[0125] S4, determining a plurality of background music segments M2 based on a plurality of audio segments included in the background music M1, and determining time information of each background music segment M2.
[0126] S5, determining the corresponding voice segment S2 from the voice data S1 based on the time information of the background music segment M2.
[0127] S6, determining the corresponding video data P2 from the video data P based on the time information of the background music segment M2, and performing text recognition on the picture frames in the video segment P2 to obtain text data T.
[0128] S7, using the background music M2, voice clip S2, video data P2 and text data T obtained in the above steps as a set of training data for training a multimodal model to obtain a background music matching model.
[0129] See also Figure 5 , which shows a schematic flow chart of the steps of a video music matching method provided in an embodiment of the present invention, the video music matching method comprises the following steps:
[0130] S501, acquiring multimodal data of a target video, wherein the multimodal data represents theme information, atmosphere information, and environment information of the target video.
[0131] S502, inputting the multimodal data and the candidate background music library into a pre-trained background music matching model to obtain the target background music that matches the target video and is determined and output by the background music matching model from the candidate background music library.
[0132] The background music matching model is trained based on the model training method proposed in any of the above embodiments.
[0133] In the embodiment of the present invention, the target multimodal data of the target video can be extracted by using the method corresponding to the method for extracting the sample multimodal data of the sample audio and video clips in the above embodiment. The target multimodal data includes target text data, target voice data, and target video data. The target text data is used to characterize the theme information of the target audio and video, the target voice data is used to characterize the atmosphere information of the target audio and video, and the video data is used to characterize the environmental information of the target audio and video.
[0134] In combination with the above embodiments, in an optional implementation manner, the embodiment of the present invention further provides another video music matching method, in which the video music matching method further includes the following steps:
[0135] Obtain background music from one or more background music libraries; add background music with a click rate greater than a click rate threshold and / or a forwarding rate greater than a forwarding rate threshold to a candidate background music library.
[0136] In an embodiment of the present invention, the corresponding background music library can be determined based on the application scenario of the background music matching model. For example, when the application scenario of the background music matching model is a short video platform, the background music library can be the background music libraries corresponding to the major short video platforms.
[0137] In an embodiment of the present invention, after obtaining the background music in the scene music library, the background music can be further screened to obtain a candidate background music library for background music matching. Specifically, the background music with a high usage rate can be determined based on the existing short videos on each short video platform, and the click-through rate and / or forwarding rate of these existing short videos can be further counted to determine the background music corresponding to the short videos with a click-through rate greater than a click-through rate threshold and / or a forwarding rate greater than a forwarding rate threshold, and add them to the candidate background music library as candidate music.
[0138] In the embodiment of the present invention, background music with a click rate greater than a click rate threshold and / or a forwarding rate greater than a forwarding rate threshold is determined from a large number of background music contained in the background music library, and is added to the candidate background music library as candidate music. Based on the candidate background music library and the background music matching model, the background music matched with multimodal data of the target video not only matches the video content of the target video, but also has a high click rate and conversion rate, and the target audio and video obtained can also have a high click rate and conversion rate.
[0139] In the embodiment of the present invention, sample audio and video clips are extracted from the film and television audio and video, and then multimodal data and background music data are extracted from the sample audio and video clips, wherein the multimodal data can characterize the theme information, atmosphere information, and environmental information of the sample audio and video. Since the background music in the film and television audio and video matches the theme information, atmosphere information, and environmental information of the corresponding video content, the multimodal data and the background music data are also matched. Based on this, the background music matching model trained by the training samples constructed in the embodiment of the present invention can also match appropriate background music based on the theme information, atmosphere information, and environmental information of the target video.
[0140] The present invention also provides a model training device, such as Figure 6 As shown, it shows a structural block diagram of a model training device provided in an embodiment of the present invention; the device includes:
[0141] The acquisition module 601 is used to acquire the video and audio data of the film and television;
[0142] A first extraction module 602, used to extract sample audio and video clips from the film and television audio and video data;
[0143] A second extraction module 603 is used to extract sample multimodal data and sample background music data from the sample audio and video clips, wherein the sample multimodal data represents the theme information, atmosphere information, and environment information of the sample audio and video;
[0144] A construction module 604, configured to construct a training sample based on the sample background music data and the sample multimodal data;
[0145] The training module 605 is used to input the sample multimodal data into the multimodal model to obtain the predicted background music data determined and output by the multimodal model from the candidate background music library; iteratively update the model parameters of the multimodal model based on the predicted background music data and the sample background music data to obtain a background music matching model.
[0146] The present invention also provides a video music matching device, such as Figure 7 As shown, it shows a structural block diagram of a video music matching device provided in an embodiment of the present invention; the device includes:
[0147] An acquisition module 701 is used to acquire multimodal data of a target video, wherein the multimodal data represents subject information, atmosphere information, and environment information of the target video;
[0148] A matching module 702 is used to input the multimodal data and the candidate background music library into a pre-trained background music matching model to obtain a target background music matched with the target video determined and output by the background music matching model from the candidate background music library;
[0149] The background music matching model is trained based on any one of the model training devices described above.
[0150] The embodiment of the present invention further provides an electronic device, such as Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 504.
[0151] Memory 803, used for storing computer programs;
[0152] The processor 801 is used to execute the program stored in the memory 503, and implements the following steps:
[0153] Acquire film and television audio and video data; extract sample audio and video clips from the film and television audio and video data; extract sample multimodal data and sample background music data from the sample audio and video clips, wherein the sample multimodal data represents the theme information, atmosphere information, and environmental information of the sample audio and video; construct training samples based on the sample background music data and the sample multimodal data; input the sample multimodal data into a multimodal model to obtain predicted background music data determined and output by the multimodal model from a candidate background music library; iteratively update the model parameters of the multimodal model based on the predicted background music data and the sample background music data to obtain a background music matching model; or,
[0154] Acquire multimodal data of a target video, wherein the multimodal data represents the theme information, atmosphere information, and environmental information of the target video; input the multimodal data and a candidate background music library into a pre-trained background music matching model to obtain a target background music that matches the target video and is determined and output by the background music matching model from the candidate background music library; the background music matching model is trained based on any of the model training methods described above.
[0155] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0156] The communication interface is used for communication between the above terminal and other devices.
[0157] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0158] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0159] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the model training method or video music matching method described in any of the above embodiments.
[0160] In another embodiment provided by the present invention, a computer program product containing instructions is also provided, which, when executed on a computer, enables the computer to execute the model training method or video music matching method described in any one of the above embodiments.
[0161] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.
[0162] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0163] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. A model training method, characterized in that: The method comprises: Get film and television audio and video data; Extracting sample audio and video clips from the film and television audio and video data; Extracting sample multimodal data and sample background music data from the sample audio and video clips, wherein the sample multimodal data represents theme information, atmosphere information, and environment information of the sample audio and video; Constructing a training sample based on the sample background music data and the sample multimodal data; Inputting the sample multimodal data into a multimodal model to obtain predicted background music data determined and output by the multimodal model from a candidate background music library; The model parameters of the multimodal model are iteratively updated based on the predicted background music data and the sample background music data to obtain a background music matching model.
2. The model training method according to claim 1, characterized in that: The sample multimodal data includes sample text data, sample voice data and sample video data. The sample text data is used to characterize the subject information of the sample audio and video, the sample voice data is used to characterize the atmosphere information of the sample audio and video, and the sample video data is used to characterize the environmental information of the sample audio and video.
3. The model training method according to claim 1, characterized in that: Extracting sample audio and video clips from the film and television audio and video data includes: Determining whether there is a background music segment in the film and television audio and video data; Based on the time point information of the segment with background music, a sample audio and video segment is extracted from the film and television audio and video data.
4. The model training method according to claim 3, characterized in that: Extracting sample multimodal data and sample background music data from the sample audio and video clips includes: Determine sample voice data and sample video data corresponding to the sample audio and video clip based on the time point information; The sample video data is subjected to text recognition to obtain sample text data, or the sample voice data is subjected to audio conversion to obtain sample text data.
5. The model training method according to claim 3, characterized in that: Extracting sample audio and video clips from the film and television audio and video data based on the time point information of the clip with background music, including: Determine the duration information of the segment with background music based on the time point information of the segment with background music; According to the duration information and the target duration information, the segment with background music is divided to obtain a plurality of sample audio and video segments.
6. A video music matching method, characterized in that: The method comprises: Acquire multimodal data of a target video, wherein the multimodal data represents subject information, atmosphere information, and environment information of the target video; Inputting the multimodal data and the candidate background music library into a pre-trained background music matching model to obtain the target background music matched with the target video determined and output by the background music matching model from the candidate background music library; The background music matching model is trained based on the model training method described in any one of claims 1 to 5.
7. The video music matching method according to claim 6, characterized in that: Before inputting the multimodal data and the candidate background music library into a pre-trained background music matching model, the method further includes: Get background music from one or more background music libraries; Background music with a click rate greater than a click rate threshold and / or a forwarding rate greater than a forwarding rate threshold is added to the candidate background music library.
8. A model training device, characterized in that: The device comprises: The acquisition module is used to obtain film and television audio and video data; A first extraction module, used to extract sample audio and video clips from the film and television audio and video data; A second extraction module is used to extract sample multimodal data and sample background music data from the sample audio and video clips, wherein the sample multimodal data represents the theme information, atmosphere information, and environment information of the sample audio and video; A construction module, used to construct a training sample based on the sample background music data and the sample multimodal data; A training module is used to input the sample multimodal data into a multimodal model to obtain predicted background music data determined and output by the multimodal model from a candidate background music library; iteratively update the model parameters of the multimodal model based on the predicted background music data and the sample background music data to obtain a background music matching model.
9. A video music matching device, comprising: An acquisition module, used to acquire multimodal data of a target video, wherein the multimodal data represents subject information, atmosphere information, and environment information of the target video; A matching module, used for inputting the multimodal data and the candidate background music library into a pre-trained background music matching model to obtain a target background music matched with the target video, which is determined and output by the background music matching model from the candidate background music library; The background music matching model is obtained by training based on the model training device described in claim 8 above.
10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the model training method described in any one of claims 1 to 5 or the video soundtrack method described in claim 6 or 7 when executing the program stored in the memory.
11. A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, which, when executed on a computer, enables the computer to execute the model training method described in any one of claims 1 to 5 or the video soundtrack method described in claim 6 or 7.
