Video score recommendation method and device based on deep learning, equipment and medium

Abstract

The invention relates to the technical field of deep learning, and discloses a video music recommendation method and device based on deep learning, equipment and a medium, and the method comprises the steps: obtaining a video to be matched with music and background music to be matched with the video to be matched with music; extracting key frame data of the video to be dubbed with music in each time period, and calculating a change value of the key frame data of the video to be dubbed with music in each time period; extracting feature information of the background music, and calculating a melody value of the background music according to the extracted feature information of the background music; and matching the change value of the key frame data of the video to be dubbed with the melody value of the background music, and recommending the background music to the video to be dubbed according to a matching result. According to the video music recommendation method based on deep learning, the efficiency of recommending background music to the video to be matched with music is effectively improved, and a large amount of labor cost is saved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method, device, equipment and medium for recommending video music based on deep learning. Background Art

[0002] With the rapid development of Internet technology, videos have gradually become the main source of information and entertainment for people in their daily lives. More and more users are participating in video creation. In order to improve the appeal of videos, users can choose background music that matches the video according to the content characteristics of the video. In the prior art, suitable background music is often matched to the video manually, that is, when making a video, the user needs to watch the video content, and then search for background music that matches the video content according to the characteristics of the video content, and finally find suitable background music in the background music library based on experience.

[0003] The effect of manually matching background music for videos is heavily dependent on the user's experience with background music. That is, users need to listen to a lot of background music and be familiar with the melodies of various background music before they can find background music that matches the video content. However, most of the time, users do not have rich experience in judging what kind of background music can fit the video content, and they do not even know how to find a background music library. As a result, the effect of recommending background music for videos is not good during the video creation process. Summary of the invention

[0004] The present invention provides a method, device, equipment and medium for recommending video music based on deep learning, so as to solve the technical problem that the effect of manually matching background music for a video to be matched with music is seriously dependent on the user's accumulated experience with background music, which results in poor effect of recommending background music for the video during the video creation process.

[0005] In a first aspect, a method for recommending video music based on deep learning is provided, comprising:

[0006] Obtaining a video to be matched with music and background music to be matched with the video to be matched with music;

[0007] Extracting key frame data in each time period of the video to be matched with music, and calculating the change value of the key frame data in each time period of the video to be matched with music;

[0008] Extracting characteristic information of the background music, and calculating a melody value of the background music according to the extracted characteristic information of the background music;

[0009] The change value of the key frame data of the video to be matched with music is matched with the melody value of the background music, and the background music is recommended to the video to be matched with music according to the matching result.

[0010] In a second aspect, a video music recommendation device based on deep learning is provided, comprising:

[0011] An acquisition module is used to acquire a video to be matched with music, and background music to be matched with the video to be matched with music;

[0012] A first calculation module is used to extract key frame data in each time period of the video to be matched with music, and calculate the change value of the key frame data in each time period of the video to be matched with music;

[0013] A second calculation module, used for extracting characteristic information of the background music, and calculating a melody value of the background music according to the extracted characteristic information of the background music;

[0014] The matching module is used to match the change value of the key frame data of the video to be matched with music with the melody value of the background music, and recommend background music to the video to be matched with music according to the matching result.

[0015] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the above-mentioned deep learning-based video soundtrack recommendation method are implemented.

[0016] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned deep learning-based video soundtrack recommendation method are implemented.

[0017] In the scheme implemented by the above-mentioned deep learning-based video music recommendation method, device, equipment and medium, the video to be matched with music and the background music to be matched with the video to be matched with music can be obtained through the client; the key frame data in each time period of the video to be matched with music is extracted, and the change value of the key frame data in each time period of the video to be matched with music is calculated; the feature information of the background music is extracted, and the melody value of the background music is calculated according to the extracted feature information of the background music; the change value of the key frame data of the video to be matched with music is matched with the melody value of the background music, and the background music is recommended to the video to be matched with music according to the matching result, and the recommended background music is fed back to the client. In the present invention, for the video to be matched with music, the feature information of the background music to be matched with the video to be matched with music can be first extracted through a deep learning model, and the melody value of the background music is calculated according to the extracted feature information of the background music, and then the change value of the key frame data of the video to be matched with music is matched with the melody value of the background music, and the background music is automatically recommended to the video to be matched with music according to the matching result, which effectively improves the efficiency of recommending background music for the video to be matched with music and saves a lot of labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0019] Figure 1 is a schematic diagram of an application environment of a video music recommendation method based on deep learning in one embodiment of the present invention;

[0020] Figure 2 is a flow chart of a method for recommending video music based on deep learning in one embodiment of the present invention;

[0021] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S20;

[0022] Figure 4 yes Figure 1 A schematic flow chart of a specific implementation of step S30;

[0023] Figure 5 yes Figure 4 A schematic flow chart of a specific implementation of step S33;

[0024] Figure 6 yes Figure 5 A flowchart of a specific embodiment of step S332;

[0025] Figure 7 for Figure 1 A flowchart of a specific embodiment of step S40;

[0026] Figure 8 is a structural schematic diagram of a video music recommendation device based on deep learning in one embodiment of the present invention;

[0027] Fig. 9 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0028] Fig.10 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The video music recommendation method based on deep learning provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, Figure 1 : is a schematic diagram of an application environment of a method for recommending video music based on deep learning in an embodiment of the present invention. Among them, the client communicates with the server through a network. The server can obtain the video to be matched with music through the client, and the background music to be matched with the video to be matched with music; extract the key frame data in each time period of the video to be matched with music, and calculate the change value of the key frame data in each time period of the video to be matched with music; extract the feature information of the background music, and calculate the melody value of the background music according to the extracted feature information of the background music; match the change value of the key frame data of the video to be matched with music with the melody value of the background music, and recommend the background music to the video to be matched with music according to the matching result, and feed the recommended background music back to the client. In the present invention, for the video to be matched with music, the feature information of the background music to be matched with the video to be matched with music can be first extracted through a deep learning model, and the melody value of the background music can be calculated according to the extracted feature information of the background music, and then the change value of the key frame data of the video to be matched with music can be matched with the melody value of the background music, and the background music can be automatically recommended to the video to be matched with music according to the matching result, which effectively improves the efficiency of recommending background music for the video to be matched with music, and saves a lot of labor costs. The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] See also Figure 2 As shown, Figure 2 1 is a flow chart of a method for recommending video music based on deep learning in one embodiment of the present invention, comprising the following steps:

[0032] S10: Obtain a video to be matched with music and background music to be matched with the video to be matched with music.

[0033] The deep learning-based video music recommendation method provided by the present invention can be applied to video production in various application scenarios. For example, on the Douyin platform, after shooting or editing a video, the user selects and determines the background music that matches the video data according to the content, style and other characteristics of the video data, so as to improve the appeal of the video and further improve the promotion efficiency of the video and the user experience.

[0034] It should be understood that in the embodiment of the present invention, when obtaining background music to be matched with the video to be accompanied by music, background music sent by other electronic devices can be received; background music can also be searched from local storage; of course, background music can also be obtained from a third-party database. The specific settings can be made according to actual needs. Here, the embodiment of the present invention only takes these three types of background music as an example for explanation, but it does not mean that the embodiments of the present application are limited to this.

[0035] S20: extracting key frame data in each time period of the video to be matched with music, and calculating the change value of the key frame data in each time period of the video to be matched with music.

[0036] After obtaining the video to be matched with music and background music, in order to match the video to be matched with background music of similar style, it is necessary to first analyze the key frame data in each time period of the video to be matched with music. In the embodiment of the present invention, the key frame data of the video to be matched with music can be extracted by computer vision technology, and the content of the video to be matched with music can be compared according to different time periods, and the style of the background music that needs to be matched with the video to be matched with music can be determined according to the change pattern of the key frame data of the video to be matched with music in different time periods. For example, if it is recognized that the appearance of a person or animal in the video to be matched with music changes regularly, and the change pattern is large, it is judged that the video to be matched with music is suitable for background music with a strong sense of rhythm.

[0037] It should be understood that to extract the key frame data of the video to be matched with music, it is necessary to determine the extraction method of the key frame data of the video to be matched with music according to different compression methods. The compression method of the video to be matched with music can be first identified by the suffix name or header information, and then the key frame of the video to be matched with music can be extracted according to the identified compression method. For example, if it is identified that the video to be matched with music is in MGPG format, it means that each frame in the video to be matched with music is a GPG picture. It is necessary to analyze each GPG picture and calculate the change value of the key frame data in each time period of the video to be matched with music. Figure 3 As stated, Figure 3 yes Figure 1 A flow chart of a specific implementation method of step S20 in the embodiment; in step S20, the key frame data in each time period of the video to be matched with music is extracted, and the change value of the key frame data in each time period of the video to be matched with music is calculated, including the following steps:

[0038] S21: Preset a dynamic change threshold between two adjacent key frames of the video to be matched with music, and identify the difference between two adjacent key frames in each time period of the video to be matched with music.

[0039] S22: Compare the difference between two adjacent key frames in each time period of the video to be matched with music with a preset dynamic change threshold to obtain a change value of the key frame data in each time period of the video to be matched with music.

[0040] For steps S21-S22, when analyzing the key frame data in each time period of the video to be matched with music, you can first preset a dynamic change threshold between two adjacent key frames of the video to be matched with music, and at the same time, judge the difference between two adjacent key frames in each time period of the video to be matched with music. The size of the difference between two adjacent key frames in each time period of the video to be matched with music is judged according to the preset dynamic change threshold, so as to analyze the change pattern of the key frame data in each time period of the video to be matched with music.

[0041] If you want to analyze the changing rules of the key frame data in each time period of the video to be matched with music, you need to compare the difference between two adjacent key frames in each time period of the video to be matched with music with the preset dynamic change threshold. If the difference between two adjacent key frames of the video to be matched with music exceeds the preset dynamic change threshold, it means that the change value between two adjacent key frames of the video to be matched with music is large. If the difference between two adjacent key frames of the video to be matched with music does not exceed the preset dynamic change threshold, it means that the change value between two adjacent key frames of the video to be matched with music is small. According to the size of the change value of the key frame data in each time period of the video to be matched with music, you can query the music style that is more suitable for the video to be matched with music. For example, if it is recognized that there are mountains and rivers in the key frame picture of the video to be matched with music, and the change value between two adjacent key frame pictures is small, it can be considered that a "soft" melody will be more suitable for the video to be matched with music.

[0042] S30: extracting characteristic information of the background music, and calculating the melody value of the background music according to the extracted characteristic information of the background music.

[0043] After calculating the change value of the key frame data in each time period of the video to be matched with music, in order to more accurately recommend background music for the video to be matched with music, it is necessary to extract the characteristic information of the background music. This solution can analyze the characteristic information of the background music from the characteristics of "beat speed", "rhythm pattern", "difference in the intensity of strong beats and weak beats" and "rhythm complexity" of the background music. Among them, "beat speed", "rhythm pattern" and "rhythm complexity" are related to the dynamics of the background music. For example, a faster beat speed is usually related to a stronger dynamic. The continuous appearance of fast quarter notes or eighth notes indicates that the background music is more dynamic. Continuous strong beats or repeated rhythm patterns may enhance the dynamics of the music. More complex and diverse rhythm patterns in the background music may increase the sense of dynamics; and "difference in the intensity of strong beats and weak beats" is related to the dynamic changes of the background music. For example, in a strong dynamic rhythm, the strong beat is usually more prominent and stronger than the weak beat, resulting in an obvious dynamic change.

[0044] In the embodiment of the present invention, in step S30, feature information of background music is extracted, which can be referred to Figure 4 , Figure 4 yes Figure 1 A specific implementation flow diagram of step S30 includes the following steps:

[0045] S31: collecting the analog audio signal of the background music, and converting the analog audio signal of the background music into a digital signal.

[0046] When extracting the characteristic information of background music, music signal processing can be performed first. It should be understood that the background music collected by the present invention is an analog audio signal, and the analog audio signal of the background music needs to be converted into a digital signal, and then the characteristic information of the background music is extracted based on the converted digital signal.

[0047] S32: extracting characteristic information of the background music converted into a digital signal through a beat detection algorithm to obtain the beat points of the background music.

[0048] After the analog audio signal of the background music is converted into a digital signal, the characteristic information of the background music after conversion into a digital signal can be extracted through beat detection algorithms such as autocorrelation function, difference function and peak detection, and the beat points of the background music can be detected by analyzing the energy and spectrum of the audio signal of the background music.

[0049] S33: performing melody digital processing on the beat points of the background music to obtain the melody value of the background music.

[0050] After obtaining the beat points of the background music, the beat points of the background music can be digitized to calculate the "melody value" of the background music, so that the background music can be matched to the soundtrack video according to the melody value. In the embodiment of the present invention, in step S33, the "melody value" of the background music is calculated, which can be referred to Figure 5 , Figure 5 yes Figure 4 The flowchart of a specific embodiment of step S33 in the embodiment includes the following steps:

[0051] S331: extracting the note sequence of the background music through a time-frequency analysis algorithm, and extracting the main pitch information of the background music through a peak detection algorithm.

[0052] S332: predicting the melody attribute label of the background music according to the note sequence of the background music and the main pitch information of the background music, and converting the melody attribute label of the background music into a digital melody value.

[0053] For step S331, the melody of the beat points of the background music is digitized, and the melody data can be extracted according to the storage format of the background music. For example, if the music is created or stored in MIDI format, the note sequence and main pitch information in the MIDI data can be directly extracted as melody data. Among them, the notes can be identified by a time-frequency analysis algorithm, and the note sequence of the background music melody can be extracted. The audio signal is converted into a spectrogram or spectrum data by a pitch conversion method, and then the main pitch information in the spectrum is extracted by a peak detection algorithm or other algorithms.

[0054] After extracting the note sequence and main pitch information of the background music in the above step S331, the melody attribute label of the background music can be predicted based on the note sequence and main pitch information of the background music. For example, the melody attribute label of the background music is predicted to be a melody label of "graceful" based on the note sequence and main pitch information of the background music, that is, the attribute label of the background music is predicted to be a melody label of "soft". After predicting the melody attribute label of the background music, the "text label" of the background music can be converted into a digital "melody value" according to a machine learning algorithm such as label mapping. For example, "graceful" is converted to "0" and "warm" is converted to "1", that is, if the attribute label of the background music is predicted to be a melody label of "soft", the melody value of the background music is 1, and if the attribute label of the background music is predicted to be a melody label of "warm", the melody value of the background music is "0".

[0055] In the above step S332, the melody label of the background music is predicted according to the note sequence and main pitch information of the background music. The note sequence and main pitch information of the background music can be input into a convolutional neural network model to obtain a prediction result of the melody label of the background music. The convolutional neural network model can be used to predict the melody label of the background music. The convolutional neural network model has a powerful feature extraction capability and can capture the relationship between the features of the background music, making the prediction result of the melody label of the background music more accurate. In this embodiment of the present invention, in step S332, the melody label of the background music is predicted, which can refer to Figure 6 , Figure 6 yes Figure 5 The flowchart of a specific embodiment of step S332 in the embodiment includes the following steps:

[0056] S3321: Input the note sequence and pitch information of the background music into the convolution layer of the convolutional neural network model, and extract the note sequence feature information of the background music and the main pitch feature information of the background music.

[0057] S3322: Input the note sequence feature information of the background music and the main pitch feature information of the background music into the feature fusion layer of the convolutional neural network model to obtain the melody target feature information of the background music.

[0058] S3323: Input the melody target feature information of the background music into the fully connected layer of the convolutional neural network model to obtain the prediction result of the melody label of the background music.

[0059] For steps S3321-S3323, to extract the note sequence and pitch information of the background music, it is necessary to input the note sequence and pitch information of the background music into the convolutional layer of the convolutional neural network model for feature extraction. The convolutional layer can capture the feature information of the background music melody.

[0060] After extracting the note sequence feature information of the background music and the main pitch feature information of the background music in step S3321, the note sequence feature information of the background music and the main pitch feature information of the background music may be feature fused to extract more complete feature information of the background music melody.

[0061] After obtaining the melody target feature information of the background music in step S3322, the melody target feature information of the background music can be input into the fully connected layer of the convolutional neural network model for prediction. The fully connected layer can convert the melody target feature map of the background music into the vector required to predict the melody value of the background music, and finally output the prediction result of the melody label of the background music.

[0062] S40: Matching the change value of the key frame data of the video to be matched with music with the melody value of the background music, and recommending background music to the video to be matched with music according to the matching result.

[0063] After the above steps S20 and S30 obtain the change value of the key frame data of the video to be matched with music and the melody value of the background music, it is necessary to match the change value of the key frame data of the video to be matched with music and the melody value of the background music, and recommend background music to the video to be matched with music according to the matching result. The present invention digitizes the melody of the background music, and after obtaining the melody value of the background music, it is necessary to match the change value of the key frame data of the video to be matched with music and the melody value of the background music. For example, if the change value of the key frame data of the video to be matched with music is small, it can be considered that a "soft" melody will be more suitable for the video to be matched with music. If the change value of the key frame data of the video to be matched with music is matched with the "soft" background music with a melody value of 0, the match will be successful. If the change value of the key frame data of the video to be matched with music is matched with the "warm" background music with a melody value of 1, the match will fail.

[0064] In the embodiment of the present invention, in step S40, the change value of the key frame data of the video to be matched with the melody value of the background music is matched, which can be referred to Figure 7 , Figure 7 yes Figure 1 The flowchart of a specific embodiment of step S40 in FIG. 1 includes the following steps:

[0065] S41: Calculate the similarity between the change value of the key frame data of the video to be matched with music and the melody value of the background music.

[0066] S42: Sort the calculated similarities between the change values ​​of the key frame data of the video to be matched with music and the melody values ​​of the background music, and recommend the background music with the highest similarity as the background music of the video to be matched with music.

[0067] For steps S41-S42, the change value of the key frame data of the video to be matched with the melody value of the background music is matched. A similarity algorithm such as a cosine similarity algorithm can be used to calculate the similarity between the change value of the key frame data of the video to be matched with the melody value of the background music.

[0068] After calculating the similarity between the change value of the key frame data of the video to be matched with music and the melody value of the background music in the above step S41, the calculated similarities can be sorted to select more suitable background music for the video to be matched with music.

[0069] It can be seen that in the above scheme, for the video to be matched with music, the feature information of the background music to be matched with the video to be matched can be first extracted through the deep learning model, and the melody value of the background music can be calculated based on the extracted feature information of the background music. Then, the change value of the key frame data of the video to be matched with the melody value of the background music can be matched, and the background music can be automatically recommended to the video to be matched based on the matching result, which effectively improves the efficiency of recommending background music for the video to be matched and saves a lot of labor costs.

[0070] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0071] In one embodiment, a video music recommendation device based on deep learning is provided, and the video music recommendation device based on deep learning corresponds to the video music recommendation method based on deep learning in the above embodiment. Figure 8 As shown, Figure 8 80 is a schematic diagram of a structure of a video music recommendation device based on deep learning in one embodiment of the present invention. The video music recommendation device based on deep learning 80 includes an acquisition module 81, a first calculation module 82, a second calculation module 83, and a matching module 84. The functional modules are described in detail as follows:

[0072] An acquisition module 81 is used to acquire a video to be matched with music, and background music to be matched with the video to be matched with music;

[0073] A first calculation module 82 is used to extract key frame data in each time period of the video to be matched with music, and calculate the change value of the key frame data in each time period of the video to be matched with music;

[0074] A second calculation module 83 is used to extract characteristic information of the background music and calculate the melody value of the background music according to the extracted characteristic information of the background music;

[0075] The matching module 84 is used to match the change value of the key frame data of the video to be matched with music with the melody value of the background music, and recommend background music to the video to be matched with music according to the matching result.

[0076] In one embodiment, the first calculation module 82 is specifically configured to:

[0077] Presetting a dynamic change threshold between two adjacent key frames of the video to be matched with music, and identifying the difference between two adjacent key frames in each time period of the video to be matched with music;

[0078] The difference between two adjacent key frames in each time period of the video to be matched with music is compared with a preset dynamic change threshold to obtain a change value of the key frame data in each time period of the video to be matched with music.

[0079] In one embodiment, the second calculation module 83 is specifically configured to:

[0080] Collecting the analog audio signal of the background music, and converting the analog audio signal of the background music into a digital signal;

[0081] Extracting characteristic information of the background music converted into a digital signal through a beat detection algorithm to obtain the beat points of the background music;

[0082] The beat points of the background music are digitally processed to obtain the melody value of the background music.

[0083] In one embodiment, the second calculation module 83 is specifically configured to:

[0084] Extracting the note sequence of the background music by using a time-frequency analysis algorithm, and extracting the main pitch information of the background music by using a peak detection algorithm;

[0085] The melody attribute label of the background music is predicted according to the note sequence of the background music and the main pitch information of the background music, and the melody attribute label of the background music is converted into a digital melody value.

[0086] In one embodiment, the second calculation module 83 is specifically configured to:

[0087] The note sequence and main pitch information of the background music are input into a convolutional neural network model to obtain a prediction result of the melody label of the background music.

[0088] In one embodiment, the second calculation module 83 is specifically configured to:

[0089] Inputting the note sequence and pitch information of the background music into the convolution layer of the convolutional neural network model, extracting the note sequence feature information of the background music and the main pitch feature information of the background music;

[0090] Inputting the note sequence feature information of the background music and the main pitch feature information of the background music into the feature fusion layer of the convolutional neural network model to obtain the melody target feature information of the background music;

[0091] The melody target feature information of the background music is input into the fully connected layer of the convolutional neural network model to obtain the prediction result of the melody label of the background music.

[0092] In one embodiment, the matching module 84 is specifically configured to:

[0093] Calculating the similarity between the change value of the key frame data of the video to be matched with music and the melody value of the background music;

[0094] The calculated similarities between the change values ​​of the key frame data of the video to be matched with music and the melody values ​​of the background music are sorted, and the background music with the highest similarity is recommended as the background music of the video to be matched with music.

[0095] The present invention provides a video music recommendation device based on deep learning. The device first extracts feature information of background music to be matched with a video to be matched with music through a deep learning model, calculates the melody value of the background music based on the extracted feature information of the background music, and then matches the change value of key frame data of the video to be matched with the melody value of the background music. The device automatically recommends background music to the video to be matched with music based on the matching result, thereby effectively improving the efficiency of recommending background music for the video to be matched with music and saving a lot of labor costs.

[0096] For the specific definition of the video music recommendation device based on deep learning, please refer to the definition of the video music recommendation method based on deep learning above, which will not be repeated here. Each module in the above-mentioned video music recommendation device based on deep learning can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0097] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown, Fig. 9 1 is a schematic diagram of the structure of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a video soundtrack recommendation method based on deep learning are implemented.

[0098] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.10 As shown, Fig.10: is another structural schematic diagram of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a client side of a video soundtrack recommendation method based on deep learning.

[0099] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0100] Obtaining a video to be matched with music and background music to be matched with the video to be matched with music;

[0101] Extracting key frame data in each time period of the video to be matched with music, and calculating the change value of the key frame data in each time period of the video to be matched with music;

[0102] Extracting characteristic information of the background music, and calculating a melody value of the background music according to the extracted characteristic information of the background music;

[0103] The change value of the key frame data of the video to be matched with music is matched with the melody value of the background music, and the background music is recommended to the video to be matched with music according to the matching result.

[0104] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0105] Obtaining a video to be matched with music and background music to be matched with the video to be matched with music;

[0106] Extracting key frame data in each time period of the video to be matched with music, and calculating the change value of the key frame data in each time period of the video to be matched with music;

[0107] Extracting characteristic information of the background music, and calculating a melody value of the background music according to the extracted characteristic information of the background music;

[0108] The change value of the key frame data of the video to be matched with music is matched with the melody value of the background music, and the background music is recommended to the video to be matched with music according to the matching result.

[0109] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0110] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0111] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0112] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A video music recommendation method based on deep learning, characterized in that: include: Obtaining a video to be matched with music and background music to be matched with the video to be matched with music; Extracting key frame data in each time period of the video to be matched with music, and calculating the change value of the key frame data in each time period of the video to be matched with music; Extracting characteristic information of the background music, and calculating a melody value of the background music according to the extracted characteristic information of the background music; The change value of the key frame data of the video to be matched with music is matched with the melody value of the background music, and the background music is recommended to the video to be matched with music according to the matching result.

2. The method for recommending video music based on deep learning according to claim 1, characterized in that: The calculating of the change value of the key frame data in each time period of the video to be matched with music includes: Presetting a dynamic change threshold between two adjacent key frames of the video to be matched with music, and identifying the difference between two adjacent key frames in each time period of the video to be matched with music; The difference between two adjacent key frames in each time period of the video to be matched with music is compared with a preset dynamic change threshold to obtain a change value of the key frame data in each time period of the video to be matched with music.

3. The video music recommendation method based on deep learning according to claim 2, characterized in that: The extracting characteristic information of the background music comprises: Collecting the analog audio signal of the background music, and converting the analog audio signal of the background music into a digital signal; Extracting characteristic information of the background music converted into a digital signal through a beat detection algorithm to obtain the beat points of the background music; The beat points of the background music are digitally processed to obtain the melody value of the background music.

4. The method for recommending video music based on deep learning according to claim 3, characterized in that: The step of performing melody digital processing on the beat points of the background music to obtain the melody value of the background music includes: Extracting the note sequence of the background music by using a time-frequency analysis algorithm, and extracting the main pitch information of the background music by using a peak detection algorithm; The melody attribute label of the background music is predicted according to the note sequence of the background music and the main pitch information of the background music, and the melody attribute label of the background music is converted into a digital melody value.

5. The method for recommending video music based on deep learning according to claim 4, characterized in that: Predicting the melody label of the background music according to the note sequence and main pitch information of the background music includes: The note sequence and main pitch information of the background music are input into a convolutional neural network model to obtain a prediction result of the melody label of the background music.

6. The method for recommending video music based on deep learning according to claim 5, characterized in that: The step of inputting the note sequence and main pitch information of the background music into a convolutional neural network model to obtain a prediction result of the melody label of the background music comprises: Inputting the note sequence and pitch information of the background music into the convolution layer of the convolutional neural network model, extracting the note sequence feature information of the background music and the main pitch feature information of the background music; Inputting the note sequence feature information of the background music and the main pitch feature information of the background music into the feature fusion layer of the convolutional neural network model to obtain the melody target feature information of the background music; The melody target feature information of the background music is input into the fully connected layer of the convolutional neural network model to obtain the prediction result of the melody label of the background music.

7. The method for recommending video music based on deep learning according to claim 6, characterized in that: The step of matching the change value of the key frame data of the video to be matched with music with the melody value of the background music, and recommending background music to the video to be matched with music according to the matching result, comprises: Calculating the similarity between the change value of the key frame data of the video to be matched with music and the melody value of the background music; The calculated similarities between the change values ​​of the key frame data of the video to be matched with music and the melody values ​​of the background music are sorted, and the background music with the highest similarity is recommended as the background music of the video to be matched with music.

8. A video music recommendation device based on deep learning, characterized in that: include: An acquisition module is used to acquire a video to be matched with music, and background music to be matched with the video to be matched with music; A first calculation module is used to extract key frame data in each time period of the video to be matched with music, and calculate the change value of the key frame data in each time period of the video to be matched with music; A second calculation module, used for extracting characteristic information of the background music, and calculating a melody value of the background music according to the extracted characteristic information of the background music; The matching module is used to match the change value of the key frame data of the video to be matched with music with the melody value of the background music, and recommend background music to the video to be matched with music according to the matching result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the video soundtrack recommendation method based on deep learning are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the video soundtrack recommendation method based on deep learning are implemented as described in any one of claims 1 to 7.

Images