Video music matching method, device, equipment and computer readable storage medium

Abstract

The present application provides a video music matching method, device, equipment and computer-readable storage medium; the method includes: obtaining a video file to be matched with music and the upload time of the video file; obtaining each video frame contained in the video file and extracting the frame features of each video frame; determining the content features and style features of the video file based on the frame features of each video frame, and determining the time features of the video file based on the upload time; determining the first candidate music matching information of the video file based on the content features, the style features and the time features; sending the first candidate music matching information to the terminal requesting the music matching. Through the present application, the accuracy of video music matching can be improved.

Description

Technical Field

[0001] The present application relates to Internet technology, and in particular to a video music matching method, device, equipment and computer-readable storage medium. Background Art

[0002] In recent years, with the rise and prosperity of short videos, video intelligent understanding and editing technology has emerged. For example, in order to increase the effect of short videos, music, cartoon avatars, etc. can be added to the produced short videos. In the related technology, there are at least the following ways to intelligently match music for videos: one is to use video tags and music tags to match for intelligent music; another is to use the cross-modal learning of the twin towers for intelligent music; and another is to directly use artificial intelligence technology to generate music, also known as automatic composition, to create a new music. The method based on video tags and music tags has high requirements for the music library. If the size or real-time performance of the music library is insufficient, the prepared music will not be obtained. The model training required by the cross-modal learning method based on the twin towers is difficult, and only the video content is considered, so the accuracy is not enough. The technology of directly using artificial intelligence technology to generate music is not mature enough, and the accuracy of the music cannot be guaranteed. Summary of the invention

[0003] The embodiments of the present application provide a video music matching method, device and computer-readable storage medium, which can improve the accuracy of video music matching.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a video music matching method, including:

[0006] Obtain the video file to be matched with music and the upload time of the video file;

[0007] Acquire each video frame contained in the video file, and extract frame features of each video frame;

[0008] Determine content features and style features of the video file based on frame features of each video frame, and determine time features of the video file based on the upload time;

[0009] Determine at least one first candidate soundtrack information of the video file based on the content feature, the style feature and the time feature;

[0010] The at least one first candidate music soundtrack information is output.

[0011] An embodiment of the present application provides a video music matching device, including:

[0012] A first acquisition module is used to acquire the video file to be matched with music and the upload time of the video file;

[0013] A feature extraction module, used to obtain each video frame contained in the video file and extract frame features of each video frame;

[0014] A first determination module, configured to determine content features and style features of the video file based on frame features of each video frame, and to determine time features of the video file based on the upload time;

[0015] A second determination module, configured to determine at least one first candidate soundtrack information of the video file based on the content feature, the style feature and the time feature;

[0016] The first output module is used to output the at least one first candidate music soundtrack information.

[0017] In some embodiments, the first determining module is further configured to:

[0018] Fusing the frame features of the respective video frames to obtain content features of the video file;

[0019] Determine a feature matrix based on frame features of each video frame, and determine an outer product of the feature matrix;

[0020] The style feature of the video file is determined based on the outer product of the feature matrix.

[0021] In some embodiments, the first determining module is further configured to:

[0022] Determine the time interval between the upload time and the preset time;

[0023] A temporal feature of the video file is determined based on the time interval.

[0024] In some embodiments, the second determining module is further configured to:

[0025] Using the classification module in the trained music network model, the content feature, the style feature and the time feature are spliced ​​to obtain spliced ​​features;

[0026] Determine at least one target soundtrack type of the video file based on the spliced ​​features;

[0027] At least one first candidate music score information of the video file is determined from a pre-built music library based on the at least one target music score type.

[0028] In some embodiments, the second determining module is further configured to:

[0029] Determine at least one piece of music information corresponding to each target soundtrack type from the music library;

[0030] At least one piece of music information corresponding to each target music soundtrack type is determined as first candidate music soundtrack information of the video file.

[0031] In some embodiments, the apparatus further comprises:

[0032] A second acquisition module is used to acquire each video data within a preset time period in the video library, and extract audio data from each video data;

[0033] A third determining module, used to determine the audio features of the audio data in each video data;

[0034] A feature clustering module, used for performing feature clustering on the audio features to obtain a clustering result, wherein the clustering result includes a plurality of clusters, each of which includes at least one audio feature;

[0035] A music library construction module is used to construct a music library of background music based on the clustering results.

[0036] In some embodiments, the music library building module is further used to:

[0037] Determine the total number of audio features included in each cluster and the feature standard deviation of each cluster;

[0038] The cluster whose total number of audio features is greater than the quantity threshold and whose feature standard deviation is less than the standard deviation threshold is determined as the target cluster;

[0039] Determine the music type corresponding to the target cluster, and add a music type identifier to the audio data corresponding to the audio features in the target cluster;

[0040] A music library of background music is constructed based on audio data to which a music type identifier is added.

[0041] In some embodiments, the apparatus further comprises:

[0042] A third acquisition module is used to acquire a preset music network model and training data, wherein the training data includes training video data and training background music of the training video data;

[0043] A prediction module, used for inputting the training video data into the music matching network model to obtain a predicted music matching type of the training video data;

[0044] A fourth determination module is used to determine the music type label of the training background music based on a pre-built music library;

[0045] The model training module is used to perform back-propagation training on the music network model based on the predicted music type and the music type label until a trained music network model is obtained.

[0046] In some embodiments, the apparatus further comprises:

[0047] A fourth acquisition module, configured to acquire an audio and video synthesis request, wherein the audio and video synthesis request carries target music information selected by the terminal from the at least one first candidate music information;

[0048] A synthesis module, configured to synthesize the video file and the target music information in response to the audio and video synthesis request to obtain a video file with music;

[0049] The second output module is used to output the video file with music.

[0050] In some embodiments, the apparatus further comprises:

[0051] a fifth acquisition module, configured to acquire a re-music arrangement request, wherein the re-music arrangement request is triggered based on the absence of target music arrangement information in the at least one first candidate music arrangement information;

[0052] a fifth determining module, configured to determine, in response to the re-music re-matching request, second candidate music information of the video file from a pre-built music library based on the at least one target music re-matching type, wherein the second candidate music re-matching information does not include the first candidate music re-matching information;

[0053] The third output module is used to output the second candidate music information.

[0054] In some embodiments, the apparatus further comprises:

[0055] a sixth acquisition module, configured to acquire a re-music arrangement request, wherein the re-music arrangement request is triggered based on the absence of target music arrangement information in the at least one first candidate music arrangement information;

[0056] A model updating module, configured to update a target music arrangement type in response to the re-music arrangement request, and obtain an updated target music arrangement type;

[0057] a sixth determining module, configured to determine third candidate soundtrack information of the video file from a pre-built music library based on the updated target soundtrack type;

[0058] The fourth output module is used to output the third candidate music information.

[0059] The present application provides a video music matching device, including:

[0060] A memory for storing executable instructions;

[0061] The processor is used to implement the video music matching method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0062] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the video music matching method provided in the embodiment of the present application.

[0063] The embodiments of the present application have the following beneficial effects:

[0064] After obtaining the video file to be matched with music and the upload time of the video file, the video file is decoded to obtain each video frame contained in the video file, and frame features of each video frame are extracted; then, content features and style features of the video file are determined based on the frame features of each video frame, and time features of the video file are determined based on the upload time, and then at least one first candidate music matching information of the video file is determined based on the content features, the style features and the time features, and the at least one first candidate music matching information is output and sent to the terminal requesting the music matching. Since the first candidate music matching information is comprehensively determined based on the content features, the style features and the time features, it can not only make it possible to make music matching recommendations at a finer granularity based on the video content, but also ensure that the recommended music takes recent popular music into consideration and is more timely. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a schematic diagram of the network architecture of the video music matching system provided in an embodiment of the present application;

[0066] Figure 2 is a schematic diagram of the structure of the server 400 provided in an embodiment of the present application;

[0067] Figure 3 This is a schematic diagram of an implementation flow of the video music matching method provided in an embodiment of the present application;

[0068] Figure 4 A schematic diagram of the implementation process of building a music library provided in an embodiment of the present application;

[0069] Figure 5 A schematic diagram of the implementation process of the training music soundtrack model provided in the embodiment of the present application;

[0070] Figure 6 A schematic diagram of another implementation flow of the video music matching method provided in an embodiment of the present application;

[0071] Figure 7 A schematic diagram of another implementation flow of the video music matching method provided in the embodiment of the present application;

[0072] Figure 8 A schematic diagram of the network model structure for video soundtracking provided in an embodiment of the present application;

[0073] Fig. 9 A schematic diagram of the implementation process of building a soundtrack library provided in an embodiment of the present application;

[0074] Fig.10 A schematic diagram of extracting image features through TSM provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0076] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0077] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0079] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0080] 1) Intelligent music matching: Automatically recommend background music that matches the short video.

[0081] 2) Video tags: a description of the semantics of each video. Each tag is a word or phrase. A video can usually be described by multiple tags. For example, for a short video of gourmet food broadcast, the corresponding video tags may be: "food broadcast", "Korean cold noodles", "beautiful broadcaster".

[0082] 3) Feature clustering: Unsupervised grouping of features so that the features in the same group are similar and the features in different groups are not similar. Each group is a cluster, and each feature is finally assigned to a cluster.

[0083] In order to better understand the video music matching method provided in the embodiment of the present application, several video music matching methods in the related art and their existing shortcomings are first described.

[0084] There are several implementation schemes for video soundtracking in related technologies:

[0085] 1. Use video tags and music tags to match for intelligent music.

[0086] When implementing this method, you need to collect a music library in advance. Each music in the library is labeled with manual labels, including style, mood, and applicable occasions. For the video provided by the uploader, first use the video label algorithm to predict the label of the video, and then match the video label with the music label in the library. For example, calculate the number of intersections between video labels and music labels to find similar music in the library. This method essentially converts the intelligent music problem into a label keyword search and matching problem.

[0087] 2. Use the cross-modal learning of Twin Towers to create video music.

[0088] The twin tower model itself is used to learn the correspondence between the two modalities of image and text, and this idea can also be used in intelligent music matching. The input includes two branches, one branch is the video input, and the other branch is the music audio input. When implementing, the video features and the music audio features are mapped to the same space, so that the similarity between the video and the music can be measured by the video and music features. Therefore, the problem of intelligent music matching for videos is transformed into the problem of finding the music features closest to the video features in the feature space.

[0089] Although the above two methods are different in implementation, both methods select a suitable video soundtrack from an existing soundtrack library. In related technologies, there is another method of directly using artificial intelligence technology to generate soundtracks, also known as automatic composition, to create a new soundtrack.

[0090] The disadvantages of the above methods include:

[0091] The method based on video and soundtrack tag matching depends on the size of the soundtrack library and the accuracy and richness of the video tag algorithm. If the library is not large enough, or the video tag prediction is inaccurate or not rich enough, it will be difficult to find matching soundtracks. On the other hand, soundtracks have their popularity periods, so the music library needs to be continuously updated and labeled.

[0092] The cross-modal learning method based on the twin towers uses a method similar to metric learning, which is difficult to train. In addition, this type of method usually only considers the video content, without taking into account the video style and release time information, which makes the soundtrack result not appropriate enough.

[0093] The technology of automatic composition is not mature enough, and it requires continuous research in cooperation between academia and industry. In addition, the long-tail phenomenon of video soundtracks is serious in reality, that is, most videos tend to use the most popular soundtracks at the time. Therefore, even if the soundtrack can be automatically composed, it is doubtful whether users will be satisfied with the automatically generated soundtrack.

[0094] Based on this, the embodiments of the present application provide a video music matching method, device, equipment and computer-readable storage medium, which can perform intelligent music matching based on video content, style and time at the same time, directly model the intelligent music matching problem as a multi-classification task, and directly predict the suitable music matching for the video based on the video to be predicted. Compared with the cross-modal learning method based on the twin towers, the present invention is simpler and more direct to train. At the same time, the music matching library can be collected from the full short video pool in an unsupervised manner without manual labeling, so the music matching library is updated quickly and can keep up with the music matching hotspots in different periods.

[0095] See also Figure 1 , Figure 1 is a schematic diagram of the network architecture of the video music matching system 100 provided in an embodiment of the present application, such as Figure 2 As shown, the video music matching system 100 includes a terminal 200, a network 300 and a server 400. The terminal 200 is connected to the server 400 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0096] The user can obtain the video file through the terminal 200. The video file can be recorded by the user through the image acquisition device in the terminal 200, received by the terminal 200 through the instant messaging application, or downloaded by the terminal 200 from the video server. When it is necessary to add music to the video, the terminal 200 sends a music request to the server 400 through the network 300. The music request carries at least the video file. The server 400 directly models the intelligent music problem as a multi-classification task, performs intelligent music based on the content features, style features and time features of the video, determines M background music (M is an integer greater than or equal to 1), and sends the M background music to the terminal 200 for the user to choose.

[0097] In some embodiments, the server 400 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 200 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, a smart car terminal, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0098] Figure 2 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, and the computer device may be Figure 1 The server 400 or terminal 200 shown in the embodiment of the present application is described by taking the computer device as the server 400 as an example.

[0099] like Figure 2 As shown, the server 400 includes: at least one processor 410, a memory 440, and at least one network interface 420. The various components in the server 400 are coupled together through a bus system 430. It can be understood that the bus system 430 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 430 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 2 Various buses are labeled as bus system 430 .

[0100] The processor 410 may be an integrated circuit chip having signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0101] The memory 440 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 440 may optionally include one or more storage devices that are physically remote from the processor 410.

[0102] The memory 440 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 440 described in the embodiments of the present application is intended to include any suitable type of memory.

[0103] In some embodiments, memory 440 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0104] Operating system 441, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0105] A network communication module 442, used to reach other computer devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0106] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 A video soundtrack device 443 stored in a memory 440 is shown, which may be software in the form of a program or plug-in, including the following software modules: a first acquisition module 4431, a feature extraction module 4432, a first determination module 4433, a second determination module 4434, and a first output module 4435. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0107] The functions of each module will be described below.

[0108] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video soundtrack method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0109] In order to better understand the video music matching method provided in the embodiment of the present application, artificial intelligence and the artificial intelligence branches involved in the embodiment of the present application are first explained.

[0110] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0111] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The embodiments of this application mainly involve two branches: speech processing technology and machine learning.

[0112] The key technologies of speech technology include automatic speech recognition technology (ASR), text-to-speech technology (TTS) and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.

[0113] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0114] The video soundtrack method provided in the embodiment of the present application will be described in combination with the exemplary application and implementation of the computer device provided in the embodiment of the present application.

[0115] The embodiment of the present application provides a method for composing music for a video. The method can be executed by a terminal or a server, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is described by taking the execution of the method by the server as an example. Figure 3 This is a schematic diagram of an implementation flow of the video music matching method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown illustrate the video music matching method provided in the embodiment of the present application.

[0116] Step S101, in response to a received music arrangement request, obtaining a video file to be arranged with music and an upload time of the video file.

[0117] The music matching request may be sent by the terminal to the server after determining the video file to be matched with music, and the music matching request at least carries the video file to be matched with music. The server parses the music matching request to obtain the video file to be matched with music. The upload time of the video file is the time when the server receives the music matching request. The server obtains the upload time of the video file by determining the time when the music matching request is received.

[0118] Step S102: acquiring each video frame contained in the video file, and extracting frame features of each video frame.

[0119] When step S102 is implemented, the video file is decoded to obtain each video frame contained in the video file. Since each video frame in the video file is temporally associated, in the embodiment of the present application, a timing shift module can be used to extract the frame features of each video frame, and an effective convolution operator (ECO) model, a time-sensitive network (TSN) and the like can also be used to extract the frame features of each video frame.

[0120] Step S103: determining content features and style features of the video file based on the frame features of the respective video frames, and determining time features of the video file based on the upload time.

[0121] When implementing step S103, the frame features of each video frame can be fused to obtain the content features of the video file, for example, the feature fusion can be performed through NetVLAD. NetVLAD is an improvement on Average Pooling, which assumes that the contribution of each frame image feature to the whole is the same, but in reality there is a lot of redundancy in video frames. The basic idea of ​​NetVLAD is that although a whole video corresponds to an action, this action has multiple behaviors and entities. Therefore, NetVLAD divides all frame features into several clusters, which can characterize multiple behaviors and entities in the action. When determining the style features, the style in the image can be summarized by calculating the Gram matrix of the features, and then style transfer can be performed based on the Gram matrix.

[0122] When the time feature of the video file is determined based on the upload time, it is determined by calculating the interval between the upload time and a preset time point.

[0123] Step S104: determining at least one first candidate music soundtrack information of the video file based on the content feature, the style feature and the time feature.

[0124] When implementing step 104, the content features, style features and time features are spliced ​​to obtain the spliced ​​features, and then the soundtrack type of the video file is predicted through a fully connected layer and a Softmax activation function, and then at least one first candidate soundtrack information of the video file is determined from a pre-built music library according to the soundtrack type. The first candidate soundtrack information may include soundtrack audio, and may also include information such as the name of the soundtrack music and the singer.

[0125] Step S105: outputting the at least one first candidate music soundtrack information.

[0126] When step S105 is implemented, the at least one first candidate music score information can be carried in the music score response and sent to the terminal requesting the music score. In some embodiments, after receiving the at least one first candidate music score information, the terminal plays the audio data of each first candidate music score information to determine whether there is a desired target music score. If there is a desired target music score, the information of the target music score is sent to the server for synthesis by the server, or the terminal itself synthesizes, and then the synthesized video is locally stored or released in the background.

[0127] In some embodiments, when the video soundtrack method is implemented by a terminal, outputting the at least one first candidate soundtrack information can be implemented by the terminal outputting the at least one first candidate soundtrack information through its own output device. For example, the identifier of each first candidate soundtrack information can be output on a display device, and the audio of each first candidate soundtrack information can be played through an audio output device, so that the user can determine whether there is a desired target soundtrack in the at least one candidate soundtrack information.

[0128] In the video music matching method provided in the embodiment of the present application, after obtaining the video file to be matched with music and the upload time of the video file, the video file is decoded to obtain each video frame contained in the video file, and the frame features of each video frame are extracted; then the content features and style features of the video file are determined based on the frame features of the each video frame, the time features of the video file are determined based on the upload time, and then the first candidate music matching information of the video file is determined based on the content features, the style features and the time features, and the first candidate music matching information is sent to the terminal requesting the music matching. Since the first candidate music matching information is comprehensively determined based on the content features, the style features and the time features, it can not only make it possible to make music matching recommendations at a finer granularity based on the video content, but also ensure that the recommended music takes recent popular music into consideration and is more timely.

[0129] In some embodiments, the above step S103 of “determining the content features and style features of the video file based on the frame features of each video frame, and determining the time features of the video file based on the upload time” can be implemented by the following steps:

[0130] Step S1031, merging the frame features of each video frame to obtain the content features of the video file.

[0131] The frame features of each video frame may be fused through a pooling operation. In an embodiment of the present application, feature fusion may also be performed through NetVLAD. In some embodiments, feature fusion may also be performed through a pooling operation, such as average pooling. Feature fusion may also be performed through NeXtVLAD to obtain content features of the video file.

[0132] Step S1032: determining a feature matrix based on frame features of each video frame, and determining an outer product of the feature matrix.

[0133] In the embodiment of the present application, it is assumed that the video file includes T video frames (T is an integer greater than 1), and the dimension of the frame feature of each video frame is 1*D (D is an integer greater than 1), then the dimension of the feature matrix composed of these T frame features is T*D. Assuming that the feature matrix is ​​X T*D, then the product of the characteristic matrix and the transpose of the characteristic matrix can be determined as the outer product of the characteristic matrix.

[0134] Step S1033: determining the style feature of the video file based on the outer product of the feature matrix.

[0135] When implementing step S1033, the Gram matrix may be determined by the outer product of the characteristic matrix using formula (1-1):

[0136] (1-1);

[0137] Expand the Gram matrix into T 2 dimensional vector. Then, the signed square root (SignedSqrt) of each element of the vector is taken to obtain the style features of the video file, that is, the dimension of the style features of the video file is 1*T 2 .

[0138] Step S1034, determining the time interval between the upload time and the preset time.

[0139] Since the popularity period of a soundtrack is usually measured in months, in an embodiment of the present application, the time interval between the upload time and the preset time can be calculated in months. For example, if the preset time is January 20, 2015 and the upload time is April 20, 2016, then the time interval between the upload time and the preset time is 16 months.

[0140] Step S1035: determining a time feature of the video file based on the time interval.

[0141] In order to make the numerical range more diverse during implementation, the time intervals t and t are used simultaneously in the embodiment of the present application. 2 , The three are determined as the time characteristics of the video file.

[0142] Continuing with the above example, when the time interval is 16 months, the time features of the video file are 16, 256, and 4.

[0143] In the above-mentioned steps S1031 to S1035, the content features of the video file are determined by fusing the frame features of each video frame, and then the frame features are used to construct a feature matrix. The style features of the video file are determined by calculating the outer product of the feature matrix, and the time features are determined based on the time interval between the upload time and the preset time, thereby providing the necessary data basis for the subsequent determination of the soundtrack information of the video file.

[0144] In some embodiments, the above step S104 "determining at least one first candidate soundtrack information of the video file based on the content feature, the style feature and the time feature" can be implemented by following the steps S1041 to S1043, which are explained below.

[0145] Step S1041, using the classification module in the trained music network model to splice the content feature, the style feature and the time feature to obtain the spliced ​​feature.

[0146] In implementation, the content features, style features and time features can be concatenated in sequence to obtain the concatenated features. Assume that the dimension of the content feature is 1*D and the dimension of the style feature is 1*T 2 , the dimension of the time feature is 3, so the dimension of the concatenated feature is 1*(D+T 2 +3).

[0147] Step S1042: determining at least one target soundtrack type of the video file based on the spliced ​​features.

[0148] When implementing step S1042, the spliced ​​features can be input into the classification model in the music network model to obtain the matching degree between the video file and each music type. The classification module of the video file includes at least one fully connected layer and a Softmax activation function.

[0149] After obtaining the matching degree between the video file and each soundtrack type, the first N soundtrack types with the highest matching degree are determined as the target soundtrack type. N is an integer greater than or equal to 1.

[0150] Step S1043: determining at least one first candidate music soundtrack information of the video file from a pre-built music library based on the at least one target music soundtrack type.

[0151] In the embodiment of the present application, this step can be implemented by the following steps S431 and S432, which are explained below.

[0152] Step S431, determining at least one piece of music information corresponding to each target music type from the music library.

[0153] The pre-selected music library includes multiple music collections, each music collection corresponds to a soundtrack type, and each music collection includes multiple music information. For example, the soundtrack type can be soothing, funny, inspirational, etc. After determining at least one target soundtrack type, at least one music information can be selected from the music collection corresponding to each target soundtrack type. In the embodiment of the present application, one music information can be selected for each target soundtrack type, or multiple music information can be selected for each target soundtrack type.

[0154] For example, there are N target soundtrack types. If each target soundtrack type selects one piece of music information, then N pieces of music information are determined in this step. If each target soundtrack type selects 2 pieces of music information, then 2N pieces of music information are determined in this step.

[0155] In some embodiments, the music set corresponding to each music type in the music library is a cluster obtained by feature clustering. After determining a target music type, when selecting one or more music information from the corresponding music set, one or more music information closest to the cluster center can be selected.

[0156] In some embodiments, each audio data in the music set corresponding to the target music type may correspond to a score representing the degree of matching with the target music type, and the higher the score, the higher the degree of matching with the target music type. In this step, one or more music information with the highest score may be selected.

[0157] Step S432: determine at least one piece of music information corresponding to each target music soundtrack type as at least one first candidate music soundtrack information of the video file.

[0158] In this step, the multiple pieces of music information determined in step S432 are determined as the first candidate music information of the video file.

[0159] In the above steps S1041 to S1043, the content features, style features and time features of the video file are spliced, and then the music information is predicted through a fully connected layer and a softmax function. In this way, compared with using only content features, style features are also considered in the embodiment of the present application, thereby improving the accuracy of music recommendation. In addition, time features are added to ensure the timeliness and popularity of the recommended music.

[0160] In some embodiments, before executing the above steps S101 to S105, it is also necessary to Figure 4 Steps S001 to S004 shown complete the music library construction.

[0161] Step S001, obtaining each video data within a preset time period in a video library, and extracting audio data from each video data.

[0162] The video library in this step may be a video library in a video publishing platform, and the preset time period may be a time period from the current time to the previous year or 18 months. After acquiring each video data within the preset time period, audio extraction is performed to obtain the audio data in each video data, and the audio data may be music or dialogue.

[0163] Step S002: determine the audio features of the audio data in each video data.

[0164] When implementing this step, the audio data may be subjected to Fourier transform to obtain the frequency spectrum of the audio data, and then the corresponding audio features may be extracted based on the frequency spectrum of the audio data. For example, the VGGish features of the audio may be extracted through a deep learning network.

[0165] Step S003: clustering the audio features to obtain clustering results.

[0166] In the embodiment of the present application, a K-means algorithm or a clustering algorithm based on mean shift (MeanShift) can be used to perform feature clustering to obtain a clustering result. The clustering result includes multiple clusters, each of which includes at least one audio feature.

[0167] The above clustering algorithm is only used for exemplary description. In actual implementation, other clustering algorithms may also be used to cluster audio features.

[0168] Step S004, constructing a music library of background music based on the clustering results.

[0169] In the embodiment of the present application, step S004 can be implemented by the following steps S0041 to S0044, which are explained below.

[0170] Step S0041: determining the total number of audio features included in each cluster and the feature standard deviation of each cluster.

[0171] When calculating the feature standard deviation of each cluster, the mean of the audio features included in the cluster may be calculated first, and then the feature standard deviation of the cluster may be determined by using each audio feature and the mean.

[0172] Step S0042: determine the cluster whose total number of audio features is greater than the quantity threshold and whose feature standard deviation is less than the standard deviation threshold as the target cluster.

[0173] Since the distribution of video soundtracks is very long-tailed, the soundtracks of most videos tend to use popular soundtracks, and some videos have no soundtracks, such as narration dialogue videos, which are not suitable for adding to the soundtrack library. Therefore, in the embodiment of the present application, a quantity threshold and a standard deviation threshold are set respectively. In this step, only the clusters whose total number of audio features is greater than the quantity threshold and whose feature standard deviation is less than the standard deviation threshold are retained. These clusters are also the target clusters.

[0174] Step S0043: determine the music type corresponding to the target cluster, and add a music type identifier to the audio data corresponding to the audio features in the target cluster.

[0175] In step S0043, the corresponding music type is determined based on the tune of each music in each target cluster, and a music type identifier is assigned to each music type, and then the music type identifier is added to the audio data corresponding to each audio feature in the target cluster.

[0176] Step S0044, constructing a music library of background music based on the audio data to which the music type identifier is added.

[0177] Through the above steps S001 to S004, the construction of the background music library is completed, and in the process of building the library, the audio data in the video library is clustered to determine the music type identification (that is, the music label) of the audio data finally added to the library. In this way, the music library is constructed in an unsupervised manner, which is more efficient than the method of manually annotating the music library.

[0178] In some embodiments, after the music library is constructed through the above steps S001 to S004, Figure 5 As shown, the preset music network model can also be trained through the following steps S011 to S014 to obtain a trained music network model.

[0179] Step S011, obtaining a preset music network model and training data.

[0180] The training data includes training video data and training background music of the training video data. The training video data may be obtained from a video library.

[0181] Step S012: input the training video data into the music matching network model to obtain the predicted music matching type of the training video data.

[0182] In some embodiments, the music network model may include a feature extraction module, a video content module, a video style module, a video time module and a classification module. When implementing step S012, the training video data may be decoded to obtain multiple training video frames, and frame features of each training video frame are extracted by the feature extraction module. Then, each frame feature is input into the video content module and the video style module to obtain the training content feature and the training style feature of the training video data, and the training time feature is determined by the video time module, and then the training content feature, the training style feature and the training time feature are input into the classification module to determine the predicted music type of the training video data.

[0183] Step S013, based on the pre-built music library, determine the music type label of the training background music.

[0184] During implementation, a search is performed from a pre-built music library to determine whether the training background music is included. If the music library includes the training background music, the soundtrack type of the training background music is determined, and the soundtrack type is determined as the soundtrack type label of the training background music.

[0185] Step S014, performing back propagation training on the music network model based on the predicted music type and the music type label until a trained music network model is obtained.

[0186] In this step, the music network model is back-propagated and trained based on the predicted music type and the music type label to adjust the parameters of the music network model until a trained music network model is obtained.

[0187] Through the above steps S011 to S014, the music problem is directly modeled as a multi-classification task, and the suitable music for the video is directly predicted based on the video to be predicted. Compared with the cross-modal learning method based on the dual towers, the training is simpler and more direct, and therefore the training efficiency is higher.

[0188] Based on the above embodiments, the present application further provides a video music matching method, which is applied to Figure 1 The network architecture shown, Figure 6 Another implementation flow diagram of the video music matching method provided in the embodiment of the present application is shown below in combination with Figure 6 The video music matching method provided in the embodiment of the present application is described.

[0189] Step S201: the terminal determines a video file to be matched with music.

[0190] The video file to be matched with music may be collected by the terminal using its own image acquisition device, may be downloaded from the network, or may be sent by other terminals.

[0191] Step S202: the terminal sends a music arrangement request to the server in response to the received music arrangement operation.

[0192] The music matching request at least carries the video file to be matched with music.

[0193] Step S203: the server obtains the video file to be matched with music and the upload time of the video file based on the received music matching request.

[0194] Step S204: the server obtains each video frame contained in the video file, and extracts frame features of each video frame.

[0195] Step S205: The server determines content features and style features of the video file based on the features of each video frame, and determines time features of the video file based on the upload time.

[0196] Step S206: The server determines at least one first candidate music score information of the video file based on the content feature, the style feature and the time feature. ;

[0197] Step S207: the server outputs the at least one first candidate music soundtrack information.

[0198] like Figure 6 As shown, when step S207 is implemented, the server sends at least one first candidate music soundtrack information to the terminal requesting the music soundtrack.

[0199] The implementation process of the above steps S203 to S207 may refer to steps S101 to S105.

[0200] Step S208: After receiving at least one first candidate music soundtrack information, the terminal plays at least one first candidate music soundtrack information.

[0201] In the embodiment of the present application, the first candidate music information includes at least the music audio, so playing at least one first candidate music information can be considered as playing the music audio. In this step, all the received first candidate music information can be played, or part of the first candidate music information can be played.

[0202] Step S209: the terminal determines whether a selection operation of target soundtrack information is received.

[0203] When the terminal receives a selection operation for the target music information, it means that the at least one first candidate music information includes the music information selected by the user, and the process enters step S210; when the terminal does not receive a selection operation for the target music information, it means that the at least one first candidate music information does not include the music information liked by the user, and the process enters step S213.

[0204] Step S210: In response to the selection operation, the terminal sends an audio and video synthesis request to the server.

[0205] The audio and video synthesis request at least includes target soundtrack information, and the target soundtrack information may include only a soundtrack identifier and may also include target soundtrack audio.

[0206] Step S211, after receiving the audio and video synthesis request, the server responds to the audio and video synthesis request and synthesizes the video file and the target music information to obtain a video file with music.

[0207] Step S212: the server outputs the video file with the music.

[0208] In actual implementation, the server outputs the video file with music, that is, the server sends the video file with music to the terminal.

[0209] In some embodiments, after step S209, steps S210 to S212 may not be performed, but the terminal itself synthesizes the selected target music information and the video file to be matched with music to obtain the matched video file. After obtaining the matched video file, the terminal may store it locally, send it to other terminals, or publish it to a video publishing platform.

[0210] Step S213: the terminal sends a re-music arrangement request to the server.

[0211] The re-music arrangement request is triggered based on the absence of target music arrangement information in the first candidate music arrangement information.

[0212] Step S214: the server receives a re-music arrangement request sent by the terminal, and in response to the re-music arrangement request, determines at least one second candidate music arrangement information of the video file from a pre-built music library based on the at least one target music arrangement type.

[0213] When step S214 is implemented, at least one second candidate soundtrack information with the highest score other than the first candidate soundtrack information can be screened out from the music set corresponding to the target soundtrack type, that is, the at least one second candidate soundtrack information does not include the first candidate soundtrack information, but the second candidate soundtrack information still belongs to the target soundtrack type.

[0214] Step S215: the server outputs the second candidate music soundtrack information.

[0215] When implementing this step, the server may send a re-music matching response carrying the second candidate music matching information to the terminal.

[0216] After step S215, the process from step S209 to step S215 may be repeated until satisfactory music information is provided to the terminal.

[0217] In the video music matching method provided in the embodiment of the present application, when the terminal determines the video file to be matched with music and requests the server to match the music, the server extracts the frame features of each video frame in the video file after obtaining the video file to be matched with music and the upload time of the video file; then, the server determines the content features and style features of the video file based on the frame features of the each video frame, determines the time features of the video file based on the upload time, and then determines the first candidate music matching information of the video file based on the content features, the style features and the time features, and sends the first candidate music matching information to the terminal requesting the music matching, so that not only can the server This allows for music recommendations to be made at a finer granularity based on the video content, and can also ensure that the recommended music takes recent popular music into consideration and is more timely. If the terminal selects target music information from multiple first candidate music information, it can request the server to perform audio and video synthesis. If the terminal does not select the target music information from multiple first candidate music information, it can request the server to re-compose the music. The server can select multiple candidate music information from the target music type again and send it to the terminal, so that the terminal can select suitable music information without manual screening, which can not only improve the music composition efficiency, but also increase the attractiveness of the video after the music composition.

[0218] In some embodiments, when it is necessary to re-accompany the video file to be accompanied by music, step S214 may not be performed, but the following steps may be performed, which are described below.

[0219] Step S314: the server updates the target music soundtrack type in response to the re-music soundtrack request, and obtains an updated target music soundtrack type.

[0220] In this step, when the server receives the re-music request, it can select the top N other music types with the highest matching degree from the matching degree between the video file and each music type, except for the target music type, and determine them as the updated target music type. N is an integer greater than or equal to 1.

[0221] Step S315: The server determines third candidate music information for the video file from a pre-built music library based on the updated target music type.

[0222] Similar to step S431, each audio data in the music set corresponding to the updated target soundtrack type may also correspond to a score representing the degree of matching with the updated target soundtrack type, and the higher the score, the higher the degree of matching with the updated target soundtrack type. In this step, one or more music information with the highest score may be selected as the third candidate music information.

[0223] Step S316: the server outputs the third music information.

[0224] In actual implementation, this step may be implemented by the server sending a re-music arrangement response carrying the third candidate music arrangement information to the terminal.

[0225] In the above steps S314 to S316, when the server receives the re-music request, it can reselect the target music type, and then determine the third candidate music information based on the re-selected target music type. This can avoid repeatedly recommending music to the user and preventing the user from selecting the desired music when the initially determined target music type is not the type the user likes, thereby improving the diversity of music selection.

[0226] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0227] Figure 7 A schematic diagram of another implementation flow of the video music matching method provided in the embodiment of the present application, the music matching method can be executed by a server, such as Figure 7 As shown, the process includes:

[0228] Step S701, receiving a video uploaded by a user.

[0229] Step S702, automatically and intelligently analyze the short video to predict the background music.

[0230] Step S703, obtain the predicted M (in Figure 7 (M=3 is taken as an example) background music.

[0231] In an embodiment of the present application, after obtaining the predicted M background music, the server sends the M background music to the user.

[0232] Step S704: determine the background music selected by the user.

[0233] Step S705, synthesize the video and upload it to the platform or provide it for users to download.

[0234] In implementation, synthesized video refers to synthesizing background music and video. After the synthesis, the synthesized video can be uploaded to a video publishing platform for other users to watch, and can also be downloaded by the user who uploaded the video file.

[0235] Figure 8 A schematic diagram of the network model structure for video soundtracking provided in an embodiment of the present application, such as Figure 8 As shown, the network model includes: feature extraction module 801, video content module 802, video style module 803, video time module 804 and classification module 805. Figure 8As shown, the video to be matched with music is input into the feature extraction module 801, and the frame features of each video frame in the video are extracted. Then, the frame features of each video frame are input into the video content module 802 and the video style module 803. The video content module 802 is used to extract the content information in the video; the video style module 803 is used to extract the style of the video; the video to be matched with music is also input into the video time module 804, and the video time module 804 is used to summarize the release time of the video as a feature. After obtaining the video content feature, video style feature and video time feature, these three features are input into the classification module 805, and the classification module 805 splices these features, and finally predicts the audio music corresponding to the video through the fully connected layer and the Softmax activation function.

[0236] The following is an example of the present application using Figure 8 The network model shown is used to illustrate the implementation process of video music.

[0237] Before performing video music matching, the network model needs to be trained, and before training the network model, a music library needs to be built. In the embodiment of the present application, background music is collected from the video library to build a music library. When implemented, the music library can be built by the server. Fig. 9 A schematic diagram of the implementation process of building a soundtrack library provided in the embodiment of the present application is shown in FIG. Fig. 9 As shown, the process includes:

[0238] Step S901: the server extracts audio.

[0239] During implementation, the audio (MP4→WAV) corresponding to each video can be extracted from the short video library.

[0240] Step S902: The server extracts VGGish audio features.

[0241] When implementing this step, the spectrum of each audio is calculated, and the VGGish feature of each audio is extracted based on the spectrum. VGGish is an audio feature based on a deep convolutional neural network, and the network structure it uses is similar to VGG-A in image classification. Based on VGG-A, the last group of convolution and confluence layers are removed, so the VGGish model has only 4 groups of convolution and confluence layers; and the dimension of the fully connected layer is reduced from 1000 to 128. Finally, the average confluence is used, and the dimension of the final audio feature is 128.

[0242] Step S903: the server performs feature clustering.

[0243] Perform feature clustering on the corresponding audio features in the video library. Assume that the number of clusters is K After clustering, for each video vi , we can get the cluster corresponding to the video c i ∈{1, 2, …, K} 。 Count the number of videos belonging to each cluster m i And the characteristic standard deviation σ under each cluster i .

[0244] Step S904: the server constructs a music library.

[0245] Since the distribution of video soundtracks is very long-tailed, most videos tend to use popular soundtracks, and some videos have no soundtracks, such as narration dialogue videos, which are not suitable for being added to the soundtrack library. Therefore, a quantity threshold and a standard deviation threshold are set respectively, and only clusters with a number of videos greater than or equal to the quantity threshold and a feature standard deviation less than or equal to the standard deviation threshold are retained.

[0246] Finally, the audio corresponding to the retained clusters constitutes the soundtrack library, and each cluster corresponds to a soundtrack. K ' indicates the number of clusters retained, and renumbers the retained clusters to 1, 2, ..., K '.

[0247] In the model training phase, for a video in the video library, firstly, the video frames are sampled from the video, and then each frame image is passed through the Temporal Shift Module (TSM) network to extract the image features of each frame, such as Fig.10 As shown in Figure 1. TSM transforms 2D convolution to approximate the effect of 3D convolution. Fig.10 As shown in , the 2D convolution operates independently in the time dimension, and the 2D convolution of each frame only uses the information of the current frame, so there is no time modeling (see Fig.10 The left part 1001 in the figure). TSM shifts some channels along the time dimension to exchange information between adjacent frames, allowing 2D convolution to perceive information from other frames. TSM shifts 1 / 4 of the channels, including offline and online shift modes: offline time shift allows 1 / 8 of the channels to shift the information of the previous frame, and 1 / 8 of the channels to shift the information of the next frame (such as Fig.10 The middle part 1002 in the middle part 1002); online time shift allows 1 / 4 of the channels to shift the information of the previous frame (such as Fig.10 The right part 1003 in FIG. Fig.10 The empty positions after the shift are filled with zeros (Zero Padding).

[0248] In the embodiment of the present application, the video feature extraction module exemplarily uses the TSM network structure, but the model structure of the deep neural network is not specifically limited and can be replaced by other effective new video model structures, such as ECO, TSN, I3D, Non-Local, SlowFast, etc.

[0249] After obtaining the features of the video frame through the feature extraction module, in an embodiment of the present application, the features of each frame can be fused through NetVLAD. NetVLAD is an improvement on Average Pooling, which assumes that the contribution of each frame image feature to the whole is the same, but in reality there is a lot of redundancy in video frames. The basic idea of ​​NetVLAD is that although an entire video corresponds to an action, this action has multiple behaviors and entities. For example, the action of "shooting" includes multiple behaviors and entities such as "basketball frame", "controlling the ball", "jumping", and "shooting". Therefore, NetVLAD divides all frame features into several clusters, which can characterize multiple behaviors and entities in the action.

[0250] It should be noted that in the embodiment of the present application, the video content module uses NetVLAD for feature extraction. In addition, other feature extraction methods may also be used, such as average convergence, NeXtVLAD, etc.

[0251] After obtaining the features of the video frame through the feature extraction module, the video style module is used to extract the style of the video. It is found that by calculating the Gram matrix of the features, the style in the image can be summarized, and then the style can be transferred based on the Gram matrix. Therefore, we extend the idea of ​​image style transfer to video.

[0252] The video frame features obtained by the feature extraction module are { x 1, x 2, …, x T}, the frame features can be expressed as a matrix: , the Gram matrix is ​​defined as the outer product of the feature matrix, as shown in formula (1-1):

[0253] (1-1);

[0254] The dimension of the Gram matrix is Then, the Gram matrix is ​​expanded into T 2 dimensional vector. Then, take the signed square root of each element of the vector, that is, execute formula (1-2) for each element in the vector:

[0255] (1-2);

[0256] Among them, sign( z ) means to take an element z The sign of , when the input is greater than 0, the function value is 1; when the input is less than 0, the function value is -1; when the input is equal to 0, the function value is 0. The result is used as the video style feature.

[0257] In some recommendation systems, video age (Example Age) is used as one of the recommendation features. The motivation is that the popularity of videos is usually short and users tend to watch newly released videos. Therefore, the video age is explicitly used as a feature so that the recommendation model can consider the age of the video when making recommendations.

[0258] Since the soundtrack has a period of popularity, even for the same video, different soundtracks are suitable for different periods of time. Machine learning algorithms, including deep learning algorithms, learn from training data, and with more historical data, they tend to predict historically suitable soundtracks, which makes the soundtrack tend to be "outdated". Therefore, the video time module uses the upload time of the video t As input, since the popularity period of the soundtrack is usually measured in months, in the embodiment of the present application, t The unit is the number of months from the time the video was uploaded to January 2005. If the upload time is before January 2005, t = 0. In addition, to make the value range more diverse, use t , t 2 , The three together serve as the video time module feature. Through this time feature, the model tends to refer to the recent video soundtrack results for prediction to ensure the popularity of the recommended soundtrack.

[0259] After obtaining the above video content, style, and time features, the classification module splices the above features together to form the final features. Then, through a fully connected layer and a Softmax activation function, the soundtrack of the video is predicted. This is a multi-classification problem. The number of categories is K '+1, where K ' is the number of music clusters selected when the music library is built. 1 means that the video is not suitable or has no corresponding music. For a video in the video library, if the audio music corresponding to the video appears in the music library, then the category corresponding to the video is the ID of the cluster of feature clustering when the music library is built; if the audio music corresponding to the video does not appear in the music library, then the video is not suitable or has no corresponding music.

[0260] In the model prediction stage, the video to be predicted is input into the trained network model to obtain the confidence that the video belongs to each soundtrack, and then sorted from high to low according to the confidence to obtain the top N The corresponding soundtrack will be fed back to the video uploader for selection.

[0261] Adding a suitable background music to a short video can make the video more attractive and provide a better viewing experience for users. For video uploaders, especially those without much experience, it is a challenge to select a suitable soundtrack for a short video in the face of a vast music library, which requires a lot of time to screen. The video soundtrack method provided by the embodiment of the present application can automatically and intelligently provide a first candidate soundtrack based on the video uploaded by the user, making it convenient for the user to choose.

[0262] In the embodiment of the present application, the soundtrack library can be collected unsupervised, which is more efficient than the method of manually annotating the soundtrack library. The popularity of soundtracks is short and the update speed is fast. The unsupervised method can store new soundtracks faster, and after storage, it can be trained to recommend new soundtracks.

[0263] Compared with the cross-modal learning method based on the dual towers, in the embodiment of the present application, the intelligent music matching problem is directly modeled as a multi-classification task, and the suitable music for the video is directly predicted based on the video to be predicted, which makes training simpler and more direct.

[0264] In the embodiment of the present application, music prediction is performed based on video content, style and time at the same time. Compared with the method of intelligent music matching based only on video content, considering the video style characteristics can make music recommendation at a finer granularity on the video content. After considering the video time, the recommended music can take into account more recent training data and be more timely.

[0265] The following is a description of an exemplary structure of a video music matching device 443 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the video music matching device 443 of the memory 440 may include:

[0266] The first acquisition module 4431 is used to acquire the video file to be matched with music and the upload time of the video file;

[0267] The feature extraction module 4432 is used to obtain each video frame contained in the video file and extract frame features of each video frame;

[0268] A first determining module 4433, configured to determine content features and style features of the video file based on frame features of each video frame, and determine time features of the video file based on the upload time;

[0269] A second determination module 4434 is used to determine first candidate soundtrack information of the video file based on the content feature, the style feature and the time feature;

[0270] The first output module 4435 is used to send the first candidate music information to the terminal requesting the music.

[0271] In some embodiments, the first determination module 4433 is further used to:

[0272] Fusing the frame features of the respective video frames to obtain content features of the video file;

[0273] Determine a feature matrix based on frame features of each video frame, and determine an outer product of the feature matrix;

[0274] The style feature of the video file is determined based on the outer product of the feature matrix.

[0275] In some embodiments, the first determination module 4433 is further used to:

[0276] Determine the time interval between the upload time and the preset time;

[0277] A temporal feature of the video file is determined based on the time interval.

[0278] In some embodiments, the second determination module 4434 is further configured to:

[0279] Using the classification module in the trained music network model, the content feature, the style feature and the time feature are spliced ​​to obtain spliced ​​features;

[0280] Determine at least one target soundtrack type of the video file based on the spliced ​​features;

[0281] At least one first candidate music score information of the video file is determined from a pre-built music library based on the at least one target music score type.

[0282] In some embodiments, the second determination module 4434 is further configured to:

[0283] Determine at least one piece of music information corresponding to each target soundtrack type from the music library;

[0284] At least one piece of music information corresponding to each target music soundtrack type is determined as first candidate music soundtrack information of the video file.

[0285] In some embodiments, the apparatus further comprises:

[0286] A second acquisition module is used to acquire each video data within a preset time period in the video library, and extract audio data from each video data;

[0287] A third determining module, used to determine the audio features of the audio data in each video data;

[0288] A feature clustering module, used for performing feature clustering on the audio features to obtain a clustering result, wherein the clustering result includes a plurality of clusters, each of which includes at least one audio feature;

[0289] A music library construction module is used to construct a music library of background music based on the clustering results.

[0290] In some embodiments, the music library building module is further used to:

[0291] Determine the total number of audio features included in each cluster and the feature standard deviation of each cluster;

[0292] The cluster whose total number of audio features is greater than the quantity threshold and whose feature standard deviation is less than the standard deviation threshold is determined as the target cluster;

[0293] Determine the music type corresponding to the target cluster, and add a music type identifier to the audio data corresponding to the audio features in the target cluster;

[0294] A music library of background music is constructed based on audio data to which a music type identifier is added.

[0295] In some embodiments, the apparatus further comprises:

[0296] A third acquisition module is used to acquire a preset music network model and training data, wherein the training data includes training video data and training background music of the training video data;

[0297] A prediction module, used for inputting the training video data into the music matching network model to obtain a predicted music matching type of the training video data;

[0298] A fourth determination module is used to determine the music type label of the training background music based on a pre-built music library;

[0299] The model training module is used to perform back-propagation training on the music network model based on the predicted music type and the music type label until a trained music network model is obtained.

[0300] In some embodiments, the apparatus further comprises:

[0301] A fourth acquisition module, configured to acquire an audio and video synthesis request, wherein the audio and video synthesis request carries target music information selected by the terminal from the at least one first candidate music information;

[0302] A synthesis module, configured to synthesize the video file and the target music information in response to the audio and video synthesis request to obtain a video file with music;

[0303] The second output module is used to output the video file with music.

[0304] In some embodiments, the apparatus further comprises:

[0305] a fifth acquisition module, configured to acquire a re-music arrangement request, wherein the re-music arrangement request is triggered based on the absence of target music arrangement information in the at least one first candidate music arrangement information;

[0306] a fifth determining module, configured to determine, in response to the re-music re-matching request, second candidate music information of the video file from a pre-built music library based on the at least one target music re-matching type, wherein the second candidate music re-matching information does not include the first candidate music re-matching information;

[0307] The third output module is used to output the second candidate music information.

[0308] In some embodiments, the apparatus further comprises:

[0309] a sixth acquisition module, configured to acquire a re-music arrangement request, wherein the re-music arrangement request is triggered based on the absence of target music arrangement information in the at least one first candidate music arrangement information;

[0310] A model updating module, configured to update a target music arrangement type in response to the re-music arrangement request, and obtain an updated target music arrangement type;

[0311] a sixth determining module, configured to determine third candidate soundtrack information of the video file from a pre-built music library based on the updated target soundtrack type;

[0312] The fourth output module is used to output the third candidate music information.

[0313] It should be noted that the description of the above video music matching device embodiment is similar to the above method description and has the same beneficial effects as the method embodiment. For technical details not disclosed in the video music matching device embodiment of this application, those skilled in the art should refer to the description of the method embodiment of this application for understanding.

[0314] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the video soundtrack method described in the embodiment of the present application.

[0315] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will be caused to execute the method provided by the present application embodiment, for example, Figure 3 , Figure 4 , Figure 5 and Figure 6 The method shown.

[0316] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0317] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0318] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0319] By way of example, executable instructions may be deployed to be executed on one computer device, or on multiple computer devices located at one site, or on multiple computer devices distributed across multiple sites and interconnected by a communication network.

[0320] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A video music matching method, characterized in that: include: Obtain the video file to be matched with music and the upload time of the video file; Acquire each video frame contained in the video file, and extract frame features of each video frame; Determine the content features and style features of the video file based on the frame features of each video frame, and determine the time features of the video file based on the upload time; the style features are determined based on the outer product of a feature matrix, and the feature matrix is ​​obtained by splicing the frame features of each video frame row by row; the time features are determined based on the time interval between the upload time and the preset time; splicing the content feature, the style feature and the time feature in columns to obtain spliced ​​features, and determining at least one first candidate soundtrack information of the video file based on the spliced ​​features; The at least one first candidate music soundtrack information is output.

2. The method according to claim 1, characterized in that The determining the content features and style features of the video file based on the features of each video frame includes: Fusing the frame features of the respective video frames to obtain content features of the video file; Determine a feature matrix based on the frame features of each video frame, and determine the outer product of the feature matrix; The style feature of the video file is determined based on the outer product of the feature matrix.

3. The method according to claim 1, characterized in that The determining the time feature of the video file based on the upload time includes: Determine the time interval between the upload time and the preset time; A temporal feature of the video file is determined based on the time interval.

4. The method according to claim 1, characterized in that The step of splicing the content feature, the style feature, and the time feature in columns to obtain spliced ​​features, and determining at least one first candidate soundtrack information of the video file based on the spliced ​​features includes: Using the classification module in the trained music network model, the content feature, the style feature and the time feature are spliced ​​in columns to obtain spliced ​​features; Determine at least one target soundtrack type of the video file based on the spliced ​​features; At least one first candidate music score information of the video file is determined from a pre-built music library based on the at least one target music score type.

5. The method according to claim 4, characterized in that The determining at least one first candidate music score information of the video file from a pre-built music library based on the at least one target music score type includes: Determine at least one piece of music information corresponding to each target soundtrack type from the music library; At least one piece of music information corresponding to each target music soundtrack type is determined as first candidate music soundtrack information of the video file.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire each video data within a preset time period in the video library, and extract audio data from each video data; Determining audio features of the audio data in each of the video data; Performing feature clustering on the audio features to obtain a clustering result, wherein the clustering result includes a plurality of clusters, each of which includes at least one audio feature; A music library of background music is constructed based on the clustering results.

7. The method according to claim 6, characterized in that The step of constructing a music library of background music based on the clustering result includes: Determine the total number of audio features included in each cluster and the feature standard deviation of each cluster; The cluster whose total number of audio features is greater than the quantity threshold and whose feature standard deviation is less than the standard deviation threshold is determined as the target cluster; Determine the music type corresponding to the target cluster, and add a music type identifier to the audio data corresponding to the audio features in the target cluster; A music library of background music is constructed based on audio data to which a music type identifier is added.

8. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire a preset music network model and training data, wherein the training data includes training video data and training background music of the training video data; Inputting the training video data into the music matching network model to obtain the predicted music matching type of the training video data; Based on the pre-built music library, determine the music type label of the training background music; The music network model is back-propagated and trained based on the predicted music type and the music type label until a trained music network model is obtained.

9. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire an audio and video synthesis request, wherein the audio and video synthesis request carries target music information selected by the terminal from the at least one first candidate music information; In response to the audio and video synthesis request, synthesizing the video file and the target soundtrack information to obtain a video file with soundtrack; Output the video file with the music.

10. The method according to claim 4 or 5, characterized in that: The method further comprises: Acquire a re-music matching request, where the re-music matching request is triggered based on the absence of target music matching information in the at least one first candidate music matching information; In response to the re-music re-matching request, determining second candidate music re-matching information of the video file from a pre-built music library based on the at least one target music re-matching type, wherein the second candidate music re-matching information does not include the first candidate music re-matching information; The second candidate music score information is output.

11. The method according to claim 4 or 5, characterized in that: The method further comprises: Acquire a re-music matching request, where the re-music matching request is triggered based on the absence of target music matching information in the at least one first candidate music matching information; In response to the re-music arrangement request, updating the target music arrangement type to obtain an updated target music arrangement type; Determining third candidate music information of the video file from a pre-built music library based on the updated target music type; The third candidate music score information is output.

12. A video music matching device, characterized in that: include: A first acquisition module is used to acquire the video file to be matched with music and the upload time of the video file; A feature extraction module, used to obtain each video frame contained in the video file and extract frame features of each video frame; A first determination module is used to determine the content features and style features of the video file based on the frame features of each video frame, and determine the time features of the video file based on the upload time; the style features are determined based on the outer product of a feature matrix, and the feature matrix is ​​obtained by splicing the frame features of each video frame row by row; the time features are determined based on the time interval between the upload time and the preset time; A second determination module is used to splice the content feature, the style feature and the time feature in columns to obtain spliced ​​features, and determine first candidate soundtrack information of the video file based on the spliced ​​features; The first output module is used to output the first candidate music score information.

13. A computer device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 11 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that: Executable instructions are stored, and when executed by a processor, the method described in any one of claims 1 to 11 is implemented.

Images