Method, device, electronic device and computer-readable medium for obtaining video soundtrack

Abstract

The present disclosure relates to a method, device, electronic device and computer-readable medium for obtaining video soundtracks, and belongs to the field of computer technology. The method includes: obtaining a video frame in a video, and identifying the main content in the video frame; determining a key frame in the video according to the main content in the video frame, and determining the main content attribute of the key frame according to the main content of the key frame; obtaining a semantic feature vector of the video according to the main content and the main content attribute of the key frame; matching the semantic feature vector of the video with the semantic feature vector of lyrics in a pre-established lyrics vector retrieval pool, and obtaining a candidate set of soundtracks for the video according to the matching results. The present disclosure can improve the matching degree between the recommended video soundtrack and the video content by matching the main content and main content attributes of the video to the corresponding video soundtrack.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method for acquiring video soundtracks, a device for acquiring video soundtracks, an electronic device, and a computer-readable medium. Background Art

[0002] With the development of the Internet, short video applications have gradually become one of the most popular applications. When sharing videos, most users will choose a suitable piece of music as the soundtrack to enhance the fun of the short video.

[0003] Currently, in related technologies, the recommendation of video soundtracks is mainly based on user behavior information, such as music that users have collected, soundtracks of videos they have clicked, etc., but the music recommended in this way may not necessarily match the user's video content.

[0004] In view of this, there is an urgent need in the art for a method for obtaining video soundtracks that can improve the matching degree between the recommended video soundtracks and the video content.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a method for obtaining video soundtracks, a device for obtaining video soundtracks, an electronic device and a computer-readable medium, thereby improving the matching degree between the recommended video soundtracks and the video content at least to a certain extent.

[0007] According to a first aspect of the present disclosure, a method for acquiring video soundtrack is provided, comprising:

[0008] Acquire a video frame in a video, and identify the main content in the video frame;

[0009] Determining a key frame in the video according to the main content in the video frame, and determining a main content attribute of the key frame according to the main content of the key frame;

[0010] Obtaining a semantic feature vector of the video according to the main content and the main content attribute of the key frame;

[0011] The semantic feature vector of the video is matched with the semantic feature vectors of lyrics in a pre-established lyrics vector search pool, and a candidate set of soundtracks for the video is obtained according to the matching results of the semantic feature vectors.

[0012] In an exemplary embodiment of the present disclosure, the identifying the main content in the video frame includes:

[0013] The video frame is input into a subject detection model trained based on multiple target loss functions to identify the subject content in the video frame.

[0014] In an exemplary embodiment of the present disclosure, determining the key frame in the video according to the main content in the video frame includes:

[0015] Dividing the video screen of the video into a plurality of regions, and determining a key region corresponding to the main content according to the region occupied by the main content in the video frame in the video screen;

[0016] When the overlapping area between the key area corresponding to the main content in the video frame and the key area corresponding to the main content in the previous video frame is less than or equal to the area overlapping threshold, or when the main content of the video frame changes compared with the previous video frame, the video frame is determined as a key frame.

[0017] In an exemplary embodiment of the present disclosure, determining the main content attribute of the key frame according to the main content of the key frame includes:

[0018] A subject recognition picture corresponding to the subject content of the key frame is obtained, and the subject recognition picture is input into a pre-trained attribute recognition model to obtain a subject content attribute corresponding to the subject content of the key frame.

[0019] In an exemplary embodiment of the present disclosure, obtaining the semantic feature vector of the video according to the main content and the main content attribute of the key frame includes:

[0020] Obtaining a vocabulary matrix corresponding to the video according to the main content and the main content attributes of the key frame;

[0021] The vocabulary matrix is ​​input into a pre-trained semantic feature extraction model to obtain a semantic feature vector of the video.

[0022] In an exemplary embodiment of the present disclosure, the step of inputting the vocabulary matrix into a pre-trained semantic feature extraction model to obtain a semantic feature vector of the video includes:

[0023] Obtaining a vocabulary sub-matrix corresponding to each of the main contents in the video according to the vocabulary matrix;

[0024] Inputting the vocabulary sub-matrices corresponding to each of the main contents into the semantic feature extraction model respectively to obtain a single main semantic feature vector corresponding to each of the main contents;

[0025] According to the single subject semantic feature vector corresponding to all the subject contents, a semantic feature vector of the video is obtained.

[0026] In an exemplary embodiment of the present disclosure, matching the semantic feature vector of the video with the semantic feature vector of lyrics in a pre-established lyrics vector search pool, and obtaining a soundtrack candidate set of the video according to the matching result of the semantic feature vector, includes:

[0027] Determine the vector distance between the semantic feature vector of the video and each semantic feature vector of lyrics in the lyrics vector retrieval pool;

[0028] A candidate soundtrack of the video is obtained according to the vector distance, and a candidate soundtrack set of the video is generated according to the candidate soundtrack of the video.

[0029] In an exemplary embodiment of the present disclosure, the method for establishing the lyrics vector retrieval pool includes:

[0030] Obtaining a song to be matched and corresponding lyrics data, and inputting the lyrics data into a pre-trained semantic feature extraction model to obtain a lyrics semantic feature vector corresponding to the song;

[0031] The lyrics vector retrieval pool is established according to the lyrics semantic feature vectors corresponding to all the songs.

[0032] In an exemplary embodiment of the present disclosure, acquiring a video frame in a video includes:

[0033] Frames are extracted from the video according to a preset time interval to obtain video frames in the video.

[0034] According to a second aspect of the present disclosure, a device for acquiring video soundtrack is provided, comprising:

[0035] A main content recognition module is configured to obtain a video frame in a video and recognize the main content in the video frame;

[0036] A subject attribute recognition module is configured to determine a key frame in the video according to the subject content in the video frame, and determine a subject content attribute of the key frame according to the subject content of the key frame;

[0037] A semantic vector determination module, configured to obtain a semantic feature vector of the video according to the main content and the main content attribute of the key frame;

[0038] The semantic vector matching module is configured to match the semantic feature vector of the video with the semantic feature vector of lyrics in a pre-established lyrics vector search pool, and obtain a candidate set of soundtracks for the video based on the matching results of the semantic feature vectors.

[0039] In an exemplary embodiment of the present disclosure, the main content identification module includes:

[0040] The subject recognition unit is configured to input the video frame into a subject detection model trained based on multiple target loss functions to recognize the subject content in the video frame.

[0041] In an exemplary embodiment of the present disclosure, the subject attribute identification module includes:

[0042] A key area determination unit is configured to divide the video screen of the video into a plurality of areas, and determine the key area corresponding to the main content according to the area occupied by the main content in the video frame in the video screen;

[0043] The key frame determination unit is configured to determine the video frame as a key frame when the overlapping area between the key area corresponding to the main content in the video frame and the key area corresponding to the main content in the previous video frame is less than or equal to an area overlap threshold, or when the main content of the video frame changes compared to the main content in the previous video frame.

[0044] In an exemplary embodiment of the present disclosure, the subject attribute identification module includes:

[0045] The attribute recognition unit is configured to obtain a subject recognition picture corresponding to the subject content of the key frame, and input the subject recognition picture into a pre-trained attribute recognition model to obtain the subject content attribute corresponding to the subject content of the key frame.

[0046] In an exemplary embodiment of the present disclosure, the semantic vector determination module includes:

[0047] A vocabulary matrix determination unit is configured to obtain a vocabulary matrix corresponding to the video according to the main content and the main content attribute of the key frame;

[0048] The semantic vector extraction unit is configured to input the vocabulary matrix into a pre-trained semantic feature extraction model to obtain a semantic feature vector of the video.

[0049] In an exemplary embodiment of the present disclosure, the semantic vector extraction unit includes:

[0050] A vocabulary submatrix determination unit is configured to obtain a vocabulary submatrix corresponding to each of the main contents in the video according to the vocabulary matrix;

[0051] A single vector extraction unit is configured to input the vocabulary sub-matrices corresponding to each of the main contents into the semantic feature extraction model to obtain a single main semantic feature vector corresponding to each of the main contents;

[0052] The comprehensive vector extraction unit is configured to execute a single subject semantic feature vector corresponding to all the subject contents to obtain a semantic feature vector of the video.

[0053] In an exemplary embodiment of the present disclosure, the semantic vector matching module includes:

[0054] A vector distance determination unit, configured to determine the vector distance between the semantic feature vector of the video and each lyric semantic feature vector in the lyric vector retrieval pool;

[0055] The candidate soundtrack determining unit is configured to obtain the candidate soundtrack of the video according to the vector distance, and generate a soundtrack candidate set of the video according to the candidate soundtrack of the video.

[0056] In an exemplary embodiment of the present disclosure, the video soundtrack acquisition device further includes a vector search pool establishment module, and the vector search pool establishment module includes:

[0057] A lyrics vector extraction unit is configured to obtain a song to be matched and corresponding lyrics data, and input the lyrics data into a pre-trained semantic feature extraction model to obtain a lyrics semantic feature vector corresponding to the song;

[0058] The retrieval pool establishing unit is configured to establish the lyrics vector retrieval pool according to the lyrics semantic feature vectors corresponding to all the songs.

[0059] In an exemplary embodiment of the present disclosure, the main content identification module includes:

[0060] The video frame acquisition unit is configured to extract frames from the video according to a preset time interval to obtain video frames in the video.

[0061] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory configured to execute executable instructions stored in the processor; wherein the processor is configured to execute any one of the above-described methods for acquiring video soundtracks by executing the executable instructions.

[0062] According to a fourth aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the method for obtaining video soundtracks described in any one of the above is implemented.

[0063] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method for obtaining video soundtracks described in any one of the above is implemented.

[0064] The exemplary embodiments of the present disclosure may have the following beneficial effects:

[0065] In the method for obtaining video soundtracks in the example implementation of the present disclosure, by identifying the main content and the corresponding main content attributes in the video, and extracting the semantic feature vector of the video based on the main content of the video and the corresponding main content attributes, the semantic feature vector of the video is then matched with each lyric semantic feature vector in the pre-established lyrics vector retrieval pool, thereby obtaining a soundtrack candidate set that matches the content of the video. The method for obtaining video soundtracks in the example implementation of the present disclosure, on the one hand, can improve the matching degree between the recommended video soundtrack and the video content by extracting the semantic features of the video based on the main content of the video and the corresponding main content attributes; on the other hand, by matching the semantic feature vector of the video with the lyric vector retrieval pool generated based on a large amount of lyrics corpus, the recommendation quality of the video soundtrack can be further improved. When using the soundtrack candidate set obtained by the method for obtaining video soundtracks in the example implementation of the present disclosure, it is easier for users to obtain soundtracks that match the video content, thereby improving the user experience.

[0066] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0068] Figure 1 A schematic diagram showing a flow chart of a method for acquiring video soundtracks according to an exemplary embodiment of the present disclosure;

[0069] Figure 2 A schematic diagram showing a process of determining key frames in a video according to an exemplary embodiment of the present disclosure;

[0070] Figure 3A schematic diagram of a process of obtaining a semantic feature vector of a video according to an exemplary embodiment of the present disclosure is shown;

[0071] Figure 4 A schematic diagram of a process of obtaining a semantic feature vector of a video through a semantic feature extraction model according to an exemplary embodiment of the present disclosure is shown;

[0072] Figure 5 A schematic diagram of a process of matching a semantic feature vector of a video in a retrieval pool according to an exemplary embodiment of the present disclosure is shown;

[0073] Figure 6 A schematic diagram showing a flow chart of a method for acquiring video soundtracks according to a specific implementation of the present disclosure;

[0074] Figure 7 A block diagram showing a device for acquiring video soundtracks according to an exemplary embodiment of the present disclosure;

[0075] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0076] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0077] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein.

[0078] The following example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0079] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0080] With the development of the Internet and the popularization of 4G technology, short video applications have gradually become one of the most popular applications. The user base and daily active users of short video applications have reached hundreds of millions, and sharing life through short videos has become the choice of most people.

[0081] The soundtrack plays a finishing touch role for short videos. Good soundtracks can make short videos more popular and help them spread quickly and become popular videos. According to statistics, up to 80% of users will choose a suitable music as the soundtrack for short videos, which further proves the importance of soundtracks for short videos. Therefore, in the development process of short video applications, how to recommend suitable video soundtracks for users is a very important task.

[0082] In some related embodiments, the recommendation of video soundtracks is mainly based on user behavior information, such as obtaining music that the user has collected, soundtracks of videos that the user has clicked, POIs (Point of Interest), or some popular soundtracks of the current application, etc., and then recommending soundtracks for short videos shot or uploaded by the user based on the above information. However, the music recommended in this way may not be consistent with the video currently shot or uploaded by the user, which may cause the user to be unable to choose satisfactory or appropriate music when adding video soundtracks, affecting the user's experience.

[0083] Based on the problems in the above-mentioned related embodiments, this exemplary embodiment first provides a method for obtaining video soundtracks. Figure 1 As shown, the method for obtaining the video soundtrack may include the following steps:

[0084] Step S110: Obtain video frames in the video and identify main content in the video frames.

[0085] Step S120: Determine key frames in the video according to the main content in the video frame, and determine main content attributes of the key frames according to the main content of the key frames.

[0086] Step S130: Obtain a semantic feature vector of the video according to the main content and main content attributes of the key frame.

[0087] Step S140: Match the semantic feature vector of the video with the semantic feature vector of lyrics in a pre-established lyrics vector search pool, and obtain a candidate set of soundtracks for the video based on the matching results of the semantic feature vectors.

[0088] In the method for obtaining video soundtracks in the example implementation of the present disclosure, by identifying the main content and the corresponding main content attributes in the video, and extracting the semantic feature vector of the video based on the main content of the video and the corresponding main content attributes, the semantic feature vector of the video is then matched with each lyric semantic feature vector in the pre-established lyrics vector retrieval pool, thereby obtaining a soundtrack candidate set that matches the content of the video. The method for obtaining video soundtracks in the example implementation of the present disclosure, on the one hand, can improve the matching degree between the recommended video soundtrack and the video content by extracting the semantic features of the video based on the main content of the video and the corresponding main content attributes; on the other hand, by matching the semantic feature vector of the video with the lyric vector retrieval pool generated based on a large amount of lyrics corpus, the recommendation quality of the video soundtrack can be further improved. When using the soundtrack candidate set obtained by the method for obtaining video soundtracks in the example implementation of the present disclosure, it is easier for users to obtain soundtracks that match the video content, thereby improving the user experience.

[0089] Next, combine Figures 2 to 5 The above steps of this exemplary embodiment are described in more detail.

[0090] In step S110 , a video frame in a video is acquired, and main content in the video frame is identified.

[0091] Video is formed by the continuous playback of coherent frames, and the number of frames displayed per second can be expressed by a certain frame rate.

[0092] In this example implementation, after acquiring a video, frames can be extracted from the video according to a preset time interval to obtain video frames in the video. For example, a frame extraction operation can be performed from the video at a preset time interval of 0.2 seconds, and 5 video frames corresponding to each second of video can be obtained. By reducing the number of video frames through the frame extraction operation, the processing efficiency of subsequent video frames can be improved.

[0093] After obtaining the video frame in the video, it is necessary to identify the main content in the video frame. The main content refers to the video subject related to the video content, such as the characters, background, objects, etc. that appear in the video. Specifically, it can include characters, animals, background, vehicles, buildings, etc.

[0094] In this example implementation, the video frame can be input into a subject detection model trained based on multiple target loss functions to identify the main content in the video frame. Among them, the subject detection model can use the YOLOv3 (You Only Look Once) object detection deep model trained with multiple targets, and the training data of the subject detection model can use public data sets and a large number of annotated pictures in online short videos. Multi-target joint training refers to training using multiple target loss functions. For example, the model can be trained and optimized using three target loss functions: border loss, target loss, and classification loss. By identifying the main content in the video frame with a subject detection model trained based on multiple target loss functions, not only the recognition speed can be improved, but also the recognition accuracy of the main content can be improved.

[0095] In step S120, key frames in the video are determined according to the main content in the video frame, and main content attributes of the key frames are determined according to the main content of the key frames.

[0096] In this example implementation, if a frame has a large change in content from the previous frame, the video frame is extracted as a key frame. For example, if the main content of two adjacent video frames differs too much and the main content occupies a small repetition in the key area of ​​the picture, the video frame is determined as a key frame.

[0097] In this example implementation, Figure 2 As shown, determining the key frames in the video according to the main content in the video frame may specifically include the following steps:

[0098] Step S210: Divide the video screen of the video into a plurality of regions, and determine the key region corresponding to the main content according to the region occupied by the main content in the video frame in the video screen.

[0099] In this example implementation, the video screen may be segmented into a plurality of regions. For each video frame, the region occupied by the main content in the video frame is obtained, and the region occupied by the main content is used as the key region corresponding to the main content.

[0100] Step S220. When the overlapping area between the key area corresponding to the main content in the video frame and the key area corresponding to the main content in the previous video frame is less than or equal to the area overlap threshold, or when the main content of the video frame changes compared with the previous video frame, the video frame is determined as a key frame.

[0101] For the same main content, if the overlap area between the key area corresponding to the main content in the current video frame and the key area corresponding to the main content in the previous video frame is less than or equal to the area overlap threshold, or the main content of the current video frame has changed compared to the main content in the previous video frame, such as from a person to an animal, the current video frame is determined as a key frame. By extracting the key frames in the video and determining the main content and the corresponding main content attributes of the key frames, compared to extracting the main content and the main content attributes from all video frames, it is possible to minimize the repeated extraction of the main content and improve the efficiency of video content recognition.

[0102] After extracting all key frames from the video frame, the main content attributes of the key frames are determined according to the main content of the key frames. The main content attributes refer to the attributes of the main content. For example, the attributes of characters may include: gender, age, appearance, skin color, etc.; the attributes of animals may include: species, fur color, body shape, etc.; the attributes of buildings may include: type (high-rise buildings, bungalows...), material, etc.; the attributes of the background may include: scene (sky, seaside, indoor, farmland...), time (morning, noon, evening, late night), etc.

[0103] In this example implementation, a subject identification picture corresponding to the subject content of the key frame can be obtained, and the subject identification picture can be input into a pre-trained attribute recognition model to obtain the subject content attribute corresponding to the subject content of the key frame. The subject identification picture refers to a picture cut out from the picture according to the corresponding subject content, rather than the entire picture.

[0104] The attribute recognition model can use a CNN (Convolutional Neural Network) multi-label deep recognition model. For each type of main content, the attributes of the main content are defined and the CNN multi-label deep recognition model of the relevant attributes is trained. The image cut out of the identified main content area is input into the model to obtain the main content attributes corresponding to the main content.

[0105] In this example implementation, the main content and main content attributes of the video are identified based on the deep neural network model, which can improve the accuracy of the main content and main content attribute identification in the key frame. If the main content and main content attribute identification are inaccurate, the subsequent recommended songs will also be inaccurate.

[0106] In step S130, a semantic feature vector of the video is obtained according to the main content and main content attributes of the key frame.

[0107] In this example implementation, the semantic feature vector refers to a feature vector that is extracted based on the main content and main content attributes and can reflect the video content, such as an embedding (vectorized) vector, and its size can be, for example, 128 dimensions.

[0108] In this example implementation, Figure 3 As shown, obtaining the semantic feature vector of the video according to the main content and main content attributes of the key frame may specifically include the following steps:

[0109] Step S310: Obtain a vocabulary matrix corresponding to the video according to the main content and main content attributes of the key frame.

[0110] According to the main content and main content attributes extracted from the key frames, the main content vocabulary sequence consisting of each main content and its related main content attributes can be obtained. The vocabulary sequences corresponding to the main content in all key frames of the video are integrated to obtain the vocabulary matrix corresponding to the video.

[0111] For example, the vocabulary matrix corresponding to a video is as follows:

[0112]

[0113] The first row of the above vocabulary matrix is ​​the name of the main content, and the following rows are the corresponding main content attributes.

[0114] Step S320: Input the vocabulary matrix into a pre-trained semantic feature extraction model to obtain a semantic feature vector of the video.

[0115] In this example implementation, the vocabulary matrix of the video can be input into a pre-trained semantic feature extraction model to obtain a semantic feature vector corresponding to the video. The semantic feature extraction model can use a semantic feature extraction model based on BERT training, which can be obtained based on a large number of lyrics training. By inputting the vocabulary matrix into the model, the semantic feature vector corresponding to the video, i.e., the semantic embedding feature vector, can be extracted.

[0116] In this example implementation, Figure 4 As shown, the vocabulary matrix is ​​input into the pre-trained semantic feature extraction model to obtain the semantic feature vector of the video, which can specifically include the following steps:

[0117] Step S410: Obtain vocabulary sub-matrices corresponding to each main content in the video according to the vocabulary matrix.

[0118] According to each column of words in the vocabulary matrix, we can get the vocabulary sub-matrix corresponding to the main content of the column, and then merge them into a sentence. For example, according to the first column in the above vocabulary matrix, we can get (person, 25-year-old handsome man, yellow race).

[0119] Step S420: Input the vocabulary sub-matrices corresponding to each main content into the semantic feature extraction model respectively to obtain a single main semantic feature vector corresponding to each main content.

[0120] The vocabulary sub-matrices corresponding to each main content in the vocabulary matrix are respectively input into the semantic feature extraction model to obtain a single main semantic feature vector corresponding to each main content, that is, a semantic embedding feature vector corresponding to each main content.

[0121] Step S430: Obtain a semantic feature vector of the video based on the single subject semantic feature vector corresponding to all the subject contents.

[0122] The single subject semantic feature vectors corresponding to all the subject contents are integrated. For example, the semantic embedding feature vectors corresponding to each subject content can be summed and then averaged. The obtained comprehensive semantic embedding is the semantic feature vector corresponding to the video.

[0123] In this example implementation, a corresponding vocabulary matrix is ​​generated based on all the main contents in the video and their corresponding main content attributes, and a single main semantic feature vector corresponding to all the main contents is integrated, which can improve the accuracy of video semantic feature extraction and thus improve the accuracy of subsequent lyrics matching results.

[0124] In step S140, the semantic feature vector of the video is matched with the semantic feature vector of lyrics in a pre-established lyrics vector search pool, and a candidate set of soundtracks for the video is obtained based on the matching results of the semantic feature vectors.

[0125] In this example implementation, a lyrics vector retrieval pool can be pre-established based on the lyrics corresponding to a large number of songs, and then the semantic feature vector of the video is matched with the semantic feature vectors of the lyrics in the lyrics vector retrieval pool to obtain a candidate set of soundtracks for the video based on the matching results.

[0126] As for the method for establishing a lyric vector retrieval pool, a large number of songs to be matched and the corresponding lyric data can be obtained, and the lyric data can be input into a pre-trained semantic feature extraction model to obtain the lyric semantic feature vector corresponding to each song, and then a lyric vector retrieval pool can be established based on the lyric semantic feature vectors corresponding to all the songs to be matched.

[0127] In the process of establishing the lyrics vector retrieval pool, for Chinese songs, the lyrics data of the songs can be directly used to extract the lyrics semantic feature vectors; for foreign songs, the lyrics can be translated into Chinese using a machine translation model and then the corresponding lyrics semantic feature vectors can be extracted; for songs without lyrics, the text label data of the song itself (such as style, scene, emotion, theme, etc.) can be used as the text input for extracting lyrics semantic feature vectors. In this way, the design of playlists under various labels can be realized, and the quality of recommended songs can be further improved by matching the semantic feature vectors of the video with the lyrics vector retrieval pool generated based on a large amount of lyrics corpus.

[0128] In this example implementation, Figure 5 As shown, the semantic feature vector of the video is matched with the semantic feature vector of lyrics in the pre-established lyrics vector retrieval pool, and a candidate set of music for the video is obtained according to the matching result of the semantic feature vector. Specifically, the following steps may be included:

[0129] Step S510: Determine the vector distance between the semantic feature vector of the video and each semantic feature vector of lyrics in the lyrics vector retrieval pool.

[0130] In a pre-established lyrics vector retrieval pool, the vector distance between the semantic feature vector of the video and the lyrics semantic feature vector in the retrieval pool is calculated. For example, the L2 (Euclidean) distance between two vectors can be calculated.

[0131] Step S520: Obtain candidate soundtracks for the video according to the vector distance, and generate a candidate soundtrack set for the video according to the candidate soundtracks for the video.

[0132] After obtaining the vector distance between the semantic feature vector of the video and the semantic feature vector of the lyrics in the search pool, the ANN search technology can be used to select a preset number of songs corresponding to the lyrics with the smallest distance as candidate soundtracks for the video, and then a soundtrack candidate set for the video is generated based on all the candidate soundtracks. By obtaining multiple songs with the highest correlation with the semantic features of the video to generate a soundtrack candidate set for the video, and then sending the generated soundtrack candidate set to the client for the user to select, it can make it easier for the user to obtain the soundtrack that matches the video content, thereby improving the user experience.

[0133] like Figure 6 The figure is a flowchart of a method for obtaining video soundtracks in a specific implementation of the present disclosure, which is an example of the above steps in this example implementation. The specific steps of the flowchart are as follows:

[0134] Step S610: Obtain short video.

[0135] Step S620: Detect the main content of the video frame.

[0136] Step S630: Extract key frames.

[0137] Step S640: Identify the main content attributes.

[0138] Step S650: Generate a vocabulary matrix.

[0139] Step S660: semantic feature extraction.

[0140] Step S670: Obtain lyrics training corpus.

[0141] Step S680: Establish a lyrics vector retrieval pool.

[0142] Step S690. ANN retrieval technology recalls matching songs.

[0143] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0144] Furthermore, the present disclosure also provides a device for obtaining video soundtracks. Figure 7 As shown, the video soundtrack acquisition device may include a main content recognition module 710, a main attribute recognition module 720, a semantic vector determination module 730, and a semantic vector matching module 740. Among them:

[0145] The main content identification module 710 is configured to obtain a video frame in a video and identify the main content in the video frame;

[0146] The subject attribute identification module 720 is configured to determine the key frames in the video according to the subject content in the video frame, and determine the subject content attributes of the key frames according to the subject content of the key frames;

[0147] The semantic vector determination module 730 is configured to obtain a semantic feature vector of the video according to the main content and main content attributes of the key frame;

[0148] The semantic vector matching module 740 is configured to match the semantic feature vector of the video with the semantic feature vector of lyrics in a pre-established lyrics vector search pool, and obtain a candidate set of soundtracks for the video based on the matching results of the semantic feature vectors.

[0149] In some exemplary embodiments of the present disclosure, the subject content identification module 710 may include a subject identification unit configured to input a video frame into a subject detection model trained based on multiple target loss functions to identify the subject content in the video frame.

[0150] In some exemplary embodiments of the present disclosure, the subject attribute identification module 720 may include a key area determination unit and a key frame determination unit.

[0151] The key area determination unit is configured to divide the video screen of the video into a plurality of areas, and determine the key area corresponding to each main content according to the area occupied by the main content in the video frame in the video screen;

[0152] The key frame determination unit is configured to determine a video frame as a key frame when an overlapping area between a key area corresponding to a main content in a video frame and a key area corresponding to a main content in a previous video frame is less than or equal to an area overlapping threshold, or when the main content of the video frame changes compared to the main content in the previous video frame.

[0153] In some exemplary embodiments of the present disclosure, the subject attribute identification module 720 may include an attribute identification unit, which is configured to obtain a subject identification picture corresponding to the subject content of the key frame, and input the subject identification picture into a pre-trained attribute recognition model to obtain the subject content attributes corresponding to the subject content of the key frame.

[0154] In some exemplary embodiments of the present disclosure, the semantic vector determination module 730 may include a vocabulary matrix determination unit and a semantic vector extraction unit.

[0155] The vocabulary matrix determination unit is configured to obtain a vocabulary matrix corresponding to the video according to the main content and main content attributes of the key frame;

[0156] The semantic vector extraction unit is configured to input the vocabulary matrix into a pre-trained semantic feature extraction model to obtain a semantic feature vector of the video.

[0157] In some exemplary embodiments of the present disclosure, the semantic vector extraction unit may include a vocabulary sub-matrix determination unit, a single vector extraction unit, and a comprehensive vector extraction unit.

[0158] The vocabulary submatrix determination unit is configured to obtain a vocabulary submatrix corresponding to each main content in the video according to the vocabulary matrix;

[0159] The single vector extraction unit is configured to input the vocabulary sub-matrices corresponding to each main content into the semantic feature extraction model respectively, and obtain the single main semantic feature vector corresponding to each main content;

[0160] The comprehensive vector extraction unit is configured to obtain a semantic feature vector of the video according to a single subject semantic feature vector corresponding to all subject contents.

[0161] In some exemplary embodiments of the present disclosure, the semantic vector matching module 740 may include a vector distance determination unit and a candidate soundtrack determination unit.

[0162] The vector distance determination unit is configured to determine the vector distance between the semantic feature vector of the video and each semantic feature vector of lyrics in the lyrics vector retrieval pool;

[0163] The candidate soundtrack determining unit is configured to obtain candidate soundtracks of the video according to the vector distance, and generate a soundtrack candidate set of the video according to the candidate soundtracks of the video.

[0164] In some exemplary embodiments of the present disclosure, the apparatus for acquiring video soundtracks may further include a vector search pool establishment module, which may include a lyrics vector extraction unit and a search pool establishment unit.

[0165] The lyrics vector extraction unit is configured as a lyrics vector extraction unit, configured to execute obtaining a song to be matched and corresponding lyrics data, and inputting the lyrics data into a pre-trained semantic feature extraction model to obtain a lyrics semantic feature vector corresponding to the song;

[0166] The retrieval pool establishing unit is configured as a retrieval pool establishing unit, and is configured to establish a lyrics vector retrieval pool according to the lyrics semantic feature vectors corresponding to all songs.

[0167] In some exemplary embodiments of the present disclosure, the main content identification module 710 may include a video frame acquisition unit configured to extract frames from the video according to a preset time interval to obtain video frames in the video.

[0168] The specific details of each module / unit in the above-mentioned video soundtrack acquisition device have been described in detail in the corresponding method embodiment part, and will not be repeated here.

[0169] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present invention is shown.

[0170] It should be noted that Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0171] like Figure 8As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage part 808 to the random access memory (RAM) 803. In the RAM 803, various programs and data required for system operation are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0172] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0173] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 809, and / or installed from a removable medium 811. When the computer program is executed by a central processing unit (CPU) 801, various functions defined in the system of the present application are executed.

[0174] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0175] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0176] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0177] It should be noted that, although several modules of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be concretized.

[0178] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0179] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for obtaining video soundtrack, characterized in that: include: Acquire a video frame in a video, and identify the main content in the video frame; Determining a key frame in the video according to the main content in the video frame, and determining a main content attribute of the key frame according to the main content of the key frame; Obtaining a vocabulary matrix corresponding to the video according to the main content and the main content attributes of the key frame; Inputting the vocabulary matrix into a pre-trained semantic feature extraction model to obtain a semantic feature vector of the video, wherein the semantic feature vector is a feature vector extracted based on the main content and the main content attribute to reflect the video content; The semantic feature vector of the video is matched with the semantic feature vectors of lyrics in a pre-established lyrics vector search pool, and a candidate set of soundtracks for the video is obtained according to the matching results of the semantic feature vectors.

2. The method for obtaining video soundtrack according to claim 1, characterized in that: The identifying the main content in the video frame includes: The video frame is input into a subject detection model trained based on multiple target loss functions to identify the subject content in the video frame.

3. The method for obtaining video soundtrack according to claim 1, characterized in that: The step of determining the key frame in the video according to the main content in the video frame comprises: Dividing the video screen of the video into a plurality of regions, and determining a key region corresponding to the main content according to the region occupied by the main content in the video frame in the video screen; When the overlapping area between the key area corresponding to the main content in the video frame and the key area corresponding to the main content in the previous video frame is less than or equal to the area overlapping threshold, or when the main content of the video frame changes compared with the previous video frame, the video frame is determined as a key frame.

4. The method for obtaining video soundtrack according to claim 1, characterized in that: The determining the main content attribute of the key frame according to the main content of the key frame includes: A subject recognition picture corresponding to the subject content of the key frame is obtained, and the subject recognition picture is input into a pre-trained attribute recognition model to obtain a subject content attribute corresponding to the subject content of the key frame.

5. The method for obtaining video soundtrack according to claim 1, characterized in that: The step of inputting the vocabulary matrix into a pre-trained semantic feature extraction model to obtain a semantic feature vector of the video includes: Obtaining a vocabulary sub-matrix corresponding to each of the main contents in the video according to the vocabulary matrix; Inputting the vocabulary sub-matrices corresponding to each of the main contents into the semantic feature extraction model respectively to obtain a single main semantic feature vector corresponding to each of the main contents; According to the single subject semantic feature vector corresponding to all the subject contents, a semantic feature vector of the video is obtained.

6. The method for obtaining video soundtrack according to claim 1, characterized in that: The step of matching the semantic feature vector of the video with the semantic feature vector of lyrics in a pre-established lyrics vector search pool, and obtaining a candidate set of soundtracks for the video according to the matching results of the semantic feature vectors, comprises: Determine the vector distance between the semantic feature vector of the video and each semantic feature vector of lyrics in the lyrics vector retrieval pool; A candidate soundtrack of the video is obtained according to the vector distance, and a candidate soundtrack set of the video is generated according to the candidate soundtrack of the video.

7. The method for obtaining video soundtrack according to claim 1, characterized in that: The method for establishing the lyrics vector retrieval pool includes: Obtaining a song to be matched and corresponding lyrics data, and inputting the lyrics data into a pre-trained semantic feature extraction model to obtain a lyrics semantic feature vector corresponding to the song; The lyrics vector retrieval pool is established according to the lyrics semantic feature vectors corresponding to all the songs.

8. The method for obtaining video soundtrack according to claim 1, characterized in that: The step of obtaining a video frame in a video includes: Frames are extracted from the video according to a preset time interval to obtain video frames in the video.

9. A device for obtaining video soundtrack, characterized in that: include: A main content recognition module is configured to obtain a video frame in a video and recognize the main content in the video frame; A subject attribute recognition module is configured to determine a key frame in the video according to the subject content in the video frame, and determine a subject content attribute of the key frame according to the subject content of the key frame; The semantic vector determination module is configured to obtain a vocabulary matrix corresponding to the video according to the main content and the main content attributes of the key frame; input the vocabulary matrix into a pre-trained semantic feature extraction model to obtain a semantic feature vector of the video, wherein the semantic feature vector is a feature vector extracted based on the main content and the main content attributes to reflect the video content; The semantic vector matching module is configured to match the semantic feature vector of the video with the semantic feature vector of lyrics in a pre-established lyrics vector search pool, and obtain a candidate set of soundtracks for the video based on the matching results of the semantic feature vectors.

10. The device for acquiring video soundtrack according to claim 9, characterized in that: The main content identification module includes: The subject recognition unit is configured to input the video frame into a subject detection model trained based on multiple target loss functions to recognize the subject content in the video frame.

11. The device for acquiring video soundtrack according to claim 9, characterized in that: The subject attribute identification module includes: A key area determination unit is configured to divide the video screen of the video into a plurality of areas, and determine the key area corresponding to the main content according to the area occupied by the main content in the video frame in the video screen; The key frame determination unit is configured to determine the video frame as a key frame when the overlapping area between the key area corresponding to the main content in the video frame and the key area corresponding to the main content in the previous video frame is less than or equal to an area overlap threshold, or when the main content of the video frame changes compared to the main content in the previous video frame.

12. The device for acquiring video soundtrack according to claim 9, characterized in that: The subject attribute identification module includes: The attribute recognition unit is configured to obtain a subject recognition picture corresponding to the subject content of the key frame, and input the subject recognition picture into a pre-trained attribute recognition model to obtain the subject content attribute corresponding to the subject content of the key frame.

13. The device for acquiring video soundtrack according to claim 9, characterized in that: The semantic vector determination module comprises: A vocabulary submatrix determination unit is configured to obtain a vocabulary submatrix corresponding to each of the main contents in the video according to the vocabulary matrix; A single vector extraction unit is configured to input the vocabulary sub-matrices corresponding to each of the main contents into the semantic feature extraction model to obtain a single main semantic feature vector corresponding to each of the main contents; The comprehensive vector extraction unit is configured to execute a single subject semantic feature vector corresponding to all the subject contents to obtain a semantic feature vector of the video.

14. The device for acquiring video soundtrack according to claim 9, characterized in that: The semantic vector matching module includes: A vector distance determination unit, configured to determine the vector distance between the semantic feature vector of the video and each lyric semantic feature vector in the lyric vector retrieval pool; The candidate soundtrack determining unit is configured to obtain the candidate soundtrack of the video according to the vector distance, and generate a soundtrack candidate set of the video according to the candidate soundtrack of the video.

15. The device for acquiring video soundtrack according to claim 9, characterized in that: The video soundtrack acquisition device further includes a vector search pool establishment module, and the vector search pool establishment module includes: A lyrics vector extraction unit is configured to obtain a song to be matched and corresponding lyrics data, and input the lyrics data into a pre-trained semantic feature extraction model to obtain a lyrics semantic feature vector corresponding to the song; The retrieval pool establishing unit is configured to establish the lyrics vector retrieval pool according to the lyrics semantic feature vectors corresponding to all the songs.

16. The device for acquiring video soundtrack according to claim 9, characterized in that: The main content identification module includes: The video frame acquisition unit is configured to extract frames from the video according to a preset time interval to obtain video frames in the video.

17. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for acquiring video soundtracks as described in any one of claims 1 to 8.

18. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for acquiring video soundtracks as claimed in any one of claims 1 to 8.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for obtaining video soundtracks according to any one of claims 1 to 8 is implemented.

Images