Method, medium, device and computing device for determining highlight segments in a video

Abstract

The embodiments of the present disclosure provide a method, medium, device and computing device for determining highlight segments in a video. When it is necessary to recommend videos to users, video segments corresponding to different scenes in the video and user features are used as model inputs, the model is used to calculate the user's preference for different video segments, and then the highlight segments are determined according to the size of the preference corresponding to different video segments.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of information technology, and more specifically, embodiments of the present disclosure relate to a method, medium, apparatus, and computing device for determining highlight segments in a video. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] At present, the video selection interface provided by Internet platforms to users usually displays highlight clips of multiple videos. If a user is attracted by a highlight clip, he or she will often further open the video corresponding to the highlight clip to watch.

[0004] In the related art, a certain segment in a video is manually judged to be likely to attract users, and then the segment is used as a highlight segment. However, this method of determining the highlight segment of a video is not effective enough. Summary of the invention

[0005] In this context, embodiments of the present disclosure are intended to provide a method, medium, apparatus, and computing device for determining highlight segments in a video, so that the highlight segments determined from the video can better meet the personalized preferences of users and more easily attract users to watch the video.

[0006] In a first aspect of the embodiments of the present disclosure, a method for determining a highlight segment in a video is provided, comprising:

[0007] Determine a video to be recommended to the user, and obtain at least two video clips corresponding to different scenes in the video;

[0008] Inputting the user features of the user and the video segment features respectively corresponding to the at least two video segments into a target model, and outputting the preference of the user for the at least two video segments respectively;

[0009] A highlight segment is determined from the at least two video segments according to the sizes of the preferences respectively corresponding to the at least two video segments.

[0010] In one embodiment of the present disclosure, obtaining at least two video clips corresponding to different scenes in the video includes:

[0011] For each unit time length of the video, extract the video frames within the unit time length;

[0012] Calculating the similarity between the video frames corresponding to each unit time length of the video, and merging several unit time lengths whose similarity is less than a specified threshold into one time period;

[0013] At least two video segments corresponding to different time periods are extracted from the video as at least two video segments corresponding to different scenes.

[0014] In one embodiment of the present disclosure, it also includes:

[0015] For each video segment of the at least two video segments, extract a plurality of key video frames from the video segment, and determine an image feature corresponding to each extracted key video frame;

[0016] The set of image features corresponding to each key video frame is used as the video segment features corresponding to the video segment.

[0017] In one embodiment of the present disclosure, extracting a plurality of key video frames from the video clip includes:

[0018] Calculate the average pixel value corresponding to multiple video frames in the video clip;

[0019] For each video frame in the video clip, if the difference between the pixel value corresponding to the video frame and the average pixel value is less than a specified value, the video frame is determined as a key video frame.

[0020] In one embodiment of the present disclosure, calculating the average pixel value corresponding to a plurality of video frames in the video clip includes:

[0021] An average value of pixel values ​​of pixel points at designated positions of multiple video frames in the video segment is calculated as an average pixel value corresponding to the multiple video frames.

[0022] In one embodiment of the present disclosure, the image feature corresponding to the key video frame is determined based on at least one of the following features:

[0023] Color features; brightness features; contrast features; clarity features; facial features corresponding to the included faces.

[0024] In one embodiment of the present disclosure, a set of image features corresponding to each key video frame is used as a video segment feature corresponding to the video segment, including:

[0025] According to the order of each key video frame in the video segment, the image features corresponding to each key video frame are arranged into an image feature sequence as the video segment features corresponding to the video segment.

[0026] In one embodiment of the present disclosure, it also includes:

[0027] For each video segment of the at least two video segments, position information of each key video frame of the video segment in the video segment is used as a video segment feature corresponding to the video segment.

[0028] In one embodiment of the present disclosure, it also includes:

[0029] Determine a text feature corresponding to the video according to at least one of an introductory text, a title text, and a subtitle text corresponding to the video;

[0030] For each video segment of the at least two video segments, the text feature corresponding to the video is used as the video segment feature corresponding to the video segment.

[0031] In one embodiment of the present disclosure, determining the text feature corresponding to the video according to at least one of the introductory text, the title text, and the subtitle text corresponding to the video includes:

[0032] Obtaining a text to be processed according to at least one of an introductory text, a title text, and a subtitle text corresponding to the video;

[0033] Segmenting the text to be processed to obtain a corresponding word sequence;

[0034] Taking a text consisting of at least two adjacent words in the word sequence as a positive text sample, to obtain a plurality of positive text samples;

[0035] Based on the multiple positive text samples, a text classification model is trained, and according to a model parameter set of the trained text classification model, text features corresponding to the video are determined.

[0036] In one embodiment of the present disclosure, the text classification model is constructed using a neural network algorithm, and the text classification model includes at least one hidden layer;

[0037] Determining text features corresponding to the video according to the trained model parameter set of the text classification model includes:

[0038] The hidden layer weight matrix corresponding to the trained text classification model is used as the text feature corresponding to the video.

[0039] In one embodiment of the present disclosure, the neural network algorithm used to construct the text classification model includes: Bert algorithm, or word2vec algorithm.

[0040] In one embodiment of the present disclosure, the user feature is used to characterize the video preference of the user.

[0041] In one embodiment of the present disclosure, the processing performed by the target model on the user feature and the video segment features respectively corresponding to the at least two video segments includes:

[0042] For each video segment of the at least two video segments, based on a self-attention mechanism, a video segment feature corresponding to the video segment is mapped to obtain a mapping feature corresponding to the video segment;

[0043] The user's preference for the video segment is calculated based on the mapping features between the user features and the video segment.

[0044] In one embodiment of the present disclosure, the target model training method includes:

[0045] Get a video sample set;

[0046] For each video sample, obtaining at least two video clip samples corresponding to different scenes in the video sample;

[0047] Inputting the user features and the video segment features corresponding to the at least two video segment samples into the target model for model training, and outputting the predicted preferences of the user for the at least two video segment samples respectively;

[0048] The predicted preference degree corresponding to the video clip sample selected as the highlight clip by the user among the at least two video clip samples is A, and the predicted preference degree set corresponding to the video clip sample not selected as the highlight clip by the user among the at least two video clip samples is B. i , i=1, ..., N, N is the number of video clip samples that are not selected as highlight clips by the user;

[0049] The training goal of the target model is to combine the predicted preference A with the predicted preference set B i The gap is maximized.

[0050] In a second aspect of the embodiments of the present disclosure, a device for determining a highlight segment in a video is provided, comprising:

[0051] An acquisition module determines a video to be recommended to a user, and acquires at least two video clips corresponding to different scenes in the video;

[0052] An analysis module, inputting the user characteristics of the user and the video segment characteristics corresponding to the at least two video segments respectively into a target model, and outputting the preference of the user for the at least two video segments respectively;

[0053] The determination module determines a highlight segment from the at least two video segments according to the sizes of the preferences respectively corresponding to the at least two video segments.

[0054] In a third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method for determining highlight segments in a video according to any embodiment of the present disclosure is implemented.

[0055] In a fourth aspect of an embodiment of the present disclosure, a computing device is provided, comprising a memory and a processor; the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement a method for determining highlight segments in a video according to any embodiment of the present disclosure when executing the computer instructions.

[0056] According to the method, medium, device and computing device for determining highlight segments in a video according to the embodiments of the present disclosure, when it is necessary to recommend videos to users, user features and video segments corresponding to different scenes in the video are used as model inputs, the model is used to calculate the user's preference for different video segments, and then the highlight segments are determined according to the size of the preference corresponding to the different video segments. The highlight segments determined in this way can better meet the user's personalized preferences and more easily attract users to watch videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0058] Figure 1 is a schematic diagram of a video selection interface exemplarily provided by the present disclosure;

[0059] Figure 2 is a flowchart of a method for determining a highlight segment in a video provided by the present disclosure;

[0060] Figure 3 is a flow chart of a method for determining video segment features corresponding to a video segment exemplarily provided by the present disclosure;

[0061] Figure 4 is a schematic diagram of a target model exemplarily provided by the present disclosure;

[0062] Figure 5 It is a flowchart of a model training method provided by the present disclosure;

[0063] Figure 6 is a structural schematic diagram of a device for determining a highlight segment in a video provided by the present disclosure;

[0064] Figure 7 is a schematic diagram of a computer-readable storage medium provided by the present disclosure;

[0065] Figure 8 is a structural schematic diagram of a computing device provided by the present disclosure;

[0066] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. The number of any element in the drawings is for example rather than limitation, and any naming is only for distinction and does not have any limiting meaning. DETAILED DESCRIPTION

[0067] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0068] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0069] According to an embodiment of the present disclosure, a method, medium, apparatus and computing device for determining highlight segments in a video are proposed.

[0070] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0071] At present, in order to better attract users to click and watch videos, Internet platforms usually play highlight clips of videos recommended to users in the video selection interface.

[0072] Figure 1 is a schematic diagram of a video selection interface provided by the present disclosure. Figure 1 As shown, the video selection interface includes several information display areas, each of which is used to display highlight clips of different videos. When a user clicks on an information display area, it can trigger the playback interface of the corresponding video. When the user browses the video selection interface, the highlight clips in each information area can be played automatically. Alternatively, when the user browses the video selection interface, each information area can start to automatically play the highlight clip in response to the user's interaction with the video selection interface. For example, when each information area in the video selection interface is initialized, it may not automatically start to play the highlight clip, but statically display a video frame contained in the highlight clip. When the user long presses the video picture in a certain information area, it can trigger the information area to start automatically playing the highlight clip.

[0073] Users can spend a short time browsing the highlight clips in the video selection interface, and use the highlight clips as a reference to determine whether the corresponding video is the video they are interested in. If a user is attracted by a highlight clip, he or she will often further open the video corresponding to the highlight clip to watch. It can be seen that whether the highlight clips determined from the video can be sufficiently attractive to users and whether they can meet the user's preferences is crucial.

[0074] The inventors have discovered that the same highlight segment of the same video has different appeal to different users. Therefore, in order to make different users more interested in the same video, different highlight segments can be shown to different users when recommending the same video to them.

[0075] In addition, the inventors have also found that the information content presented in the video can often be divided into different scenes, and the process of playing the video is actually the process of switching between different video clips corresponding to different scenes. The main factor that attracts users to watch the video is often the information content of a certain scene in the video. Therefore, the highlight clips can be determined in the video clips corresponding to different scenes of the video.

[0076] The method for determining highlight segments in a video provided by the present disclosure uses user features and video segments corresponding to different scenes in the video as model inputs when it is necessary to recommend videos to users, calculates the user's preference for different video segments using the model, and then determines the highlight segments according to the size of the preference corresponding to the different video segments. The highlight segments determined in this way can better meet the user's personalized preferences and are more likely to attract users to watch videos.

[0077] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0078] Figure 2 : is a flowchart of a method for determining a highlight segment in a video provided by the present disclosure, comprising the following steps:

[0079] S200: Determine a video to be recommended to a user, and obtain at least two video clips corresponding to different scenes in the video.

[0080] In some embodiments, the video to be recommended to the user needs to meet certain quality requirements.

[0081] For example, a video containing only one or a few video frames (which may be referred to as a static video) can provide very limited information to the user and is a video that does not meet the quality requirements. Therefore, the static video may not be recommended to the user.

[0082] The present disclosure provides a method for identifying static videos. A plurality of scattered video frames (e.g., video frames are extracted at equal intervals) in a video to be identified can be extracted, and the similarity between the extracted video frames can be calculated. If the calculated similarity is greater than a first preset threshold, it can be determined that the video frames are very similar and the video to be identified is a static video.

[0083] For another example, a video that contains multiple black video frames at the beginning or end (which may be referred to as a black screen video) provides a poor viewing experience to users and is also a video that does not meet quality requirements. Therefore, the black screen video may not be recommended to users, or the black screen video may be optimized (for example, several consecutive black video frames at the beginning and / or end may be deleted) before being recommended to users.

[0084] The present disclosure provides a method for identifying a black screen video. Extract several consecutive video frames at the beginning of a video to be identified, calculate the average value of the pixel values ​​of the pixels in each extracted video frame, and if the difference between the calculated average value and the black pixel value is less than a second preset threshold, determine that the corresponding video frame is a black video frame, and if the difference between the calculated average value and the black pixel value is not less than a second specified threshold, determine that the corresponding video frame is not a black video frame. If several consecutive video frames at the beginning of the video are all black video frames, then it can be determined that the video to be identified is a black screen video. In addition, several consecutive video frames at the end of the video to be identified can also be extracted, and a similar method can be used to determine whether the several consecutive video frames at the end are black video frames, and then determine whether it is a black screen video.

[0085] In the case where it is necessary to use the video selection interface to recommend videos to users, for each video to be recommended to the user, Figure 2 For the convenience of description, take how to determine the highlight segment of a single video as an example. Figure 2 The method flow shown is described.

[0086] It should be noted that the information content presented by the video may only include one scene. In this case, a video clip can be directly extracted from the video as a highlight clip without applying Figure 2 The method shown.

[0087] If the information content presented by the video contains at least two different scenes, then at least two video clips corresponding to the different scenes can be extracted from the video. The at least two video clips can be video clips corresponding to all the scenes contained in the information content presented by the video, or can be video clips corresponding to some of the scenes contained in the information content presented by the video.

[0088] In some embodiments, the video itself may contain tags for distinguishing different scenes. For example, the video producer may add a watermark to the last video frame corresponding to each scene in the video. For another example, the last video frame corresponding to each scene in the video is a black video frame. In this way, at least two video clips corresponding to different scenes in the video can be directly extracted based on the tags.

[0089] In some embodiments, if the video itself does not contain markers for distinguishing different scenes, image processing technology may be used to separate video segments corresponding to different scenes from the video.

[0090] For example, for each unit duration of the video, one or more video frames within the unit duration can be extracted; then, the similarity between the video frames corresponding to each unit duration of the video is calculated, and several unit durations with a similarity less than a specified threshold are merged into one time period; then, at least two video segments corresponding to different time periods are extracted from the video as at least two video segments corresponding to different scenes.

[0091] It should be noted that when calculating the similarity between the video frames corresponding to each unit time length of the video, if there are multiple video frames corresponding to a unit time length, the multiple video frames corresponding to the unit time length can be averaged or merged first to obtain a video frame corresponding to the unit time length. Thereafter, the similarity between the video frames corresponding to each unit time length is calculated.

[0092] Usually, the similarity between video frames in the same scene is small enough, and the unit durations corresponding to the video frames with small similarity are often continuous and can be combined into one time period. Naturally, different time periods in the video can correspond to different scenes.

[0093] S202: Input the user features of the user and the video segment features corresponding to the at least two video segments respectively into a target model, and output the user's preference for the at least two video segments respectively.

[0094] In some embodiments, the segment features corresponding to the video segment may be determined based on information of the video segment in the image dimension. For example, the video segment features corresponding to the video segment may be determined based on image information of all video frames included in the video segment.

[0095] For another example, the video segment features corresponding to the video segment can be determined based on the image information of several key frames contained in the video segment. For each of the at least two video segments obtained from the video, multiple key video frames can be extracted from the video segment, and the image features corresponding to each extracted key video frame can be determined; then the set of image features corresponding to each key video frame is used as the video segment features corresponding to the video segment.

[0096] The present disclosure provides a method for extracting key frames from a video clip. The average pixel value corresponding to a plurality of video frames in the video clip can be calculated; then for each video frame in the video clip, if the difference between the pixel value corresponding to the video frame and the average pixel value is less than a specified value, the video frame is determined as a key video frame.

[0097] The pixel value corresponding to the video frame may be the pixel value of a specified position (such as a central pixel) or the average pixel value of all pixels. The average pixel value corresponding to multiple video frames may be the average of the pixel values ​​corresponding to the multiple video frames.

[0098] The image feature of the video frame may be determined based on at least one of the following features:

[0099] Color features; brightness features; contrast features; clarity features; facial features corresponding to the included faces.

[0100] In addition, not only can the segment features corresponding to the video segment be determined based on the information of the video segment in the image dimension, but also the segment features corresponding to the video segment can be determined based on the information of the video in the text dimension. In other words, the segment features corresponding to the video segment can include not only image features but also text features.

[0101] It should be noted here that if the text features corresponding to the acquired video and the image features corresponding to the video clip are not in the same feature space, it is necessary to map the text features corresponding to the video and the image features corresponding to the video clip to the same feature space through dense operation, and then combine them into the video clip features corresponding to the video clip.

[0102] In some embodiments, the text features corresponding to the video can be determined based on at least one of the introductory text, title text, and subtitle text corresponding to the video; and then, for each of the at least two video segments obtained from the video, the text features corresponding to the video are used as the video segment features corresponding to the video segment.

[0103] For example, at least one of the introductory text, title text, and subtitle text corresponding to the video may be mapped into a text code as a text feature corresponding to the video.

[0104] For another example, the text to be processed can be obtained based on at least one of the introductory text, title text, and subtitle text corresponding to the video; the text to be processed is segmented to obtain a corresponding word sequence; a text composed of at least two adjacent words in the word sequence is used as a positive text sample to obtain multiple positive text samples; a text classification model is trained based on the multiple positive text samples, and the text features corresponding to the video are determined based on the model parameter set of the trained text classification model. Among them, the text composed of at least two non-adjacent words in the word sequence can be used as a negative text sample, or some unrelated words can be obtained to form negative text samples.

[0105] Since the text classification model is essentially trained based on the text semantics corresponding to the video, the text classification model trained in this way can learn a model parameter set that matches the text semantics of the video. The text features corresponding to the video determined based on such a model parameter set can better represent the abstract text semantics at a deeper level than the surface text semantics corresponding to the video.

[0106] The above-mentioned text classification model can be constructed by using a neural network algorithm (such as Bert algorithm or word2vec algorithm), and such a text classification model includes at least one hidden layer. The hidden layer weight matrix corresponding to the trained text classification model can be used as the text feature corresponding to the video.

[0107] In other embodiments, for each video segment obtained from the video, a text feature corresponding to the video segment may be determined according to the subtitle text corresponding to the video segment as the video segment feature corresponding to the video segment.

[0108] The user features in step S202 may be any attributes related to the user, such as age, gender, region, etc. In addition, the user features may also include features for characterizing the user's video preferences (determined based on statistical information such as the length of time the user watches videos, the style, type, and region of production of the videos they like to watch). Internet platforms usually gradually describe the user's user portrait in the process of providing services to users, and obtain more and more accurate user portraits. The user portrait can describe the user from multiple dimensions, especially the user's video preferences. Therefore, the user portrait can be used as a user feature.

[0109] It should be noted that for some users, the attribute information obtained from these users may be relatively small, or the time for providing services to these users may not be long enough (resulting in inadequate description of user portraits), which means that the distribution of user characteristics of these users is relatively sparse. In this case, dense operations can be performed on the user characteristics of these users to make the distribution of user characteristics of these users dense.

[0110] It should also be noted that if the user features and the video clip features are not in the same feature space, it is also necessary to map the user features and the video clip features to one feature space through a dense operation.

[0111] Figure 3 is a flow chart of a method for determining video segment features corresponding to a video segment provided by the present disclosure. Figure 3 , obtain two video clips corresponding to different scenes from the video, namely video clip A and video clip B. Then determine the three key frames of video clip A and determine the image features corresponding to each key frame; determine the three key frames of video clip B and determine the image features corresponding to each key frame. Then, the image features corresponding to the three key frames of video clip A and the text features corresponding to the video are combined into the clip features corresponding to video clip A. Similarly, the image features corresponding to the three key frames of video clip B and the text features corresponding to the video are combined into the clip features corresponding to video clip B.

[0112] The target model described in the present disclosure is a model for predicting the user's preference for a video segment based on input user features and video segment features. When constructing the target model, the structure of the target model can be defined, that is, the calculation function used by the target model can be defined, and the calculation function is used to calculate the user's preference for the video segment based on the user features and the video segment features.

[0113] The target model usually needs to be trained before a better prediction effect can be obtained. The process of training the target model is actually the process of optimizing the parameters in the calculation function (called model parameters).

[0114] Various algorithmic ideas can be used to define the calculation function of the target model, which is used to calculate the user's preference for the video segment according to the user characteristics and the video segment characteristics, and all should be within the scope of the disclosure. Here, an exemplary target model is provided, and the processing of the user characteristics and the video segment characteristics corresponding to the at least two video segments by the target model may include:

[0115] For each video clip of at least two video clips in the video, based on the self-attention mechanism, the video clip features corresponding to the video clip are mapped to obtain the mapping features corresponding to the video clip; according to the user features and the mapping features corresponding to the video clip, the user's preference for the video clip is calculated.

[0116] It should be noted here that the self-attention mechanism is a neural network algorithm that is good at capturing the internal correlation of data and increasing the weight of key information within the data. After the self-attention mechanism is used to map the video clip features corresponding to the video clip, the mapping features obtained can better characterize the internal correlation of the video clip features and increase the weight of the key information contained in the video clip features. For example, if the video clip features contain image features corresponding to multiple key frames, then the self-attention mechanism can capture the correlation between these key frames and increase the weight of image features corresponding to some key frames that are more representative of the video clip.

[0117] The video clip features are processed by the self-attention mechanism to obtain mapping features, which can be regarded as optimized video clip features. The user's preference for the video clip calculated based on the user features and the optimized video clip features can better reflect the user's preference for the video clip.

[0118] In addition, in an embodiment of building a target model based on a self-attention mechanism, for each video segment in the at least two video segments, the position information of each key video frame of the video segment in the video segment can be used as a video segment feature corresponding to the video segment. The position information here can refer to the order of the key video frames in each video frame in the video segment. Providing the position information of each key video frame to the self-attention mechanism can also help the self-attention mechanism better capture the correlation between the key video frames.

[0119] In addition, in the embodiment of constructing the target model based on the self-attention mechanism, the image features corresponding to each key video frame can be arranged into an image feature sequence according to the order of each key video frame in the video clip (from front to back, or from back to front), as the video clip features corresponding to the video clip. It is also possible to add the text features corresponding to the video to the image feature sequence corresponding to each key video frame to obtain a feature sequence. Since the feature sequence already contains the order of each key video frame, it is equivalent to containing the position information of each key video frame in the video clip.

[0120] Figure 4 Schematic diagram of a target model provided by the present disclosure. Figure 4As shown, the target model includes an input layer, a self-attention mechanism layer, a pooling layer, and an output layer. The input layer is used to carry the user features input to the target model and the video clip features corresponding to at least two video clips; the self-attention mechanism layer is used to map the video clip features corresponding to at least two video clips to obtain the mapping features corresponding to at least two video clips. The pooling layer is used to reduce the dimension of the data, save the amount of calculation, and improve the calculation efficiency. The output layer is used to calculate the user's preference for at least two video clips based on the mapping features corresponding to the user features and at least two video clips.

[0121] S204: Determine a highlight segment from the at least two video segments according to the sizes of the preferences respectively corresponding to the at least two video segments.

[0122] In some embodiments, a video segment with the highest preference may be determined as a highlight segment.

[0123] In other embodiments, the at least two video clips may be sorted in descending order of preference, and the first few video clips may be determined as highlight clips. When actually displaying to the user, multiple highlight clips may be played in sequence, or one highlight clip may be selected for display.

[0124] In addition, the training method of the target model adopted in the present disclosure is explained.

[0125] Figure 5 It is a flowchart of a model training method provided by the present disclosure, which includes the following steps:

[0126] S500: Obtain a video sample set.

[0127] S502: For each video sample, obtain at least two video segment samples corresponding to different scenes in the video sample.

[0128] S504: Input the user features and the video segment features corresponding to the at least two video segment samples into the target model for model training, and output the predicted preferences of the user for the at least two video segment samples.

[0129] In order to distinguish the application stage and the training stage of the target model, the video used in the training stage is called a video sample. The processing of each video sample is similar to the processing of the video to be recommended to the user in the application stage. You can refer to the previous text for understanding and will not repeat it here.

[0130] In some embodiments, dedicated target models can be trained for different users, so that when training a dedicated target model for a certain user, only the user features of the user can be used as model input. In other words, the target model only learns the user's preference for each video segment in each video sample.

[0131] In other embodiments, a universal target model may be trained for each user, and thus, when training the universal target model, the user characteristics of each user need to be used as model input. In other words, the target model is made to learn the preference of each user for each video segment in each video sample for each user.

[0132] The target model can be trained using supervised learning. The preference can be defined as a probability value between 0 and 1, and the preference is positively correlated with the user's preference for the video clip. Each video clip sample of the same video sample can be annotated in advance, and the video clip sample selected by the user as a highlight clip can be annotated as "preference is 1", indicating that it is completely liked; and the video clip sample not selected by the user as a highlight clip can be annotated as "preference is 0", indicating that it is completely disliked.

[0133] Before starting to train the target model, you need to set the training goal of the target model.

[0134] In an embodiment of training dedicated target models for different users, the training goal can be defined as minimizing, for each video segment sample of the same video sample, the gap between the user's predicted preference for the video segment sample output by the model and the labeled preference for the video segment sample.

[0135] In an embodiment of training dedicated target models for different users, the predicted preference corresponding to the video segment sample selected as the highlight segment by the user among at least two video segment samples of the same video sample is recorded as A, and the predicted preference set corresponding to the video segment sample not selected as the highlight segment by the user among the at least two video segment samples is recorded as B, and B is composed of B i , i=1, ..., N, where N is the number of video clip samples that are not selected as highlight clips by the user. Then, the training goal of the target model can be defined as: i The gap is maximized.

[0136] In the embodiment of training a common target model for all users, assume that the number of users is M, user j represents the jth user, j=1, 2, ..., M. The predicted preference corresponding to the video segment selected as the highlight segment by user j in at least two video segment samples of the same video sample is defined as A jThe predicted preference set corresponding to the video segments that are not selected as highlight segments by user j in the at least two video segment samples is defined as B j , B j By B ji , i=1, ..., N, where N is the number of video clip samples that are not selected as highlight clips by the user. Then, the training objective of the target model can be defined as: j With B ji The gap is maximized.

[0137] Based on the above-mentioned embodiment of training a common target model for all users, the present disclosure provides a more specific embodiment. In this embodiment:

[0138] For multiple video segment samples of the same video sample, the video segment samples may be combined in pairs to obtain a pair of video segment samples. For example, the first video segment sample of the same video sample may be combined with the second video segment sample, the second video segment sample may be combined with the third video segment sample, and the N-1th video segment sample may be combined with the Nth video segment sample.

[0139] Furthermore, for each user, in the process of training the target model, a user can be made to choose one of a pair of video clip samples. One video clip sample selected by the user as the highlight clip is marked as "preference 1", and the other video clip sample is marked as "preference 0". For each user, the user features of the user and a pair of video clip samples are used as a set of model inputs and input into the model for training, and the labels of the pair of video clip samples are used as the model output corresponding to the set of model inputs. In this way, the model training process can learn a user's preference for a pair of video clip samples, and the goal of model training is to maximize the difference in a user's preference for a pair of video clip samples.

[0140] The loss function is given as an example here:

[0141]

[0142] Where loss represents the loss function, u represents a single user, s represents a video clip sample selected as a highlight clip when a single user chooses one of a pair of video clip samples, and w represents a video clip sample not selected as a highlight clip when a single user chooses one of a pair of video clip samples. D represents the data domain containing each user and each pair of video clip samples. u represents the user characteristics of a single user, q s represents the video clip feature corresponding to s, q wrepresents the video clip feature corresponding to w. σ and α are coefficients that can be specified based on experience. The dot product of p and q represents the preference. is a regularization term used to prevent overfitting during training.

[0143] The smaller the value of the above loss function is, the greater the sum of the preference differences between a pair of video clip samples representing the same video sample (each video sample) of the user (each user) is from the perspective of the entire data domain.

[0144] Correspondingly, if the target model is trained using the above method, then when the target model is used to calculate the preference of each video clip in the video to be recommended, it is also necessary to input each video clip of the video to be recommended into the target model in pairs for pairwise comparison. Finally, after pairwise comparison, the winning video clip is determined as the highlight clip.

[0145] Figure 6 : is a schematic diagram of a structure of a device for determining a highlight segment in a video provided by the present disclosure, comprising:

[0146] An acquisition module 601 determines a video to be recommended to a user, and acquires at least two video clips corresponding to different scenes in the video;

[0147] The analysis module 602 inputs the user characteristics of the user and the video segment characteristics corresponding to the at least two video segments into a target model, and outputs the preference of the user for the at least two video segments respectively;

[0148] The determination module 603 determines a highlight segment from the at least two video segments according to the sizes of the preferences respectively corresponding to the at least two video segments.

[0149] The acquisition module 601 extracts the video frames within each unit time length of the video; calculates the similarity between the video frames corresponding to each unit time length of the video, and merges several unit time lengths whose similarity is less than a specified threshold into one time period; and extracts at least two video segments corresponding to different time periods from the video as at least two video segments corresponding to different scenes.

[0150] The acquisition module 601 extracts multiple key video frames from each video segment of the at least two video segments, and determines the image features corresponding to each extracted key video frame; and uses the set of image features corresponding to each key video frame as the video segment features corresponding to the video segment.

[0151] The acquisition module 601 calculates the average pixel value corresponding to multiple video frames in the video clip; for each video frame in the video clip, if the difference between the pixel value corresponding to the video frame and the average pixel value is less than a specified value, the video frame is determined as a key video frame.

[0152] The acquisition module 601 calculates an average value of pixel values ​​of pixel points at designated positions of multiple video frames in the video clip as an average pixel value corresponding to the multiple video frames.

[0153] The image features corresponding to the key video frames are determined based on at least one of the following features:

[0154] Color features; brightness features; contrast features; clarity features; facial features corresponding to the included faces.

[0155] The acquisition module 601 arranges the image features corresponding to each extracted key video frame into an image feature sequence from first to last according to the order of each extracted key video frame in the video segment sample as the video segment features corresponding to the video segment sample.

[0156] The acquisition module 601 uses, for each video segment of the at least two video segments, position information of each key video frame of the video segment in the video segment as a video segment feature corresponding to the video segment.

[0157] The acquisition module 601 determines the text features corresponding to the video according to at least one of the introductory text, title text and subtitle text corresponding to the video; and for each video segment of the at least two video segments, uses the text features corresponding to the video as the video segment features corresponding to the video segment.

[0158] The acquisition module 601 obtains a text to be processed based on at least one of the introductory text, title text, and subtitle text corresponding to the video; performs word segmentation on the text to be processed to obtain a corresponding word sequence; takes a text consisting of at least two adjacent words in the word sequence as a positive text sample to obtain multiple positive text samples; trains a text classification model based on the multiple positive text samples, and determines text features corresponding to the video based on a model parameter set of the trained text classification model.

[0159] The text classification model is constructed using a neural network algorithm, and the text classification model includes at least one hidden layer;

[0160] The acquisition module 601 uses the hidden layer weight matrix corresponding to the trained text classification model as the text feature corresponding to the video.

[0161] The neural network algorithms used to construct the text classification model include: Bert algorithm, or word2vec algorithm.

[0162] The user features are used to characterize the video preferences of the user.

[0163] The processing performed by the target model on the user feature and the video segment features respectively corresponding to the at least two video segments includes:

[0164] For each video segment of the at least two video segments, based on a self-attention mechanism, a video segment feature corresponding to the video segment is mapped to obtain a mapping feature corresponding to the video segment;

[0165] The user's preference for the video segment is calculated based on the mapping features between the user features and the video segment.

[0166] The training method of the target model includes:

[0167] Get a video sample set;

[0168] For each video sample, obtaining at least two video clip samples corresponding to different scenes in the video sample;

[0169] Inputting user features of several users and video segment features corresponding to the at least two video segment samples into a target model for model training, and outputting the preference of each user for the at least two video segment samples;

[0170] The set of video clip samples selected as highlight clips by user i among the at least two video clip samples is defined as A i The set of video clip samples that are not selected as highlight clips by user i among the at least two video clip samples is defined as B i , i = 1, ..., N, N is the number of the plurality of users; the training objective of the target model is to maximize the output of the target model for user i to A i The preference of user i for B i The gap between preferences.

[0171] It should be noted that, although several units / modules or subunits / modules of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0172] Figure 770 is a schematic diagram of a computer-readable storage medium provided by the present disclosure. A computer program is stored on the medium 70. When the program is executed by a processor, the method for determining highlight segments in a video according to any embodiment of the present disclosure is implemented.

[0173] The present disclosure also provides a computing device, including a memory and a processor; the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the method for determining highlight segments in a video in any embodiment of the present disclosure when executing the computer instructions.

[0174] Figure 8 is a schematic diagram of a computing device provided by the present disclosure, such as Figure 8 As shown, the computing device 130 may include, but is not limited to: a processor 131 , a memory 132 , and a bus 133 connecting different system components (including the memory 132 and the processor 131 ).

[0175] The memory 132 stores computer instructions, which can be executed by the processor 131, so that the processor 131 can execute the method for determining highlight segments in a video according to any embodiment of the present disclosure. The memory 132 may include a random access memory unit RAM 1321, a cache memory unit 1322, and / or a read-only memory unit ROM 1323. The memory 132 may also include: a program tool 1325 having a set of program modules 1324, the program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, and one or more combinations of these program modules may include the implementation of a network environment.

[0176] The bus 133 may include, for example, a data bus, an address bus, and a control bus. The computing device 130 may also communicate with an external device 135 through an I / O interface 134. The external device 135 may be, for example, a keyboard, a Bluetooth device, etc. The computing device 130 may also communicate with one or more networks through a network adapter 136. For example, the network may be a local area network, a wide area network, a public network, etc. Figure 8 As shown, the network adapter 136 can also communicate with other modules of the computing device 130 via the bus 133 .

[0177] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0178] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A method for determining a highlight segment in a video, comprising: Determine a video to be recommended to the user, and obtain at least two video clips corresponding to different scenes in the video; Inputting the user features of the user and the video segment features corresponding to the at least two video segments into the target model, and outputting the user's preference for the at least two video segments respectively; the user features include features for characterizing the user's video preferences; the target model is used to calculate the user's preference for the video segment based on the user features and the video segment features in the same feature space; the video segment features include image features corresponding to multiple key video frames in the video segment; Determining a highlight segment from the at least two video segments according to the preference levels respectively corresponding to the at least two video segments; The training method of the target model includes: for each video sample in the video sample set, obtaining at least two video clip samples corresponding to different scenes in the video sample; The user features and the video clip features corresponding to the at least two video clip samples are input into the target model for model training, and the predicted preferences of the user for the at least two video clip samples are output; wherein the predicted preference corresponding to the video clip sample selected as the highlight clip by the user among the at least two video clip samples is A, and the predicted preference set corresponding to the video clip sample not selected as the highlight clip by the user among the at least two video clip samples is B i , i=1, ..., N, N is the number of video clip samples that are not selected as highlight clips by the user; The training goal of the target model is to combine the predicted preference A with the predicted preference set B i The gap is maximized.

2. The method of claim 1, wherein obtaining at least two video clips corresponding to different scenes in the video comprises: For each unit time length of the video, extract the video frames within the unit time length; Calculating the similarity between the video frames corresponding to each unit time length of the video, and merging several unit time lengths whose similarity is less than a specified threshold into one time period; At least two video segments corresponding to different time periods are extracted from the video as at least two video segments corresponding to different scenes.

3. The method of claim 1, further comprising: For each video segment of the at least two video segments, extract a plurality of key video frames from the video segment, and determine an image feature corresponding to each extracted key video frame; The set of image features corresponding to each key video frame is used as the video segment features corresponding to the video segment.

4. The method of claim 3, wherein extracting a plurality of key video frames from the video clip comprises: Calculate the average pixel value corresponding to multiple video frames in the video clip; For each video frame in the video clip, if the difference between the pixel value corresponding to the video frame and the average pixel value is less than a specified value, the video frame is determined as a key video frame.

5. The method of claim 4, wherein calculating the average pixel value corresponding to a plurality of video frames in the video clip comprises: An average value of pixel values ​​of pixel points at designated positions of multiple video frames in the video segment is calculated as an average pixel value corresponding to the multiple video frames.

6. The method of claim 3, wherein the image feature corresponding to the key video frame is determined based on at least one of the following features: Color characteristics; Brightness characteristics; Contrast characteristics; Clarity features; The facial features corresponding to the included faces.

7. The method of claim 3, wherein the set of image features corresponding to each key video frame is used as the video segment features corresponding to the video segment, comprising: According to the order of each key video frame in the video segment, the image features corresponding to each key video frame are arranged into an image feature sequence as the video segment features corresponding to the video segment.

8. The method of claim 3, further comprising: For each video segment of the at least two video segments, position information of each key video frame of the video segment in the video segment is used as a video segment feature corresponding to the video segment.

9. The method of claim 3, further comprising: Determine a text feature corresponding to the video according to at least one of an introductory text, a title text, and a subtitle text corresponding to the video; For each video segment of the at least two video segments, the text feature corresponding to the video is used as the video segment feature corresponding to the video segment.

10. The method of claim 9, wherein determining the text feature corresponding to the video according to at least one of the introductory text, the title text, and the subtitle text corresponding to the video comprises: Obtaining a text to be processed according to at least one of an introductory text, a title text, and a subtitle text corresponding to the video; Segmenting the text to be processed to obtain a corresponding word sequence; Taking a text consisting of at least two adjacent words in the word sequence as a positive text sample, to obtain a plurality of positive text samples; Based on the multiple positive text samples, a text classification model is trained, and according to a model parameter set of the trained text classification model, text features corresponding to the video are determined.

11. The method of claim 10, wherein the text classification model is constructed using a neural network algorithm, and the text classification model includes at least one hidden layer; Determining text features corresponding to the video according to the trained model parameter set of the text classification model includes: The hidden layer weight matrix corresponding to the trained text classification model is used as the text feature corresponding to the video.

12. The method of claim 11, wherein the neural network algorithm used to construct the text classification model comprises: Bert algorithm, or word2vec algorithm.

13. The method of claim 1, wherein the target model processes the user feature and the video segment features corresponding to the at least two video segments respectively, comprising: For each video segment of the at least two video segments, based on a self-attention mechanism, a video segment feature corresponding to the video segment is mapped to obtain a mapping feature corresponding to the video segment; The user's preference for the video segment is calculated based on the mapping features between the user features and the video segment.

14. A device for determining a highlight segment in a video, comprising: An acquisition module determines a video to be recommended to a user, and acquires at least two video clips corresponding to different scenes in the video; The analysis module inputs the user features of the user and the video segment features corresponding to the at least two video segments into the target model, and outputs the user's preference for the at least two video segments respectively; the user features include features for characterizing the user's video preferences; the target model is used to calculate the user's preference for the video segment based on the user features and the video segment features in the same feature space; the video segment features include image features corresponding to multiple key video frames in the video segment; a determination module, determining a highlight segment from the at least two video segments according to the preference levels respectively corresponding to the at least two video segments; The training method of the target model includes: for each video sample in the video sample set, obtaining at least two video clip samples corresponding to different scenes in the video sample; The user features and the video clip features corresponding to the at least two video clip samples are input into the target model for model training, and the predicted preferences of the user for the at least two video clip samples are output; wherein the predicted preference corresponding to the video clip sample selected as the highlight clip by the user among the at least two video clip samples is A, and the predicted preference set corresponding to the video clip sample not selected as the highlight clip by the user among the at least two video clip samples is B i , i=1, ..., N, N is the number of video clip samples that are not selected as highlight clips by the user; The training goal of the target model is to combine the predicted preference A with the predicted preference set B i The gap is maximized.

15. The device as claimed in claim 14, wherein the acquisition module extracts the video frames within each unit time length of the video; calculates the similarity between the video frames corresponding to each unit time length of the video, and merges several unit time lengths whose similarity is less than a specified threshold into one time period; and extracts at least two video segments corresponding to different time periods from the video as at least two video segments corresponding to different scenes.

16. In the device as described in claim 14, the acquisition module extracts multiple key video frames from each video segment of the at least two video segments, and determines the image features corresponding to each extracted key video frame; and uses the set of image features corresponding to each key video frame as the video segment features corresponding to the video segment.

17. In the device as described in claim 16, the acquisition module calculates the average pixel value corresponding to multiple video frames in the video clip; for each video frame in the video clip, if the difference between the pixel value corresponding to the video frame and the average pixel value is less than a specified value, the video frame is determined as a key video frame.

18. The device of claim 17, wherein the acquisition module calculates an average value of pixel values ​​of pixel points at designated positions of a plurality of video frames in the video clip as an average pixel value corresponding to the plurality of video frames.

19. The apparatus of claim 16, wherein the image feature corresponding to the key video frame is determined based on at least one of the following features: Color characteristics; Brightness characteristics; Contrast characteristics; Clarity features; The facial features corresponding to the included faces.

20. The device as claimed in claim 16, wherein the acquisition module arranges the image features corresponding to each extracted key video frame into an image feature sequence from first to last according to the order of each extracted key video frame in the video clip sample as the video clip features corresponding to the video clip sample.

21. The device as claimed in claim 16, wherein the acquisition module uses, for each video segment of the at least two video segments, position information of each key video frame of the video segment in the video segment as the video segment feature corresponding to the video segment.

22. In the device as described in claim 16, the acquisition module determines the text features corresponding to the video based on at least one of the introductory text, title text, and subtitle text corresponding to the video; and for each video segment of the at least two video segments, uses the text features corresponding to the video as the video segment features corresponding to the video segment.

23. In the device as described in claim 22, the acquisition module obtains the text to be processed based on at least one of the introductory text, title text, and subtitle text corresponding to the video; performs word segmentation on the text to be processed to obtain a corresponding word sequence; takes a text composed of at least two adjacent words in the word sequence as a positive text sample to obtain multiple positive text samples; trains a text classification model based on the multiple positive text samples, and determines the text features corresponding to the video based on the model parameter set of the trained text classification model.

24. The device of claim 23, wherein the text classification model is constructed using a neural network algorithm, and the text classification model includes at least one hidden layer; The acquisition module uses the hidden layer weight matrix corresponding to the trained text classification model as the text feature corresponding to the video.

25. The apparatus of claim 24, wherein the neural network algorithm used to construct the text classification model comprises: Bert algorithm, or word2vec algorithm.

26. The apparatus of claim 14, wherein the target model processes the user feature and the video segment features corresponding to the at least two video segments respectively, comprising: For each video segment of the at least two video segments, based on a self-attention mechanism, a video segment feature corresponding to the video segment is mapped to obtain a mapping feature corresponding to the video segment; The user's preference for the video segment is calculated based on the mapping features between the user features and the video segment.

27. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

28. A computing device, comprising a memory and a processor; the memory is used to store computer instructions executable on the processor, and the processor is used to implement any one of the methods of claims 1 to 13 when executing the computer instructions.

Images