Video segment extraction method, video segment extraction device and storage medium

Abstract

The present disclosure relates to a video segment extraction method, a video segment extraction device and a computer-readable storage medium. The video segment extraction method includes: obtaining a video, and dividing the video into multiple segments. The multiple segments are input into a pre-trained scoring model to obtain the score of each segment, wherein the scoring model is trained based on a data pair consisting of a positive segment and a negative segment, and the data pair consisting of the positive segment and the negative segment is obtained based on a labeled segment annotated with a target attribute, and the target attribute includes an attribute that characterizes the segment as a target segment or a non-target segment. The target segment is extracted from the multiple segments based on the score of each segment. The scoring model in the present disclosure is trained based on the labeled segments that are accurately annotated and have clear target attributes, so that each segment of the acquired video can be accurately scored, and the differences between the segments can be distinguished, which helps to quickly extract the target segment that the user wants to watch, thereby improving the user experience.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video segment extraction method, a video segment extraction device, and a computer-readable storage medium. Background Art

[0002] With the rapid development of network technology, a large number of videos have been generated, greatly enriching people's daily lives. However, due to time and other factors, when users cannot watch the entire video, they hope to quickly obtain the main content of the video by browsing some clips of the video (such as highlights) in a short time.

[0003] In the related art, a model for extracting video clips is trained and used to extract target clips from a video, such as extracting the highlights of a video. However, the data for training a model for extracting video clips currently contains a large amount of noise data, and a complex network structure is required to improve the accuracy of clip selection, resulting in the user being unable to quickly and accurately obtain the required clips when using the trained model to extract video clips, affecting the user experience. In addition, due to the excessive amount of noise data, the training is difficult and the training cost is high, and it is impossible to quickly obtain an effective video clip extraction model in a short time. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a video segment extraction method, a video segment extraction device and a computer-readable storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for extracting video segments is provided, comprising: obtaining a video, and dividing the video into a plurality of segments. The plurality of segments are input into a pre-trained scoring model to obtain scores for each of the segments, wherein the scoring model is trained based on a data pair consisting of a positive segment and a negative segment, and the data pair consisting of the positive segment and the negative segment is obtained based on a labeled segment labeled with a target attribute, wherein the target attribute includes an attribute that characterizes a segment as a target segment or a non-target segment. A target segment is extracted from the plurality of segments based on the scores of the respective segments.

[0006] In one embodiment, the data pair consisting of the positive segment and the negative segment is obtained based on the marked segment marked with the target attribute in the following manner: a sample video including one or more marked segments is obtained. Based on the target attributes marked by the one or more marked segments and the non-marked segments included in the sample video, a data pair consisting of the positive segment and the negative segment is obtained, wherein the possibility that the positive segment becomes the target segment is greater than the possibility that the negative segment becomes the target segment.

[0007] In another embodiment, based on the target attributes annotated by the one or more annotated segments and the non-annotated segments included in the sample video, a data pair consisting of a positive segment and a negative segment is obtained, including: if the target attributes annotated by the one or more annotated segments include attributes that characterize the segment as a target segment, the one or more annotated segments are used as positive segments, and some segments are extracted from the non-annotated segments included in the sample video as negative segments, to obtain one or more data pairs consisting of a positive segment and a negative segment. Or if the target attributes annotated by the one or more annotated segments include attributes that characterize the segment as a non-target segment, the one or more annotated segments are used as negative segments, and some segments are extracted from the non-annotated segments included in the sample video as positive segments, to obtain one or more data pairs consisting of a positive segment and a negative segment. Alternatively, if the target attributes annotated by the one or more annotated segments include attributes characterizing the segment as a target segment and attributes characterizing the segment as a non-target segment, the annotated segment annotated with the attributes characterizing the target segment is taken as a positive segment, the annotated segment annotated with the attributes characterizing the non-target segment is taken as a negative segment, and partial segments are extracted from the non-annotated segments included in the sample video to obtain one or more data pairs consisting of positive segments and negative segments; wherein the data pairs include at least one of the following data pairs or a combination thereof: a data pair consisting of the positive segment and the negative segment, a data pair consisting of the positive segment and the partial segment as negative segments, or a data pair consisting of the negative segment and the partial segment as positive segments.

[0008] In another embodiment, the scoring model is trained based on a data pair consisting of a positive segment and a negative segment in the following manner: extracting a feature vector of the positive segment and a feature vector of the negative segment in the data pair consisting of the positive segment and the negative segment based on a video understanding model. The feature vector of the positive segment and the feature vector of the negative segment are input into a twin neural network to obtain a score of the positive segment and a score of the negative segment, and back propagation is performed using a sorting loss to train a trained twin neural network; wherein the twin neural network includes two multi-layer perceptron models with shared parameters; and the scoring model is a multi-layer perceptron model of the trained twin neural network.

[0009] In another embodiment, the multiple fragments are input into a pre-trained scoring model to obtain scores for the individual fragments, including: inputting the multiple fragments into a video understanding model, extracting feature vectors of each of the multiple fragments based on the video understanding model, and inputting the feature vectors of each fragment into a multi-layer perceptron model to obtain scores for each fragment.

[0010] In another embodiment, dividing the video into multiple segments includes: using a video shot boundary detection method to divide the video into multiple segments to obtain multiple segments of the video; or dividing the video into multiple segments based on a sliding window to obtain multiple segments of the video.

[0011] According to a second aspect of an embodiment of the present disclosure, there is provided a video segment extraction device, comprising: an acquisition unit, for acquiring a video and dividing the video into a plurality of segments. An evaluation unit, for inputting the plurality of segments into a pre-trained scoring model to obtain scores for each of the segments, wherein the scoring model is trained based on a data pair consisting of a positive segment and a negative segment, and the data pair consisting of the positive segment and the negative segment is obtained based on a labeled segment labeled with a target attribute, wherein the target attribute includes an attribute that characterizes a segment as a target segment or a non-target segment. An extraction unit, for extracting a target segment from the plurality of segments based on the scores of the respective segments.

[0012] In one embodiment, the data pair consisting of the positive segment and the negative segment is obtained based on the marked segment marked with the target attribute in the following manner: a sample video including one or more marked segments is obtained. Based on the target attributes marked by the one or more marked segments and the non-marked segments included in the sample video, a data pair consisting of the positive segment and the negative segment is obtained, wherein the possibility that the positive segment becomes the target segment is greater than the possibility that the negative segment becomes the target segment.

[0013] In another embodiment, the data pairs consisting of the positive and negative segments are obtained based on the target attributes marked by the one or more marked segments and the non-marked segments included in the sample video in the following manner: if the target attributes marked by the one or more marked segments include attributes that characterize the segments as target segments, the one or more marked segments are used as positive segments, and some segments are extracted from the non-marked segments included in the sample video as negative segments, to obtain one or more data pairs consisting of the positive and negative segments. Or if the target attributes marked by the one or more marked segments include attributes that characterize the segments as non-target segments, the one or more marked segments are used as negative segments, and some segments are extracted from the non-marked segments included in the sample video as positive segments, to obtain one or more data pairs consisting of the positive and negative segments. Alternatively, if the target attributes annotated by the one or more annotated segments include attributes characterizing the segment as a target segment and attributes characterizing the segment as a non-target segment, the annotated segment annotated with the attributes characterizing the target segment is taken as a positive segment, the annotated segment annotated with the attributes characterizing the non-target segment is taken as a negative segment, and partial segments are extracted from the non-annotated segments included in the sample video to obtain one or more data pairs consisting of positive segments and negative segments; wherein the data pairs include at least one of the following data pairs or a combination thereof: a data pair consisting of the positive segment and the negative segment, a data pair consisting of the positive segment and the partial segment as negative segments, or a data pair consisting of the negative segment and the partial segment as positive segments.

[0014] In another embodiment, the scoring model is trained based on a data pair consisting of a positive segment and a negative segment in the following manner: extracting a feature vector of the positive segment and a feature vector of the negative segment in the data pair consisting of the positive segment and the negative segment based on a video understanding model. The feature vector of the positive segment and the feature vector of the negative segment are input into a twin neural network to obtain a score of the positive segment and a score of the negative segment, and back propagation is performed using a sorting loss to train a trained twin neural network; wherein the twin neural network includes two multi-layer perceptron models with shared parameters; and the scoring model is a multi-layer perceptron model of the trained twin neural network.

[0015] In another embodiment, the evaluation unit inputs the multiple fragments into a pre-trained scoring model to obtain scores for the individual fragments in the following manner: inputs the multiple fragments into a video understanding model, extracts feature vectors of each of the multiple fragments based on the video understanding model, and inputs the feature vectors of each fragment into a multi-layer perceptron model to obtain scores for each fragment.

[0016] In another embodiment, the acquisition unit divides the video into multiple segments in the following manner: using a video shot boundary detection method to divide the video into multiple segments to obtain multiple segments of the video; or dividing the video into multiple segments based on a sliding window to obtain multiple segments of the video.

[0017] According to a third aspect of the embodiments of the present disclosure, a video segment extraction device is provided, comprising: a memory for storing instructions, and a processor for calling the instructions stored in the memory to execute any one of the above-mentioned video segment extraction methods.

[0018] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed by a processor, any one of the above-mentioned video segment extraction methods is executed.

[0019] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: In the present disclosure, the scoring model is trained based on annotated segments with clear target attributes. In the training process, there is less noise data and higher accuracy, which makes the network structure of the scoring model simple and can converge faster, thereby speeding up the training process and helping to reduce training costs. And because the network structure of the scoring model is simple, when using the scoring model to score each segment of the acquired video, the calculation time can be shortened, which helps to quickly extract the target segment that the user wants to watch, thereby improving the user experience.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] Figure 1 The figure is a flowchart of a method for extracting video segments according to an exemplary embodiment.

[0023] Figure 2 A schematic diagram of segment division is shown according to an exemplary embodiment.

[0024] Figure 3 is another schematic diagram of segment division according to an exemplary embodiment.

[0025] Figure 4 The figure is a schematic diagram of a fragment display according to an exemplary embodiment.

[0026] Figure 5is another schematic diagram of fragment display according to an exemplary embodiment.

[0027] Figure 6 The figure is a flow chart of a method for obtaining a data pair according to an exemplary embodiment.

[0028] Figure 7 The figure is a schematic diagram of labeling a sample video according to an exemplary embodiment.

[0029] Figure 8 The diagram is another schematic diagram of sample video annotation according to an exemplary embodiment.

[0030] Fig. 9 The diagram is another schematic diagram of sample video annotation according to an exemplary embodiment.

[0031] Fig.10 The figure is a flowchart of a scoring model training method according to an exemplary embodiment.

[0032] Fig.11 The figure is a block diagram of a video segment extraction device according to an exemplary embodiment.

[0033] Fig.12 It is a block diagram of another video segment extraction device according to an exemplary embodiment. DETAILED DESCRIPTION

[0034] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0035] The video clip extraction method provided by the embodiment of the present disclosure is applied to the scene of extracting target clips, such as the scene of extracting wonderful clips. For example, the scene of generating wonderful clips of user videos in the mobile phone album and displaying them to the user as preview images can also be a scene of obtaining wonderful clips of short videos in a short video application to generate gif images and display them to the user as video covers.

[0036] Among them, the target segment extraction usually adopts the model-based extraction method, so it is necessary to train the model for extracting video segments.

[0037] In the related art, the model training methods for extracting video clips mainly include the following three methods: matching-based method, classification-based method and sorting-based method. The present disclosure embodiment takes the training method for extracting highlights from a video as an example to illustrate the above-mentioned model training methods.

[0038] The matching method is to predefine multiple specified highlight types, match the multiple specified highlight types with the acquired video content, determine the matching degree between the video content and each specified highlight type, and then calculate the highlight level of each segment in the video, so as to take the segment with the highest highlight level as the highlight segment of the video. However, since the highlight types need to be defined in advance and only the segments related to the predefined highlight types can be calculated in each segment, the types of videos that can be used to extract highlight segments are limited.

[0039] The classification-based method is to collect a large number of exciting clips and not exciting clips as training sets in advance, and then train a binary classification model that can distinguish between exciting and not exciting clips. Therefore, when used, it can determine which clips in the video are exciting clips and which clips are not exciting clips. However, this method cannot clearly determine the difference in the degree of excitement between different clips, and then distinguish which clip among the exciting clips is the most exciting.

[0040] In the video clip extraction model training method mentioned above, the training data is difficult to label, and the acquired data is not accurate enough. In addition, the training data quality is poor, and a complex model is required, making training more difficult.

[0041] In view of this, an embodiment of the present disclosure provides a video clip extraction method. The video clip extraction model used in the video clip extraction method can be easily labeled during training to obtain high-quality training data, and a simple multi-layer perceptron model can be used, and the model training is not difficult.

[0042] The video segment extraction method provided in the present disclosure uses the method of inputting the video segments into a scoring model to obtain the scores of each segment, and extracting the target segment based on the scores of each segment. The scoring model used is obtained by training based on data pairs of positive segments and negative segments that are pre-labeled with target attributes. Based on the labeled segments, it helps to reduce the interference of noise data and improve the accuracy of training data during the training process. There is no need to use other network structures to improve the accuracy of training data, thereby making the network structure of the scoring model simple, the training difficulty is low, and it can converge quickly, thereby speeding up the training process and helping to reduce training costs. And because the network structure of the scoring model is simple, when using the scoring model to score each segment of the acquired video, it can shorten the time for calculating the score of each segment, thereby helping users to quickly extract the target segment they want to watch and improve the user experience.

[0043] Figure 1 is a flowchart of a video segment extraction method according to an exemplary embodiment. Figure 1 As shown, the video segment extraction method includes the following steps S11 to S13.

[0044] In step S11, a video is acquired and divided into a plurality of segments.

[0045] In the disclosed embodiment, in a video, there are both target segments to be extracted and non-target segments. The target segments have target segment attributes, and the non-target segments also have non-target segment attributes. For example, when the target segment is a highlight segment, the video has both the relatively most highlight segment and the relatively least highlight segment. Different users have different target segments. In order to facilitate the rapid extraction of the target segment required by the user based on the acquired video, the video is divided into multiple segments, and then the score of each segment can be obtained through the scoring model, which is conducive to rapid extraction.

[0046] Usually, the target segment can be one or more segments in the video and has target segment attributes. For example, a highlight segment is one or more relatively short segments in the video, and the content is more exciting and attractive than the content of other segments. For example, taking a basketball game video as an example, the dunk, smash and other segments in the video are the highlight segments in the basketball game video, while the ordinary dribbling and other segments are non-highlight segments, and the black screen and animation during the camera switching are the least exciting segments.

[0047] The embodiments of the present disclosure may use a variety of methods to divide the segments in a video.

[0048] In one embodiment, a video shot boundary detection method may be used to detect the shot boundary of a video, and then the shot boundary is used as a dividing point to divide the video into multiple segments. This helps to protect the coherence and integrity of the content of each segment. For example Figure 2As shown, the current video is detected to have three shot boundaries by the shot boundary detection method, and the video is divided into four segments a, b, c, and d based on the detected three shot boundaries. In the present disclosure, the video shot boundary detection method may include: an absolute frame difference method, a color histogram method, or a perceptual hashing method. By the absolute frame difference method, the difference between the brightness sum of pixels of adjacent image frames can be compared, and then when the difference between the brightness sum of pixels of adjacent image frames is greater than a specified threshold, it is considered that the shot picture has a sudden change, thereby determining that the current position belongs to the shot boundary. By the color histogram method, the color histogram of the image can be used as an image feature, and the histogram can be used to measure the similarity between adjacent images. Then, when the similarity is lower than a specified threshold, it is considered that the shot picture has a sudden change, thereby determining that the current position belongs to the shot boundary. By the perceptual hashing method, the similarity between two frames of images can be determined, and then when the similarity of the images is lower than a specified threshold, it is considered that the shot picture has a sudden change, thereby determining that the current position belongs to the shot boundary.

[0049] In another embodiment, a sliding window with a fixed duration is used to slide on the video with a specified step length, and the range covered by the sliding window each time it slides is a segment. Figure 3 As shown, each time the sliding window slides, when the end position of the sliding window slides to the start position of the sliding window, a video segment is generated. In one example, if the remaining duration of the last division of the video is less than the duration of the sliding window, the segment is still retained. By dividing the segments by sliding windows, the segment division duration can be customized, so that the target segment can meet the time length required by the user, thereby helping to improve the user experience.

[0050] In step S12, multiple segments are input into a pre-trained scoring model to obtain a score for each segment.

[0051] In the disclosed embodiment, the divided segments are scored by a trained scoring model to obtain the scores of each segment. The scoring model can score according to the wonderfulness of the content of each segment. For different segments of the same video, the higher the score output by the scoring model, the more wonderful the content is. The relative size between the scores can be characterized as the difference in relative wonderfulness between the contents of each segment. By obtaining the scores of each segment, the relative wonderfulness between the contents of each segment can be intuitively distinguished, which helps to quickly extract the target segment required by the user.

[0052] In the present disclosure, the scoring model is trained based on data pairs consisting of positive segments and negative segments. In the process of obtaining the data pairs, the obtained annotated segments are annotated based on the segment content, and then it is determined whether the annotated segment belongs to a positive segment or a negative segment. The target attribute includes an attribute that characterizes the segment as a target segment or a non-target segment. In one example, the target segment may be the most exciting segment in the video; the non-target segment is the least exciting segment in the video. Therefore, based on the obtained data pairs, the difference between the positive segment and the negative segment can be clearly distinguished, so that when training the scoring model, the scoring model can quickly learn the segment features between different attributes. Furthermore, based on the attribute annotation of the annotated segments, the accuracy of the training data can be improved, and the interference of noise data on model training can be reduced, which helps to improve the cleanliness of the training data, make the structure of the training model simpler, and do not need to use an additional network model to calculate the reliability of the annotated segments. Thereby, the training process of the scoring model can be accelerated to converge and help save costs.

[0053] In step S13, a target segment is extracted from the plurality of segments based on the scores of the respective segments.

[0054] In the embodiment of the present disclosure, according to the needs of the user, based on the obtained scores of each segment, the target segment required by the user is extracted.

[0055] In the disclosed embodiment, the target segment is taken as an example for explanation. Extracting the highlight segments based on the scores of multiple segments in the video helps the user to quickly understand the highlight content of the video through the extracted target segment. For example, if the user needs the most highlight segment in the video, the segment with the highest score can be used as the target segment according to the scores of each segment. In one example, if the user needs multiple highlight segments, the segments can be sorted based on the corresponding scores, and multiple segments with relatively high scores can be used as target segments.

[0056] For example, there are 6 segments, namely, segment 1, segment 2, segment 3, segment 4, segment 5, and segment 6. The segments are sorted based on their corresponding scores. Segment 4 has the highest score, followed by segment 3, segment 2, segment 6, segment 5, and segment 1. According to user needs, when 3 selected segments need to be obtained, segment 4 with the highest score and relatively high segments 3 and 2 are extracted to meet the user's needs.

[0057] In the embodiment of the present disclosure, according to the actual application, after the target segment is extracted, the extracted target segment can be displayed to the user. The display method may include: using the extracted segment as the highlight of the video and storing it separately. Or storing the extracted segment as a gif image, or replacing the original cover of the video. In the embodiment of the present disclosure, a collection method can be used to store multiple segments in a unified manner.

[0058] In an implementation scenario, the target segment is a highlight segment as an example for description. Figure 4 As shown, based on the video obtained by the mobile phone, the highest-scoring segment in the video is extracted through the scoring model, and then the highlight segment of the video is generated in the mobile phone album and displayed to the user as a preview of the video. The segment can be a short video stored separately, or a generated gif image. When displayed as a preview of the video, it can include a short video, a gif image, or a short video and a gif image at the same time. When the user swipes up through the mobile phone album, information such as the highlight moments of the video appears. Among them, the highlight moment is the segment with the highest score in the video. This helps users quickly understand the wonderful content of the video. When there are too many wonderful segments, the extracted multiple segments can be stored in the highlight collection of the video, which is convenient for users to view at one time.

[0059] In another implementation scenario, the target segment is still taken as an example for explanation. When the video obtained by the terminal is a short video, the highlight segment of the video can be generated into a GIF image and displayed to the user as the video cover, thereby helping to attract the user to click and watch the video. Figure 5 As shown in the figure, there are 4 video covers displayed on the terminal application software page. For each video cover, the scoring model can extract the highest-scoring segment in the video, generate a gif image from the most exciting segment, and then replace the original video cover, so that users can quickly understand the exciting content of the video through the cover, thereby attracting users to click to watch the video.

[0060] Through the above-mentioned video clip extraction method, the scores of each clip in the video can be quickly obtained through a well-trained and simple scoring model, and then the target clip required by the user can be extracted according to the score, thereby improving the user experience. In addition, the scoring model is trained based on the data of positive and negative clips pre-labeled with target attributes during the training process. There is less noise data in the training data, so that the scoring accuracy of the scoring model obtained after training is higher, which is more helpful to distinguish the degree of excitement between the contents of each clip, and thus helps to extract accurately.

[0061] The present disclosure embodiment will describe the training process of the scoring model below.

[0062] Before training the scoring model, data pairs suitable for scoring model training are obtained in advance, so that when training the scoring model, the scoring model can distinguish the differences in the wonderfulness between different video contents, so as to improve the accuracy of scoring. And the data pairs used for training are obtained based on the annotated segments of the sample video with the target attributes annotated, which helps to avoid the mixing of noise data and improve the quality of training data, thereby reducing the difficulty of training and speeding up the training process.

[0063] Figure 6 is a flow chart of a method for obtaining a data pair according to an exemplary embodiment. Figure 6 As shown, the method for obtaining a data pair includes the following steps S21 to S22.

[0064] In step S21, a sample video including one or more labeled segments is obtained.

[0065] In one embodiment, before training the scoring model, a certain amount of sample videos is acquired in advance to obtain a sample video set so that the scoring model can have sufficient training data for training.

[0066] In step S22, based on the target attributes annotated by one or more annotated segments and the non-annotated segments included in the sample video, a data pair consisting of a positive segment and a negative segment is obtained.

[0067] In the sample video set, each sample video has one or more annotated segments and unannotated segments. The annotated segments in each sample video are annotated based on the content to clarify the target attributes of each annotated segment. Based on the annotated target attributes, each annotated segment and each unannotated segment are obtained into a data pair consisting of a positive segment and a negative segment. Among them, the possibility of a positive segment becoming a target segment is greater than the possibility of a negative segment becoming a target segment. Based on the difference between the positive segment and the negative segment, it is convenient for the scoring model to accurately distinguish the feature differences between the target segment and the non-target segment and the unannotated segment, which helps to improve the accuracy of the scoring model.

[0068] In one example, in order to facilitate the scoring model to better distinguish the differences in the degree of excitement between different clips in the same video, when obtaining the data pair, the positive clip and the negative clip can come from the same sample video, so that the relative score difference between the clips can be obtained, which helps to distinguish the excitement between the clips in the same video and helps to make full use of the sample video. For example: for a dunk highlight video, each dunk clip is a highlight clip. By using the data obtained from the positive clip and the negative clip from the same sample video to train the scoring model, the relative size of the scores of each dunk clip can be obtained, which helps to distinguish the relatively more exciting dunk clips, thereby facilitating the extraction of the target video.

[0069] The embodiments of the present disclosure are described below by taking the target segment as a wonderful video segment as an example.

[0070] Among them, when marking the target attribute in the sample video as the marked segment of the target segment attribute, according to the content of the sample video, the most exciting segment in the sample video is regarded as the segment with the target segment attribute, that is, the segment is more exciting and attractive than the content at other moments in the same video. Then the start and end times of the segment are marked to obtain the marked segment with the target segment attribute. When marking the non-target attribute in the sample video as the marked segment of the target segment attribute, according to the content of the sample video, the least exciting segment in the sample video is regarded as the segment with the non-target segment attribute, that is, the segment is less exciting and less attractive than the content at other moments in the same video. Then the start and end times of the segment are marked to obtain the marked segment with the non-target segment attribute.

[0071] In one example, the sample video may include one or more annotated segments whose target attributes are target segment attributes, and unannotated segments. When obtaining data pairs, one or more annotated segments may be used as positive segments, and some segments may be extracted from the unannotated segments included in the sample video as negative segments. If there is only one annotated segment in the sample video, and the duration of the unannotated segment is similar to that of the annotated segment, the annotated segment may be directly used as the positive segment, and the unannotated segment may be used as the negative segment, so as to obtain the data pairs required for training. For example: Figure 7As shown, segment 2 is a labeled segment whose target attribute is the target segment attribute, and segment 1 and segment 3 are non-labeled segments. Then, when obtaining data pairs, data pairs in which segment 2 is a positive segment and segment 1 is a negative segment and data pairs in which segment 2 is a positive segment and segment 3 is a negative segment can be obtained. If there is only one labeled segment in the sample video, but the duration of the non-labeled segment is too long, the non-labeled segment can be divided into multiple sub-non-labeled segments within a specified duration range, and then multiple data pairs in which the labeled segment is a positive segment and the sub-non-labeled segment is a negative segment can be obtained. This helps to reduce the difficulty of labeling, and a large number of training data pairs can be obtained by labeling a small number of labeled segments. For example: the sample video duration is 60 seconds, of which the labeled segment is 10 seconds and the non-labeled segment is 50 seconds. In order to facilitate the acquisition of a large number of training data pairs, the non-labeled segment can be divided into multiple sub-non-labeled segments with a duration close to that of the labeled segment. If the non-labeled segment is divided into multiple sub-non-labeled segments of no more than 10 seconds, at least 5 sub-non-labeled segments can be obtained, for example: sub-non-labeled segment 1, sub-non-labeled segment 2, sub-non-labeled segment 3, sub-non-labeled segment 4, sub-non-labeled segment 5. Then, 5 pairs of data for fractional model training can be obtained: the labeled segment is a positive segment and the sub-non-labeled segment 1 is a negative segment; the labeled segment is a positive segment and the sub-non-labeled segment 2 is a negative segment; the labeled segment is a positive segment and the sub-non-labeled segment 3 is a negative segment; the labeled segment is a positive segment and the sub-non-labeled segment 4 is a negative segment; the labeled segment is a positive segment and the sub-non-labeled segment 5 is a negative segment.

[0072] In another example, the sample video may include one or more annotated segments whose target attributes are non-target segment attributes, and unannotated segments. When obtaining data pairs, one or more annotated segments may be used as negative segments, and some segments may be extracted from the unannotated segments included in the sample video as positive and negative segments. If there is only one annotated segment in the sample video, and the duration of the unannotated segment is similar to that of the annotated segment, the annotated segment may be directly used as the negative segment and the unannotated segment as the positive segment to obtain the data pairs required for training. For example: Figure 8 As shown, segment 3 is a labeled segment whose target attribute is a non-target segment attribute, and segments 1 and 2 are non-labeled segments. Then, when obtaining data pairs, data pairs in which segment 1 is a positive segment and segment 3 is a negative segment and data pairs in which segment 2 is a positive segment and segment 3 is a negative segment can be obtained. If there is only one labeled segment in the sample video, but the duration of the non-labeled segment is too long, the non-labeled segment can be divided into multiple sub-non-labeled segments within a specified duration range, and multiple data pairs in which the labeled segment is a negative segment and the sub-non-labeled segment is a positive segment can be obtained. This helps to reduce the difficulty of labeling, and a large number of training data pairs can be obtained by labeling a small number of labeled segments.

[0073] In another example, the sample video may include one or more annotated segments whose target attributes are target segment attributes, one or more annotated segments whose target attributes are non-target segment attributes, and non-annotated segments. The data pair may include at least one or a combination of the following data pairs: a data pair consisting of a positive segment and a negative segment, a data pair consisting of a positive segment and a part of the segment as a negative segment, or a data pair consisting of a negative segment and a part of the segment as a positive segment. That is, when obtaining the data pair, if the annotated segment with the annotation representing the target segment attributes is taken as the positive segment, then the annotated segment with the annotation representing the non-target segment attributes is taken as the negative segment, or a part of the segment extracted from the non-annotated segment is taken as the negative segment. If the annotated segment with the annotation representing the non-target segment attributes is taken as the negative segment, then the annotated segment with the annotation representing the target segment attributes is taken as the positive segment, or a part of the segment extracted from the non-annotated segment is taken as the positive segment. For example: Fig. 9 As shown, segment 2 is a labeled segment that represents the target segment attribute, segment 3 is a labeled segment whose target attribute is a non-target segment attribute, and segment 1 is a non-labeled segment. Then, when obtaining data pairs, we can obtain data pairs where segment 2 is a positive segment and segment 1 is a negative segment; data pairs where segment 2 is a positive segment and segment 3 is a negative segment; and data pairs where segment 1 is a positive segment and segment 3 is a negative segment.

[0074] By obtaining labeled training data pairs, the generation of noise data can be effectively reduced and the interference of noise data can be avoided, which in turn helps to improve the cleanliness of the training data and make the scoring model structure simple. There is no need to adopt other network models or add other parameters to improve the reliability of the training data. The training difficulty is low, which helps to speed up the convergence of the scoring model during the training process.

[0075] Fig.10 is a flowchart of a scoring model training method according to an exemplary embodiment. In the present disclosure, the scoring model is obtained by training the data pairs consisting of positive segments and negative segments obtained above. Fig.10 As shown, the scoring model training method includes the following steps S31 to S32.

[0076] In step S31, a feature vector of a positive segment and a feature vector of a negative segment in a data pair consisting of a positive segment and a negative segment are extracted based on a video understanding model.

[0077] In an embodiment of the present disclosure, the feature vectors of the positive segment and the negative segment in the data pair can be extracted by a video understanding model, and then the feature vector of the positive segment and the feature vector of the negative segment are obtained. The feature vector of the positive segment and the feature vector of the negative segment are used as inputs of a scoring model to train the scoring model to obtain the scores of the positive segment and the negative segment. By extracting feature vectors of the positive segment and the negative segment, it is helpful for the scoring model to learn the similarities and differences between the contents of different segments during the training process, so as to improve the accuracy of the scoring. The video understanding model may include: a three-dimensional convolutional neural network (C3D), an inflated convolutional network (Inflated 3D ConvNet, or a time segment network (Temporal Segment Networks, TSN), which is not limited in the present disclosure. When using the video understanding model to extract feature vectors, the output vector of the previous layer of the classification layer of the video understanding model can be used as the feature vector of the input segment.

[0078] In step S32, the positive segment feature vector and the negative segment feature vector are input into the twin neural network to obtain the scores of the positive segments and the negative segments, and the sorting loss is used for back propagation to obtain a trained twin neural network; wherein the twin neural network includes two multi-layer perceptron models with shared parameters; and the scoring model is a multi-layer perceptron model of the trained twin neural network.

[0079] In the disclosed embodiment, the scoring model can be obtained by a multi-layer perceptron model. By copying a multi-layer perceptron model and sharing parameters, a twin neural network is obtained, and a trained scoring model is obtained by training the twin neural network. When training according to a data pair consisting of a positive segment and a negative segment, the acquired positive segment feature vector and negative segment feature vector can be used as input at the same time to obtain the score of the positive segment and the score of the negative segment, and the loss value is calculated according to the scores of the positive segment and the negative segment, so as to train the twin neural network through the back propagation algorithm. The number of hidden layers of the multi-layer perceptron can be set to 2 to 4. For example: if the dimension of the video feature vector is N, the input dimension of the first fully connected layer of the multi-layer perceptron is N, and the output dimension of the last fully connected layer is 1. During training, the feature vector of the positive segment and the feature vector of the negative segment in each group of data pairs are sequentially input into the twin neural network constructed by the multi-layer perceptron model to obtain the score of each segment.

[0080] The ranking loss function is used to back-propagate the scores output by the scoring model to adjust the parameters and weight ratios in the twin neural network, thereby improving the accuracy of the scoring model and accelerating the convergence rate of the scoring model during training. In one example, the feature vector of the positive segment can be recorded as S + , the eigenvector of the negative segment is S -, then the sorting loss can be expressed as follows: L(S + ,S - )=max(0,1-MLP(S + )+MLP(S - )), where MLP(S + ) and MLP(S - ) represent the output scores of the twin neural network for positive and negative segments respectively.

[0081] Through the above-mentioned scoring model training method, the cleanliness of the training data can be improved based on the annotation of the annotated fragments, thereby effectively avoiding the mixing of noise data, thereby improving the quality of the training data. And based on high-quality training data, when training the scoring model, a simple multi-layer perceptron model can be directly used for training, which is conducive to reducing the difficulty of training, facilitating the rapid acquisition of an effective scoring model, and helping to save training costs.

[0082] In one embodiment, when the number of samples of the data pairs used for training is small, the already trained video understanding model can be used to extract the feature vector of the labeled segment, and then the obtained feature vector of the labeled segment is input into the multi-layer perceptron model to train the scoring model. When the number of samples of the data pairs used for training is large enough, the scoring model can include the video understanding model and the multi-layer perceptron model, and then the video understanding model and the multi-layer perceptron model are trained together to improve the matching degree between the two and improve the accuracy of the video understanding model and the multi-layer perceptron model.

[0083] In an implementation scenario, after the scoring model is trained, the obtained scoring model can directly score the acquired video clips. Among them, scoring the video clips through the scoring model can be understood as a branch network of the trained twin neural network obtaining the score corresponding to the clip. For different clips in the same video, the relative size of the scores output by the scoring model can be regarded as the relative difference in the degree of excitement between different clips. That is, in the same video, the higher the score of the clip, the higher its degree of excitement. In order to facilitate the extraction of the wonderful clips in the video, it is first necessary to divide the acquired video into multiple clips, and then extract the feature vectors of each clip through the trained video understanding model, and input the feature vectors of each clip into the trained multi-layer perceptron for scoring to obtain the scores corresponding to each clip. According to the scores corresponding to each clip, the clips are sorted according to the scores, and the highest score is the most exciting clip of the video. The clip with the highest score can be extracted, which helps users to quickly obtain the most exciting content in the video in a short time, thereby improving the user experience. Several clips with higher scores may also be used as several highlights of the video, thereby helping users to quickly obtain the main content of the video in a short time and understand the main content and plot of the video, thereby saving unnecessary time.

[0084] Based on the same concept, the embodiment of the present disclosure also provides a video segment extraction device.

[0085] It is understandable that in order to achieve the above functions, the video clip extraction device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0086] Fig.11 FIG. 1 is a block diagram of a video segment extraction device according to an exemplary embodiment. Fig.11 The video segment extraction device 100 includes an acquisition unit 101, an evaluation unit 102 and an extraction unit 103.

[0087] The acquisition unit 101 is used to acquire a video and divide the video into multiple segments.

[0088] The evaluation unit 102 is used to input multiple fragments into a pre-trained scoring model to obtain a score for each fragment, wherein the scoring model is trained based on a data pair consisting of a positive fragment and a negative fragment, and the data pair consisting of the positive fragment and the negative fragment is obtained based on a labeled fragment labeled with a target attribute, and the target attribute includes an attribute that characterizes the fragment as a target fragment or a non-target fragment.

[0089] The extraction unit 103 is configured to extract a target segment from the multiple segments based on the scores of the individual segments.

[0090] In one embodiment, a data pair consisting of a positive segment and a negative segment is obtained based on annotated segments annotated with target attributes in the following manner: a sample video including one or more annotated segments is obtained. Based on the target attributes annotated by the one or more annotated segments and the non-annotated segments included in the sample video, a data pair consisting of a positive segment and a negative segment is obtained, wherein the possibility that the positive segment becomes the target segment is greater than the possibility that the negative segment becomes the target segment.

[0091] In another embodiment, the data pairs consisting of positive segments and negative segments are obtained based on the target attributes marked by one or more marked segments and the non-marked segments included in the sample video in the following manner: if the target attributes marked by one or more marked segments include attributes that characterize the segments as target segments, then the one or more marked segments are used as positive segments, and some segments are extracted from the non-marked segments included in the sample video as negative segments, to obtain one or more data pairs consisting of positive segments and negative segments. Or if the target attributes marked by one or more marked segments include attributes that characterize the segments as non-target segments, then the one or more marked segments are used as negative segments, and some segments are extracted from the non-marked segments included in the sample video as positive segments, to obtain one or more data pairs consisting of positive segments and negative segments. Alternatively, if the target attributes annotated by one or more annotated segments include attributes characterizing the segment as a target segment and attributes characterizing the segment as a non-target segment, the annotated segment annotated with the attributes characterizing the target segment is taken as a positive segment, the annotated segment annotated with the attributes characterizing the non-target segment is taken as a negative segment, and partial segments are extracted from the non-annotated segments included in the sample video to obtain one or more data pairs consisting of positive segments and negative segments; wherein the data pairs include at least one or a combination of the following data pairs: a data pair consisting of a positive segment and a negative segment, a data pair consisting of a positive segment and a partial segment as a negative segment, or a data pair consisting of a negative segment and a partial segment as a positive segment.

[0092] In another embodiment, the scoring model is trained based on the data pair consisting of positive segments and negative segments in the following manner: extract the feature vector of the positive segment and the feature vector of the negative segment in the data pair consisting of the positive segment and the negative segment based on the video understanding model. Input the positive segment feature vector and the negative segment feature vector into the twin neural network to obtain the score of the positive segment and the score of the negative segment, and use the sorting loss for back propagation to train the trained twin neural network; wherein the twin neural network includes two multi-layer perceptron models with shared parameters; the scoring model is a multi-layer perceptron model of the trained twin neural network

[0093] In another embodiment, the evaluation unit 102 inputs the multiple clips into a pre-trained scoring model to obtain the score of each clip in the following manner: inputs the multiple clips into a video understanding model, extracts the feature vector of each clip in the multiple clips based on the video understanding model, and inputs the feature vector of each clip into a multi-layer perceptron model to obtain the score of each clip.

[0094] In another embodiment, the acquisition unit 101 divides the video into multiple segments in the following manner: using a video shot boundary detection method to divide the video into multiple segments to obtain multiple segments of the video; or dividing the video into multiple segments based on a sliding window to obtain multiple segments of the video.

[0095] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0096] Fig.12 2 is a block diagram of a video segment extraction device 200 according to an exemplary embodiment. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0097] Reference Fig.12 The video segment extraction device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0098] The processing component 202 generally controls the overall operation of the video segment extraction device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 202 may include one or more modules to facilitate the interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate the interaction between the multimedia component 208 and the processing component 202.

[0099] The memory 204 is configured to store various types of data to support the operation of the video segment extraction device 200. Examples of such data include instructions for any application or method operating on the video segment extraction device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0100] The power component 206 provides power to various components of the video segment extraction device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the video segment extraction device 200.

[0101] The multimedia component 208 includes a screen that provides an output interface between the video segment extraction device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the video segment extraction device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0102] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the video segment extraction device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0103] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0104] The sensor assembly 214 includes one or more sensors for providing various aspects of status assessment for the video clip extraction device 200. For example, the sensor assembly 214 can detect the open / closed state of the video clip extraction device 200, the relative positioning of components, such as the display and keypad of the video clip extraction device 200, the sensor assembly 214 can also detect the position change of the video clip extraction device 200 or a component of the video clip extraction device 200, the presence or absence of user contact with the video clip extraction device 200, the orientation or acceleration / deceleration of the video clip extraction device 200, and the temperature change of the video clip extraction device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0105] The communication component 216 is configured to facilitate wired or wireless communication between the video segment extraction device 200 and other devices. The video segment extraction device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0106] In an exemplary embodiment, the video segment extraction device 200 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0107] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the video segment extraction device 200 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0108] It is further understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar thereto. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0109] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0110] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0111] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0112] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0113] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video segment extraction method, characterized in that: The video clip extraction method comprises: Acquire a video, and divide the video into multiple segments; Inputting the plurality of segments into a pre-trained scoring model to obtain a score for each segment, wherein the scoring model is trained based on a data pair consisting of a positive segment and a negative segment, and the data pair consisting of the positive segment and the negative segment is obtained based on annotated segments annotated with target attributes, wherein the target attributes include attributes characterizing a segment as a target segment or a non-target segment; Extracting a target segment from the multiple segments based on the scores of the individual segments; The step of inputting the plurality of segments into a pre-trained scoring model to obtain a score for each segment includes: The multiple segments are input into a video understanding model, feature vectors of each segment of the multiple segments are extracted based on the video understanding model, and the feature vectors of each segment are input into a multi-layer perceptron model to obtain scores of each segment, wherein the scoring model is determined based on a twin network, and the twin neural network includes a multi-layer perceptron model with shared parameters.

2. The video segment extraction method according to claim 1, characterized in that: The data pair consisting of the positive segment and the negative segment is obtained based on the labeled segment labeled with the target attribute in the following manner: Obtaining a sample video including one or more annotated segments; Based on the target attributes annotated by the one or more annotated segments and the unannotated segments included in the sample video, a data pair consisting of a positive segment and a negative segment is obtained, wherein the possibility of the positive segment becoming the target segment is greater than the possibility of the negative segment becoming the target segment.

3. The video segment extraction method according to claim 2, characterized in that: Based on the target attributes annotated by the one or more annotated segments and the non-annotated segments included in the sample video, a data pair consisting of a positive segment and a negative segment is obtained, including: If the target attributes annotated by the one or more annotated segments include an attribute characterizing that the segment is a target segment, the one or more annotated segments are used as positive segments, and some segments are extracted from the non-annotated segments included in the sample video as negative segments, to obtain one or more data pairs consisting of positive segments and negative segments; or If the target attributes annotated by the one or more annotated segments include an attribute characterizing that the segment is a non-target segment, the one or more annotated segments are used as negative segments, and some segments are extracted from the non-annotated segments included in the sample video as positive segments, to obtain one or more data pairs consisting of positive segments and negative segments; or If the target attributes annotated by the one or more annotated segments include attributes characterizing the segment as a target segment and attributes characterizing the segment as a non-target segment, the annotated segment annotated with the attributes characterizing the target segment is taken as a positive segment, the annotated segment annotated with the attributes characterizing the non-target segment is taken as a negative segment, and partial segments are extracted from the non-annotated segments included in the sample video to obtain one or more data pairs consisting of positive segments and negative segments; wherein the data pairs include at least one of the following data pairs or a combination thereof: a data pair consisting of the positive segment and the negative segment, a data pair consisting of the positive segment and the partial segment as negative segments, or a data pair consisting of the negative segment and the partial segment as positive segments.

4. The video segment extraction method according to claim 2 or 3, characterized in that: The scoring model is trained based on data pairs consisting of positive segments and negative segments in the following manner: Extracting feature vectors of positive segments and feature vectors of negative segments from data pairs consisting of positive segments and negative segments based on a video understanding model; The positive segment feature vector and the negative segment feature vector are input into the twin neural network to obtain the score of the positive segment and the score of the negative segment, and the sorting loss is used for back propagation to obtain a trained twin neural network; wherein the twin neural network includes two multi-layer perceptron models with shared parameters; and the scoring model is a multi-layer perceptron model of the trained twin neural network.

5. The video segment extraction method according to claim 1, characterized in that: The step of dividing the video into a plurality of segments comprises: Using a video shot boundary detection method to divide the video into multiple segments to obtain the multiple segments of the video; or The video is divided into a plurality of segments based on a sliding window to obtain a plurality of segments of the video.

6. A video clip extraction device, characterized in that: The video segment extraction device comprises: An acquisition unit, used for acquiring a video and dividing the video into a plurality of segments; an evaluation unit, configured to input the plurality of segments into a pre-trained scoring model to obtain a score of each segment, wherein the scoring model is trained based on a data pair consisting of a positive segment and a negative segment, and the data pair consisting of the positive segment and the negative segment is obtained based on annotated segments annotated with target attributes, wherein the target attributes include attributes characterizing a segment as a target segment or a non-target segment; An extraction unit, configured to extract a target segment from the multiple segments based on the scores of the individual segments; The evaluation unit inputs the multiple segments into a pre-trained scoring model to obtain the score of each segment in the following manner: The multiple segments are input into a video understanding model, feature vectors of each segment of the multiple segments are extracted based on the video understanding model, and the feature vectors of each segment are input into a multi-layer perceptron model to obtain scores of each segment, wherein the scoring model is determined based on a twin network, and the twin neural network includes a multi-layer perceptron model with shared parameters.

7. The video segment extraction device according to claim 6, characterized in that: The data pair consisting of the positive segment and the negative segment is obtained based on the labeled segment labeled with the target attribute in the following manner: Obtaining a sample video including one or more annotated segments; Based on the target attributes annotated by the one or more annotated segments and the unannotated segments included in the sample video, a data pair consisting of a positive segment and a negative segment is obtained, wherein the possibility of the positive segment becoming the target segment is greater than the possibility of the negative segment becoming the target segment.

8. The video segment extraction device according to claim 7, characterized in that: The data pair consisting of the positive segment and the negative segment is obtained based on the target attributes annotated by the one or more annotated segments and the non-annotated segments included in the sample video in the following manner: If the target attributes annotated by the one or more annotated segments include an attribute characterizing that the segment is a target segment, the one or more annotated segments are used as positive segments, and some segments are extracted from the non-annotated segments included in the sample video as negative segments, to obtain one or more data pairs consisting of positive segments and negative segments; or If the target attributes annotated by the one or more annotated segments include an attribute characterizing that the segment is a non-target segment, the one or more annotated segments are used as negative segments, and some segments are extracted from the non-annotated segments included in the sample video as positive segments, to obtain one or more data pairs consisting of positive segments and negative segments; or If the target attributes annotated by the one or more annotated segments include attributes characterizing the segment as a target segment and attributes characterizing the segment as a non-target segment, the annotated segment annotated with the attributes characterizing the target segment is taken as a positive segment, the annotated segment annotated with the attributes characterizing the non-target segment is taken as a negative segment, and partial segments are extracted from the non-annotated segments included in the sample video to obtain one or more data pairs consisting of positive segments and negative segments; wherein the data pairs include at least one of the following data pairs or a combination thereof: a data pair consisting of the positive segment and the negative segment, a data pair consisting of the positive segment and the partial segment as negative segments, or a data pair consisting of the negative segment and the partial segment as positive segments.

9. The video segment extraction device according to claim 7 or 8, characterized in that: The scoring model is trained based on data pairs consisting of positive segments and negative segments in the following manner: Extracting feature vectors of positive segments and feature vectors of negative segments from data pairs consisting of positive segments and negative segments based on a video understanding model; The positive segment feature vector and the negative segment feature vector are input into the twin neural network to obtain the score of the positive segment and the score of the negative segment, and the sorting loss is used for back propagation to obtain a trained twin neural network; wherein the twin neural network includes two multi-layer perceptron models with shared parameters; and the scoring model is a multi-layer perceptron model of the trained twin neural network.

10. The video segment extraction device according to claim 6, characterized in that: The acquisition unit divides the video into multiple segments in the following manner: Using a video shot boundary detection method, the video is divided into a plurality of segments to obtain the plurality of segments of the video; or The video is divided into a plurality of segments based on a sliding window to obtain a plurality of segments of the video.

11. A video segment extraction device, wherein: The video segment extraction device comprises: a memory for storing instructions; and The processor is used to call the instructions stored in the memory to execute the video segment extraction method as described in any one of claims 1-5.

12. A computer-readable storage medium storing instructions, wherein when the instructions are executed by a processor, the video segment extraction method according to any one of claims 1 to 5 is executed.

Images