A method and apparatus for extracting a video

Abstract

The application provides a video extraction method and device, applied to the technical field of video analysis, after obtaining a target video, determining structured information of the target video, based on transition information of the target video, cutting the target video into multiple initial video segments, each initial video segment corresponding to a shot, based on the structured information of the target video, merging and filtering the multiple initial video segments through multiple deep learning methods, obtaining multiple final video segments, each final video segment corresponding to a scene whose face proportion time length and overall time length satisfy a first preset condition, determining a target video segment from each final video segment through action recognition, and performing extraction, the target video segment being a video segment in each final video segment that satisfies a second preset condition, the target video segment with high information carrying capacity can be efficiently extracted from the target video, and a large amount of human resources is saved.

Description

Technical Field

[0001] This invention belongs to the field of video analysis technology, and particularly relates to a method and apparatus for extracting video data. Background Technology

[0002] In recent years, with the successive maturity of mobile terminals, mobile communications, and platforms, many users have been able to make full use of their spare time to shoot and watch short videos. How to efficiently convert existing resources into short videos with high information carrying capacity is a very important issue.

[0003] In current technology, video extraction mainly relies on manual editing, which often consumes a lot of human resources and is inefficient. Summary of the Invention

[0004] In view of the above problems, this application proposes a video extraction method and apparatus, which can save a lot of manpower and improve efficiency. The specific solution is as follows:

[0005] A method for extracting video, comprising:

[0006] Acquire the target video;

[0007] The structured information of the target video is determined, and the structured information of the target video includes audio type information, facial information and text information at each time point in the target video;

[0008] Based on the transition information of the target video, the target video is divided into multiple initial video segments, each of which corresponds to a shot;

[0009] Based on the structured information of the target video, the multiple initial video segments are merged and filtered to obtain multiple final video segments, and the face proportion duration and overall duration of each final video segment meet the first preset condition.

[0010] The target video segment is determined from each of the final video segments and extracted. The target video segment is the video segment that meets the second preset condition among the final video segments.

[0011] Optionally, based on the structured information of the target video, the multiple initial video segments are merged and filtered to obtain multiple final video segments, including:

[0012] Based on the structured information of each of the initial video segments, the initial video segments are merged to obtain a first merging result, which includes multiple first video segments.

[0013] Based on the structured information of each of the first video segments, the first video segments in the first merging result are merged to obtain a second merging result, which includes multiple second video segments.

[0014] Based on the structured information of each of the second shot segments, the second video segments in the second merging result are merged to obtain a third merging result, which includes multiple third video segments;

[0015] The third video segments in the third merging result are filtered to obtain multiple final video segments.

[0016] Optionally, the step of merging the initial video segments based on their structured information to obtain a first merging result includes:

[0017] Determine the current video segment and the next initial video segment adjacent to the current video segment from the plurality of initial video segments;

[0018] Based on the structured information of the current video segment and the structured information of the next initial video segment, determine whether the current video segment and the next initial video segment are continuous;

[0019] If the current video segment is continuous with the next initial video segment, the current video segment and the next initial video segment are merged to form a new current video segment. If the current video segment is not continuous with the next initial video segment, the next initial video segment is used as the new current video segment. The merging step continues until the last initial video segment is processed, and the first merging result is obtained.

[0020] Optionally, the attributes of the structured information of the initial video segment include a set of IDs; determining whether the current video segment and the subsequent initial video segment are continuous based on the structured information of the current video segment and the structured information of the subsequent initial video segment includes:

[0021] Determine whether the set of IDs of the structured information of the current video segment is consistent with the set of IDs of the structured information of the next initial video segment;

[0022] If they match, then the current video segment and the next initial video segment are determined to be continuous;

[0023] If there is a discrepancy, it is determined that the current video segment and the subsequent initial video segment are not continuous.

[0024] Optionally, the step of merging the first video segments in the first merging result based on the structured information of each of the first video segments to obtain the second merging result includes:

[0025] Determine the current video segment and the next adjacent first video segment from the plurality of first video segments;

[0026] Based on the last structured information of the current video segment and the first structured information of the next first video segment, determine whether the current video segment and the next first video segment are continuous;

[0027] If the current video segment is continuous with the next first video segment, the current video segment and the next first video segment are merged to form a new current video segment. If the current video segment is not continuous with the next first video segment, the next first video segment is used as the new current video segment, and the merging step continues until the last first video segment is processed, resulting in a second merging result.

[0028] Optionally, the attributes of the structured information of the first video segment include a start time; determining whether the current video segment and the next first video segment are continuous based on the last structured information of the current video segment and the first structured information of the next first video segment includes:

[0029] Based on the start time of the last structured information of the current video segment and the start time of the first structured information of the next first video segment, a time interval is determined, and it is determined whether the time interval is less than a first preset threshold.

[0030] If the time interval is less than the first preset threshold, then it is determined whether the type of the last structured information of the current video segment is consistent with the type of the first structured information of the next first video segment;

[0031] If they match, then the current video segment and the next first video segment are determined to be continuous;

[0032] If they are inconsistent or the time interval is not less than the first preset threshold, then determine whether the longest duration percentage of the face information of the current video segment and the longest duration percentage of the face information of the next first video segment are consistent.

[0033] If the longest duration facial information of the current video segment is consistent with the longest duration facial information of the next first video segment, then the current video segment and the next first video segment are determined to be continuous.

[0034] If the longest duration facial information of the current video segment is inconsistent with the longest duration facial information of the next first video segment, then it is determined that the current video segment and the next first video segment are not continuous.

[0035] Optionally, the step of merging the second video segments in the second merging result based on the structured information of each second shot segment to obtain a third merging result includes:

[0036] A target second video segment is determined from the second merging result, wherein the target second video segment is the second video segment in the second merging result that satisfies the third preset condition;

[0037] Determine the current video segment and the next target second video segment adjacent to the current video segment from the target second video segment;

[0038] Based on the structured information of the current video segment and the structured information of the next target second video segment, determine whether the current video segment and the next target second video segment are continuous;

[0039] If the current video segment is continuous with the next initial video segment, then the current video segment and the next target second video segment, as well as the second video segments in between, are merged, and the merged result is taken as the new current video segment. If the current video segment is not continuous with the next initial video segment, then the next target second video segment is taken as the new current video segment, and the merging step continues until the last target second video segment is processed, resulting in the third merging result.

[0040] Optionally, the structured information of the second video segment includes audio type information and facial information at each time point within the second video segment; determining whether the current video segment and the subsequent target second video segment are continuous based on the structured information of the current video segment and the structured information of the next target second video segment includes:

[0041] Based on the audio type information of the current video segment and the audio type information of the next target second video segment, determine the audio type similarity;

[0042] Based on the facial information of the current video segment and the facial information of the second video segment of the next target, the similarity between the people is determined;

[0043] Determine whether the audio type similarity and the person similarity meet a second preset threshold;

[0044] If the audio type similarity and the person similarity meet the second preset threshold, then the current video segment and the next target second video segment are determined to be continuous;

[0045] If the audio type similarity and the person similarity do not meet the second preset threshold, then it is determined that the current video segment and the next target second video segment are not continuous.

[0046] Optionally, the structured information of the final video segment includes audio type information at each time point within the final video segment, wherein the audio type information includes music and / or dialogue; the step of determining and extracting the target video segment from each of the final video segments includes:

[0047] Based on the structured information of the final video segment, action recognition is performed on the final video segment to obtain a target video segment. The target video segment is the final video segment that satisfies the following conditions: the audio type information at each time point is music and the action recognition is the target category; or the final video segment that satisfies the following conditions: the audio type information at each time point is music and dialogue, the overall duration is greater than a fourth preset threshold, the action recognition is the target category, and the category confidence of the action recognition is greater than or equal to a fifth preset threshold.

[0048] A video extraction device, comprising:

[0049] The acquisition unit is used to acquire the target video;

[0050] The determining unit is used to determine the structured information of the target video, wherein the structured information of the target video includes audio type information, facial information and text information at each time point in the target video;

[0051] The segmentation unit is used to segment the target video into multiple initial video segments based on the transition information of the target video, with each initial video segment corresponding to a shot;

[0052] The processing unit is used to merge and filter the multiple initial video segments based on the structured information of the target video to obtain multiple final video segments, wherein the face proportion duration and overall duration of each final video segment meet the first preset condition.

[0053] An extraction unit is used to determine and extract a target video segment from each of the final video segments, wherein the target video segment is a video segment from each of the final video segments that meets a second preset condition.

[0054] Based on the above technical solution, the video extraction method and apparatus provided by the present invention, after acquiring the target video, determines the structured information of the target video, and based on the transition information of the target video, divides the target video into multiple initial video segments, each initial video segment corresponding to a shot. Based on the structured information of the target video, multiple initial video segments are merged and filtered using various deep learning methods to obtain multiple final video segments. Each final video segment corresponds to a scene whose face proportion duration and overall duration meet a first preset condition. The target video segment is determined from each final video segment through action recognition and extracted. The target video segment is the video segment among each final video segment that meets a second preset condition. This method can efficiently extract target video segments with high information carrying capacity from the target video and saves a lot of human resources. Attached Figure Description

[0055] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0056] Figure 1 This is a schematic flowchart of a video extraction method disclosed in an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram illustrating the relationship between a shot, transition, and scene according to an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram illustrating a method for merging and filtering multiple initial video segments based on structured information of a target video to obtain multiple final video segments, as disclosed in an embodiment of the present invention.

[0059] Figure 4 This is a schematic diagram of a method for merging initial video segments based on structured information of each initial video segment to obtain a first merging result, as disclosed in an embodiment of the present invention.

[0060] Figure 5 This is a schematic diagram illustrating the process of a method for determining whether a current video segment and a subsequent initial video segment are continuous based on the structured information of the current video segment and the structured information of the subsequent initial video segment, as disclosed in an embodiment of the present invention.

[0061] Figure 6 This is a schematic diagram of a method disclosed in an embodiment of the present invention for merging each first video segment in a first merging result based on the structured information of each first video segment to obtain a second merging result;

[0062] Figure 7 This is a schematic diagram of a method for determining whether a current video segment and a subsequent first video segment are continuous based on the last structured information of the current video segment and the first structured information of the subsequent first video segment, as disclosed in an embodiment of the present invention.

[0063] Figure 8 This is a schematic diagram of a method disclosed in an embodiment of the present invention for merging each second video segment in a second merging result based on the structured information of each second shot segment to obtain a third merging result;

[0064] Figure 9 This is a schematic diagram illustrating the process of a method for determining whether a current video segment and a subsequent target second video segment are continuous based on the structured information of the current video segment and the structured information of the subsequent target second video segment, as disclosed in an embodiment of the present invention.

[0065] Figure 10 This is a schematic diagram of the structure of a video extraction device disclosed in an embodiment of the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of the invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0068] To save significant human resources and improve efficiency, this invention provides a video extraction method. The video extraction method provided by this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] Please see the appendix Figure 1 , Figure 1This is a flowchart illustrating a video extraction method provided by an embodiment of the present invention. The method may include the following steps:

[0070] Step S101: Obtain the target video.

[0071] In this application, the target video can be a music variety show, including singing competitions, band competitions, music galas or concerts, etc.

[0072] Step S102: Determine the structured information of the target video. The structured information of the target video includes audio type information, facial information, and text information at each time point in the target video.

[0073] In this application, the structured information of the target video includes three main attributes: ID set, type, and start time.

[0074] Step S103: Based on the transition information of the target video, the target video is divided into multiple initial video segments, each of which corresponds to a shot.

[0075] It should be noted that a shot is a continuous shot or sequence between two edits or between edits; a transition is the transition or change between shots.

[0076] In this application, based on each frame of the target video, the changes in the content of the preceding and following frames are compared, and rapid switching between video scenes is detected to obtain the time points of rapid switching, i.e., the transition information of the target video, thereby dividing the target video into multiple initial video segments.

[0077] Step S104: Based on the structured information of the target video, merge and filter multiple initial video segments to obtain multiple final video segments. The face proportion duration and overall duration of each final video segment meet the first preset condition.

[0078] In this application, multiple initial video segments are merged once or multiple times using audio classification technology, face recognition technology, and optical character recognition technology to obtain multiple video segments. Each video segment corresponds to a scene. The multiple video segments are then filtered to obtain multiple video segments that meet the first preset conditions of the corresponding face proportion duration and overall duration, which are the final video segments.

[0079] For easier understanding, please refer to Figure 2 , Figure 2 This is a diagram illustrating the relationship between shots, transitions, and scenes. Shots are segmented through transitions, and scenes can be composed of scenes or shots.

[0080] It should be noted that audio classification technology refers to classifying sounds and predicting their categories; facial recognition technology refers to using deep learning to detect and recognize facial visual features in images for identity verification; and optical character recognition technology refers to the process of converting text images into machine-readable text formats.

[0081] Step S105: Determine the target video segment from each final video segment and extract it. The target video segment is the video segment from each final video segment that meets the second preset condition.

[0082] In this application, action recognition is performed on each final video segment to determine the target video segment and extract it. The specific implementation method will be described in detail through the following embodiments, and will not be described here.

[0083] In summary, the video extraction method and apparatus provided by this invention, after acquiring the target video, determines the structured information of the target video, and based on the transition information of the target video, divides the target video into multiple initial video segments, each initial video segment corresponding to a shot. Based on the structured information of the target video, multiple initial video segments are merged and filtered using various deep learning methods to obtain multiple final video segments. Each final video segment corresponds to a scene whose face proportion duration and overall duration meet a first preset condition. The target video segment is determined from each final video segment through action recognition and extracted. The target video segment is the video segment among each final video segment that meets a second preset condition. This method can efficiently extract target video segments with high information carrying capacity from the target video and saves a significant amount of human resources.

[0084] Based on the embodiments disclosed in the present invention above, in another embodiment of the present invention, the specific implementation method of step S104, which merges and filters multiple initial video segments based on the structured information of the target video to obtain multiple final video segments, is described in detail.

[0085] As one possible implementation method, please refer to the appendix. Figure 3 This is a schematic diagram illustrating a method disclosed in this invention for merging and filtering multiple initial video segments based on structured information of a target video to obtain multiple final video segments. The method may include the following steps:

[0086] Step S201: Based on the structured information of each initial video segment, merge each initial video segment to obtain a first merge result, which includes multiple first video segments.

[0087] In this application, each initial video segment is traversed, and the initial video segments that meet the conditions are merged based on the structured information of each initial video segment to obtain the first merging result. The specific implementation method will be described in detail through the following embodiments, and will not be described here.

[0088] Step S202: Based on the structured information of each first video segment, merge each first video segment in the first merging result to obtain a second merging result, which includes multiple second video segments.

[0089] In this application, the first video segments are traversed, and the first video segments that meet the conditions are merged based on the structured information of each first video segment to obtain the second merging result. The specific implementation method will be described in detail through the following embodiments, and will not be described here.

[0090] Step S203: Based on the structured information of each second shot segment, merge each second video segment in the second merging result to obtain a third merging result, which includes multiple third video segments.

[0091] In this application, the second video segments are traversed, and the second video segments that meet the conditions are merged based on the structured information of each second video segment to obtain a third merging result. The specific implementation method will be described in detail through the following embodiments, and will not be described here.

[0092] Step S204: Filter each third video segment in the third merging result to obtain multiple final video segments.

[0093] Based on the embodiments disclosed in the present invention above, in another embodiment of the present invention, the specific implementation method of step S201, which merges each initial video segment based on the structured information of each initial video segment to obtain a first merging result, is described in detail.

[0094] As one possible implementation method, please refer to the appendix. Figure 4 This is a schematic diagram illustrating a method disclosed in this invention for merging initial video segments based on structured information to obtain a first merged result. The method may include the following steps:

[0095] Step S301: Determine the current video segment and the next initial video segment adjacent to the current video segment from multiple initial video segments.

[0096] In this application, the first initial video segment among multiple initial video segments is taken as the current video segment.

[0097] For ease of understanding, here is an example: There are initial video segments 1, 2, 3, 4, and 5. Initial video segment 1 is the current video segment, and initial video segment 2 is the next initial video segment adjacent to the current video segment.

[0098] Step S302: Based on the structured information of the current video segment and the structured information of the next initial video segment, determine whether the current video segment and the next initial video segment are continuous.

[0099] In this application, the determination of whether the current video segment and the next initial video segment are continuous is based on the audio type information, face information and text information at each time point in the current video segment and the audio type information, face information and text information of the next initial video segment. The specific implementation method will be described in detail through the following embodiments, and will not be described here.

[0100] For ease of understanding, an example is given below: Based on the structured information of initial video segment 1 and initial video segment 2, determine whether initial video segment 1 and initial video segment 2 are continuous.

[0101] Step S303: If the current video segment is continuous with the next initial video segment, then the current video segment and the next initial video segment are merged to form a new current video segment. If the current video segment is not continuous with the next initial video segment, then the next initial video segment is used as the new current video segment. The merging step continues until the last initial video segment is processed, and the first merging result is obtained.

[0102] In this application, after the new current video segment is determined, the merging process continues until the last initial video segment has been processed.

[0103] For ease of understanding, the following example is given: If initial video segment 1 and initial video segment 2 are continuous, then initial video segment 1 and initial video segment 2 are merged and used as the new current video segment. The merging step continues until initial video segment 5 is processed, resulting in the first merged result, which is first video segment 1, first video segment 2, first video segment 3, and first video segment 4. If initial video segment 1 and initial video segment 2 are not continuous, then initial video segment 2 is used as the new current video segment. The merging step continues until initial video segment 5 is processed, resulting in the first merged result, which is first video segment 1, first video segment 2, first video segment 3, first video segment 4, and first video segment 5.

[0104] Based on the embodiments disclosed in the present invention above, in another embodiment of the present invention, the attributes of the structured information of the initial video segment include an ID set; the specific implementation of step S302, which determines whether the current video segment and the next initial video segment are continuous based on the structured information of the current video segment and the structured information of the next initial video segment, is described in detail.

[0105] As one possible implementation method, please refer to the appendix. Figure 5 This is a schematic diagram illustrating a method disclosed in this invention for determining whether a current video segment and a subsequent initial video segment are continuous, based on structured information of the current video segment and structured information of the subsequent initial video segment. The method may include the following steps:

[0106] Step S401: Determine whether the ID set of the structured information of the current video segment is consistent with the ID set of the structured information of the next initial video segment.

[0107] In this application, the structured information of the current video segment includes audio type information, facial information, and text information at each time point within the current video segment, and the structured information of the subsequent initial video segment includes audio type information, facial information, and text information at each time point within the subsequent initial video segment.

[0108] Step S402: If they match, then determine that the current video segment is continuous with the next initial video segment.

[0109] In this application, if at least one of the ID set of audio type information, the ID set of face information, and the ID set of text information is consistent, then the current video segment is determined to be continuous with the next initial video segment.

[0110] Step S403: If there is a discrepancy, it is determined that the current video segment is not continuous with the next initial video segment.

[0111] In this application, if the ID set of audio type information, the ID set of face information, and the ID set of text information are all inconsistent, it is determined that the current video segment is not continuous with the subsequent initial video segment.

[0112] Based on the embodiments disclosed in the present invention above, in another embodiment of the present invention, the specific implementation method of step S202, which merges each first video segment in the first merging result based on the structured information of each first video segment, to obtain the second merging result, is described in detail.

[0113] As one possible implementation method, please refer to the appendix. Figure 6This is a schematic diagram illustrating a method disclosed in this invention for merging first video segments in a first merging result based on structured information of each first video segment to obtain a second merging result. The method may include the following steps:

[0114] Step S501: Determine the current video segment and the next adjacent first video segment from a plurality of first video segments.

[0115] In this application, the first video segment among a plurality of first video segments is taken as the current video segment.

[0116] For ease of understanding, here is an example: There are first video clip 1, first video clip 2, first video clip 3, and first video clip 4. First video clip 1 is the current video clip, and first video clip 2 is the next first video clip adjacent to the current video clip.

[0117] Step S502: Based on the last structured information of the current video segment and the first structured information of the next first video segment, determine whether the current video segment and the next first video segment are continuous.

[0118] In this application, the determination of whether the current video segment and the next first video segment are continuous is based on the audio type information, face information and text information at each time point in the current video segment and the audio type information, face information and text information of the next first video segment. The specific implementation method will be described in detail through the following embodiments, and will not be described here.

[0119] For ease of understanding, the following example is given: Based on the structured information of the first video segment 1 and the structured information of the first video segment 2, it is determined whether the first video segment 1 and the first video segment 2 are continuous.

[0120] Step S503: If the current video segment is continuous with the next first video segment, then merge the current video segment and the next first video segment to obtain the new current video segment. If the current video segment is not continuous with the next first video segment, then take the next first video segment as the new current video segment and continue to perform the merging step until the last first video segment is processed to obtain the second merging result.

[0121] In this application, after the new current video segment is determined, the merging process continues until the last first video segment has been processed.

[0122] For ease of understanding, the following example is given: If the first video segment 1 and the first video segment 2 are continuous, then the first video segment 1 and the first video segment 2 are merged and used as the new current video segment. The merging step continues until the initial video segment 4 is processed, resulting in the second merging result, which is the second video segment 1, the second video segment 2, and the second video segment 3. If the first video segment 1 and the first video segment 2 are not continuous, then the first video segment 2 is used as the new current video segment. The merging step continues until the initial video segment 5 is processed, resulting in the second merging result, which is the second video segment 1, the second video segment 2, the second video segment 3, and the second video segment 4.

[0123] Based on the embodiments disclosed in the present invention above, in another embodiment of the present invention, the attributes of the structured information of the first video segment include the start time; the specific implementation of step S502, which determines whether the current video segment and the next first video segment are continuous based on the last structured information of the current video segment and the first structured information of the next first video segment, is described in detail.

[0124] As one possible implementation method, please refer to the appendix. Figure 7 This is a schematic diagram illustrating a method disclosed in this invention for determining whether a current video segment and a subsequent first video segment are continuous based on the last structured information of the current video segment and the first structured information of the next first video segment. The method may include the following steps:

[0125] Step S601: Based on the start time of the last structured information of the current video segment and the start time of the first structured information of the next first video segment, determine the time interval and determine whether the time interval is less than the first preset threshold.

[0126] In this application, it is determined whether the time interval between the last audio type information of the current video segment and the audio type information of the next first video segment is less than a first preset threshold; it is determined whether the time interval between the last face information of the current video segment and the face information of the next first video segment is less than a first preset threshold; and it is determined whether the time interval between the last text information of the current video segment and the text information of the next first video segment is less than a first preset threshold.

[0127] Step S602: If the time interval is less than the first preset threshold, determine whether the type of the last structured information of the current video segment is consistent with the type of the first structured information of the next first video segment.

[0128] In this application, the type of the last audio type information of the current video segment and the type of the audio type information of the next first video segment are determined, the type of the last face information of the current video segment and the type of the face information of the next first video segment are determined, and the type of the last text information of the current video segment and the type of the text information of the next first video segment are determined. If at least one of them is consistent, then the type of the last structured information of the current video segment is consistent with the type of the first structured information of the next first video segment; otherwise, they are inconsistent.

[0129] Step S603: If they match, then determine that the current video segment is continuous with the next first video segment.

[0130] Step S604: If they are inconsistent or the time interval is not less than the first preset threshold, then determine whether the longest duration percentage of the face information in the current video segment is consistent with the longest duration percentage of the face information in the next first video segment.

[0131] Step S605: If the longest duration facial information of the current video segment is consistent with the longest duration facial information of the next first video segment, then the current video segment and the next first video segment are determined to be continuous.

[0132] Step S606: If the longest duration facial information of the current video segment is inconsistent with the longest duration facial information of the next first video segment, then it is determined that the current video segment and the next first video segment are not continuous.

[0133] Based on the embodiments disclosed in the present invention above, in another embodiment of the present invention, the specific implementation method of step S203, which merges each second video segment in the second merging result based on the structured information of each second shot segment, to obtain the third merging result, is described in detail.

[0134] As one possible implementation method, please refer to the appendix. Figure 8 This is a schematic diagram illustrating a method disclosed in this invention for merging second video segments in a second merging result based on structured information of each second shot segment to obtain a third merging result. The method may include the following steps:

[0135] Step S701: Determine the target second video segment from the second merging result. The target second video segment is the second video segment in the second merging result that meets the third preset condition.

[0136] In this application, the face appearance duration and overall duration of each second video segment are determined, and second video segments with a face appearance duration and / or overall duration greater than a third preset threshold are selected from each second video segment and used as target second video segments.

[0137] For ease of understanding, the following example is given: Given second video clip 1, second video clip 2, second video clip 3, and second video clip 4, the second video clips with a face appearance duration ratio and / or overall duration greater than a third preset threshold are selected. That is, second video clip 1, second video clip 3, and second video clip 4 are selected as target second video clip 1, second video clip 2, and target second video clip 3.

[0138] Step S702: Determine the current video segment and the next target second video segment adjacent to the current video segment from the target second video segment.

[0139] In this application, the first target second video segment among multiple target second video segments is taken as the current video segment.

[0140] For ease of understanding, here is an example: There are target second video clip 1, target second video clip 2, and target second video clip 3. Target second video clip 1 is the current video clip, and target second video clip 2 is the next target second video clip adjacent to the current video clip.

[0141] Step S703: Based on the structured information of the current video segment and the structured information of the next target second video segment, determine whether the current video segment and the next target second video segment are continuous.

[0142] In this application, the determination of whether the current video segment is continuous with the next first video segment is based on the audio type information and facial information at each time point in the current video segment and the audio type information and facial information of the next second video segment. The specific implementation method will be described in detail through the following embodiments, and will not be described here.

[0143] For ease of understanding, the following example is given: Based on the structured information of target second video segment 1 and target second video segment 2, determine whether target second video segment 1 and target second video segment 2 are continuous.

[0144] Step S704: If the current video segment is continuous with the next initial video segment, merge the current video segment and the next target second video segment and the second video segment in between, and use the merged result as the new current video segment. If the current video segment is not continuous with the next initial video segment, use the next target second video segment as the new current video segment, and continue to perform the merging step until the last target second video segment is processed, and obtain the third merging result.

[0145] In this application, after the new current video segment is determined, the merging process continues until the last first video segment has been processed.

[0146] For ease of understanding, the following example is given: If target second video segment 1 and target second video segment 2 are continuous, then target second video segment 1 and target second video segment 2 are merged and used as the new current video segment. The merging process continues until target second video segment 3 is processed, resulting in the third merging result. The first merging result is target second video segment 1 and target second video segment 2. If target second video segment 1 and target second video segment 2 are not continuous, then target second video segment 2 is used as the new current video segment. The merging process continues until target second video segment 3 is processed, resulting in the third merging result. The third merging result is target second video segment 1, target second video segment 2, and target second video segment 3.

[0147] Based on the embodiments disclosed in the present invention above, in another embodiment of the present invention, the structured information of the two video segments includes audio type information and face information at each time point within the second video segment; the specific implementation method of step S703, which determines whether the current video segment and the next target second video segment are continuous based on the structured information of the current video segment and the structured information of the next target second video segment, is described in detail.

[0148] As one possible implementation method, please refer to the appendix. Figure 9 This is a schematic diagram illustrating the process of a method disclosed in this invention for determining whether a current video segment and a subsequent target second video segment are continuous based on structured information of the current video segment and structured information of the next target second video segment. The method may include the following steps:

[0149] Step S801: Determine the audio type similarity based on the audio type information of the current video segment and the audio type information of the next target second video segment.

[0150] Step S802: Determine the similarity between the people based on the facial information of the current video segment and the facial information of the second video segment of the next target.

[0151] Step S803: Determine whether the audio type similarity and the person similarity meet the second preset threshold.

[0152] Step S804: If the audio type similarity and the person similarity meet the second preset threshold, then the current video segment and the next target second video segment are determined to be continuous.

[0153] Step S805: If the audio type similarity and the person similarity do not meet the second preset threshold, then it is determined that the current video segment and the next target second video segment are not continuous.

[0154] Based on the embodiments disclosed in the present invention above, in another embodiment of the present invention, the structured information of the final video segment includes audio type information at each time point within the final video segment, and the audio type information includes music and / or dialogue; the specific implementation method of step S105 for determining and extracting the target video segment from each final video segment is described in detail.

[0155] As one possible implementation, action recognition can be performed on the final video segment based on the structured information of the final video segment to obtain the target video segment. The target video segment is a final video segment that satisfies the audio type information of music at each time point and the action recognition is the target category; or a final video segment that satisfies the audio type information of music and dialogue at each time point, the overall duration is greater than a fourth preset threshold, the action recognition is the target category, and the category confidence of the action recognition is greater than or equal to a fifth preset threshold.

[0156] In this application, an action recognition model can be used to determine the type of action and filter out the final video clips of target categories such as singing and dancing. Considering that the accuracy of action recognition is affected by the duration of the scene, it is more reasonable to configure the confidence level of action classification in combination with the duration of the scene. That is, the longer the time, the lower the confidence level should be, so as to filter out scenes with insufficient confidence. The confidence level of the action recognition category can be compared with the fifth preset threshold of 0.8.

[0157] The methods described in the above-disclosed embodiments of the present invention are detailed. The methods of the present invention can be implemented by various forms of devices. Therefore, the present invention also discloses a video extraction device. Specific embodiments are given below for detailed description.

[0158] Please see the appendix Figure 10 , Figure 10 This is a schematic diagram of a video extraction device disclosed in an embodiment of this application. The device may include:

[0159] Acquisition unit 11 is used to acquire the target video.

[0160] The determining unit 12 is used to determine the structured information of the target video, which includes audio type information, facial information, and text information at each time point in the target video.

[0161] The segmentation unit 13 is used to segment the target video into multiple initial video segments based on the transition information of the target video, with each initial video segment corresponding to a shot.

[0162] Processing unit 14 is used to merge and filter the multiple initial video segments based on the structured information of the target video to obtain multiple final video segments, wherein the face proportion duration and overall duration of each final video segment meet the first preset condition.

[0163] Extraction unit 15 is used to determine and extract target video segments from each of the final video segments, wherein the target video segment is a video segment from each of the final video segments that meets a second preset condition.

[0164] As one possible implementation, the processing unit 14 includes:

[0165] The first merging subunit is used to merge the initial video segments based on the structured information of each initial video segment to obtain a first merging result, which includes multiple first video segments.

[0166] The second merging subunit is used to merge each of the first video segments in the first merging result based on the structured information of each of the first video segments to obtain a second merging result, the second merging result including multiple second video segments.

[0167] The third merging subunit is used to merge each of the second video segments in the second merging result based on the structured information of each of the second shot segments to obtain a third merging result, which includes multiple third video segments.

[0168] The filtering subunit is used to filter each of the third video segments in the third merging result to obtain multiple final video segments.

[0169] As one possible implementation, the first merging subunit includes:

[0170] The first determining subunit is used to determine the current video segment and the next initial video segment adjacent to the current video segment from the plurality of initial video segments.

[0171] The second determining subunit is used to determine whether the current video segment and the next initial video segment are continuous based on the structured information of the current video segment and the structured information of the next initial video segment.

[0172] The first processing subunit is configured to, if the current video segment is continuous with the next initial video segment, merge the current video segment and the next initial video segment to obtain a new current video segment; if the current video segment is not continuous with the next initial video segment, use the next initial video segment as the new current video segment, and continue to execute the merging step until the last initial video segment is processed to obtain a first merging result.

[0173] As one possible implementation, the attributes of the structured information of the initial video segment include an ID set; the second determining subunit is specifically used to: determine whether the ID set of the structured information of the current video segment is consistent with the ID set of the structured information of the next initial video segment; if they are consistent, then determine that the current video segment and the next initial video segment are continuous; if they are inconsistent, then determine that the current video segment and the next initial video segment are not continuous.

[0174] As one possible implementation, the second merging subunit includes:

[0175] The third determining subunit is used to determine the current video segment and the next adjacent first video segment from the plurality of first video segments.

[0176] The fourth determining subunit is used to determine whether the current video segment and the next first video segment are continuous based on the last structured information of the current video segment and the first structured information of the next first video segment.

[0177] The second processing subunit is configured to, if the current video segment is continuous with the next first video segment, merge the current video segment and the next first video segment to obtain a new current video segment; if the current video segment is not continuous with the next first video segment, take the next first video segment as the new current video segment, and continue to execute the merging step until the last first video segment is processed to obtain a second merging result.

[0178] As one possible implementation, the structured information of the first video segment includes a start time. The fourth determining subunit is specifically used to: determine a time interval based on the start time of the last structured information of the current video segment and the start time of the first structured information of the next first video segment, and determine whether the time interval is less than a first preset threshold. If the time interval is less than the first preset threshold, determine whether the type of the last structured information of the current video segment is consistent with the type of the first structured information of the next first video segment. If they are consistent, determine that the current video segment and the next first video segment are continuous. If they are inconsistent or the time interval is not less than the first preset threshold, determine whether the longest duration percentage face information of the current video segment is consistent with the longest duration percentage face information of the next first video segment. If the longest duration percentage face information of the current video segment is consistent with the longest duration percentage face information of the next first video segment, determine that the current video segment and the next first video segment are continuous. If the longest duration percentage face information of the current video segment is inconsistent with the longest duration percentage face information of the next first video segment, determine that the current video segment and the next first video segment are not continuous.

[0179] As one possible implementation, the third merging subunit includes:

[0180] The fifth determining subunit is used to determine the target second video segment from the second merging result, wherein the target second video segment is the second video segment in the second merging result that meets the third preset condition.

[0181] The sixth determining subunit is used to determine the current video segment and the next target second video segment adjacent to the current video segment from the target second video segment.

[0182] The seventh determining subunit is used to determine whether the current video segment and the next target second video segment are continuous based on the structured information of the current video segment and the structured information of the next target second video segment.

[0183] The third processing subunit is configured to, if the current video segment is continuous with the next initial video segment, merge the current video segment and the next target second video segment and the second video segment in between, and use the merged result as the new current video segment; if the current video segment is not continuous with the next initial video segment, use the next target second video segment as the new current video segment, and continue to execute the merging step until the last target second video segment is processed, and obtain the third merging result.

[0184] As one possible implementation, the structured information of the second video segment includes audio type information and facial information at each time point within the second video segment; the seventh determining subunit is specifically used to: determine audio type similarity based on the audio type information of the current video segment and the audio type information of the next target second video segment; determine person similarity based on the facial information of the current video segment and the facial information of the next target second video segment; determine whether the audio type similarity and the person similarity meet a second preset threshold; if the audio type similarity and the person similarity meet the second preset threshold, then determine that the current video segment and the next target second video segment are continuous; if the audio type similarity and the person similarity do not meet the second preset threshold, then determine that the current video segment and the next target second video segment are not continuous.

[0185] As one possible implementation, the structured information of the final video segment includes audio type information at each time point within the final video segment, the audio type information including music and / or dialogue; the extraction unit is specifically used to: perform action recognition on the final video segment based on the structured information of the final video segment to obtain a target video segment, the target video segment being the final video segment that satisfies the requirement that the audio type information at each time point is music and the action recognition is of the target category; or the final video segment that satisfies the requirement that the audio type information at each time point is music and dialogue, the overall duration is greater than a fourth preset threshold, the action recognition is of the target category and the category confidence of the action recognition is greater than or equal to a fifth preset threshold.

[0186] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0187] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0188] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present invention, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, portable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0189] In summary, the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the above embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting video, characterized in that, include: Acquire the target video; The structured information of the target video is determined, and the structured information of the target video includes audio type information, facial information and text information at each time point in the target video; Based on the transition information of the target video, the target video is divided into multiple initial video segments, each of which corresponds to a shot; Based on the ID set of the structured information of each initial video segment, the initial video segments are merged to obtain a first merging result, which includes multiple first video segments. Based on the start time of the structured information of each of the first video segments, the first video segments in the first merging result are merged to obtain a second merging result, which includes multiple second video segments. Based on the audio type information and facial information at each time point in the structured information of each second video segment, the second video segments in the second merging result are merged to obtain a third merging result, which includes multiple third video segments. Filter each of the third video segments in the third merging result to obtain multiple final video segments, and the scene in which the face proportion duration and the overall duration of each final video segment meet the first preset condition. Based on the structured information of the final video segment, action recognition is performed on the final video segment to obtain the target video segment, which is then extracted. The target video segment is the video segment among the final video segments that meets the second preset condition.

2. The method according to claim 1, characterized in that, The process of merging the initial video segments based on their structured information to obtain a first merging result includes: Determine the current video segment and the next initial video segment adjacent to the current video segment from the plurality of initial video segments; Based on the structured information of the current video segment and the structured information of the next initial video segment, determine whether the current video segment and the next initial video segment are continuous; If the current video segment is continuous with the next initial video segment, the current video segment and the next initial video segment are merged to form a new current video segment. If the current video segment is not continuous with the next initial video segment, the next initial video segment is used as the new current video segment. The merging step continues until the last initial video segment is processed, and the first merging result is obtained.

3. The method according to claim 2, characterized in that, The structured information of the initial video segment includes an ID set; determining whether the current video segment and the subsequent initial video segment are continuous based on the structured information of the current video segment and the structured information of the subsequent initial video segment includes: Determine whether the set of IDs of the structured information of the current video segment is consistent with the set of IDs of the structured information of the next initial video segment; If they match, then the current video segment and the next initial video segment are determined to be continuous; If there is a discrepancy, it is determined that the current video segment and the subsequent initial video segment are not continuous.

4. The method according to claim 1, characterized in that, The step of merging the first video segments in the first merging result based on the structured information of each first video segment to obtain the second merging result includes: Determine the current video segment and the next adjacent first video segment from the plurality of first video segments; Based on the last structured information of the current video segment and the first structured information of the next first video segment, determine whether the current video segment and the next first video segment are continuous; If the current video segment is continuous with the next first video segment, the current video segment and the next first video segment are merged to form a new current video segment. If the current video segment is not continuous with the next first video segment, the next first video segment is used as the new current video segment, and the merging step continues until the last first video segment is processed, resulting in a second merging result.

5. The method according to claim 4, characterized in that, The structured information attributes of the first video segment include a start time; determining whether the current video segment and the subsequent first video segment are continuous based on the last structured information of the current video segment and the first structured information of the subsequent first video segment includes: Based on the start time of the last structured information of the current video segment and the start time of the first structured information of the next first video segment, a time interval is determined, and it is determined whether the time interval is less than a first preset threshold. If the time interval is less than the first preset threshold, then it is determined whether the type of the last structured information of the current video segment is consistent with the type of the first structured information of the next first video segment; If they match, then the current video segment and the next first video segment are determined to be continuous; If they are inconsistent or the time interval is not less than the first preset threshold, then determine whether the longest duration percentage of the face information of the current video segment and the longest duration percentage of the face information of the next first video segment are consistent. If the longest duration facial information of the current video segment is consistent with the longest duration facial information of the next first video segment, then the current video segment and the next first video segment are determined to be continuous. If the longest duration facial information of the current video segment is inconsistent with the longest duration facial information of the next first video segment, then it is determined that the current video segment and the next first video segment are not continuous.

6. The method according to claim 1, characterized in that, The third merging result is obtained by merging the second video segments in the second merging result based on the structured information of each second video segment, including: A target second video segment is determined from the second merging result, wherein the target second video segment is the second video segment in the second merging result that satisfies the third preset condition; Determine the current video segment and the next adjacent target second video segment from the target second video segment; Based on the structured information of the current video segment and the structured information of the next target second video segment, determine whether the current video segment and the next target second video segment are continuous; If the current video segment is continuous with the next target second video segment, then the current video segment, the next target second video segment, and the second video segments in between are merged, and the merged result is used as the new current video segment. If the current video segment is not continuous with the next target second video segment, then the next target second video segment is used as the new current video segment, and the merging step continues until the last target second video segment is processed, resulting in the third merging result.

7. The method according to claim 6, characterized in that, The structured information of the second video segment includes audio type information and facial information at each time point within the second video segment; determining whether the current video segment and the subsequent target second video segment are continuous based on the structured information of the current video segment and the structured information of the next target second video segment includes: Based on the audio type information of the current video segment and the audio type information of the next target second video segment, determine the audio type similarity; Based on the facial information of the current video segment and the facial information of the second video segment of the next target, the similarity between the people is determined; Determine whether the audio type similarity and the person similarity meet a second preset threshold; If the audio type similarity and the person similarity meet the second preset threshold, then the current video segment and the next target second video segment are determined to be continuous; If the audio type similarity and the person similarity do not meet the second preset threshold, then it is determined that the current video segment and the next target second video segment are not continuous.

8. The method according to claim 1, characterized in that, The structured information of the final video segment includes audio type information at each time point within the final video segment, and the audio type information includes music and / or dialogue. The step of performing action recognition on the final video segment based on the structured information of the final video segment to obtain the target video segment includes: Based on the structured information of the final video segment, action recognition is performed on the final video segment to obtain a target video segment. The target video segment is the final video segment that satisfies the following conditions: the audio type information at each time point is music and the action recognition is the target category; or the final video segment that satisfies the following conditions: the audio type information at each time point is music and dialogue, the overall duration is greater than a fourth preset threshold, the action recognition is the target category, and the category confidence of the action recognition is greater than or equal to a fifth preset threshold.

9. A video extraction device, characterized in that, include: The acquisition unit is used to acquire the target video; The determining unit is used to determine the structured information of the target video, wherein the structured information of the target video includes audio type information, facial information and text information at each time point in the target video; The segmentation unit is used to segment the target video into multiple initial video segments based on the transition information of the target video, with each initial video segment corresponding to a shot; The processing unit is configured to merge the initial video segments based on the ID set of the structured information of each initial video segment to obtain a first merging result, wherein the first merging result includes multiple first video segments; Based on the start time of the structured information of each of the first video segments, the first video segments in the first merging result are merged to obtain a second merging result, which includes multiple second video segments. Based on the audio type information and facial information at each time point in the structured information of each second video segment, the second video segments in the second merging result are merged to obtain a third merging result, which includes multiple third video segments. Filter each of the third video segments in the third merging result to obtain multiple final video segments, and the scene in which the face proportion duration and the overall duration of each final video segment meet the first preset condition. The extraction unit is used to perform action recognition on the final video segment based on the structured information of the final video segment to obtain a target video segment, and to extract the target video segment, wherein the target video segment is a video segment among the final video segments that meets the second preset condition.

Images