Video segmentation method, device, apparatus and storage medium

Abstract

Embodiments of the present disclosure relate to a video segmentation method and device, equipment and a storage medium, wherein the method comprises: performing first detection processing of a target event on at least a part of video frames in a video based on first modal data of the part of video frames to obtain a first detection result; performing second detection processing of the target event on the part of video frames based on second modal data of the part of video frames to obtain a second detection result; determining a target frame in which the target event occurs from the part of video frames based on the first detection result and the second detection result; and segmenting the video based on the target frame to obtain a video segment in which the target event occurs. The scheme provided by the embodiments of the present disclosure can improve the accuracy and precision of video segmentation.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a video segmentation method, apparatus, device, and storage medium. Background Technology

[0002] Video segmentation is an important technique in video analysis, especially for videos featuring spoken individuals (such as news videos and live streams). Video segmentation allows for the extraction of spoken segments from the video, facilitating analysis and summarization of the content. However, most existing video segmentation methods rely on single-modal data (such as audio, images, or subtitles), resulting in limited data diversity and lower accuracy. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a video segmentation method, apparatus, device, and storage medium.

[0004] A first aspect of this disclosure provides a video segmentation method, the method comprising: performing a first detection process on the portion of video frames based on first modal data of at least a portion of video frames to obtain a first detection result; performing a second detection process on the portion of video frames based on second modal data of the target event to obtain a second detection result; determining a target frame from the portion of video frames where the target event occurs based on the first detection result and the second detection result; and segmenting the video based on the target frame to obtain a video segment where the target event occurs.

[0005] A second aspect of this disclosure provides a video segmentation method, the method comprising:

[0006] Based on the first modal data of at least a portion of the video frames in the live video, speaker detection processing is performed on the portion of the video frames to obtain a first detection result; based on the second modal data of the portion of the video frames, speaker detection processing is performed on the portion of the video frames to obtain a second detection result; based on the first detection result and the second detection result, a target frame in the portion of the video frames is determined to have someone speaking; based on the target frame, the live video is segmented to obtain video segments in the live video where someone is speaking.

[0007] A third aspect of this disclosure provides a video segmentation method, the method comprising:

[0008] Speaker detection processing is performed on the image data of at least a portion of the video frames to obtain a first detection result; human voice detection processing is performed on the audio data of the portion of the video frames to obtain a second detection result; based on the first and second detection results, target frames in the portion of the video frames are identified as having someone speaking; the video is segmented based on the target frames to obtain video segments in which someone is speaking.

[0009] A fourth aspect of this disclosure provides a video segmentation apparatus, the apparatus comprising:

[0010] The first detection module is used to perform first detection processing on the target event of at least a portion of the video frames based on the first modal data of the video frames, and obtain a first detection result.

[0011] The second detection module is used to perform second detection processing on the target event of the video frame based on the second modal data of the video frame in that part of the video, and obtain the second detection result.

[0012] The determination module is used to determine the target frame from the portion of video frames where the target event occurred, based on the first detection result and the second detection result.

[0013] The first segmentation module is used to segment the video based on the target frame to obtain the video segment in which the target event occurred.

[0014] A fifth aspect of this disclosure provides a video segmentation apparatus, the apparatus comprising:

[0015] The first detection module is used to perform speaker detection processing on at least a portion of the video frames in the live video based on the first modal data of the portion of the video frames, and obtain the first detection result.

[0016] The second detection module is used to perform speaker detection processing on the video frames based on the second modal data of the video frames to obtain the second detection result.

[0017] The determination module is used to determine the target frame from the video frame segment where someone is speaking, based on the first detection result and the second detection result.

[0018] The segmentation module is used to segment the live video based on the target frame to obtain video segments in which someone is speaking.

[0019] A sixth aspect of this disclosure provides a video segmentation apparatus, the apparatus comprising:

[0020] The first detection module is used to perform speaker detection processing on the image data of at least a portion of the video frames in the video to obtain the first detection result.

[0021] The second detection module is used to perform human voice detection processing on the audio data of this part of the video frame to obtain the second detection result.

[0022] The determination module is used to determine the target frame in which someone is speaking from the part of the video frames based on the first detection result and the second detection result.

[0023] The segmentation module is used to segment the video based on the target frame to obtain video segments in which someone is speaking.

[0024] A seventh aspect of this disclosure provides a computer device comprising: a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method of any one of the first to third aspects described above.

[0025] An eighth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of the first to third aspects described above.

[0026] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0027] In this embodiment, a first detection process for a target event is performed on at least a portion of video frames based on first modal data, yielding a first detection result. A second detection process for the target event is then performed on the same portion of video frames based on second modal data, yielding a second detection result. Based on the first and second detection results, a target frame from which the target event occurs is determined from the remaining video frames. The video is then segmented based on the target frame to obtain video segments in which the target event occurs. Because this embodiment can analyze video from multiple perspectives based on multimodal data, it provides more accurate video analysis results compared to single-modal video segmentation methods. Therefore, accurate video segmentation results can be obtained based on the accurate video analysis results, improving the accuracy of video segmentation. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0029] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating a video segmentation method provided in an embodiment of this disclosure;

[0031] Figure 2 This is a schematic diagram of a video frame provided in an embodiment of this disclosure;

[0032] Figure 3 This is a schematic diagram of a face detection result provided in an embodiment of this disclosure;

[0033] Figure 4 This is a schematic diagram illustrating a method for determining a target frame according to an embodiment of this disclosure;

[0034] Figure 5 This is a schematic diagram illustrating another method for determining a target frame provided in an embodiment of this disclosure;

[0035] Figure 6 This is a schematic diagram of a video provided in an embodiment of this disclosure;

[0036] Figure 7 This is a flowchart of another video segmentation method provided in this disclosure embodiment;

[0037] Figure 8 This is a flowchart illustrating yet another video segmentation method provided in this embodiment of the disclosure;

[0038] Figure 9 This is a flowchart illustrating yet another video segmentation method provided in this embodiment of the disclosure;

[0039] Figure 10 This is a flowchart illustrating yet another video segmentation method provided in this embodiment of the disclosure;

[0040] Figure 11 This is a schematic diagram of the structure of a video segmentation device provided in an embodiment of this disclosure. Detailed Implementation

[0041] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0042] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0043] To clearly understand the technical solutions of the embodiments of this disclosure, the technical terms involved in the embodiments of this disclosure are explained below:

[0044] Modality: A source or form of data is called a modality. The video described in this disclosure may include at least two of the following modal data: audio, subtitles, and images.

[0045] Video segmentation refers to dividing a video into smaller, more detailed video segments.

[0046] Speaker detection processing refers to the process of identifying and processing individuals speaking in an image based on model recognition technology. This embodiment of the disclosure uses speaker detection processing technology to detect individuals speaking in a video frame, thereby obtaining the probability that someone is speaking within the video frame.

[0047] Voice detection processing refers to the use of voiceprint detection technology to detect human voices in the audio of a video. This embodiment of the disclosure uses voice detection processing technology to detect audio in video frames and obtain the probability that someone is speaking in the video frame.

[0048] Related technologies typically segment videos based on single-modal data, and the methods generally include the following:

[0049] I. Video Segmentation Methods Based on Audio Data

[0050] The main method uses voiceprint technology to detect human voices in the audio of videos, thereby segmenting video segments containing human voices. However, noise has a significant impact on this method; when the noise in the audio is high, it may lead to inaccurate detection results, which in turn reduces the accuracy of video segmentation.

[0051] II. Video Segmentation Method Based on Subtitles

[0052] This method primarily extracts themes from subtitles in videos and clusters video frames according to these themes to obtain multiple video segments with different themes. However, this method is highly sensitive to the accuracy of theme extraction and clustering; errors in theme extraction or inaccurate clustering will negatively impact the accuracy of video segmentation.

[0053] III. Image-based video segmentation methods

[0054] This method primarily uses image feature detection to determine the differences in image features between consecutive frames in a video, and then segments the video based on these differences. However, this method is affected by the accuracy of feature detection; when the accuracy of feature detection is low, the accuracy of video segmentation also decreases.

[0055] As can be seen from the methods provided by related technologies, regardless of which method is used, some problems will arise when analyzing and segmenting videos using single-modal data. These problems will lead to a decrease in the accuracy of video segmentation. Therefore, there is an urgent need for a new video segmentation method to improve the accuracy of video segmentation.

[0056] To address the aforementioned deficiencies in related technologies, this disclosure provides a video segmentation scheme. This scheme can analyze video from multiple perspectives based on multimodal data, yielding more accurate video analysis results compared to single-modal video segmentation methods. Further segmentation based on this analysis improves the accuracy of video segmentation.

[0057] To better understand the technical solutions of the embodiments of this disclosure, the technical solutions of the embodiments of this disclosure will be described below in conjunction with exemplary embodiments.

[0058] Figure 1 This is a flowchart illustrating a video segmentation method provided in an embodiment of this disclosure, which can be executed by a computer device. Figure 1 As shown, the method includes:

[0059] Step 101: Based on the first modal data of at least a portion of the video frames in the video, perform first detection processing on the target event of the portion of the video frames to obtain the first detection result.

[0060] The video in this embodiment can be understood as, for example, a video containing images of people and their speech, such as a live video or a news video. The images of people may be carried on some or all of the video frames, and the speech may be carried on audio and / or subtitles within the video.

[0061] The first modal data in this embodiment can be understood as image data. This image data can be extracted from video frames of a video.

[0062] In this embodiment, the target event can be understood as the event of someone speaking. Accordingly, the first detection process can be specifically defined as detecting whether someone is speaking in a video frame, and the first detection result can be specifically defined as the probability that someone is speaking in the corresponding video frame. Specifically, in one embodiment, the first detection process can include the following operations: performing face detection processing on at least a portion of the video frames to determine video frames containing faces (hereinafter referred to as first video frames) and video frames not containing faces (hereinafter referred to as second video frames); based on the face detection result, performing speaker detection processing on the first video frames containing faces to obtain the probability that someone is speaking in each first video frame. In one exemplary embodiment, when performing speaker detection processing on each first video frame, a preset face recognition model can be first used to perform face recognition processing on the first video to obtain face recognition boxes on the first video frame, and then a preset speaker recognition model can be used to perform speaker detection processing on the face recognition boxes of the first video frame. Further, for the second video frames not containing faces, this embodiment exemplarily determines the probability that someone is speaking in the second video frame to be 0. For example, Figure 2 This is a schematic diagram of a video frame provided in an embodiment of this disclosure, such as... Figure 2 As shown, video frames 1, 2, 3, and 4 are consecutive video frames from the same video. Assume that a face detection model detects faces in video frames 1 and 2, but not in video frames 3 and 4. Therefore, the probability of someone speaking in video frames 3 and 4 can be set to 0. For video frames 1 and 2, a pre-trained speaker detection model can be used to obtain the probabilities of someone speaking in video frame 1 and 2, respectively. Figure 2 This is merely an example and is not the only limitation of this disclosure.

[0063] Furthermore, this embodiment also provides a method for correcting face detection results. For example, Figure 3 This is a schematic diagram of a face detection result provided in an embodiment of this disclosure, as shown below. Figure 3As shown, video frames 1, 2, 3, and 4 are consecutive video frames in the same video. The face detection results for video frames 1, 3, and 4 indicate that faces are present; the face detection result for video frame 2 indicates that faces are not present. Since the time interval between consecutive video frames is very short, the image in video frame 2 should be at least the same as the image in video frame 1 or 3. However, the actual detection results show that faces are detected in video frames 1 and 3, but not in video frame 2. This indicates that the face detection result for video frame 2 is incorrect, meaning a false detection occurred. To address this, this embodiment corrects the face detection result of video frame 2 to indicate that a face was detected, assigning a preset probability value to video frame 2 as the probability that someone is speaking in video frame 2. Alternatively, the probability of someone speaking in video frames 1 or 3 can be assigned to video frame 2 as the probability that someone is speaking in video frame 2. Correcting the face detection result of intermediate frames using the face detection results of preceding and following frames can reduce the false detection rate and improve the accuracy of video segmentation. Of course, this is only an example... Figure 3 The examples provided are for illustrative purposes only and are not the sole limitation of this disclosure.

[0064] Step 102: Based on the second modal data of this part of the video frames, perform second detection processing on the target event of this part of the video frames to obtain the second detection result.

[0065] In one embodiment of this invention, the second modal data can be exemplarily understood as audio data. Correspondingly, the second detection process can specifically be human voice detection processing. The second detection result can specifically be the probability that someone is speaking in a video frame. Specifically, in one embodiment of this invention, when performing the second detection process, audio data from at least a portion of the video frames can be used as input to a preset voiceprint recognition model. The voiceprint recognition model is then used to perform human voice detection processing on the audio data in each video frame to obtain the probability that someone is speaking in each video frame.

[0066] In another embodiment of this invention, the second modal data may further be specifically subtitles. Correspondingly, the second detection process may be specifically performed on text detection within a preset area of ​​the video frame. The second detection result may be specifically the probability of detecting text on the video frame. In this manner, the probability of detecting text in the video frame can be equated to the probability of someone speaking in the video frame.

[0067] Of course, the above two implementation methods are only two preferred methods of this embodiment, and not all implementation methods of this disclosure.

[0068] Step 103: Based on the first detection result and the second detection result, determine the target frame from this part of the video frames where the target event occurred.

[0069] For example, the target event corresponding to this embodiment is specifically the event of someone speaking, and the target frame in this embodiment can be specifically a video frame in which the detection result is that someone is speaking.

[0070] This embodiment can use at least one of the following methods to determine the target frame:

[0071] Example, Figure 4 This is a schematic diagram illustrating a method for determining a target frame according to an embodiment of this disclosure, as shown below. Figure 4 As shown, in one determination method, when at least one of the first detection result and the second detection result corresponding to the video frame has a probability greater than a preset threshold, the video frame is determined to be a target frame; otherwise, the video frame is determined to be a non-target frame.

[0072] Example, Figure 5 This is a schematic diagram illustrating another method for determining the target frame provided in this embodiment of the disclosure, as shown below. Figure 5 As shown, in another determination method, the first detection result and the second detection result of the video frame can be weighted and summed, and the video frame whose weighted summation result is greater than a preset threshold can be determined as the target frame.

[0073] Of course, the above two determination methods are only two preferred methods provided in this embodiment, and are not the only limitation of this disclosure.

[0074] This embodiment obtains the target frame in the video by comprehensively analyzing the first and second detection results, which can ensure the accuracy of the obtained target frame and improve the accuracy of video segmentation.

[0075] Step 104: Segment the video based on the target frame to obtain the video segment in which the target event occurred.

[0076] Specifically, when the number of non-target frames between two target frames in a video is less than a preset number, these two target frames can be segmented into the same video segment, thus avoiding false detections from affecting video segmentation. When the number of non-target frames between two target frames in a video is greater than a preset number, these two target frames can be segmented into two different video segments. For example, Figure 6 This is a schematic diagram of a video provided in an embodiment of this disclosure, such as... Figure 6 As shown, Figure 6 It includes (m+n) consecutive video frames, where m and n are positive integers. Target frames 1 to 50 and 51 to m are consecutive target frames, while non-target frames 1 to n are consecutive video frames between target frames 50 and 51. If n is greater than a preset number, then in the... Figure 6When segmenting the video, based on the position of the target frames, target frames 1 to 50 can be divided into one video segment, and target frames 51 to m can be divided into another video segment, thus obtaining two video segments in which the target events occur. Of course, this is only an example and is not the only limitation of this disclosure.

[0077] In this embodiment, a first detection process for a target event is performed on at least a portion of the video frames based on first modal data, yielding a first detection result. A second detection process for the target event is then performed on the same portion of the video frames based on second modal data, yielding a second detection result. Based on the first and second detection results, a target frame from this portion of the video frames is determined where the target event occurred. The video is then segmented based on the target frame to obtain video segments where the target event occurred. Because this embodiment can analyze the video from multiple perspectives based on multimodal data, it yields more accurate video analysis results compared to single-modal video segmentation methods. Therefore, based on accurate video analysis results, accurate video segmentation results can be obtained, improving the accuracy of video segmentation.

[0078] Figure 7 This is a flowchart of another video segmentation method provided in this disclosure embodiment, such as... Figure 7 As shown, the method includes:

[0079] Step 701: Based on the first modal data of at least a portion of the video frames in the live video, perform speaker detection processing on the portion of the video frames to obtain the first detection result.

[0080] Step 702: Based on the second modal data of this part of the video frames, perform speaker detection processing on this part of the video frames to obtain the second detection result.

[0081] Step 703: Based on the first detection result and the second detection result, determine the target frame from this part of the video frames where someone is speaking.

[0082] Step 704: Segment the live video based on the target frame to obtain video segments in which someone is speaking.

[0083] The live video referred to in this embodiment includes images of the anchor and / or other individuals, as well as audio recordings and / or subtitles of the anchor and / or other individuals' speech. The first modal data can specifically be one of image, audio, or subtitles. When the first modal data is an image, the speaker detection processing in this embodiment can be understood as performing image recognition processing on the video frame using a preset speaker recognition model to obtain the probability that someone is speaking in the video frame. When the first modal data is audio, the speaker detection processing in this embodiment can be understood as performing voice detection processing on the audio data in the video frame using a preset voiceprint recognition model to obtain the probability that someone is speaking in the video frame. When the first modal data is video subtitles, the speaker detection processing in this embodiment can be understood as performing text detection processing on a preset area of ​​the video frame to obtain the probability that there are subtitles in the video frame. In this case, the probability of having subtitles in the video can be equated to the probability that someone is speaking in the video frame.

[0084] The second modal data and the first modal data referred to in this embodiment are two different modal data. For example, when the first modal data is image data, the second modal data can be audio data or subtitles; when the first modal data is audio data, the second modal data can be image data or subtitles; and when the first modal data is subtitles, the second modal data can be image data or audio data.

[0085] When the second modal data is image data, a preset speaker recognition model is used to obtain the probability that someone is speaking in the video frame. When the second modal data is audio data, a preset voiceprint recognition model is used to perform human voice detection processing on the audio data of the video frame to obtain the probability that someone is speaking in the video frame. When the second modal data is subtitles, text detection processing is performed on a preset area of ​​the video frame to obtain the probability that there are subtitles in the video frame. In this case, the probability that there are subtitles in the video frame is equivalent to the probability that someone is speaking in the video frame.

[0086] Figure 7 The specific implementation method and beneficial effects of the embodiments are as follows: Figure 1 The implementation examples are similar and will not be repeated here.

[0087] Figure 8 This is a flowchart illustrating another video segmentation method provided in this disclosure, as shown in the embodiment. Figure 8 As shown, the method includes:

[0088] Step 801: Perform speaker detection processing on the image data of at least a portion of the video frames in the video to obtain the first detection result.

[0089] Step 802: Perform human voice detection processing on the audio data of this part of the video frame to obtain the second detection result.

[0090] Step 803: Based on the first detection result and the second detection result, determine the target frame from this part of the video frames where someone is speaking.

[0091] Step 804: Segment the video based on the target frame to obtain video segments in which someone is speaking.

[0092] Figure 8 The specific implementation method and beneficial effects of the embodiments are as follows: Figure 1 The implementation examples are similar and will not be repeated here.

[0093] Figure 9 This is a flowchart illustrating another video segmentation method provided in this disclosure, as shown in the embodiment. Figure 9 As shown, after obtaining video segments, this embodiment can further segment the video segments using the following method:

[0094] Step 901: Convert the audio data in the video clip into text data, and perform topic extraction processing on the text data.

[0095] Step 902: Segment the video clips according to the themes in the text data.

[0096] Specifically, by Figure 1 As can be seen from existing video segmentation methods, each video frame in the resulting video segment contains the target event, meaning each frame includes human audio. However, this audio may not actually be produced by the same person. To address this, this embodiment further provides a video segmentation method. This method converts the audio data of each video frame in the video segment into text data, performs word segmentation on the text data to obtain keywords, and determines the theme of each video frame using these keywords. Then, by performing theme clustering on consecutive video frames in the video segment, consecutive video frames belonging to the same theme are obtained, thereby segmenting these video frames into a single video segment, thus achieving theme-based video segmentation.

[0097] In this embodiment, after obtaining the initially segmented video segments, the video segments are further segmented based on the topic extraction method and the topic clustering method, which can improve the accuracy of video segmentation.

[0098] Figure 10 This is a flowchart illustrating another video segmentation method provided in this disclosure, as shown in the embodiment. Figure 10 As shown, after obtaining video segments, this embodiment can further segment the video segments using the following method:

[0099] Step 1001: Cluster the speakers in the video clips.

[0100] Step 1002: Segment the video segments based on the clustering results.

[0101] Specifically, based on Figure 1 After obtaining the video segment using the segmentation method in this embodiment, face recognition and speaker recognition processing are performed on each video frame in the video segment to obtain the speaker's face recognition bounding box on each video frame. Then, face clustering processing is performed on consecutive video frames in the video segment to identify consecutive video frames with the same speaker. By segmenting these video frames into the same video segment, speaker-based video segmentation is achieved.

[0102] In this embodiment, after obtaining the initially segmented video segments, the video segments are further segmented based on face recognition and face clustering methods, which can improve the accuracy of video segmentation.

[0103] It is worth noting that the above Figure 9 Examples and Figure 10 The methods provided in the embodiments can also be used in combination; that is, after obtaining the video segments, they can be used separately. Figure 9 and Figure 10 The method in this embodiment further segments the video clips. Its execution method is similar to... Figure 9 and Figure 10 The implementation examples are similar and will not be repeated here.

[0104] Figure 11 This is a schematic diagram of a video segmentation device provided in an embodiment of this disclosure. This device can be understood as the computer device in the above embodiments or a functional module within the computer device described above. Specifically, as... Figure 11 As shown, the video segmentation device 110 provided in this embodiment includes:

[0105] The first detection module 111 is used to perform first detection processing on the target event of at least a portion of the video frames based on the first modal data of the video frames, and obtain a first detection result.

[0106] The second detection module 112 is used to perform second detection processing on the target event of the video frame based on the second modal data of the video frame to obtain a second detection result.

[0107] The determination module 113 is used to determine the target frame in the part of the video frames where the target event occurs, based on the first detection result and the second detection result.

[0108] The first segmentation module 114 is used to segment the video based on the target frame to obtain the video segment in which the target event occurs.

[0109] In one implementation, the target event in this embodiment includes a speaking event; the first detection result and the second detection result are used to indicate the probability that someone speaks in the video frame.

[0110] In one implementation, the first modal data is image data.

[0111] In one embodiment, the first detection module 111 is specifically used for:

[0112] Face detection processing is performed on at least a portion of the video frames to obtain a first video frame containing a face and a second video frame not containing a face; speaker detection processing is performed on the first video frame to obtain the probability that someone speaks in each first video frame; for the second video frame, the probability that someone speaks in the second video frame is determined to be 0.

[0113] In one embodiment, the first detection module 111 is further configured to:

[0114] Face recognition processing is performed on the first video frame to obtain a face recognition bounding box on the first video frame; speaker detection processing is then performed within the face recognition bounding box.

[0115] In one implementation, the second modal data is audio data.

[0116] In one implementation, the second detection module 112 is specifically used to: perform human voice detection processing on the audio data of the video frame to obtain the probability that someone speaks in each video frame.

[0117] In one embodiment, the determining module 113 is specifically used to: for each video frame in the part of video frames, perform weighted summation on the first detection result and the second detection result corresponding to the video frame to obtain a weighted summation result; and determine the video frame in the part of video frames whose weighted summation result is greater than a preset threshold as the target frame.

[0118] In one implementation, the first segmentation module 114 is specifically used for:

[0119] If the number of non-target frames between two target frames is less than a preset number, the two target frames are split into the same video segment; if the number of non-target frames between two target frames is greater than the preset number, the two target frames are split into two different video segments.

[0120] In one embodiment, the device 110 may further include:

[0121] The topic extraction module is used to convert audio data in video clips into text data and perform topic extraction processing on the text data.

[0122] The second segmentation module is used to segment video clips based on the themes in the text data.

[0123] In one embodiment, the device 110 may further include:

[0124] The clustering module is used to cluster speakers in video clips.

[0125] The third segmentation module is used to segment video segments based on the clustering results.

[0126] The device provided in this embodiment is capable of performing... Figures 1-8 The methods in any of the embodiments are similar in execution and beneficial effects, and will not be described again here.

[0127] This disclosure also provides a video segmentation device, which includes:

[0128] The first detection module is used to perform speaker detection processing on at least a portion of the video frames in the live video based on the first modal data of the portion of the video frames, and obtain the first detection result.

[0129] The second detection module is used to perform speaker detection processing on the video frames based on the second modal data of the video frames to obtain the second detection result.

[0130] The determination module is used to determine the target frames in this part of the video frames in which someone is speaking, based on the first detection result and the second detection result.

[0131] The segmentation module is used to segment the live video based on the target frame to obtain video segments in which someone is speaking.

[0132] The device provided in this embodiment is capable of performing... Figure 9 The method described in the embodiments is similar in its execution and beneficial effects, and will not be repeated here.

[0133] This disclosure also provides a video segmentation device, which includes:

[0134] The first detection module is used to perform speaker detection processing on the image data of at least a portion of the video frames in the video to obtain the first detection result.

[0135] The second detection module is used to perform human voice detection processing on the audio data of this part of the video frame to obtain the second detection result.

[0136] The determination module is used to determine the target frames in this part of the video frames in which someone is speaking, based on the first detection result and the second detection result.

[0137] The segmentation module is used to segment the video based on the target frame to obtain video segments in which someone is speaking.

[0138] The device provided in this embodiment is capable of performing... Figure 10 The method described in the embodiments is similar in its execution and beneficial effects, and will not be repeated here.

[0139] This disclosure also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it can achieve... Figures 1-10 The methods in any of the embodiments are similar in execution and beneficial effects, and will not be described again here.

[0140] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can achieve... Figures 1-10 The methods in any of the embodiments are similar in execution and beneficial effects, and will not be described again here.

[0141] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0142] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video segmentation method, characterized in that, include: Based on the first modal data of at least a portion of the video frames, a first detection process for target events is performed on the at least a portion of the video frames to obtain a first detection result; The target event is detected by performing a second detection process on the at least a portion of the video frames based on the second modal data of the at least a portion of the video frames, and a second detection result is obtained. Based on the first detection result and the second detection result, the target frame in which the target event occurs is determined from the at least a subset of video frames; The video is segmented based on the target frame to obtain video segments in which the target event occurs. The first detection result and the second detection result are used to indicate the probability of the target event occurring in the video frame; The step of determining the target frame from the at least a subset of video frames where the target event occurs, based on the first detection result and the second detection result, includes: The target frame is obtained based on the probability analysis of the first detection result and the second detection result.

2. The method according to claim 1, characterized in that, The target events include speech events.

3. The method according to claim 2, characterized in that, The first modal data is image data.

4. The method according to claim 3, characterized in that, The first detection processing of target events on at least a portion of video frames based on first modal data of at least a portion of video frames to obtain a first detection result includes: Face detection processing is performed on the images of at least a portion of the video frames to obtain a first video frame including a face and a second video frame not including a face; Perform speaker detection processing on the first video frame to obtain the probability that someone speaks in the first video frame; The probability that someone is speaking in the second video frame is determined to be 0.

5. The method according to claim 4, characterized in that, The speaker detection processing of the first video frame includes: Perform face recognition processing on the first video frame to obtain a face recognition bounding box on the first video frame; Speaker detection is performed within the face recognition frame.

6. The method according to claim 2, characterized in that, The second modal data is audio data.

7. The method according to claim 6, characterized in that, The second detection processing of the target event on the at least a portion of the video frames based on the second modal data of the at least a portion of the video frames to obtain a second detection result includes: Human voice detection processing is performed on the audio data of at least a portion of the video frames to obtain the probability that someone speaks in each video frame.

8. The method according to claim 2, characterized in that, The step of determining the target frame from the at least a subset of video frames where the target event occurs, based on the first detection result and the second detection result, includes: For each video frame in the at least part of the video frames, the first detection result and the second detection result corresponding to the video frame are weighted and summed to obtain the corresponding weighted summation result; The video frames in the at least part of the video frames whose weighted summation result is greater than a preset threshold are determined as target frames.

9. The method according to any one of claims 1-8, characterized in that, The step of segmenting the video based on the target frame to obtain video segments in which the target event occurs includes: If the number of non-target frames between two target frames is less than a preset number, the two target frames are split into the same video segment. If the number of non-target frames between two target frames is greater than the preset number, the two target frames are split into two different video segments.

10. The method according to any one of claims 1-8, characterized in that, After obtaining the video segment, the method further includes: The audio data in the video clip is converted into text data, and the text data is then processed by topic extraction. The video segment is segmented based on the theme in the text data.

11. The method according to any one of claims 1-8, characterized in that, After obtaining the video segment, the method further includes: Cluster the speakers in the video clips; The video segments are segmented based on the clustering results.

12. A video segmentation method, characterized in that, include: Speaker detection processing is performed on at least a portion of the video frames based on first modal data from at least a portion of the video frames in the live video to obtain a first detection result; Speaker detection processing is performed on the at least a portion of the video frames based on the second modal data of the at least a portion of the video frames to obtain a second detection result; Based on the first detection result and the second detection result, target frames in which someone is speaking are determined from at least a portion of the video frames; The live video is segmented based on the target frame to obtain video segments in which someone speaks. The first detection result and the second detection result are used to indicate the probability that someone is speaking in the video frame; The step of determining the target frame from the at least a subset of video frames where someone is speaking, based on the first detection result and the second detection result, includes: The target frame is obtained based on the probability analysis of the first detection result and the second detection result.

13. A video segmentation method, characterized in that, include: Speaker detection processing is performed on the image data of at least a portion of the video frames to obtain a first detection result; Human voice detection processing is performed on the audio data of at least a portion of the video frames to obtain a second detection result; Based on the first detection result and the second detection result, target frames in which someone is speaking are determined from at least a portion of the video frames; The video is segmented based on the target frame to obtain video segments in which someone is speaking; The first detection result and the second detection result are used to indicate the probability that someone is speaking in the video frame; The step of determining the target frame from the at least a subset of video frames where someone is speaking, based on the first detection result and the second detection result, includes: The target frame is obtained based on the probability analysis of the first detection result and the second detection result.

14. A video segmentation device, characterized in that, include: The first detection module is used to perform a first detection process on the at least a portion of video frames based on the first modal data of the at least a portion of video frames in the video, and obtain a first detection result. The second detection module is used to perform a second detection process on the target event of the at least a portion of the video frames based on the second modal data of the at least a portion of the video frames, and obtain a second detection result; The determining module is configured to determine, based on the first detection result and the second detection result, the target frame from the at least a subset of video frames in which the target event occurred; The first segmentation module is used to segment the video based on the target frame to obtain video segments in which the target event occurs; The first detection result and the second detection result are used to indicate the probability of the target event occurring in the video frame; The step of determining the target frame from the at least a subset of video frames where the target event occurs, based on the first detection result and the second detection result, includes: The target frame is obtained based on the probability analysis of the first detection result and the second detection result.

15. A video segmentation device, characterized in that, include: The first detection module is used to perform speaker detection processing on the at least a portion of video frames based on the first modal data of the at least a portion of video frames in the live video, and obtain a first detection result; The second detection module is used to perform speaker detection processing on the at least a portion of video frames based on the second modal data of the at least a portion of video frames, and obtain a second detection result; The determining module is used to determine, based on the first detection result and the second detection result, a target frame in which someone is speaking from the at least a portion of video frames; The segmentation module is used to segment the live video based on the target frame to obtain video segments in the live video where someone is speaking; The first detection result and the second detection result are used to indicate the probability that someone is speaking in the video frame; The step of determining the target frame from the at least a subset of video frames where someone is speaking, based on the first detection result and the second detection result, includes: The target frame is obtained based on the probability analysis of the first detection result and the second detection result.

16. A video segmentation device, characterized in that, include: The first detection module is used to perform speaker detection processing on the image data of at least a portion of the video frames in the video to obtain a first detection result; The second detection module is used to perform human voice detection processing on the audio data of at least a portion of the video frames to obtain a second detection result; The determining module is used to determine, based on the first detection result and the second detection result, a target frame in which someone is speaking from the at least a portion of video frames; The segmentation module is used to segment the video based on the target frame to obtain video segments in which someone is speaking. The first detection result and the second detection result are used to indicate the probability that someone is speaking in the video frame; The step of determining the target frame from the at least a subset of video frames where someone is speaking, based on the first detection result and the second detection result, includes: The target frame is obtained based on the probability analysis of the first detection result and the second detection result.

17. A computer device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 1-13.

18. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-13.

Images