Video segmentation and single question explanation video acquisition method and device, and electronic device

Abstract

The present disclosure provides a video segmentation and single question explanation video acquisition method and device, electronic equipment and storage medium, comprising: performing video segmentation according to each timestamp corresponding to each image frame in a target video and each image feature to obtain a plurality of video segments of the target video; determining segment features and segment times of each video segment according to each timestamp corresponding to each image frame in each video segment and each image feature, and determining a segment sequence composed of each video segment arranged in sequence according to each segment time corresponding to each video segment; determining a segment set composed of at least three continuous video segments in the segment sequence according to each segment feature and each segment time corresponding to each video segment in the segment sequence; and merging each video segment according to the segment set in the segment sequence to obtain each target sub-video of the target video. Therefore, the present disclosure can improve the accuracy of video segmentation and ensure the content integrity of each target sub-video after segmentation.

Description

Technical Field

[0001] This invention relates to the field of multimedia data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for video segmentation and single-topic explanation videos. Background Technology

[0002] With the rapid development of short videos, people prefer watching high-density, engaging short videos to long videos. The number of videos on social media platforms is exploding, making it a meaningful and challenging task to divide a long video into multiple sub-videos containing complete storylines based on changes in scenes, actions, and events.

[0003] Video understanding is a crucial technique in video segmentation, while event boundary detection is a significant area of ​​focus within video understanding. Specifically, event boundary detection involves identifying event boundary points within a long video based on its content, thereby dividing the video into multiple complete and independent sub-videos. Current research focuses on film scene segmentation, event boundary detection, and performance-based videos, with applications in short video platforms, video retrieval, surveillance analysis, and educational settings.

[0004] Furthermore, with the popularization of internet education, online teaching videos have greatly promoted the dissemination of high-quality teaching resources and are very popular among users. With the widespread availability of pre-recorded teaching video products, the number of online teaching video resources has increased dramatically. Currently, most online teaching videos are long videos segmented based on knowledge points, making it difficult for users to find individual video explanations of topics of interest from the vast database of long videos.

[0005] In view of this, there is an urgent need for a technology that can perform accurate video segmentation on long videos, especially long educational videos. Summary of the Invention

[0006] In view of this, the present disclosure provides a method, apparatus, electronic device, and storage medium for video segmentation and acquisition of single-topic explanation videos, so as to at least partially solve the above problems.

[0007] According to a first aspect of this disclosure, a video segmentation method is provided, comprising: performing video segmentation based on timestamps and image features corresponding to each image frame in a target video to obtain multiple video segments of the target video; determining segment features and segment time of each video segment based on timestamps and image features corresponding to each image frame in each video segment, and determining a segment sequence composed of sequentially arranged video segments based on the segment times corresponding to each video segment; determining at least one segment set in the segment sequence based on the segment features and segment times corresponding to each video segment in the segment sequence, wherein each segment set is composed of at least three consecutive video segments in the segment sequence; merging at least a portion of each video segment based on at least one segment set in the segment sequence, and obtaining each target sub-video of the target video based on the merging result of each video segment in the segment sequence.

[0008] According to a second aspect of this disclosure, a method for obtaining single-question explanation videos is provided, comprising: obtaining a long teaching video; segmenting the long teaching video to obtain single-question explanation videos corresponding to each test question; wherein the segmentation operation of the long teaching video is implemented using the video segmentation method described in the first aspect above.

[0009] According to a third aspect of this disclosure, a video segmentation apparatus is provided, comprising: a video segmentation module, configured to perform video segmentation based on timestamps and image features corresponding to each image frame in a target video to obtain multiple video segments of the target video; a segment processing module, configured to determine segment features and segment time of each video segment based on timestamps and image features corresponding to each image frame in each video segment, and to determine a segment sequence composed of sequentially arranged video segments based on the segment time corresponding to each video segment; and a segment merging module, configured to determine at least one segment set in the segment sequence based on segment features and segment time corresponding to each video segment in the segment sequence, and to merge at least a portion of each video segment based on at least one segment set in the segment sequence, and to obtain each target sub-video of the target video based on the merging result of each video segment in the segment sequence; wherein each segment set is composed of at least three consecutive video segments in the segment sequence.

[0010] According to a fourth aspect of this disclosure, a device for acquiring single-question explanation videos is provided, comprising: an acquisition module for acquiring a long teaching video; and a segmentation module for performing video segmentation on the long teaching video to obtain single-question explanation videos corresponding to each test question; wherein the segmentation module is the video segmentation method described in the first aspect above, or performs video segmentation on the long teaching video using the video segmentation device described in the third aspect above.

[0011] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the video segmentation method described in the first aspect above, or to perform the single-topic explanation video acquisition method described in the second aspect above.

[0012] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are configured to cause the computer to perform a video segmentation method as described in the first aspect, or to perform a single-topic explanation video acquisition method as described in the second aspect.

[0013] In summary, the video segmentation schemes provided in this disclosure first perform video segmentation by comparing the image features corresponding to each image frame in the target video to obtain multiple video segments of the target video. Then, the segment features of each non-adjacent video segment are compared, and the video segments are selectively merged to obtain each target sub-video of the target video. In this way, this disclosure can ensure the integrity of the video content of each target sub-video after segmentation and effectively improve the accuracy of the video segmentation results. Attached Figure Description

[0014] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0015] Figure 1 This is a flowchart illustrating a video segmentation method as an exemplary embodiment of the present disclosure.

[0016] Figure 2 This is a flowchart illustrating a video segmentation method as an exemplary embodiment of this disclosure.

[0017] Figure 3 This is a flowchart illustrating a video segmentation method as an exemplary embodiment of this disclosure.

[0018] Figure 4 This is a flowchart illustrating a method for acquiring single-topic explanation videos, which is an exemplary embodiment of this disclosure.

[0019] Figure 5 This is a structural block diagram of video segmentation and transposition, which is an exemplary embodiment of this disclosure.

[0020] Figure 6 This is a structural block diagram of a single-topic explanation video acquisition device, which is an exemplary embodiment of this disclosure.

[0021] Figure 7 This is a schematic diagram of the architecture of an electronic device that is an exemplary embodiment of this disclosure. Detailed Implementation

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0024] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0025] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative and not restrictive. Those skilled in the art should understand that, unless explicitly stated in the context, they should be understood as "one or more". The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0026] As described in the background section above, online education has accumulated a large number of high-quality course videos. However, most of the current teaching videos are long videos segmented based on knowledge points, making it difficult for users to quickly find videos explaining specific topics of interest from a vast database of long videos. Therefore, it is very valuable and beneficial to students to find a way to segment the large number of high-quality long videos in online education into complete and independent short videos explaining specific topics. This can not only reuse existing video resources and maintain their value, but also disseminate knowledge and help students strengthen their understanding of knowledge and develop problem-solving skills.

[0027] Current video segmentation technology mainly combines OCR text recognition technology. However, in the actual process of explaining a single question, teachers often slide the window up and down. This operation can easily lead to the complete explanation process of the same question being incorrectly segmented into multiple sub-videos.

[0028] In view of this, the present disclosure provides a video segmentation scheme that can improve the problems existing in the prior art. The specific embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0029] Figure 1 The flowchart of a video segmentation method as an exemplary embodiment of this disclosure is shown, and it mainly includes the following steps:

[0030] Step S102: Perform video segmentation based on the timestamps and image features corresponding to each image frame in the target video to obtain multiple video segments of the target video.

[0031] Optionally, the target video may include instructional videos.

[0032] Optionally, frame extraction processing can be performed on the target video according to a given frame extraction frequency to obtain multiple image frames of the target video and the timestamps corresponding to each image frame.

[0033] For example, the OpenCV toolkit can be used to extract frames from the target video at a rate of 1 frame per second (i.e., a frame extraction frequency of 1 frame / second) to obtain multiple image frames from the target video. It should be noted that the frame extraction frequency is not limited to the example above, and can be arbitrarily adjusted according to actual needs (such as video segmentation accuracy), and this disclosure does not impose any restrictions on this.

[0034] Optionally, an image frame sequence consisting of sequentially arranged image frames can be obtained based on the timestamps corresponding to each image frame.

[0035] Optionally, the adjacent similarity value between any two adjacent image frames in the image frame sequence can be calculated to obtain the adjacent similarity value corresponding to each image frame. The timestamp of the image frame with an adjacent similarity value less than a preset adjacent similarity threshold is determined as the video segmentation point. Based on the determined video segmentation points, the image frame sequence is segmented to obtain each video segment of the target video.

[0036] In this embodiment, each video segment contains at least one image frame.

[0037] Step S104: Based on the timestamps and image features corresponding to each image frame in each video segment, determine the segment features and segment time of each video segment, and based on the segment times corresponding to each video segment, determine the segment sequence composed of the sequentially arranged video segments.

[0038] Optionally, the segment features of each video segment can be determined based on the image features corresponding to each image frame contained in each video segment.

[0039] In this embodiment, the image features of an image frame can be determined based on the text features and image features of the image frame.

[0040] Optionally, the image frame can be fed into a trained neural network to predict the text region and its corresponding text features in the image frame, as well as the image region and its corresponding image features in the image frame, thereby determining the image features of the image frame.

[0041] Optionally, the start time and / or end time of each video segment can be determined based on the timestamps corresponding to each image frame contained in each video segment, and the segment time of each video segment can be determined based on the start time and / or end time of each video segment.

[0042] Step S106: Determine at least one set of segments in the segment sequence based on the segment features and time of each video segment in the segment sequence.

[0043] In this embodiment, each segment set consists of at least three consecutive video segments in the segment sequence, the segment duration of the segment set meets a preset merging duration threshold, and the first and last similarity values ​​of the segment set meet a preset merging similarity threshold.

[0044] In this embodiment, the segment duration of the segment set is determined by the segment duration of the start and end segments of each video segment constituting the segment set, and the beginning-end similarity value of the segment set is determined by the segment features of the start and end segments of each video segment constituting the segment set.

[0045] Step S108: Based on at least one set of segments in the segment sequence, merge at least a portion of each video segment, and based on the merging result of each video segment in the segment sequence, obtain each target sub-video of the target video.

[0046] Optionally, based on each segment set in the segment sequence, video segments belonging to the same segment set in the segment sequence can be merged to obtain the merged segment corresponding to each segment set, and based on the unmerged video segments in the segment sequence and the merged segment corresponding to each segment set, each target sub-video of the target video can be obtained.

[0047] Specifically, each target sub-video in the target video can be determined based on the start and end times of the unmerged video segments in the segment sequence, and the start and end times of each merged segment.

[0048] In summary, the video segmentation method provided in this embodiment first performs video segmentation based on the timestamps and image features of each image frame in the target video to obtain multiple video segments of the target video. Then, based on the segment features and segment timestamps of each video segment, at least a portion of each video segment is merged to obtain each target sub-video of the target video. Therefore, this disclosure can improve the accuracy of video segmentation processing and ensure the integrity of the segmented video content.

[0049] Figure 2 This is a flowchart illustrating a video segmentation method as an exemplary embodiment of this disclosure. This embodiment is an exemplary implementation of step S102 described above. As shown in the figure, this embodiment mainly includes the following steps:

[0050] Step S202: Determine the image frame sequence composed of sequentially arranged image frames based on the timestamps corresponding to each image frame.

[0051] Specifically, the image frames can be arranged in order from earliest to latest according to the timestamps corresponding to each image frame to obtain the image frame sequence.

[0052] Step S204: Based on the image features corresponding to each image frame, calculate the similarity between any image frame in the image frame sequence and its adjacent previous image frame to obtain the adjacent similarity values ​​corresponding to each image frame.

[0053] Alternatively, the image features corresponding to each image frame can be determined in the following ways:

[0054] For any current image frame in each image frame, the text region and image region of the current image frame are determined by identifying the object information in the current image frame. The text features corresponding to the text region and the image features corresponding to the image region are determined by identifying the text region and the image features corresponding to the image region. Based on the text features and image features, the image features of the current image frame are determined.

[0055] In one embodiment, when the target video is an instructional video, the test question object in the current image frame can be identified, and based on the identification result of the test question object, the text region and image region in the current image frame can be determined.

[0056] For example, the test question object displayed in the current image frame can be determined by identifying the test question code in the current image frame, and the graphic layout information of the test question object (e.g., graphic and text arranged vertically, graphic and text arranged horizontally, graphic and text mixed arrangement, graphic and text wrapping, etc.) can be obtained from a given test question library to determine the text area and image area in the current image frame. By means of this technique, the recognition results of the text area and image area of ​​the current image frame can be improved.

[0057] It should be noted that in other embodiments, the text region and image region in the current image frame can also be determined by directly identifying the image portion and text portion in the current image frame.

[0058] Optionally, any image frame in the image frame sequence can be designated as the current image frame, and the image frame preceding the current image frame in the image frame sequence can be designated as the preceding image frame. For example, if the nth image frame in the image frame sequence is designated as the current image frame, then the (n-1)th image frame in the image frame sequence is the preceding image frame.

[0059] Optionally, the text similarity value of the current image frame is determined based on the text features of the current image frame and the text features of the previous image frame; the image similarity value of the current image frame is determined based on the image features of the current image frame and the image features of the previous image frame; and the adjacent similarity value of the current image frame is determined based on the text similarity value and the image similarity value.

[0060] Optionally, the text similarity value and image similarity value of the current image frame can be determined using Euclidean distance calculation, but this is not a limitation. Other similarity calculation methods (such as cosine similarity calculation) can also be used to determine the text similarity value and image similarity value of the current image frame. This disclosure does not impose any restrictions on this.

[0061] Step S206: Based on the adjacent similarity values ​​corresponding to each image frame, determine the timestamp of the image frame with an adjacent similarity value less than the preset adjacent similarity threshold as the video segmentation point.

[0062] For example, if the target video is an instructional video and contains explanations of multiple test questions, if the adjacent similarity value of the current image frame is less than the preset adjacent similarity threshold, it means that the probability that the current image frame and the previous image frame belong to the same test question explanation is extremely low. Therefore, the timestamp of the current image frame can be determined as the video segmentation point.

[0063] Step S208: Segment each image frame in the image frame sequence according to each video segmentation point to obtain each video segment of the target video.

[0064] In this embodiment, each video segment obtained contains at least one image frame.

[0065] In summary, this embodiment, by comparing the adjacency similarity between two adjacent image frames, can perform preliminary segmentation of the target video when a significant change in image information occurs between the current image frame and the preceding image frame (i.e., the adjacency similarity is less than a preset adjacency similarity threshold). This ensures that video content belonging to different themes or plots within the target video can be accurately segmented. For example, it can ensure that explanations of different test questions in an instructional video can be accurately segmented.

[0066] Furthermore, by combining and comparing image features and text features between two adjacent image frames, this embodiment can improve the accuracy of the determination of video segmentation points, thereby improving the accuracy of the segmentation results of each video segment.

[0067] Figure 3 A flowchart illustrating a video segmentation method according to another exemplary embodiment of the present disclosure is shown. This embodiment illustrates an exemplary implementation of step S106 described above.

[0068] As mentioned above, in actual single-question explanations, the operation of sliding the window up and down frequently occurs. For example, if the complete information of the current question needs to be displayed on two pages, the window will often be slid up and down during the explanation of the current question. Specifically, during the explanation of the first page of the current question, the window is slid down to switch from the first page to the second page and temporarily interweave the explanation. After the interweaving is completed, the window is slid up to return from the second page to the first page and continue the explanation. However, this window sliding operation will result in a low similarity value between the current image frame and the previous image frame, causing the complete explanation of the same question to be incorrectly divided into multiple sub-videos.

[0069] In view of this, this disclosure specifically designs a method that, after performing segmentation processing on the target video, compares the similarity values ​​between non-adjacent video segments to merge multiple video segments that originally belonged to the same test question, thereby improving the content integrity of the segmented video.

[0070] The following will combine Figure 3 This document describes in detail the merging and processing scheme for the various video segments disclosed herein. As shown in the figure, this embodiment mainly includes the following steps:

[0071] Step S302: Determine any video segment in the segment sequence as the current segment.

[0072] Optionally, a video segment can be selected as the current segment based on the sequentially arranged video segments in the segment sequence.

[0073] Step S304: Based on the segment time of the current segment and the segment time of each video segment, determine the interval duration of each video segment relative to the current segment, and determine each video segment whose interval duration meets the preset merging duration threshold and is not adjacent to the current segment as candidate segments.

[0074] Optionally, the segment time of each video segment can be determined based on the segment start time and / or segment end time of each video segment.

[0075] For example, the interval duration of each video segment relative to the current segment can be determined based on the difference between the start time of the current segment and the start time of each video segment; or, the interval duration of each video segment relative to the current segment can be determined based on the difference between the end time of the current segment and the end time of each video segment; furthermore, the interval duration of each video segment relative to the current segment can also be determined based on the difference between the start time of the current segment and the end time of each video segment, or based on the difference between the end time of the current segment and the start time of each video segment.

[0076] In this embodiment, a preset merging duration threshold can be determined based on the actual application scenario of the target video. For example, if the target video is an instructional video, the window's up-and-down scrolling time usually does not exceed three minutes; therefore, the preset merging duration threshold can be set to 3 minutes.

[0077] Step S306: Based on the segment features of the current segment and the segment features of each candidate segment, determine the merged similarity value of each candidate segment relative to the current segment, and determine the candidate segment with the largest merged similarity value that meets the preset merged similarity threshold and the longest interval between them as the target segment.

[0078] Optionally, the segment features of each video segment can be determined based on the image features corresponding to each image frame contained in each video segment. Based on the segment features of the current segment and the segment features of each candidate segment, the merged similarity value of each candidate segment relative to the current segment can be calculated. If the merged similarity value between the candidate segment and the current segment is higher than the preset merged similarity threshold, it means that a window up-and-down sliding operation may have occurred between the current segment and the candidate segment.

[0079] Furthermore, considering that multiple window up-and-down sliding operations may occur in a short period of time during the explanation of the same test question, the candidate segment with the longest interval can be selected from the candidate segments whose similarity value is higher than the preset merging similarity threshold and determined as the target segment. In other words, there may be one or more window up-and-down sliding operations during the interval between the current segment and the target segment.

[0080] Step S308: Take one of the current segment and the target segment as the starting segment, and take the other of the current segment and the target segment as the ending segment to determine a segment set.

[0081] Specifically, one of the current segment and the target segment can be used as the starting segment, the other of the current segment and the target segment can be used as the ending segment, and at least one video segment in the segment sequence located between the starting segment and the ending segment can be used as an intermediate segment, thus determining a segment set consisting of the starting segment, at least one intermediate segment, and the ending segment.

[0082] In summary, this embodiment calculates the similarity value between two non-adjacent video segments to determine a segment set consisting of at least three video segments that should belong to the same theme or plot. By subsequently merging the video segments in the segment set, it can effectively solve the problem of the explanation of the same test question being incorrectly divided into multiple video segments due to the sliding window, thereby ensuring the integrity of the segmented video content.

[0083] Figure 4 The flowchart of a method for acquiring single-topic explanation videos, which is an exemplary embodiment of this disclosure, mainly includes the following steps:

[0084] Step S402: Obtain the long instructional video.

[0085] In this embodiment, the long instructional video includes explanation videos for multiple test questions.

[0086] Step S404: Divide the long teaching video into segments to obtain a single-question explanation video corresponding to each test question.

[0087] In this embodiment, the above-mentioned Figures 1 to 3 The examples of video segmentation methods shown demonstrate the segmentation operation of long teaching videos.

[0088] In summary, the embodiments of this disclosure can obtain a complete explanation video of a single question from a long teaching video, avoiding the problem that the explanation content of the same question is split into multiple videos due to the up and down scrolling operation of the window during the question explanation process, thus ensuring the integrity of the content of the single question explanation video after splitting.

[0089] Figure 5 The following is a structural block diagram of a video segmentation device as an exemplary embodiment of the present disclosure. As shown in the figure, the video segmentation device 500 of this embodiment mainly includes a video segmentation module 502, a segment processing module 504, and a segment merging module 506.

[0090] The video segmentation module 502 is used to perform video segmentation based on the timestamps and image features corresponding to each image frame in the target video, thereby obtaining multiple video segments of the target video.

[0091] The segment processing module 504 is used to determine the segment features and segment time of each video segment based on the timestamps and image features corresponding to each image frame in each video segment, and to determine the segment sequence composed of the sequentially arranged video segments based on the segment times corresponding to each video segment.

[0092] The segment merging module 506 is used to determine at least one set of segments in the segment sequence based on the segment features and segment times corresponding to each video segment in the segment sequence, and to merge at least a portion of each video segment based on the at least one set of segments in the segment sequence, and to obtain each target sub-video of the target video based on the merging result of each video segment in the segment sequence, wherein each segment set consists of at least three consecutive video segments in the segment sequence.

[0093] Optionally, the video segmentation module 502 is further configured to: determine an image frame sequence composed of sequentially arranged image frames based on the timestamps corresponding to each image frame; calculate the similarity between any image frame in the image frame sequence and its adjacent preceding image frame based on the image features corresponding to each image frame, and obtain the adjacent similarity values ​​corresponding to each image frame; determine the timestamps of image frames with adjacent similarity values ​​lower than a preset adjacent similarity threshold as video segmentation points based on the adjacent similarity values ​​corresponding to each image frame; and perform video segmentation based on each video segmentation point to obtain video segments of the target video.

[0094] Optionally, the video segmentation module 502 is further configured to: for any current image frame in each image frame, identify object information in the current image frame, and determine the text region and image region of the current image frame; identify the text region and image region of the current image frame, and determine the text features of the text region and the image features of the image region; and determine the image features of the current image frame based on the text features and the image features.

[0095] Optionally, the video segmentation module 502 is further configured to: determine any one image frame in the image frame sequence as the current image frame, and determine the image frame in the image frame sequence that precedes the current image frame as the preceding image frame; determine the text similarity value of the current image frame based on the text features of the current image frame and the text features of the preceding image frame; determine the image similarity value of the current image frame based on the image features of the current image frame and the image features of the preceding image frame; and determine the adjacent similarity value of the current image frame based on the text similarity value and the image similarity value.

[0096] Optionally, the segment processing module 504 is further configured to: determine the segment features of each video segment based on the image features corresponding to each image frame contained in each video segment; determine the segment start time and / or segment end time of each video segment based on the timestamps corresponding to each image frame contained in each video segment; and determine the segment time of each video segment based on the segment start time and / or segment end time of each video segment.

[0097] Optionally, the segment duration of the segment set meets a preset merging duration threshold, and the beginning-end similarity value of the segment set meets a preset merging similarity threshold; the segment duration of the segment set is determined by the segment time of the beginning and end segments in each video segment constituting the segment set; the beginning-end similarity value of the segment set is determined by the segment features of the beginning and end segments in each video segment constituting the segment set.

[0098] Optionally, the segment merging module 506 is further configured to: determine any video segment in the segment sequence as the current segment; determine the interval duration of each video segment relative to the current segment based on the segment time of the current segment and the segment time of each video segment, and determine each video segment whose interval duration meets a preset merging duration threshold and is not adjacent to the current segment as a candidate segment; determine the merging similarity value of each candidate segment relative to the current segment based on the segment features of the current segment and the segment features of each candidate segment, and determine the candidate segment with the largest merging similarity value that meets a preset merging similarity threshold and the largest interval duration as the target segment; and determine a segment set by taking one of the current segment and the target segment as the starting segment and the other of the current segment and the target segment as the ending segment.

[0099] Optionally, the segment merging module 506 is further configured to: merge video segments belonging to the same segment set in the segment sequence according to each segment set in the segment sequence, to obtain a merged segment corresponding to each segment set; and obtain each target sub-video of the target video according to the unmerged video segments in the segment sequence and the merged segment corresponding to each segment set.

[0100] Figure 6 The figure shows a structural block diagram of a single-topic explanation video acquisition device as an exemplary embodiment of the present disclosure. The single-topic explanation video acquisition device 600 of this embodiment mainly includes an acquisition module 602 and a segmentation module 604.

[0101] Module 602 is used to acquire long instructional videos.

[0102] The segmentation module 604 is used to perform video segmentation on the long teaching video to obtain a single-question explanation video corresponding to each test question. The segmentation module performs video segmentation on the long teaching video using the video segmentation method or the video segmentation device described in the embodiments of this disclosure.

[0103] This disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to cause the computer to execute the video segmentation method described in the exemplary embodiments of this disclosure, or to execute the single-topic explanation video acquisition method described in the exemplary embodiments of this disclosure.

[0104] An exemplary embodiment of this disclosure provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program, when executed by the at least one processor, causing the electronic device to perform a video segmentation method according to various exemplary embodiments of this disclosure, or to perform a single-topic explanation video acquisition method according to various exemplary embodiments of this disclosure.

[0105] Please refer to Figure 7The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0106] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0107] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, output unit 704, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to electronic device 700. Input unit 706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 706 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 704 may include, but is not limited to, disks and optical discs. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0108] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the video segmentation method or single-topic explanation video acquisition method described above can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. In some embodiments, the computing unit 701 can be configured by any other suitable means (e.g., by means of firmware) to perform the video segmentation method or single-topic explanation video acquisition method described above.

[0109] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0110] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0111] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0113] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0114] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0115] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this disclosure can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this disclosure.

[0116] The above embodiments are only used to illustrate the embodiments of this disclosure, and are not intended to limit the embodiments of this disclosure. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this disclosure. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this disclosure, and the patent protection scope of the embodiments of this disclosure should be defined by the claims.

Claims

1. A video segmentation method, characterized in that, include: Video segmentation is performed based on the timestamps and image features corresponding to each image frame in the target video to obtain multiple video segments of the target video; Based on the timestamps and image features corresponding to each image frame in each video segment, the segment features and segment time of each video segment are determined, and based on the segment times corresponding to each video segment, the segment sequence composed of the sequentially arranged video segments is determined. Based on the segment features and time of each video segment in the segment sequence, at least one segment set in the segment sequence is determined, wherein each segment set consists of at least three consecutive video segments in the segment sequence; the segment duration of the segment set meets a preset merging duration threshold, and the first-to-last similarity value of the segment set meets a preset merging similarity threshold; Based on at least one set of segments in the segment sequence, at least a portion of each video segment is merged, and based on the merging result of each video segment in the segment sequence, each target sub-video of the target video is obtained.

2. The method according to claim 1, characterized in that, The step of performing video segmentation based on the timestamps and image features corresponding to each image frame in the target video to obtain multiple video segments of the target video includes: Based on the timestamps corresponding to each image frame, determine the image frame sequence consisting of the sequentially arranged image frames; Based on the image features corresponding to each image frame, the similarity between any image frame in the image frame sequence and its adjacent previous image frame is calculated to obtain the adjacent similarity values ​​corresponding to each image frame. Based on the adjacent similarity values ​​corresponding to each image frame, the timestamps of image frames with adjacent similarity values ​​lower than the preset adjacent similarity threshold are determined as video segmentation points; Video segmentation is performed based on each video segmentation point to obtain video segments of the target video.

3. The method according to claim 2, characterized in that, The image features corresponding to each image frame can be determined in the following ways: For any current image frame in each image frame, Identify object information in the current image frame to determine the text region and image region of the current image frame; Identify the text region and image region of the current image frame, and determine the text features of the text region and the image features of the image region; Based on the text features and the image features, the image features of the current image frame are determined.

4. The method according to claim 2, characterized in that, The step of calculating the similarity between any image frame in the image frame sequence and its adjacent preceding image frame based on the image features corresponding to each image frame, and obtaining the adjacent similarity values ​​corresponding to each image frame, includes: Any image frame in the image frame sequence is determined as the current image frame, and the image frame in the image frame sequence that precedes the current image frame is determined as the preceding image frame. The text similarity value of the current image frame is determined based on the text features of the current image frame and the text features of the previous image frame. The image similarity value of the current image frame is determined based on the image features of the current image frame and the image features of the previous image frames. Based on the text similarity value and the image similarity value, the adjacent similarity value of the current image frame is determined.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the segment features and segment time of each video segment based on the timestamps and image features corresponding to each image frame in each video segment includes: Based on the image features corresponding to each image frame contained in each video segment, the segment features of each video segment are determined. Based on the timestamps corresponding to each image frame contained in each video segment, determine the segment start time and / or segment end time of each video segment, and determine the segment time of each video segment based on the segment start time and / or segment end time of each video segment.

6. The method according to claim 1, characterized in that, The duration of a segment in the segment set is determined by the segment duration of the start and end segments of each video segment constituting the segment set. The similarity value between the beginning and end of the segment set is determined by the segment features of the beginning and end segments of each video segment constituting the segment set.

7. The method according to claim 1 or 6, characterized in that, The step of determining at least one set of segments in the segment sequence based on the segment features and segment times corresponding to each video segment in the segment sequence includes: Select any one of the video segments in the segment sequence as the current segment; Based on the segment time of the current segment and the segment time of each video segment, the interval duration of each video segment relative to the current segment is determined, and each video segment whose interval duration meets the preset merging duration threshold and is not adjacent to the current segment is determined as a candidate segment. Based on the segment features of the current segment and the segment features of each candidate segment, the merged similarity value of each candidate segment relative to the current segment is determined, and the candidate segment with the largest merged similarity value that meets the preset merged similarity threshold and has the longest interval is determined as the target segment. A segment set is determined by taking one of the current segment and the target segment as the starting segment and the other of the current segment and the target segment as the ending segment.

8. The method according to claim 1, characterized in that, The step of merging at least a portion of each video segment based on at least one set of segments in the segment sequence, and obtaining each target sub-video of the target video based on the merging result of each video segment in the segment sequence, includes: Based on each segment set in the segment sequence, merge the video segments belonging to the same segment set in the segment sequence to obtain the merged segment corresponding to each segment set; Based on the unmerged video segments in the segment sequence and the merged segments corresponding to each segment set, the target sub-videos of the target video are obtained.

9. A method for obtaining single-topic explanation videos, characterized in that, include: Get the long instructional videos; The teaching video is segmented to obtain a single-question explanation video corresponding to each test question. The segmentation operation of the teaching video is implemented using the video segmentation method as described in any one of claims 1 to 8.

10. A video segmentation device, characterized in that, include: The video segmentation module is used to perform video segmentation based on the timestamps and image features corresponding to each image frame in the target video, thereby obtaining multiple video segments of the target video. The segment processing module is used to determine the segment features and segment time of each video segment based on the timestamps and image features corresponding to each image frame in each video segment, and to determine the segment sequence composed of the sequentially arranged video segments based on the segment times corresponding to each video segment. The segment merging module is used to determine at least one set of segments in the segment sequence based on the segment features and segment times corresponding to each video segment in the segment sequence, and to merge at least a portion of each video segment based on the at least one set of segments in the segment sequence, and to obtain each target sub-video of the target video based on the merging result of each video segment in the segment sequence. Each segment set consists of at least three consecutive video segments from the segment sequence. The segment duration of the segment set meets a preset merging duration threshold, and the first-to-last similarity value of the segment set meets a preset merging similarity threshold.

11. A device for acquiring single-topic explanation videos, characterized in that, include: The acquisition module is used to acquire long instructional videos; The segmentation module is used to perform video segmentation on the long teaching video to obtain a single-question explanation video corresponding to each test question in the long teaching video; The segmentation module performs video segmentation on the teaching long video using the video segmentation method as described in any one of claims 1 to 8, or using the video segmentation device as described in claim 10.

12. An electronic device, comprising: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the video segmentation method as described in any one of claims 1-8, or to perform the single-topic explanation video acquisition method as described in claim 9.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the video segmentation method as described in any one of claims 1-8, or to execute the single-topic explanation video acquisition method as described in claim 9.

Images