Video segmentation method, recommendation method, system, device and medium

Abstract

The application relates to a video segmentation method, a recommendation method, a system, a device and a medium. The segmentation method comprises the following steps: obtaining a target video; performing voice recognition on the target video to obtain a plurality of subtitle sentences corresponding to the target video and starting and ending moments of the subtitle sentences in the target video; calculating time intervals between each subtitle sentence and a previous subtitle sentence according to the starting and ending moments of each subtitle sentence in the target video, and fusing each subtitle sentence and the corresponding time interval to obtain a plurality of fusion sentences arranged in sequence; identifying and extracting each chapter name from the plurality of fusion sentences based on a sequence labeling algorithm, and determining a corresponding chapter starting moment; and segmenting the target video into a plurality of chapter videos according to each chapter name and the corresponding chapter starting moment. The application improves the accuracy of video segmentation.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a video segmentation method, recommendation method, system, device, and medium. Background Technology

[0002] With the increasing richness of video content and the diversification of user needs, how to accurately segment long video content into short video segments with independent semantics has become an increasingly important issue. In many cases, users tend to prefer videos containing specific knowledge points rather than long, drawn-out videos. Current technologies typically segment videos based on their theme, overall tags, or time stamps, but this approach cannot achieve fine-grained video segmentation. Therefore, there is a need to provide a video segmentation method, recommendation method, system, device, and medium. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a video segmentation method, recommendation method, system, device and medium that improves the problem of low precision in video segmentation in the prior art.

[0004] To achieve the above and other related objectives, the present invention provides a video segmentation method, the segmentation method comprising: acquiring a target video; performing speech recognition on the target video to obtain multiple subtitle statements corresponding to the target video and their start and end times in the target video; calculating the time interval between each subtitle statement and its preceding subtitle statement based on the start and end times of each subtitle statement in the target video, and fusing each subtitle statement and its corresponding time interval to obtain multiple fused statements arranged in sequence; identifying and extracting each chapter name from the multiple fused statements based on a sequence labeling algorithm, and determining the corresponding chapter start time; and segmenting the target video into multiple chapter videos based on each chapter name and its corresponding chapter start time.

[0005] In one embodiment of the present invention, the step of performing speech recognition on the target video to obtain multiple subtitle statements corresponding to the target video and their start and end times in the target video includes: extracting audio information from the target video; performing speech recognition on the audio information to convert the audio information into text information; performing sentence segmentation processing on the text information based on a speech activity detection algorithm to obtain multiple subtitle statements in the target video, and determining the start and end times of each subtitle statement in the target video.

[0006] In one embodiment of the present invention, the step of calculating the time interval between each subtitle statement and its preceding subtitle statement based on the start and end times of each subtitle statement in the target video, and fusing each subtitle statement and its corresponding time interval to obtain a plurality of fused statements arranged in sequence includes: calculating the time interval between a subtitle statement and its preceding subtitle statement based on the start time of the subtitle statement and the end time of its preceding subtitle statement, and performing word embedding processing on the time interval; performing word segmentation and word embedding processing on the subtitle statements to obtain a word embedding vector sequence of the subtitle statements; and fusing the word embedding time interval and the word embedding vector sequence of the subtitle statements to generate fused statements.

[0007] In one embodiment of the present invention, the step of identifying and extracting chapter names from multiple fused sentence sequences based on sequence labeling algorithms and determining the corresponding chapter start time includes: for each fused sentence: inputting the fused sentence into the encoder network of the sequence labeling model to capture the linguistic and temporal features of the fused sentence and generating a comprehensive feature of the fused sentence; inputting the comprehensive feature into the first branch network of the sequence labeling model to determine the type of the corresponding subtitle sentence; wherein the type of the subtitle sentence is a start sentence or a non-start sentence; inputting the comprehensive feature into the second branch network of the sequence labeling model to obtain the recognition degree of each word in the corresponding subtitle sentence, and concatenating words with recognition degrees greater than or equal to a preset second threshold in sequence as the corresponding chapter name; determining the chapter start time corresponding to the chapter name according to the type of subtitle sentence corresponding to the chapter name.

[0008] In one embodiment of the present invention, determining the chapter start time corresponding to the chapter name based on the type of subtitle statement corresponding to the chapter name includes: if the type of subtitle statement corresponding to the chapter name has a start statement, then the start time of the start statement in the target video is taken as the chapter start time corresponding to the chapter name; if the type of subtitle statement corresponding to the chapter name does not have a start statement, then the start time of the preceding start statement closest to the subtitle statement is extracted from the subtitle statements arranged in chronological order, and this is taken as the chapter start time corresponding to the chapter name.

[0009] In one embodiment of the present invention, after segmenting the target video into multiple chapter videos according to each chapter name and the corresponding chapter start time, the method further includes: concatenating the title and chapter name of the target video to generate combined text corresponding to the chapter name, and extracting features from the combined text to generate semantic vectors corresponding to the chapter names.

[0010] In one embodiment of the present invention, the step of concatenating the title and chapter name of the target video to generate combined text corresponding to the chapter name, and extracting features from the combined text to generate a semantic vector corresponding to the chapter name includes: concatenating the title and chapter name of the target video to generate combined text of the chapter name; inputting the generated combined text into a word embedding model to capture the corresponding semantic features in the combined text and generate a semantic vector corresponding to the chapter name.

[0011] In one embodiment of the present invention, a video recommendation method is also provided. The recommendation method includes: acquiring videos of interest; performing speech recognition on the videos of interest to obtain multiple subtitle statements corresponding to the videos of interest and their start and end times in the videos of interest; calculating the time interval between each subtitle statement and its preceding subtitle statement based on the start and end times of each subtitle statement in the videos of interest, and fusing each subtitle statement and its corresponding time interval to obtain multiple fused statements arranged in sequence; identifying and extracting chapter names from the multiple fused statements based on a sequence labeling algorithm; concatenating the title and chapter name of the videos of interest to generate corresponding combined text, and performing feature extraction on the combined text to generate corresponding semantic vectors of interest; calculating the similarity between the semantic vectors of interest and each semantic vector to be recommended in the video library, and selecting and recommending corresponding chapter videos from the video library based on the similarity calculation results; wherein the videos stored in the video library are segmented by any one of the segmentation methods described above.

[0012] In one embodiment of the present invention, the step of calculating the similarity between the semantic vector of interest and each semantic vector to be recommended in the video library, and determining the chapter video to be recommended and its chapter start time from the video library based on the similarity calculation results, includes: calculating the cosine similarity between the semantic vector of interest and each semantic vector to be recommended in the video library; determining whether the calculated cosine similarity is greater than a preset similarity threshold, and selecting the semantic vector to be recommended corresponding to the similarity greater than the similarity threshold; and searching for the corresponding chapter video from the video library based on the selected semantic vector to be recommended for recommendation.

[0013] In one embodiment of the present invention, the step of finding the corresponding chapter video from the video library based on the selected semantic vector to be recommended and making recommendations includes: finding the corresponding chapter video from the video library based on the selected semantic vector to be recommended and using it as the chapter video to be recommended; sorting the chapter videos to be recommended according to their respective cosine similarity; and displaying the sorted chapter videos to be recommended.

[0014] In one embodiment of the present invention, a video recommendation system is also provided. The system includes: a video acquisition module for acquiring a target video; a text extraction module for performing speech recognition on the target video to obtain multiple subtitle statements corresponding to the target video and their start and end times in the target video; a word embedding module for calculating the time interval between each subtitle statement and its preceding subtitle statement based on the start and end times of each subtitle statement in the target video, and fusing each subtitle statement and its corresponding time interval to obtain multiple fused statements arranged in sequence; a chapter extraction module for identifying and extracting each chapter name from the multiple fused statements based on a sequence labeling algorithm, and determining the corresponding chapter start time; and a video segmentation module for segmenting the target video into multiple chapter videos based on each chapter name and its corresponding chapter start time.

[0015] In one embodiment of the present invention, an electronic device is also provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device enables the video segmentation method or the recommended video method described above.

[0016] In one embodiment of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a computer processor, causes the computer to perform any of the video segmentation methods or recommended video methods described above.

[0017] As described above, the video segmentation method, recommendation method, system, device, and medium of the present invention have the following beneficial effects: By performing speech recognition on the target video, multiple subtitle statements and their corresponding start and end times are generated, and the time interval between the subtitle statements is calculated. The time interval is then fused with the subtitle statements, resulting in a fused statement that not only has semantic information but also time interval information between adjacent statements. Since a larger time interval indicates a weaker correlation between two statements, a sequence labeling algorithm can more accurately extract relevant chapter names and start times based on semantic and time interval information. This allows for the segmentation of a complete target video into multiple chapter videos based on these chapter names and start times. This method of the present invention performs video segmentation at the chapter level, improving the accuracy of video segmentation and enabling the automated construction of video chapter information, including extracting chapter names and corresponding start times from the video. This allows for more accurate video segmentation, facilitating the subsequent recommendation of chapters highly relevant to the user's viewed video. Attached Figure Description

[0018] Figure 1A flowchart illustrating a video segmentation method provided in an embodiment of the present invention;

[0019] Figure 2 A flowchart illustrating the video recommendation method provided in this embodiment of the invention.

[0020] Figure 3 The diagram shown is a structural block diagram of a video segmentation system provided in an embodiment of the present invention;

[0021] Figure 4 The diagram shown is a structural schematic of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0022] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0023] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0024] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0025] Existing video recommendation technologies primarily recommend content at the overall video level. For example, when a user clicks on an instructional video about "deep learning" on an online learning platform, the platform typically recommends other videos on "deep learning" by different speakers, based on the user's information and the platform's resource data. However, this approach is limited to recommending videos as a whole and neglects the details within longer videos, especially those that are typically divided into multiple chapters.

[0026] The inventors discovered that long videos can be broken down into multiple chapter videos at a finer granular level. For example, a video titled "Automotive Engine Structure and Repair" might contain a chapter on "Ignition Control Systems," while another video titled "Automotive Electrical and Electronic Equipment" might contain a chapter on "The Working Principle of Ignition Systems." These two chapters are related. After watching one video, a user might want to learn more about that chapter. However, current technology lacks the fine-grained segmentation and analysis of these chapters within a video, making it impossible to proactively recommend other chapter videos related to a specific chapter to the user. Existing video recommendation technologies primarily rely on global features such as video category, tags, description, and author information, neglecting the hierarchical structure of chapters within the video. Therefore, video recommendations are limited to presenting the entire video to the user, without breaking down the video content into chapters or independent segments for recommendation. This holistic recommendation approach is therefore unsuitable for meeting a user's precise needs for a specific chapter. In summary, existing technologies, when segmenting and recommending videos, fail to perform fine-grained segmentation and content analysis based on chapters, and cannot proactively recommend other videos related to the chapters that a user is currently interested in. While watching a video, a user may develop an interest in a specific chapter, but existing technologies cannot recommend other related video segments containing that chapter from a segmentation perspective.

[0027] This invention provides a video recommendation method. By performing speech recognition on a target video, multiple subtitle statements and their corresponding start and end times are generated. The time intervals between the subtitle statements are calculated, and these time intervals are fused with the subtitle statements. The resulting fused statements contain not only semantic information but also the time interval information between adjacent statements. Since a larger time interval indicates a weaker correlation between two statements, a sequence labeling algorithm can more accurately extract relevant chapter names and start times based on the semantic and time interval information. This allows for the segmentation of a complete target video into multiple chapter videos based on these chapter names and start times. This method performs video segmentation at the chapter level, automating the construction of video chapter information, including extracting chapter names and corresponding start times. This enables more accurate video segmentation, facilitating the subsequent recommendation of chapters highly relevant to the user's viewed videos.

[0028] Please see Figure 1 The video recommendation method includes the following steps:

[0029] S11. Obtain the target video.

[0030] The target video refers to a complete video that needs to be split into several parts. This target video may contain multiple chapters. The method described in this invention can split a long, complete video into several chapter videos for the user to choose from. The target video can be obtained from a specified storage location, video library, or certain online resource platforms. The target video's format can be parsed to ensure data consistency during subsequent processing. Furthermore, if the target video's format does not belong to a preset category, it can be converted to a processable format.

[0031] S12. Perform speech recognition on the target video to obtain multiple subtitle statements corresponding to the target video and their start and end times in the target video.

[0032] The audio track is extracted from the target video to generate an audio file. The audio file is then processed using speech recognition technology to convert it into corresponding subtitles, and timestamps are generated for the start and end times of each subtitle in the target video.

[0033] In one embodiment of the present invention, the step of performing speech recognition on the target video to obtain multiple subtitle statements corresponding to the target video and their start and end times in the target video includes:

[0034] Extract the audio information from the target video;

[0035] The audio information is subjected to speech recognition and converted into text information;

[0036] The text information is segmented based on a speech activity detection algorithm to obtain multiple subtitle statements in the target video, and the start and end times of each subtitle statement in the target video are determined.

[0037] To extract subtitles from a target video, the video first needs to be parsed. Video processing tools can be used to extract the corresponding audio information. Then, speech recognition is used to process the audio information and convert it into text. As an example, the extracted audio information can be input into a pre-trained language model. By analyzing the audio information frame by frame, the complete text information corresponding to the target video can be generated. This text information consists of multiple consecutive subtitle statements arranged sequentially according to their appearance in the video. A Voice Activity Detection (VAD) algorithm detects the start and stop times of speech activity in the audio information, dividing the complete text information into several subtitle statements and generating timestamps for the start and end times of each subtitle statement in the video. These generated subtitle statements are then arranged according to their appearance in the target video to form a subtitle statement sequence. This subtitle statement sequence is presented as a string, which is formed by concatenating all the previously obtained subtitle statements in chronological order, with adjacent subtitle statements separated by the [CLS] delimiter to distinguish them. The subtitle sequence includes each subtitle statement arranged sequentially in the target video, along with its corresponding start and end times.

[0038] S13. Based on the start and end times of each subtitle statement in the target video, calculate the time interval between each subtitle statement and its preceding subtitle statement, and merge each subtitle statement and its corresponding time interval to obtain multiple merged statements arranged in sequence.

[0039] Considering that the time interval between adjacent subtitle statements reflects the pause between them and demonstrates the semantic coherence of adjacent statements, thus aiding in subsequent chapter division, after obtaining the subtitle statement sequence as described above, the time interval between the current subtitle statement and the previous subtitle statement is calculated one by one, and this time interval is associated with the current subtitle statement. Through a fusion method, each subtitle statement and its corresponding time interval are merged into a complete whole, serving as the fused statement corresponding to that subtitle statement.

[0040] In one embodiment of the present invention, the step of calculating the time interval between each subtitle statement and its preceding subtitle statement based on the start and end times of each subtitle statement in the target video, and fusing each subtitle statement and its corresponding time interval to obtain a plurality of sequentially arranged fused statements includes:

[0041] Based on the start time of the subtitle statement and the end time of its preceding subtitle statement, the time interval between the subtitle statement and its preceding subtitle statement is calculated, and word embedding processing is performed on the time interval.

[0042] The subtitle sentences are segmented and embedded to obtain a sequence of word embedding vectors for the subtitle sentences.

[0043] The time interval after word embedding and the word embedding vector sequence of the subtitle sentence are fused together to generate a fused sentence.

[0044] The difference between the start time of each subtitle statement and the end time of its preceding subtitle statement is calculated to obtain the time interval between these two subtitle statements, and this time interval is associated with the current subtitle statement. Considering that a larger time interval may indicate lower content coherence between adjacent statements, suggesting that the current subtitle statement may represent the start of a new chapter, word embedding is first performed on the time intervals to enable the model to understand their content. Specifically, the time intervals can be normalized, for example, scaled to between 0 and 1, and then mapped to a high-dimensional vector T of a preset dimension through an embedding layer. Furthermore, each subtitle statement undergoes token segmentation, breaking down a complete statement into several basic tokens arranged according to their order. A token is a smaller unit that divides text data; the granularity of token segmentation includes character granularity, word granularity, and sub-word granularity, which can be adaptively selected based on the model's training needs. A pre-trained language model is used to perform word embedding on all tokens of each subtitle statement, resulting in a token embedding vector sequence for the corresponding subtitle statement; each token embedding vector corresponds to one token. The time interval after word embedding is fused with the word embedding vector of the corresponding subtitle sentence to obtain the fused sentence of the subtitle sentence. All the fused sentences are arranged in sequence to form the fused sentence sequence.

[0045] As one example, for a specific target video, the subtitles and their corresponding start and end times are shown in Table 1 below:

[0046] Table 1. Examples of subtitle phrases and corresponding times.

[0047] [CLS] Okay, now let's learn the second part. Start time: 00:00:44,060, End time: 00:00:46,565 [CLS] is an overview of the development of educational psychology; in this section... Start time: 00:00:47,420, End time: 00:00:52,035 [CLS] We'll start by introducing a case study. Start time: 00:00:52,890, End time: 00:00:59,355 [CLS] So let's have the learners discuss who the founder of educational psychology is. Start time: 00:01:00,020, End time: 00:01:02,275

[0048] For the time interval between the first statement and the second statement, it is the difference between the start time (00:00:47,420) of the second statement and the end time (00:00:46,565) of the first statement, which is 855 milliseconds. Perform token segmentation on the second statement, generating the result "[CLS]-就-是-教-育-心-理-学-的-发-展-概-况-,-在-这-一-部-分", obtaining 19 tokens. Use the pre-trained language model to perform word embedding processing on these tokens, obtaining the token embedding vector sequence [E0, E1, …, E18] of this subtitle statement, where E0 is the token embedding vector of "[CLS]" and E18 is the token embedding vector of "分". Fuse the embedding vector T of the time interval and E0 through concatenation, obtaining the fused embedding vector E0, = [E0, T]. Since the time interval between tokens in the same subtitle statement is set to zero, the token embedding vectors at other positions can be expressed as Ei, = [Ei, 0]. In this way, the fused statement of this subtitle statement can be obtained as [E0,, E1, …, E18,]. Among them, E0, contains both time interval information and semantic information.

[0049] S14. Based on the sequence labeling algorithm, identify and extract the names of each chapter from multiple fused statements, and determine the corresponding starting time of each chapter.

[0050] Input the sequence of fused statements that integrate semantic and time interval information into the fine-tuned pre-trained dual-branch sequence labeling model. By extracting the features of each statement, and through the first branch, determine whether the current subtitle statement is the starting statement of a chapter. If so, mark the starting time of this subtitle statement as the starting time of the corresponding chapter. Further, also label the tokens in the subtitle statement through the second branch, so as to identify which tokens constitute the chapter name. In this way, the names of each chapter and the corresponding starting times of the target video can be extracted for video segmentation. It can be understood that the sequence labeling model can be based on various neural network architectures, including but not limited to the LSTM-based model and the Transformer architecture-based model such as BERT, as long as it can play the role of sequence labeling. Preferably, in order to improve the recognition accuracy of the model, the sequence labeling model is BERT.

[0051] In an embodiment of the present invention, the method of identifying and extracting the names of each chapter from multiple fused statement sequences based on the sequence labeling algorithm and determining the corresponding starting time of each chapter includes:

[0052] For each fused statement:

[0053] The fused sentences are input into the encoder network of the sequence labeling model to capture the linguistic and temporal features of the fused sentences and generate the comprehensive features of the fused sentences.

[0054] The comprehensive features are input into the first branch network of the sequence labeling model to determine the type of the corresponding subtitle statement; wherein, the type of the subtitle statement is either a start statement or a non-start statement;

[0055] The comprehensive features are input into the second branch network of the sequence labeling model to obtain the recognition degree of each word in the corresponding subtitle sentence, and the word elements with recognition degree greater than or equal to the preset second threshold are concatenated in sequence as the corresponding chapter name;

[0056] Determine the chapter start time based on the type of subtitle text corresponding to the chapter title.

[0057] The following explanation uses the BERT sequence labeling model as an example. The last layer of the model has two branch networks, each outputting a different sequence label. For each fused sentence: it is first input into the encoder network of the sequence labeling model. By capturing the linguistic and temporal features of the fused sentence, a comprehensive feature is generated. This comprehensive feature fully integrates the semantic and time interval information of the subtitle sentence.

[0058] The generated comprehensive features are input into the first branch network of the sequence labeling model to determine whether a subtitle statement is the start or non-starting statement of a chapter. Specifically, the first branch network comprehensively analyzes temporal and linguistic features, primarily using the comprehensive features at the [CLS] position to determine whether a subtitle statement is a starting statement and obtains a probability value. A label is then generated based on this probability value. Specifically, if the probability value is greater than or equal to a preset first threshold, the subtitle statement is considered a starting statement, and label 1 is generated; otherwise, it is considered a non-starting statement, and label 0 is generated. Through the first branch network, all starting statements can be extracted from all subtitle statements in the target video, and each starting statement can be considered the start of a chapter, thus dividing the entire target video into multiple chapter videos.

[0059] Furthermore, the comprehensive features are input into the second branch network of the sequence labeling model. For each word in the same subtitle statement, its recognition score as a component of the chapter title is obtained. This score is compared with a preset second threshold. Words with a score greater than or equal to the second threshold are labeled with a 1, while those less than the threshold are labeled with a 0. All words labeled 1 in this subtitle statement are selected and concatenated in their original order to form the chapter title. It's understood that if the recognition score of all words in a subtitle statement is less than the second threshold, it means that the subtitle statement does not contain a chapter title, and the next subtitle statement can be identified. Through the second branch network, relevant chapter titles can be automatically extracted from all subtitle statements in the target video, helping users quickly understand the video content. Furthermore, after obtaining the chapter title, the start time of the current chapter can be determined based on the type of subtitle statement corresponding to that chapter title. By associating each chapter title with a specific time point in the video, subsequent segmentation processing is facilitated.

[0060] In one embodiment of the present invention, determining the chapter start time corresponding to the chapter name based on the type of subtitle statement corresponding to the chapter name includes:

[0061] If the type of subtitle statement corresponding to the chapter name has a start statement, then the start time of the target video shall be used as the chapter start time corresponding to the chapter name.

[0062] If the type of subtitle statement corresponding to the chapter name does not have a starting statement, then extract the start time of the previous starting statement that is closest to the subtitle statement from the subtitle statements arranged in chronological order, and use it as the chapter start time corresponding to the chapter name.

[0063] Since there are multiple chapters in a target video, after inputting a complete sequence of merged sentences into the sequence labeling model, multiple chapter names and start times will be generated. To establish the association between the two, it is necessary to determine the start time of the chapter corresponding to the chapter name according to the type of the subtitle sentence corresponding to the chapter name. Specifically, if there is a start sentence in the type of subtitle sentence corresponding to the chapter name, the start time of this start sentence in the target video is used as the start time of the chapter corresponding to the current chapter name, establishing the corresponding relationship between the chapter name and the chapter start time, so that the start time of the chapter corresponds to the actual start time in the target video. On the contrary, if there is no start sentence in the type of subtitle sentence corresponding to the chapter name, it is necessary to search for the previous subtitle sentence that was determined to be a start sentence and is closest to the current subtitle sentence among all subtitle sentences sorted by time, and use the start time of this start sentence in the target video as the start time of the chapter corresponding to the current chapter name. Thus, even if the type of the current subtitle sentence is not a chapter start sentence, the start time of the chapter can still be accurately determined according to the nearest chapter start point.

[0064] As an example, the string input to the sequence labeling model is "[CLS]Okay, then next we will learn the second part [CLS]which is the development overview of educational psychology. In this part [CLS]we will first introduce a case [CLS]where learners discuss who is the founder of educational psychology". For the first branch network, for each separator [CLS] of the merged sentence, it outputs the corresponding label. Assuming the label of the first separator [CLS] is 1 and the labels of the remaining separators [CLS] are 0, it means that the first subtitle sentence is a chapter start sentence, and the start time of the corresponding subtitle sentence 00:00:44,060 is used as the start time. For the second branch network, for each token, it outputs the corresponding label. Assuming that in the second subtitle sentence, the labels of the tokens "ed", "u", "ca", "tio", "n", "al", "psy", "cho", "lo", "gy", "de", "ve", "lop", "me", "nt", "ov", "er", "vi", "ew" are all 1 and the labels of the remaining tokens are 0, then these tokens are concatenated in order to obtain the chapter name "development overview of educational psychology". Since the subtitle sentence corresponding to the chapter name is not a start sentence, the nearest start sentence to it is searched, and the start time (00:00:44,060) of this subtitle sentence is associated with the chapter name as the start time of this chapter. Therefore, for the above string, the corresponding chapter name is "development overview of educational psychology" and the chapter start time is "00:00:44,060".

[0065] S15. According to each chapter name and the corresponding chapter start time, the target video is segmented into multiple chapter videos.

[0066] The start times of all chapters in the target video are sorted chronologically. The content between the start time of each chapter and the start time of the next chapter is used as the current chapter's video. This method divides a complete target video into several independent chapter videos, each corresponding to a specific chapter's content and associated with a chapter name. These are then saved to the video library for later retrieval.

[0067] In one embodiment of the present invention, after segmenting the target video into multiple chapter videos based on each chapter name and the corresponding chapter start time, the method further includes: concatenating the title and chapter name of the target video to generate combined text corresponding to the chapter name, and extracting features from the combined text to generate a semantic vector corresponding to the chapter name.

[0068] To facilitate recommendations of relevant video chapters that users may be interested in, after segmenting the target video into several chapter videos, the title of the target video and the name of each chapter are concatenated to obtain the combined text of the corresponding chapter. Word embedding technology is used to extract features from these combined texts, generating a semantic vector for each chapter name. This semantic vector contains not only the overall semantic information of the video title but also the specific content of the chapter name. A mapping relationship is established between the generated semantic vectors and information such as the corresponding chapter name, the title of the target video, the segmented chapter videos, and the chapter start time, and this mapping is saved to the video library for easy retrieval and recommendation of relevant chapter videos later.

[0069] In one embodiment of the present invention, the step of concatenating the title and chapter name of the target video to generate combined text corresponding to the chapter name, and extracting features from the combined text to generate a semantic vector corresponding to the chapter name, includes:

[0070] The title and chapter name of the target video are concatenated to generate a combined text of the chapter name;

[0071] The generated combined text is input into the word embedding model to capture the corresponding semantic features in the combined text and generate the semantic vector corresponding to the chapter name.

[0072] For each target video, the title and the name of each chapter are concatenated according to a preset format to obtain the combined text of the corresponding chapter. For example, if the title of the target video is "Solid Foundations—Student Guidance: Learning Theory," and one chapter is titled "Behaviorism Theory," then the concatenated combined text would be "《Solid Foundations—Student Guidance: Learning Theory》-Behaviorism Theory". This combined text is then input into a word embedding model to capture its semantic features, mapping the text to a high-dimensional vector to generate the semantic vector of the chapter name. The word embedding model can be any model capable of text-to-vector conversion; no specific limitation is made here. In this way, each chapter name is represented as a vector with rich semantic information, which can be used for subsequent tasks such as similarity calculation, chapter association analysis, and content recommendation.

[0073] Please see Figure 2 The present invention also provides a video recommendation method, the recommendation method comprising the following steps:

[0074] S21. Obtain videos of interest;

[0075] S22. Perform speech recognition on the video of interest to obtain multiple subtitle statements corresponding to the video of interest and their start and end times in the video of interest;

[0076] S23. Based on the start and end times of each subtitle statement in the video of interest, calculate the time interval between each subtitle statement and its preceding subtitle statement, and merge each subtitle statement and its corresponding time interval to obtain multiple merged statements arranged in sequence.

[0077] S24. Based on sequence labeling algorithms, identify and extract chapter names from multiple fused sentences;

[0078] S25. Concatenate the title and chapter name of the video of interest to generate the corresponding combined text, and extract features from the combined text to generate the corresponding semantic vector of interest;

[0079] S26. Calculate the similarity between the semantic vector of interest and each semantic vector to be recommended in the video library. Based on the similarity calculation results, select and recommend the corresponding chapter video from the video library; wherein, the videos stored in the video library are segmented using any of the segmentation methods described above.

[0080] When a user watches a video, the entire video is analyzed as a longer video of interest. Speech recognition is performed on the video of interest to extract subtitles and the start and end times of each subtitle within the entire video of interest. Based on the time information of each subtitle, the time interval between the current subtitle and the previous subtitle is calculated, and the current subtitle and the corresponding time interval are fused to generate a series of sequentially arranged fused sentences. These fused sentences are sequentially input into a sequence labeling model, and the second branch network obtains all chapter names of the entire video of interest. The title of the video of interest is concatenated with each chapter name to generate combined text for each chapter name. These combined texts are then used for feature extraction using word embedding techniques to obtain an N-dimensional semantic vector of interest for each combined text. This semantic vector of interest includes semantic information from the title of the video of interest and semantic information from the chapter names. The similarity between each semantic vector of interest and the semantic vectors of segmented videos to be recommended in the video library (i.e., the semantic vectors to be recommended) is calculated. Based on the calculation results, the recommended chapter videos corresponding to each chapter name in the video of interest are determined to achieve accurate recommendations. Furthermore, to achieve personalized recommendations, allowing users to see related recommended videos while watching a video, in one embodiment of the invention, after displaying the sorted recommended chapter videos and their chapter start times, the currently viewed chapter name can be determined from all chapter names of the identified videos of interest based on the playback duration of the videos the user is currently watching. Then, corresponding recommended chapter videos are recommended from the video library based on this chapter name. It is understood that when video segmentation is performed, the information ultimately saved in the video library is the complete video and the start times of each chapter within the video. In this case, when recommending related videos, to facilitate quick location of the desired chapter video for the user, the corresponding chapter video is selected and recommended from the video library based on similarity calculation results, and its chapter start time is displayed. This allows users to either directly click on the corresponding chapter video to watch, or quickly locate related chapters within a complete video content by their chapter start times, providing users with flexible and diverse viewing methods.

[0081] In one embodiment of the present invention, the step of calculating the similarity between the semantic vector of interest and each semantic vector to be recommended in the video library, and determining the chapter video to be recommended and its chapter start time from the video library based on the similarity calculation results, includes:

[0082] Calculate the cosine similarity between the interest semantic vector and each semantic vector to be recommended in the video library;

[0083] Determine whether the calculated cosine similarity is greater than a preset similarity threshold, and select the semantic vectors to be recommended that have a similarity greater than the similarity threshold;

[0084] From the video library, the corresponding chapter videos are found and recommended based on the selected semantic vectors to be recommended.

[0085] The cosine similarity is calculated between the semantic vector of interest and the semantic vector of each video to be recommended in the video library, which consists of the video title and chapter name. Cosine similarity measures the similarity between two vectors in the semantic space; the closer the value is to 1, the more similar the semantics. It is determined whether each calculated cosine similarity exceeds a similarity threshold (e.g., 0.5), and semantic vectors with similarities higher than the threshold are selected. The corresponding chapter videos and their start times are then filtered from the video library and used as recommended chapter videos corresponding to the current chapter name in the video of interest. It should be noted that there is a one-to-one correspondence between the semantic vector of interest and each chapter video in the video of interest. During the user's video viewing, the system can also perform real-time similarity calculations based on the currently viewed chapter using the corresponding semantic vector of interest and recommend related chapter videos. This allows for the recommendation of chapter videos highly relevant to the user's current learning content, greatly improving the user's learning experience and significantly reducing ineffective search time.

[0086] In one embodiment of the present invention, the step of calculating the similarity between the semantic vector of interest and each semantic vector to be recommended in the video library, and selecting and recommending the corresponding chapter video from the video library based on the similarity calculation results, includes:

[0087] From the video library, the corresponding chapter videos are found based on the selected semantic vectors to be recommended, and these are used as the chapter videos to be recommended.

[0088] The recommended chapter videos are sorted according to their cosine similarity.

[0089] Displays the sorted videos for each chapter to be recommended.

[0090] After retrieving relevant videos for recommended chapters from the video library, considering that a higher cosine similarity indicates a semantic similarity between the two videos, and to facilitate quick navigation for users to relevant recommended chapter videos, the videos for a specific chapter from the videos of interest are further sorted according to their similarity from highest to lowest. These sorted videos are then presented to the user on the display interface, allowing them to quickly see the recommended chapters most relevant to their currently viewed content.

[0091] As an example, the title of the video of interest is "Solid Foundations - Student Guidance: Learning Theory." Through the process described above, one of the chapter titles is extracted as "Behavioralism Theory" (chapter start time: 00:05:27,520). When a user browses the "Behavioralism Theory" section of the video "Solid Foundations - Student Guidance: Learning Theory," the cosine similarity between the semantic vector of "Behavioralism Theory" and the semantic vectors of each video to be recommended is calculated. Related recommended resources with a similarity greater than a similarity threshold (e.g., 0.5) are then selected, as shown in Table 2. These recommended resources include the video title and the corresponding chapter title. After the user clicks on these recommendations, they will be redirected to the starting playback position of the relevant knowledge point in the corresponding video.

[0092] Table 2 List of recommended chapter videos

[0093]

[0094] Please see Figure 3 The video recommendation system 100 includes: a video acquisition module 110, a text extraction module 120, a word embedding module 130, a chapter extraction module 140, and a video segmentation module 150. The video acquisition module 110 is used to acquire the target video. The text extraction module 120 is used to perform speech recognition on the target video to obtain multiple subtitle statements corresponding to the target video and their start and end times in the target video. The word embedding module 130 is used to calculate the time interval between each subtitle statement and its preceding subtitle statement based on the start and end times of each subtitle statement in the target video, and to fuse each subtitle statement and its corresponding time interval to obtain multiple fused statements arranged in sequence. The chapter extraction module 140 is used to identify and extract each chapter name from the multiple fused statements based on a sequence labeling algorithm, and to determine the corresponding chapter start time. The video segmentation module 150 is used to segment the target video into multiple chapter videos based on each chapter name and its corresponding chapter start time.

[0095] Specific limitations regarding the video recommendation system can be found in the limitations of the video recommendation method described above, and will not be repeated here. Each module in the aforementioned video recommendation system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware format within or independently of the processor in a computer device, or stored in software format in the memory of a computer device, so that the processor can invoke the corresponding operations of each module.

[0096] It should be noted that, in order to highlight the innovative aspects of this invention, this embodiment does not include modules that are not closely related to solving the technical problems proposed by this invention, but this does not mean that there are no other modules in this embodiment.

[0097] Please see Figure 4 The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a video recommendation program.

[0098] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 12 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 1. Furthermore, the memory 12 can include both internal and external storage units of the electronic device 1. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 1, such as recommended video codes, but also to temporarily store data that has been output or will be output.

[0099] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 1, connecting various components of the electronic device 1 via various interfaces and lines. It executes programs or modules (such as video recommendation programs) stored in the memory 12 and calls data stored in the memory 12 to perform various functions and process data of the electronic device 1.

[0100] The processor 13 executes the operating system of the electronic device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the recommended method for the aforementioned video.

[0101] For example, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a video acquisition module 110, a text extraction module 120, a word embedding module 130, a chapter extraction module 140, and a video segmentation module 150.

[0102] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium, which can be non-volatile or volatile. The software functional module, stored in the storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to perform some functions of the recommended video method described in the various embodiments of this application.

[0103] In summary, the video recommendation method, system, device, and medium disclosed in this invention perform speech recognition on a target video to generate multiple subtitle statements and their corresponding start and end times. The time intervals between the subtitle statements are calculated, and these time intervals are fused with the subtitle statements. The resulting fused statements possess not only semantic information but also the time interval information between adjacent statements. Since a larger time interval indicates a weaker correlation between two statements, a sequence labeling algorithm can more accurately extract relevant chapter names and start times based on the semantic and time interval information. This allows for the segmentation of a complete target video into multiple chapter videos based on these chapter names and start times. This method performs video segmentation at the chapter level, automating the construction of video chapter information, including extracting chapter names and corresponding start times. This enables more accurate video segmentation, facilitating the subsequent recommendation of highly relevant chapters to users. Furthermore, it establishes a correlation between chapters in all video data in the video library from a relevance perspective. When recommending content related to a chapter, the recommendations are sorted according to relevance, providing users with personalized and accurate video recommendations, significantly reducing search time and improving user experience. Therefore, this invention effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0104] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method of segmenting a video, characterized by, The segmentation method comprises: acquiring a target video; performing speech recognition on the target video to obtain a plurality of subtitle sentences corresponding to the target video and starting and ending time points of the subtitle sentences in the target video; calculating time intervals between each subtitle sentence and its previous subtitle sentence according to the starting and ending time points of each subtitle sentence in the target video, and fusing each subtitle sentence and the corresponding time interval to obtain a plurality of fusion sentences arranged in sequence; based on a sequence labeling algorithm, identifying and extracting each chapter name from the plurality of fusion sentences, and determining the corresponding chapter starting time point; segmenting the target video into a plurality of chapter videos according to each chapter name and the corresponding chapter starting time point; based on a sequence labeling algorithm, identifying and extracting each chapter name from the plurality of fusion sentences, and determining the corresponding chapter starting time point, comprising: for each fusion sentence: input the fusion sentence into the encoder network of the sequence labeling model to capture the language features and time features of the fusion sentence, and generate comprehensive features of the fusion sentence; input the comprehensive features into the first branch network of the sequence labeling model to determine the type of the corresponding subtitle sentence; wherein the type of the subtitle sentence is a starting sentence or a non-starting sentence; input the comprehensive features into the second branch network of the sequence labeling model to obtain the recognition degree of each word element in the corresponding subtitle sentence, and sequentially splice the word elements with a recognition degree greater than or equal to a preset second threshold to obtain the corresponding chapter name; determine the chapter starting time point corresponding to the chapter name according to the type of the subtitle sentence corresponding to the chapter name.

2. The video segmentation method of claim 1, wherein, The speech recognition on the target video to obtain a plurality of subtitle sentences corresponding to the target video and starting and ending time points of the subtitle sentences in the target video, comprising: extracting audio information in the target video; performing speech recognition on the audio information to convert the audio information into text information; based on a voice activity detection algorithm, performing sentence breaking processing on the text information to obtain a plurality of subtitle sentences in the target video, and determining the starting and ending time points of each subtitle sentence in the target video.

3. The method of claim 1, wherein, The calculation of the time interval between each subtitle sentence and its previous subtitle sentence according to the starting and ending time points of each subtitle sentence in the target video, and the fusion of each subtitle sentence and the corresponding time interval to obtain a plurality of fusion sentences arranged in sequence, comprising: based on the starting time point of the subtitle sentence and the ending time point of its previous subtitle sentence, calculating the time interval between the subtitle sentence and its previous subtitle sentence, and performing word embedding processing on the time interval; performing word segmentation and word embedding processing on the subtitle sentence to obtain a word embedding vector sequence of the subtitle sentence; fuse the word embedding time interval and the word embedding vector sequence of the subtitle sentence to generate a fusion sentence.

4. The method of claim 1, wherein, The determination of the chapter starting time point corresponding to the chapter name according to the type of the subtitle sentence corresponding to the chapter name, comprising: If the type of the subtitle sentence corresponding to the chapter name exists a start sentence, the start sentence is taken as the chapter start time corresponding to the chapter name at the start time of the target video; If the type of the subtitle sentence corresponding to the chapter name does not exist a start sentence, the start time of the previous start sentence closest to the subtitle sentence is extracted from the subtitle sentences arranged in time sequence, and the start time is taken as the chapter start time corresponding to the chapter name.

5. The method of claim 1, wherein After the target video is divided into multiple chapter videos according to each chapter name and the corresponding chapter start time, the method further comprises: splicing the title of the target video and the chapter name to generate a combined text corresponding to the chapter name, and performing feature extraction on the combined text to generate a semantic vector corresponding to the chapter name.

6. The method of claim 5, wherein, The method of splicing the title of the target video and the chapter name of the target video to generate a combined text corresponding to the chapter name, and performing feature extraction on the combined text to generate a semantic vector corresponding to the chapter name, comprises: Splicing the title of the target video and the chapter name of the target video to generate a combined text of the chapter name; Inputting the generated combined text into a word embedding model to capture the corresponding semantic features in the combined text, and generating a semantic vector corresponding to the chapter name.

7. A method of recommending a video, characterized by, The recommendation method comprises: Obtaining a video of interest; Performing voice recognition on the video of interest to obtain multiple subtitle sentences corresponding to the video of interest and the start and end times of the subtitle sentences in the video of interest; According to the start and end times of each subtitle sentence in the video of interest, calculating the time interval between each subtitle sentence and the previous subtitle sentence, and fusing each subtitle sentence and the corresponding time interval to obtain multiple fusion sentences arranged in sequence; Based on a sequence labeling algorithm, chapter names are identified and extracted from the multiple fusion sentences; Splicing the title of the video of interest and the chapter name to generate a corresponding combined text, and performing feature extraction on the combined text to generate a corresponding semantic vector of interest; Calculating the similarity between the semantic vector of interest and each semantic vector to be recommended in the video library, and selecting and recommending the corresponding chapter video from the video library based on the similarity calculation result; wherein the videos saved in the video library are divided by the division method of any one of claims 1-6; Based on a sequence labeling algorithm, each chapter name is identified and extracted from the multiple fusion sentence sequences, comprising: For each fusion sentence: Input the fusion sentence into the encoder network of the sequence labeling model to capture the language features and time features of the fusion sentence, and generate comprehensive features of the fusion sentence; Input the comprehensive features into the first branch network of the sequence labeling model to determine the type of the corresponding subtitle sentence; wherein the type of the subtitle sentence is a start sentence or a non-start sentence; Input the comprehensive features into the second branch network of the sequence labeling model to obtain the recognition degree of each word element in the corresponding subtitle sentence, and sequentially splice the word elements with a recognition degree greater than or equal to a preset second threshold to obtain the corresponding chapter name. 8.The method of claim 7, wherein, The similarity of the semantic vector of interest and each to-be-recommended semantic vector in the video library is calculated respectively, and corresponding chapter videos are selected and recommended from the video library based on the similarity calculation results, including: The cosine similarity of the semantic vector of interest and each to-be-recommended semantic vector in the video library is calculated respectively; It is judged whether the calculated cosine similarity is greater than a preset similarity threshold, and the to-be-recommended semantic vector corresponding to the similarity greater than the similarity threshold is selected; According to the selected to-be-recommended semantic vector, corresponding chapter videos are searched from the video library for recommendation. 9.The method of claim 8, wherein, The similarity of the semantic vector of interest and each to-be-recommended semantic vector in the video library is calculated respectively, and corresponding chapter videos are selected and recommended from the video library based on the similarity calculation results, including: According to the selected to-be-recommended semantic vector, corresponding chapter videos are searched from the video library for recommendation. The to-be-recommended chapter videos are sorted according to their respective cosine similarities. The sorted to-be-recommended chapter videos are displayed.

10. A system for segmenting a video, characterized by The system comprises: A video acquisition module is configured to acquire a target video. A text extraction module is configured to perform voice recognition on the target video to obtain a plurality of subtitle sentences corresponding to the target video and starting and ending time points of the subtitle sentences in the target video. A word embedding module is configured to calculate a time interval between each subtitle sentence and a previous subtitle sentence according to the starting and ending time points of each subtitle sentence in the target video, and fuse each subtitle sentence and the corresponding time interval to obtain a plurality of fused sentences arranged in sequence. A chapter extraction module is configured to identify and extract each chapter name from the plurality of fused sentences based on a sequence labeling algorithm, and determine a corresponding chapter starting time point. A video segmentation module is configured to segment the target video into a plurality of chapter videos according to each chapter name and the corresponding chapter starting time point. The system comprises: For each fused sentence: The fused sentence is input into an encoder network of a sequence labeling model to capture language features and time features of the fused sentence, and generate comprehensive features of the fused sentence. The comprehensive features are input into a first branch network of the sequence labeling model to determine the type of the corresponding subtitle sentence; wherein the type of the subtitle sentence is a starting sentence or a non-starting sentence. The comprehensive features are input into a second branch network of the sequence labeling model to obtain the recognition degree of each word element in the corresponding subtitle sentence, and the word elements with a recognition degree greater than or equal to a preset second threshold are sequentially spliced to obtain the corresponding chapter name. The chapter starting time point corresponding to the chapter name is determined according to the type of the subtitle sentence corresponding to the chapter name.

11. An electronic device, comprising: The electronic device comprises: One or more processors; A storage device is configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the video segmentation method of any one of claims 1 to 6 or the video recommendation method of any one of claims 7 to 9.

12. A computer-readable storage medium, characterized in that, A computer program product, comprising a computer readable medium having stored thereon a computer program, the computer program comprising instructions which, when executed on at least one processor of a computer, cause the computer to carry out the method according to any one of claims 1 to 6 or the method according to any one of claims 7 to 9.

Images