Video segmentation method, recommendation method, system, equipment and medium

Abstract

The invention relates to a video segmentation method, a video recommendation method, a video segmentation system, video recommendation equipment and a medium. The segmentation method comprises the steps of obtaining a target video; performing voice recognition on the target video to obtain a plurality of subtitle statements corresponding to the target video and starting and ending moments of the subtitle statements in the target video; according to the starting moment and the ending moment of each subtitle statement in the target video, calculating a time interval between each subtitle statement and a previous subtitle statement, and fusing each subtitle statement and the corresponding time interval to obtain a plurality of fusion statements arranged in sequence; based on a sequence labeling algorithm, identifying and extracting each chapter name from the plurality of fused statements, and determining a corresponding chapter starting moment; and according to each chapter name and the corresponding chapter starting moment, segmenting the target video into a plurality of chapter videos. According to the invention, the accuracy of video segmentation is improved.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video segmentation method, a recommendation method, a system, a device and a medium. Background Art

[0002] With the increasing richness of video content and the diversification of user needs, how to accurately segment long video content into short video clips with independent semantics has become an issue of increasing concern. In many cases, users tend to prefer videos with specific knowledge points rather than a whole long video. In the prior art, videos are usually split according to the theme, overall label or time tag of the video, but this method cannot achieve fine video segmentation. Therefore, it is necessary to provide a video segmentation method, recommendation method, system, device and medium. Summary of the invention

[0003] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a video segmentation method, a recommendation method, a system, a device and a medium, which improve the problem of low precision of video segmentation in the prior art.

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention provides a video segmentation method, which includes: obtaining a target video; performing speech recognition on the target video to obtain multiple subtitle sentences corresponding to the target video and their start and end times in the target video; calculating the time interval between each subtitle sentence and its previous subtitle sentence according to the start and end times of each subtitle sentence in the target video, and fusing each subtitle sentence and the corresponding time interval to obtain a plurality of fused sentences arranged in sequence; based on a sequence labeling algorithm, identifying and extracting each chapter name from the plurality of fused sentences, and determining the corresponding chapter start time; and segmenting the target video into a plurality of chapter videos according to each chapter name and the corresponding chapter start time.

[0005] In one embodiment of the present invention, performing speech recognition on the target video to obtain multiple subtitle sentences corresponding to the target video and their start and end times in the target video includes: extracting audio information in the target video; performing speech recognition on the audio information to convert the audio information into text information; performing sentence segmentation processing on the text information based on a voice activity detection algorithm to obtain multiple subtitle sentences in the target video, and determining the start and end time of each subtitle sentence in the target video.

[0006] In one embodiment of the present invention, the time interval between each subtitle sentence and its previous subtitle sentence is calculated according to the start and end time of each subtitle sentence in the target video, and each subtitle sentence and the corresponding time interval are merged to obtain a plurality of fused sentences arranged in sequence, including: based on the start time of the subtitle sentence and the end time of the previous subtitle sentence, the time interval between the subtitle sentence and its previous subtitle sentence is calculated, and the time interval is word-embedded; the subtitle sentence is word-embedded and word-embedded to obtain a word-embedded vector sequence of the subtitle sentence; the time interval after word embedding and the word-embedded vector sequence of the subtitle sentence are merged to generate a fused sentence.

[0007] In one embodiment of the present invention, the sequence labeling algorithm is based on identifying and extracting each chapter name from multiple fused sentence sequences, and determining the corresponding chapter start time, including: for each fused sentence: inputting the fused sentence into the encoder network of the sequence labeling model, capturing the language features and time features of the fused sentence, and generating a comprehensive feature of the fused sentence; inputting the comprehensive feature into the first branch network of the sequence labeling model to determine the type of the corresponding subtitle sentence; wherein the type of the subtitle sentence is a start sentence or a non-start sentence; inputting the comprehensive feature into the second branch network of the sequence labeling model to obtain the recognition degree of each word unit in the corresponding subtitle sentence, and sequentially splicing the word units with a recognition degree greater than or equal to a preset second threshold as the corresponding chapter name; and determining the chapter start time corresponding to the chapter name according to the type of the subtitle sentence corresponding to the chapter name.

[0008] In one embodiment of the present invention, determining the chapter start time corresponding to the chapter name according to the type of the subtitle sentence corresponding to the chapter name includes: if the type of the subtitle sentence corresponding to the chapter name has a start sentence, then using the start time of the target video as the chapter start time corresponding to the chapter name; if the type of the subtitle sentence corresponding to the chapter name does not have a start sentence, then extracting the start time of the previous start sentence that is closest to the subtitle sentence from the subtitle sentences arranged in chronological order, and using it as the chapter start time corresponding to the chapter name.

[0009] In one embodiment of the present invention, after the target video is divided into multiple chapter videos based on the chapter names and the corresponding chapter start times, it also includes: splicing the title and chapter names of the target video to generate a combined text corresponding to the chapter names, and performing feature extraction on the combined text to generate a semantic vector corresponding to the chapter names.

[0010] In one embodiment of the present invention, the title and chapter name of the target video are concatenated to generate a combined text corresponding to the chapter name, and features are extracted from the combined text to generate a semantic vector corresponding to the chapter name, including: concatenating the title of the target video and the chapter name of the target video to generate a combined text of the chapter name; inputting the generated combined text into a word embedding model to capture the corresponding semantic features in the combined text and generate a semantic vector corresponding to the chapter name.

[0011] In one embodiment of the present invention, a video recommendation method is also provided, the recommendation method comprising: obtaining an interested video; performing speech recognition on the interested video to obtain multiple subtitle sentences corresponding to the interested video and their start and end times in the interested video; calculating the time interval between each subtitle sentence and its previous subtitle sentence according to the start and end times of each subtitle sentence in the interested video, and fusing each subtitle sentence and the corresponding time interval to obtain a plurality of fused sentences arranged in sequence; identifying and extracting chapter names from the plurality of fused sentences based on a sequence labeling algorithm; concatenating the title and chapter name of the interested video to generate a corresponding combined text, and performing feature extraction on the combined text to generate a corresponding interested semantic vector; performing similarity calculation on the interested semantic vector and each semantic vector to be recommended in a video library, and selecting and recommending the corresponding chapter video from the video library based on the similarity calculation results; wherein the video stored in the video library is segmented by any of the segmentation methods described above.

[0012] In one embodiment of the present invention, similarity calculation is performed on the semantic vector of interest and each semantic vector to be recommended in the video library, and the chapter video to be recommended and the starting time of the chapter are determined from the video library based on the similarity calculation result, including: respectively calculating the cosine similarity of the semantic vector of interest and each semantic vector to be recommended in the video library; judging whether the calculated cosine similarity is greater than a preset similarity threshold, and selecting the semantic vector to be recommended whose similarity is greater than the similarity threshold; from the video library, searching for the corresponding chapter video according to the selected semantic vector to be recommended for recommendation.

[0013] In one embodiment of the present invention, the method of searching for corresponding chapter videos from the video library according to the selected semantic vector to be recommended for recommendation includes: searching for corresponding chapter videos from the video library according to the selected semantic vector to be recommended, and using them as chapter videos to be recommended; sorting the chapter videos to be recommended according to their respective cosine similarities; and displaying the sorted chapter videos to be recommended.

[0014] In one embodiment of the present invention, a video recommendation system is also provided, the system comprising: a video acquisition module, used to acquire a target video; a text extraction module, used to perform speech recognition on the target video, and obtain multiple subtitle sentences corresponding to the target video and their start and end times in the target video; a word embedding module, used to calculate the time interval between each subtitle sentence and its previous subtitle sentence according to the start and end times of each subtitle sentence in the target video, and to fuse each subtitle sentence and the corresponding time interval to obtain a plurality of fused sentences arranged in sequence; a chapter extraction module, used to identify and extract each chapter name from the plurality of fused sentences based on a sequence labeling algorithm, and to determine the corresponding chapter start time; a video segmentation module, used to segment the target video into a plurality of chapter videos according to each chapter name and the corresponding chapter start time.

[0015] In one embodiment of the present invention, an electronic device is also provided, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements any of the above-mentioned video segmentation methods or video recommendation methods.

[0016] In one embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes any of the above-mentioned video segmentation methods or video recommendation methods.

[0017] As described above, a video segmentation method, recommendation method, system, device and medium of the present invention have the following beneficial effects: by performing speech recognition on the target video, a plurality of subtitle sentences and their corresponding start and end times are generated, and the time interval between the subtitle sentences is calculated, and the time interval is fused with the subtitle sentences, and the obtained fused sentence has not only semantic information, but also time interval information between adjacent sentences. Since the larger the time interval, the smaller the association between the two sentences, the sequence labeling algorithm can be used to more accurately extract the relevant chapter names and start times according to the semantic information and time interval information, so that a complete target video is segmented according to these chapter names and start times, and divided into multiple chapter videos. This method of the present invention performs video segmentation from the chapter granularity, improves the accuracy of video segmentation, and realizes the automatic construction of video chapter information, including extracting the chapter names and corresponding start time points in the video. Thereby, more accurate video segmentation can be achieved, which is convenient for users to push chapters that are highly relevant to the videos they watch. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1A schematic flow chart of a video segmentation method provided by an embodiment of the present invention;

[0019] Figure 2 A flowchart of a video recommendation method provided by an embodiment of the present invention

[0020] Figure 3 Shown is a structural block diagram of a video segmentation system provided by an embodiment of the present invention;

[0021] Figure 4 Shown is a structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0023] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0024] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0025] Existing video recommendation technologies mainly recommend content from the overall level of the video. For example, when a user clicks on a teaching video about "deep learning" on an online learning platform, the platform will usually recommend other speakers who may be interested in "deep learning"-related videos to the user based on the user's information and resource data on the platform. In fact, this method is limited to the overall recommendation of the video, and ignores the chapter details within the long video, especially for those long videos, whose content can generally be divided into multiple chapter segments.

[0026] The inventors found that long videos can be split from a more fine-grained level to form multiple chapter videos. For example, in a video titled "Automobile Engine Structure and Maintenance", there is a chapter called "Ignition Control System". In another video titled "Automobile Electrical and Electronic Equipment", there is a chapter called "Working Principle of Ignition System". There is a certain correlation between the two chapters. After watching one of the videos, the user may want to understand this chapter more comprehensively. However, the prior art lacks the refined segmentation and analysis of these chapters within the video, so it is impossible to actively recommend other chapter videos related to specific chapters to users. Existing video recommendation technologies are mainly based on global features such as video categories, tags, descriptions, author information, etc. for recommendation, while ignoring the hierarchy of chapters within the video. Therefore, when recommending videos, it is limited to recommending the entire video to users, without realizing the segmentation of the video content, decomposing the long video into chapters or independent segments and recommending them. Therefore, this overall recommendation method is not suitable for meeting the user's precise needs for a specific chapter. In summary, the existing technology fails to perform fine-grained segmentation and content analysis on the video based on chapters when segmenting and recommending videos, and fails to actively recommend other chapter videos related to the chapter that the user is currently interested in. When watching a video, the user may be interested in a specific chapter in the video, but the existing technology cannot recommend other related video clips containing the chapter to the user from the perspective of video segmentation.

[0027] The present invention provides a video recommendation method, which generates multiple subtitle sentences and their corresponding start and end times by performing speech recognition on a target video, and calculates the time interval between the subtitle sentences, and fuses the time interval with the subtitle sentences. The obtained fused sentence has not only semantic information, but also time interval information between adjacent sentences. Since the larger the time interval, the smaller the association between the two sentences, the sequence labeling algorithm can more accurately extract the relevant chapter names and start times according to the semantic information and the time interval information, so as to segment a complete target video according to these chapter names and start times, and divide it into multiple chapter videos. The method of the present invention performs video segmentation from the chapter granularity, realizes the automatic construction of the chapter information of the video, including extracting the chapter name and the corresponding start time point in the video. Thereby, more accurate video segmentation can be achieved, which is convenient for the subsequent push of chapters highly related to the video watched by the user.

[0028] See also Figure 1 ,The video recommendation method includes the following steps:

[0029] S11. Obtain target video.

[0030] The target video refers to a complete video that needs to be split and processed. There may be multiple chapters in the target video. The method described in the present invention can split a long complete long video into several chapter videos for users to choose. The target video can be obtained from a specified storage location, a video library or some online resource platform, and the target video can be format parsed to ensure the consistency of data during subsequent processing. Furthermore, if the format of the target video does not belong to the preset type, it can be format converted and other processing to convert the target video into a processable format.

[0031] S12: Perform speech recognition on the target video to obtain multiple subtitle sentences corresponding to the target video and their start and end times in the target video.

[0032] The audio track is separated from the target video to generate an audio file. The audio file is processed by speech recognition technology, converted into corresponding subtitle sentences, and the timestamps of the start and end times in the target video are generated for each subtitle sentence.

[0033] In one embodiment of the present invention, performing speech recognition on the target video to obtain a plurality of subtitle sentences corresponding to the target video and their start and end times in the target video includes:

[0034] Extracting audio information from the target video;

[0035] Performing speech recognition on the audio information to convert the audio information into text information;

[0036] The text information is segmented based on a voice activity detection algorithm to obtain a plurality of subtitle sentences in the target video, and the start and end time of each subtitle sentence in the target video is determined.

[0037] In order to extract subtitle sentences from the target video, the target video needs to be parsed first. The corresponding audio information can be extracted from the target video using video processing tools. The audio information is processed using speech recognition to convert it into text information. As an example, the extracted audio information can be input into a pre-trained language model, and the complete text information corresponding to the target video is generated by analyzing the audio information frame by frame. The text information consists of multiple continuous subtitle sentences arranged in sequence according to the order in which they appear in the video. Based on the Voice Activity Detection (VAD) algorithm, the start and stop times of the voice activity in the audio information are detected, so that the entire complete text information is divided into several subtitle sentences, and the timestamps of the start and end times of each subtitle sentence in the video are generated. The generated multiple subtitle sentences are arranged according to the order in which they appear in the target video, thereby forming a subtitle sentence sequence. It can be understood that the subtitle sentence sequence is presented in the form of a string, which is formed by splicing all the subtitle sentences obtained above according to the time sequence, and the adjacent subtitle sentences are separated by a separator [CLS] to distinguish different sentences. The subtitle sentence sequence includes each subtitle sentence arranged in sequence in the target video and the corresponding start and end times.

[0038] S13, calculating the time interval between each subtitle sentence and its previous subtitle sentence according to the start and end time of each subtitle sentence in the target video, and fusing each subtitle sentence and the corresponding time interval to obtain a plurality of fused sentences arranged in sequence.

[0039] Considering that the time interval between adjacent subtitle sentences can reflect the pause time between the two sentences, it reflects the semantic coherence of adjacent sentences, which can assist in the subsequent chapter division. Therefore, after obtaining the subtitle sentence sequence in the above manner, the time interval between the current subtitle sentence and the previous subtitle sentence is calculated one by one, and the time interval is associated with the current subtitle sentence. By fusion, each subtitle sentence and the corresponding time interval are merged into a complete whole as the fused sentence corresponding to the subtitle sentence.

[0040] In one embodiment of the present invention, the time interval between each subtitle sentence and its previous subtitle sentence is calculated according to the start and end time of each subtitle sentence in the target video, and each subtitle sentence and the corresponding time interval are merged to obtain a plurality of merged sentences arranged in sequence, including:

[0041] Based on the start time of the subtitle sentence and the end time of the previous subtitle sentence, calculate the time interval between the subtitle sentence and the previous subtitle sentence, and perform word embedding processing on the time interval;

[0042] Perform word unit segmentation and word embedding on the subtitle sentence to obtain a word unit embedding vector sequence of the subtitle sentence;

[0043] The time interval after word embedding and the word element embedding vector sequence of the subtitle sentence are fused to generate a fused sentence.

[0044] The difference between the start time of each subtitle sentence and the end time of the previous subtitle sentence is calculated to obtain the time interval between the two subtitle sentences, and the time interval is associated with the current subtitle sentence. Considering that the larger the time interval, the lower the content coherence between the two adjacent sentences may be, indicating that the current subtitle sentence may represent the beginning of a new chapter. In order to allow the model to understand the content of the time interval, the time interval is first processed by word embedding. Specifically, the time interval can be normalized, for example, the interval is scaled to between 0 and 1, and the normalized time interval is mapped to a high-dimensional vector T of a preset dimension through an embedding layer. In addition, each subtitle sentence is also subjected to word element segmentation, and a complete sentence is split into several basic word elements (tokens) arranged according to the order. Among them, word elements are text data divided into smaller units, and the division granularity of word elements includes character granularity, word granularity, and subword granularity, etc., which can be adaptively selected by technicians in this field based on the training needs of the model. All word elements of each subtitle sentence are processed by word embedding through a pre-trained language model to obtain a word element embedding vector sequence of the corresponding subtitle sentence; wherein each word element embedding vector corresponds to a word element. The time interval after word embedding is fused with the word element embedding vector of the corresponding subtitle sentence to obtain the fused sentence of the subtitle sentence. All fused sentences are arranged in order to form a fused sentence sequence.

[0045] As one example, for a certain target video, the subtitle sentences and the corresponding start and end times are obtained as shown in Table 1:

[0046] Table 1 Example table of subtitle sentences and corresponding time

[0047] [CLS]Okay, let’s study the second part. Start time: 00:00:44,060, End time: 00:00:46,565 [CLS] is an overview of the development of educational psychology. Start time: 00:00:47,420, End time: 00:00:52,035 [CLS] Let us first introduce a case Start time: 00:00:52,890, End time: 00:00:59,355 [CLS] Then it is the learners who discuss who is the founder of educational psychology Start time: 00:01:00,020, End time: 00:01:02,275

[0048] For the time interval between the first statement and the second statement, it is the difference between the start time (00:00:47,420) of the second statement and the end time (00:00:46,565) of the first statement, which is 855 milliseconds. Perform token segmentation on the second statement to generate the result "[CLS]-就-是-教-育-心-理-学-的-发-展-概-况-,-在-这-一-部-分", obtaining 19 tokens. Use a pre-trained language model to perform word embedding processing on these tokens to obtain the token embedding vector sequence [E0, E1, …, E18] of this subtitle statement, where E0 is the token embedding vector of "[CLS]" and E18 is the token embedding vector of "分". Fuse the embedding vector T of the time interval and E0 by concatenation to obtain the fused embedding vector E0, = [E0, T]. Since the time interval between tokens in the same subtitle statement is set to zero, the token embedding vectors at other positions can be represented as Ei, = [Ei, 0]. In this way, the fused statement of this subtitle statement can be obtained as [E0,, E1, …, E18,]. Among them, E0, contains both time interval information and semantic information.

[0049] S14. Based on the sequence labeling algorithm, identify and extract each chapter name from multiple fused statements, and determine the corresponding chapter start time.

[0050] Input the fused statement sequence fused with semantic and time interval information into a fine-tuned pre-trained double-branch sequence labeling model. By extracting the features of each statement, and through the first branch, determine whether the current subtitle statement is the start statement of a chapter. If so, mark the start time of this subtitle statement as the start time of the corresponding chapter. Further, also label the tokens in the subtitle statement through the second branch, so as to identify which tokens constitute the chapter name. In this way, each chapter name and the corresponding start time of the target video can be extracted for video segmentation. It can be understood that the sequence labeling model can be based on various neural network architectures, including but not limited to the LSTM-based model and the Transformer architecture-based model such as BERT, as long as it can play the role of sequence labeling. Preferably, in order to improve the recognition accuracy of the model, the sequence labeling model is BERT.

[0051] In an embodiment of the present invention, the method of identifying and extracting each chapter name from multiple fused statement sequences based on the sequence labeling algorithm and determining the corresponding chapter start time includes:

[0052] For each fused statement:

[0053] Input the fused sentence into the encoder network of the sequence labeling model to capture the language features and time features of the fused sentence and generate the comprehensive features of the fused sentence;

[0054] Inputting the comprehensive feature into the first branch network of the sequence labeling model to determine the type of the corresponding subtitle sentence; wherein the type of the subtitle sentence is an initial sentence or a non-initial sentence;

[0055] Inputting the comprehensive features into the second branch network of the sequence labeling model to obtain the recognition degree of each word in the corresponding subtitle sentence, and sequentially splicing the words whose recognition degree is greater than or equal to a preset second threshold value as the corresponding chapter name;

[0056] According to the type of the subtitle sentence corresponding to the chapter name, the chapter start time corresponding to the chapter name is determined.

[0057] The following uses the BERT model as an example to illustrate the sequence labeling model. The last layer of the model has two branch networks, each of which outputs a sequence label. For each fused sentence: First, it is input into the encoder network of the sequence labeling model, and a comprehensive feature is generated by capturing the language features and time features of the fused sentence. This comprehensive feature fully integrates the semantic information and time interval information of the subtitle sentence.

[0058] The generated comprehensive features are input into the first branch network of the sequence labeling model, and it can be determined whether the subtitle sentence is the starting sentence or non-starting sentence of the chapter. Specifically, the first branch network comprehensively analyzes the time features and language features, mainly through the comprehensive features of the separator [CLS] position to determine whether the subtitle sentence is the starting sentence, and obtains a probability value, and generates a label according to the probability value. Specifically, when the probability value is greater than or equal to the preset first threshold, the subtitle sentence is considered to be a starting sentence, and a label 1 is generated; otherwise, the subtitle sentence is considered to be a non-starting sentence, and a label 0 is generated. Through the first branch network, all starting sentences can be extracted from all subtitle sentences of the target video, and each starting sentence can be regarded as the beginning of a chapter, so that the entire target video can be divided into multiple chapter videos.

[0059] Further, the comprehensive features are input into the second branch network of the sequence labeling model, and for each word in the same subtitle sentence, the recognition degree of the word as a component of the chapter name is obtained, and the recognition degree is compared with the preset second threshold value, and the label corresponding to the word greater than or equal to the second threshold value is marked as 1, otherwise, the label corresponding to the word less than the second threshold value is marked as 0. All the words with a label of 1 in this subtitle sentence are selected, and spliced ​​according to the original order to form the chapter name of the chapter. It can be understood that if the recognition degree of all words in the subtitle sentence is less than the second threshold value, it means that the subtitle sentence does not contain the chapter name, and the next subtitle sentence can be identified. Through the second branch network, the relevant chapter names can be automatically extracted from all subtitle sentences of the target video, thereby helping users to quickly understand the video content. Further, after obtaining the chapter name, according to the type of subtitle sentence corresponding to the chapter name, the chapter start time corresponding to the current chapter name can be determined. By associating each chapter name with a specific time point in the video, it is convenient for subsequent splitting processing.

[0060] In one embodiment of the present invention, determining the chapter start time corresponding to the chapter name according to the type of the subtitle sentence corresponding to the chapter name includes:

[0061] If the type of subtitle sentence corresponding to the chapter name has a start sentence, use the start sentence as the chapter start time corresponding to the chapter name at the start time of the target video;

[0062] If there is no start sentence for the type of subtitle sentence corresponding to the chapter name, the start time of the previous start sentence closest to the subtitle sentence is extracted from the subtitle sentences arranged in chronological order, and is used as the chapter start time corresponding to the chapter name.

[0063] Since there are multiple chapters in a target video, after inputting a complete sequence of merged sentences into the sequence labeling model, multiple chapter names and start times will be generated. To establish the association between the two, it is necessary to determine the start time of the chapter corresponding to the chapter name according to the type of subtitle sentence corresponding to the chapter name. Specifically, if there is a start sentence in the type of subtitle sentence corresponding to the chapter name, the start time of this start sentence in the target video is used as the start time of the chapter corresponding to the current chapter name, establishing the corresponding relationship between the chapter name and the chapter start time, so that the start time of the chapter corresponds to the actual start time in the target video. Conversely, if there is no start sentence in the type of subtitle sentence corresponding to the chapter name, it is necessary to find the previous subtitle sentence that is judged as a start sentence and is closest to the current subtitle sentence from all subtitle sentences sorted by time, and use the start time of this start sentence in the target video as the start time of the chapter corresponding to the current chapter name. Thus, even if the type of the current subtitle sentence is not a chapter start sentence, the start time of the chapter can still be accurately determined according to the nearest chapter start point.

[0064] As an example, the string input to the sequence labeling model is "[CLS]Okay, then next we will learn the second part [CLS]which is the development overview of educational psychology. In this part [CLS]we will first introduce a case [CLS]where learners discuss who is the founder of educational psychology". For the first branch network, for each separator [CLS] of the merged sentence, it outputs the corresponding label. Assuming the label of the first separator [CLS] is 1 and the labels of the remaining separators [CLS] are 0, it means that the first subtitle sentence is the chapter start sentence, and the start time 00:00:44,060 of the corresponding subtitle sentence is used as the start time. For the second branch network, for each token, it outputs the corresponding label. Assuming that in the second subtitle sentence, the labels of the tokens "ed", "u", "ca", "tio", "nal", "psy", "chol", "ogy", "de", "vel", "opm", "ent", "ove", "rvi", "ew" are all 1 and the labels of the remaining tokens are 0, then these tokens are concatenated in order to obtain the chapter name "the development overview of educational psychology". Since the subtitle sentence corresponding to the chapter name is not a start sentence, the nearest start sentence to it is found, and the start time (00:00:44,060) of this subtitle sentence is associated with the chapter name as the start time of this chapter. Therefore, for the above string, the corresponding chapter name is "the development overview of educational psychology" and the chapter start time is "00:00:44,060".

[0065] S15. According to each chapter name and the corresponding chapter start time, segment the target video into multiple chapter videos.

[0066] Sort all chapter start times in the target video in chronological order, and use the content between the chapter start time of each chapter and the chapter start time of the next chapter as the chapter video of the current chapter. In this way, a complete target video is divided into several independent chapter videos, each of which corresponds to a specific chapter content and the chapter name, and is saved in the video library for subsequent use.

[0067] In one embodiment of the present invention, after dividing the target video into multiple chapter videos based on the chapter names and the corresponding chapter start times, it also includes: splicing the title and chapter names of the target video to generate a combined text corresponding to the chapter names, and performing feature extraction on the combined text to generate a semantic vector corresponding to the chapter names.

[0068] In order to recommend relevant chapter videos of interest to users, after the target video is divided into several chapter videos, the title of the target video and each chapter name are concatenated to obtain the combined text of the corresponding chapter. The word embedding technology is used to extract features from these combined texts and generate semantic vectors corresponding to each chapter name. Among them, the semantic vector contains not only the overall semantic information of the video title, but also the specific content of the chapter name. A mapping relationship is established between the generated semantic vector and the corresponding chapter name, the title of the target video, the segmented chapter video, and the chapter start time, and the information is saved in the video library, so as to facilitate the subsequent rapid call and recommendation of the relevant chapter videos.

[0069] In one embodiment of the present invention, the title and chapter name of the target video are concatenated to generate a combined text corresponding to the chapter name, and features are extracted from the combined text to generate a semantic vector corresponding to the chapter name, including:

[0070] Concatenate the title of the target video and the chapter name of the target video to generate a combined text of the chapter name;

[0071] The generated combined text is input into the word embedding model to capture the corresponding semantic features in the combined text and generate the semantic vector corresponding to the chapter name.

[0072] For each target video, the title of the target video and each chapter name of the target video are spliced ​​according to the preset format requirements to obtain the combined text of the corresponding chapter. For example, the title of the target video is "Solid Foundation-Student Guidance: Learning Theory", and one of the chapter names is: Behaviorism Theory, then the combined text after splicing can be ""Solid Foundation-Student Guidance: Learning Theory"-Behaviorism Theory". The combined text is input into the word embedding model to capture the semantic features in the combined text, and the text is mapped into a high-dimensional vector to generate a semantic vector for the chapter name. Among them, the word embedding model can be any model that can realize text-vector conversion, and is not limited here. In this way, each chapter name is represented as a vector with rich semantic information, which can be used for subsequent tasks such as similarity calculation, chapter association analysis, and content recommendation.

[0073] See also Figure 2 The present invention also provides a video recommendation method, which includes the following process:

[0074] S21, obtaining a video of interest;

[0075] S22, performing speech recognition on the video of interest to obtain a plurality of subtitle sentences corresponding to the video of interest and their start and end times in the video of interest;

[0076] S23, calculating the time interval between each subtitle sentence and its previous subtitle sentence according to the start and end time of each subtitle sentence in the video of interest, and fusing each subtitle sentence and the corresponding time interval to obtain a plurality of fused sentences arranged in sequence;

[0077] S24, based on a sequence labeling algorithm, identifying and extracting chapter names from multiple fused sentences;

[0078] S25, concatenating the title and chapter name of the video of interest to generate a corresponding combined text, and performing feature extraction on the combined text to generate a corresponding semantic vector of interest;

[0079] S26, calculating similarity between the semantic vector of interest and each semantic vector to be recommended in the video library, and based on the similarity calculation results, selecting and recommending corresponding chapter videos from the video library; wherein the videos stored in the video library are segmented using any of the segmentation methods described above.

[0080] When a user watches a video, the complete video is analyzed as a longer video of interest. Speech recognition is performed on the video of interest to extract the subtitle sentences and the start and end times of each subtitle sentence in the entire video of interest. According to the time information of each subtitle sentence, the time interval between the current subtitle sentence and the previous subtitle sentence is calculated, and the current subtitle sentence and the corresponding time interval are fused to generate a series of fused sentences arranged in sequence. These fused sentences are input into the sequence labeling model in sequence, and all chapter names of the entire video of interest are obtained through the second branch network. The title of the video of interest is spliced ​​with each chapter name respectively to generate a combined text of each chapter name. These combined texts are feature extracted by word embedding technology to obtain the N-dimensional semantic vector of interest corresponding to each combined text. This semantic vector of interest includes the semantic information in the title of the video of interest and the semantic information of the chapter name. The similarity between each semantic vector of interest and the semantic vector of the segmented video to be recommended in the video library (that is, the semantic vector to be recommended) is calculated, and the chapter video to be recommended corresponding to each chapter name in the video of interest is determined according to the calculation result, so as to achieve accurate recommendation. Further, in order to achieve personalized recommendation, the user can be presented with the video content to be recommended when watching the video. In one embodiment of the present invention, after displaying the sorted chapter videos to be recommended and their chapter start times, the chapter name currently being watched can be determined from all the chapter names of the above-identified videos of interest according to the playback time of the video of interest currently being watched by the user, and the corresponding chapter video to be recommended can be recommended from the video library according to the chapter name. It can be understood that when the video is segmented, the information finally saved in the video library is the complete video and the start time of each chapter in the video. At this time, the related video is recommended, in order to facilitate the user to quickly locate the required chapter video, the corresponding chapter video will be selected and recommended from the video library according to the similarity calculation result and its chapter start time will be displayed. In this way, when the user watches, he can directly click on the corresponding chapter video to watch, and can also quickly locate the associated chapter in a complete video content through the chapter start time to watch, presenting a flexible and diverse viewing method for the user.

[0081] In one embodiment of the present invention, the method of calculating similarities between the semantic vector of interest and the semantic vectors to be recommended in the video library, and determining the chapter videos to be recommended and the chapter start times thereof from the video library based on the similarity calculation results, comprises:

[0082] Calculating the cosine similarity of the interest semantic vector and each semantic vector to be recommended in the video library respectively;

[0083] Determine whether the calculated cosine similarity is greater than a preset similarity threshold, and select the semantic vector to be recommended corresponding to the similarity greater than the similarity threshold;

[0084] From the video library, the corresponding chapter video is searched for recommendation according to the selected semantic vector to be recommended.

[0085] The cosine similarity between the semantic vector of interest and the semantic vector to be recommended, which is composed of "video title + chapter name" of all the videos to be recommended in the video library, is calculated respectively. Cosine similarity is used to measure the similarity between two vectors in the semantic space. The closer the value is to 1, the closer the semantics are. It can be judged whether each calculated cosine similarity exceeds the similarity threshold (such as 0.5), and the semantic vector to be recommended with a similarity higher than the similarity threshold is selected, and the chapter videos and their starting times corresponding to these selected semantic vectors to be recommended are screened out from the video library as the recommended chapter videos corresponding to the current chapter name in the video of interest. It should be noted that the semantic vector of interest corresponds to each chapter video in the video of interest one by one. In the process of watching videos by users, the corresponding semantic vector of interest can also be used to perform similarity calculation in real time and recommend related chapter videos according to the chapter currently watched by the user, so that chapter videos highly related to the user's current learning content can be recommended, which greatly improves the user's learning experience and greatly reduces the user's invalid search time.

[0086] In one embodiment of the present invention, the calculating similarity between the semantic vector of interest and each semantic vector to be recommended in the video library, and selecting and recommending the corresponding chapter video from the video library based on the similarity calculation result, comprises:

[0087] From the video library, searching for corresponding chapter videos according to the selected semantic vectors to be recommended, and using the corresponding chapter videos as the chapter videos to be recommended;

[0088] Sort the chapter videos to be recommended according to their cosine similarities;

[0089] Display the sorted chapter videos to be recommended.

[0090] After finding the relevant recommended chapter videos from the video library, considering that the higher the cosine similarity, the closer the two video contents are in semantics, in order to facilitate users to quickly navigate to the relevant recommended chapter videos, after obtaining the recommended chapter videos of a certain chapter in the video of interest, the recommended chapter videos will be sorted in order from high to low similarity. The sorted recommended chapter videos are presented to users on the display interface so that users can see the recommended chapters with the highest relevance to the content they are currently watching more quickly.

[0091] As an example, the title of the video of interest is "Solid Foundation - Student Guidance: Learning Theory", and through the above process, one of the chapters is extracted as "Behaviorism Theory" (the chapter starts at 00:05:27,520). When the user browses the "Behaviorism Theory" section of the video "Solid Foundation - Student Guidance: Learning Theory", by calculating the cosine similarity between the semantic vector of "Behaviorism Theory" and the semantic vectors to be recommended in the video library, the relevant recommended resources with a similarity greater than the similarity threshold (such as 0.5) are selected, as shown in Table 2. These recommended resources include the video title and the corresponding chapter name. After the user clicks on these recommended contents, they will jump to the starting playback position of the relevant knowledge point of the corresponding video.

[0092] Table 2 List of chapter videos to be recommended

[0093]

[0094] See also Figure 3 , the video recommendation system 100 includes: a video acquisition module 110, a text extraction module 120, a word embedding module 130, a chapter extraction module 140 and a video segmentation module 150. The video acquisition module 110 is used to acquire a target video. The text extraction module 120 is used to perform speech recognition on the target video to obtain multiple subtitle sentences corresponding to the target video and their starting and ending times in the target video. The word embedding module 130 is used to calculate the time interval between each subtitle sentence and its previous subtitle sentence according to the starting and ending times of each subtitle sentence in the target video, and to fuse each subtitle sentence and the corresponding time interval to obtain multiple fused sentences arranged in sequence. The chapter extraction module 140 is used to identify and extract each chapter name from multiple fused sentences based on a sequence labeling algorithm, and determine the corresponding chapter starting time. The video segmentation module 150 is used to segment the target video into multiple chapter videos based on each chapter name and the corresponding chapter starting time.

[0095] For the specific definition of the video recommendation system, please refer to the definition of the video recommendation method above, which will not be repeated here. Each module in the above video recommendation system can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in hardware format, or stored in the memory in the computer device in software format, so that the processor can call the operations corresponding to the above modules.

[0096] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0097] See also Figure 4 The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a video recommendation program.

[0098] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The memory 12 may be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1. The memory 12 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 12 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as the recommended code of the video, etc., but also can be used to temporarily store data that has been output or is to be output.

[0099] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect various components of the entire electronic device 1, and executes or executes programs or modules (such as video recommendation programs, etc.) stored in the memory 12, and calls data stored in the memory 12 to execute various functions of the electronic device 1 and process data.

[0100] The processor 13 executes the operating system and various installed applications of the electronic device 1. The processor 13 executes the applications to implement the steps in the above-mentioned video recommendation method.

[0101] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a video acquisition module 110, a text extraction module 120, a word embedding module 130, a chapter extraction module 140, and a video segmentation module 150.

[0102] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium, and the computer-readable storage medium can be non-volatile or volatile. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of the video recommendation method described in each embodiment of the present application.

[0103] In summary, the present invention discloses a method, system, device and medium for recommending a video. By performing speech recognition on a target video, a plurality of subtitle sentences and their corresponding start and end times are generated, and the time interval between the subtitle sentences is calculated. The time interval is fused with the subtitle sentences, and the fused sentence obtained has not only semantic information, but also time interval information between adjacent sentences. Since the larger the time interval is, the smaller the correlation between the two sentences is, the sequence labeling algorithm can be used to more accurately extract the relevant chapter names and start times according to the semantic information and the time interval information, so that a complete target video is segmented according to these chapter names and start times, and divided into multiple chapter videos. This method of the present invention performs video segmentation from the chapter granularity, and realizes the automatic construction of the chapter information of the video, including extracting the chapter name and the corresponding start time point in the video. Thus, more accurate video segmentation can be achieved, which is convenient for users to push chapters that are highly related to the videos they watch. In addition, the chapters of all video data in the video library are associated from the perspective of relevance. When recommending content related to the chapter, it is sorted according to the level of relevance, providing users with personalized and accurate video recommendations, greatly shortening the user's search time, and improving the user experience. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0104] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A video segmentation method, characterized in that: The segmentation method comprises: Get the target video; Performing speech recognition on the target video to obtain a plurality of subtitle sentences corresponding to the target video and their start and end times in the target video; According to the start and end time of each subtitle sentence in the target video, the time interval between each subtitle sentence and its previous subtitle sentence is calculated, and each subtitle sentence and the corresponding time interval are merged to obtain a plurality of merged sentences arranged in sequence; Based on the sequence labeling algorithm, identify and extract each chapter name from multiple fused sentences, and determine the corresponding chapter start time; The target video is divided into a plurality of chapter videos according to the names of the chapters and the corresponding starting times of the chapters.

2. The video segmentation method according to claim 1, characterized in that: The performing speech recognition on the target video to obtain a plurality of subtitle sentences corresponding to the target video and their start and end times in the target video includes: Extracting audio information from the target video; Performing speech recognition on the audio information to convert the audio information into text information; The text information is segmented based on a voice activity detection algorithm to obtain a plurality of subtitle sentences in the target video, and the start and end time of each subtitle sentence in the target video is determined.

3. The video segmentation method according to claim 1, characterized in that: The method of calculating the time interval between each subtitle sentence and its previous subtitle sentence according to the start and end time of each subtitle sentence in the target video, and fusing each subtitle sentence and the corresponding time interval to obtain a plurality of fused sentences arranged in sequence includes: Based on the start time of the subtitle sentence and the end time of the previous subtitle sentence, calculate the time interval between the subtitle sentence and the previous subtitle sentence, and perform word embedding processing on the time interval; Perform word unit segmentation and word embedding on the subtitle sentence to obtain a word unit embedding vector sequence of the subtitle sentence; The time interval after word embedding and the word element embedding vector sequence of the subtitle sentence are fused to generate a fused sentence.

4. The video segmentation method according to claim 1, characterized in that: The sequence labeling algorithm is used to identify and extract each chapter name from multiple fused sentence sequences, and determine the corresponding chapter start time, including: For each fused statement: Input the fused sentence into the encoder network of the sequence labeling model to capture the language features and time features of the fused sentence and generate the comprehensive features of the fused sentence; Inputting the comprehensive feature into the first branch network of the sequence labeling model to determine the type of the corresponding subtitle sentence; wherein the type of the subtitle sentence is an initial sentence or a non-initial sentence; Inputting the comprehensive features into the second branch network of the sequence labeling model to obtain the recognition degree of each word in the corresponding subtitle sentence, and sequentially splicing the words whose recognition degree is greater than or equal to a preset second threshold value as the corresponding chapter name; According to the type of the subtitle sentence corresponding to the chapter name, the chapter start time corresponding to the chapter name is determined.

5. The video segmentation method according to claim 4, characterized in that: The step of determining the chapter start time corresponding to the chapter name according to the type of the subtitle sentence corresponding to the chapter name includes: If the type of subtitle sentence corresponding to the chapter name has a start sentence, use the start sentence as the chapter start time corresponding to the chapter name at the start time of the target video; If there is no start sentence for the type of subtitle sentence corresponding to the chapter name, the start time of the previous start sentence closest to the subtitle sentence is extracted from the subtitle sentences arranged in chronological order, and is used as the chapter start time corresponding to the chapter name.

6. The video segmentation method according to claim 1, characterized in that: After dividing the target video into multiple chapter videos according to the chapter names and the corresponding chapter start times, the method further includes: splicing the title and chapter names of the target video to generate a combined text corresponding to the chapter names, and performing feature extraction on the combined text to generate a semantic vector corresponding to the chapter names.

7. The video segmentation method according to claim 6, characterized in that: The step of concatenating the title and chapter name of the target video to generate a combined text corresponding to the chapter name, and extracting features from the combined text to generate a semantic vector corresponding to the chapter name includes: Concatenate the title of the target video and the chapter name of the target video to generate a combined text of the chapter name; The generated combined text is input into the word embedding model to capture the corresponding semantic features in the combined text and generate the semantic vector corresponding to the chapter name.

8. A video recommendation method, characterized in that: The recommended methods include: Get the video you are interested in; Performing speech recognition on the video of interest to obtain a plurality of subtitle sentences corresponding to the video of interest and their start and end times in the video of interest; According to the start and end time of each subtitle sentence in the video of interest, the time interval between each subtitle sentence and its previous subtitle sentence is calculated, and each subtitle sentence and the corresponding time interval are merged to obtain a plurality of merged sentences arranged in sequence; Based on the sequence labeling algorithm, chapter names are identified and extracted from multiple fused sentences; Concatenate the title and chapter name of the video of interest to generate a corresponding combined text, and perform feature extraction on the combined text to generate a corresponding semantic vector of interest; The semantic vector of interest is respectively calculated for similarity with each semantic vector to be recommended in the video library, and based on the similarity calculation results, the corresponding chapter video is selected and recommended from the video library; wherein the videos stored in the video library are segmented by the segmentation method described in any one of claims 1-7.

9. The video recommendation method according to claim 8, characterized in that: The step of calculating similarities between the semantic vector of interest and the semantic vectors to be recommended in the video library, and selecting and recommending corresponding chapter videos from the video library based on the similarity calculation results, comprises: Calculating the cosine similarity of the interest semantic vector and each semantic vector to be recommended in the video library respectively; Determine whether the calculated cosine similarity is greater than a preset similarity threshold, and select the semantic vector to be recommended corresponding to the similarity greater than the similarity threshold; From the video library, the corresponding chapter video is searched for recommendation according to the selected semantic vector to be recommended.

10. The video recommendation method according to claim 9, characterized in that: The step of searching for corresponding chapter videos from the video library according to the selected semantic vector to be recommended for recommendation comprises: From the video library, searching for corresponding chapter videos according to the selected semantic vectors to be recommended, and using the corresponding chapter videos as the chapter videos to be recommended; Sort the chapter videos to be recommended according to their cosine similarities; Display the sorted chapter videos to be recommended.

11. A video segmentation system, characterized in that: The system comprises: A video acquisition module is used to acquire a target video; A text extraction module, used to perform speech recognition on the target video to obtain a plurality of subtitle sentences corresponding to the target video and their start and end times in the target video; A word embedding module, for calculating the time interval between each subtitle sentence and its previous subtitle sentence according to the start and end time of each subtitle sentence in the target video, and fusing each subtitle sentence and the corresponding time interval to obtain a plurality of fused sentences arranged in sequence; The chapter extraction module is used to identify and extract the names of each chapter from multiple fused sentences based on a sequence labeling algorithm, and determine the corresponding chapter start time; The video segmentation module is used to segment the target video into multiple chapter videos according to the names of each chapter and the corresponding chapter start time.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the video segmentation method described in any one of claims 1 to 7 or the video recommendation method described in any one of claims 8 to 10.

13. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the video segmentation method described in any one of claims 1 to 7 or the video recommendation method described in any one of claims 8 to 10.

Images