Video catalog generation, video playback, and deep learning model training methods

Abstract

The present disclosure provides a video directory generation method, relates to the technical field of artificial intelligence, and particularly relates to the fields of deep learning, computer vision, image processing and video processing. The specific implementation scheme is: determining target text data from subtitles or voice of a target video; performing paragraph division on the target text data to obtain at least one text paragraph; generating a text summary of the text paragraph; and generating a directory of the target video according to the text summaries of the at least one text paragraph respectively. The present disclosure also provides a deep learning model training method, a video playing method and device, an electronic device and a storage medium.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, computer vision, image processing and video processing. More specifically, the present disclosure provides a video directory generation method, a deep learning model training method, a video playing method, an apparatus, an electronic device and a storage medium. BACKGROUND

[0002] Videos often have redundancy in content, which leads to a long time and effort to obtain the main information of the video. Generally, the main information of the video can be extracted by extracting a video summary. For example, meaningful segments or partial frame images are extracted from the video to synthesize a new video as a video summary of the original video. SUMMARY

[0003] The present disclosure provides a video directory generation method, a deep learning model training method, a video playing method, an apparatus, an electronic device and a storage medium.

[0004] According to a first aspect, a video directory generation method is provided, which includes: determining target text data from a subtitle or a voice of a target video; performing paragraph division on the target text data to obtain at least one text paragraph; generating a text summary of the text paragraph; and generating a directory of the target video according to the text summary of each of the at least one text paragraph.

[0005] According to a second aspect, a deep learning model training method is provided, which includes: inputting sample splicing data into a deep learning model to obtain an output paragraph identifier at each splicing position in the sample splicing data, the splicing position having a label paragraph identifier, the paragraph identifier indicating one of segmentation and non-segmentation; determining a loss of the deep learning model according to the output paragraph identifier at the splicing position and the label paragraph identifier; and adjusting a parameter of the deep learning model according to the loss.

[0006] According to a third aspect, a video playing method is provided, which includes: in response to obtaining a target video, displaying the target video and a directory of the target video in association, wherein the directory includes at least one text summary, the at least one text summary respectively corresponding to at least one video node of the target video, and the directory is determined according to a video directory generation method; and in response to a text summary in the directory being clicked, jumping to a video node corresponding to the text summary for playing.

[0007] According to a fourth aspect, a video directory generation apparatus is provided, which comprises: a text determination module configured to determine target text data from a subtitle or a speech of a target video; a paragraph division module configured to divide the target text data into at least one text paragraph; an abstract generation module configured to generate a text abstract of the text paragraph; and a directory generation module configured to generate a directory of the target video according to the text abstract of each of the at least one text paragraph.

[0008] According to a fifth aspect, a deep learning model training apparatus is provided, which comprises: a data processing module configured to input sample splicing data into a deep learning model to obtain an output paragraph identifier at each splicing position in the sample splicing data, the splicing position having a label paragraph identifier, the paragraph identifier indicating one of segmentation and non-segmentation; a loss calculation module configured to determine a loss of the deep learning model according to the output paragraph identifier and the label paragraph identifier at the splicing position; and an adjustment module configured to adjust a parameter of the deep learning model according to the loss.

[0009] According to a sixth aspect, a video playing apparatus is provided, which comprises: a display module configured to, in response to obtaining a target video, display the target video and a directory of the target video in association, wherein the directory comprises at least one text abstract, the at least one text abstract corresponding to at least one video node of the target video respectively, and the directory is determined according to a video directory generation apparatus; and a playing module configured to, in response to a text abstract in the directory being clicked, jump to a video node corresponding to the text abstract to play.

[0010] According to a seventh aspect, an electronic device is provided, which comprises: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a method according to the present disclosure.

[0011] According to an eighth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, the computer instructions being used to cause a computer to perform a method according to the present disclosure.

[0012] According to a ninth aspect, a computer program product is provided, which comprises a computer program stored in at least one of a readable storage medium and an electronic device, and the computer program, when executed by a processor, implements a method according to the present disclosure.

[0013] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0015] Figure 1 is an exemplary system architecture schematic diagram according to one embodiment of the present disclosure, which can apply at least one of a video directory generation method, a training method of a deep learning model, and a video playing method;

[0016] Figure 2 is a flowchart of a video directory generation method according to one embodiment of the present disclosure;

[0017] Figure 3 is a schematic diagram of a video directory generation method according to one embodiment of the present disclosure;

[0018] Figure 4 is a method schematic diagram of generating a text summary according to one embodiment of the present disclosure;

[0019] Figure 5 is a schematic diagram of a target video and a directory associated according to one embodiment of the present disclosure;

[0020] Figure 6 is a flowchart of a video playing method according to one embodiment of the present disclosure;

[0021] Figure 7 is a flowchart of a training method of a deep learning model according to one embodiment of the present disclosure;

[0022] Figure 8 is a schematic diagram of a training method of a deep learning model according to one embodiment of the present disclosure;

[0023] Figure 9 is a block diagram of a video directory generation device according to one embodiment of the present disclosure;

[0024] Figure 10 is a block diagram of a video playing device according to one embodiment of the present disclosure;

[0025] Figure 11 is a block diagram of a training device of a deep learning model according to one embodiment of the present disclosure;

[0026] Figure 12 is a block diagram of an electronic device for at least one of a video directory generation method, a training method of a deep learning model, and a video playing method according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are meant to be exemplary in nature, and include various details intended to facilitate understanding of the present disclosure. Thus, it should be apparent to those skilled in the art that various modifications and changes can be made in the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, it should be apparent to those skilled in the art from this description that certain features of the embodiments described herein can be interchanged, although not explicitly described as such. It is therefore intended that the scope of the present disclosure should only be limited by the appended claims.

[0028] Video summary is to extract the main information of the video from the perspective of video and picture. For example, the difference between images is determined based on the visual features of each frame of the video, and the key frames are selected from the video according to the difference to synthesize the video summary. However, the selection of key frames has problems such as complex calculation and difficult object extraction, and the visual features cannot represent the understanding of the video content. Therefore, the video summary has limited ability to extract information, especially in complex scenes, and the video summary cannot well obtain the main information of the video.

[0029] Therefore, the present disclosure provides a video directory generation method from the perspective of natural language, which utilizes the text information in the video to realize the division of core paragraphs and the condensation of content, and summarizes the theme content of each segment in the video through short sentences to form a video directory, thereby facilitating users to quickly understand the main content of the video.

[0030] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0031] In the technical solutions of the present disclosure, the authorization or consent of the user is obtained before the user's personal information is acquired or collected.

[0032] Figure 1 is an exemplary system architecture schematic diagram according to at least one of the video directory generation method, the training method of the deep learning model, and the video playing method according to an embodiment of the present disclosure. It should be noted that, Figure 1 The system architecture shown is only an example of a system architecture that can apply the embodiments of the present disclosure to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0033] As Figure 1 shown, the system architecture 100 according to the embodiment can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used as a medium to provide a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired and / or wireless communication links, etc.

[0034] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, 103 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop portable computers, etc.

[0035] At least one of the video catalog generation method and the deep learning model training method provided by the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the video catalog generation apparatus and the deep learning model training apparatus provided by the embodiments of the present disclosure can generally be arranged in the server 105. The video catalog generation method and the deep learning model training method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the video catalog generation apparatus and the deep learning model training apparatus provided by the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105.

[0036] The video playing method provided by the embodiments of the present disclosure can generally be executed by the terminal devices 101-103. Accordingly, the video playing apparatus provided by the embodiments of the present disclosure can generally be arranged in the terminal devices 101-103.

[0037] Figure 2 is a flowchart of a video catalog generation method according to one embodiment of the present disclosure.

[0038] As shown in Figure 2 The video catalog generation method 200 includes operation S210-operation S240.

[0039] In operation S210, target text data is determined from a subtitle or a voice of a target video.

[0040] The target text data of the target video is from a subtitle of the target video, or is obtained by performing voice recognition on a voice of the target video.

[0041] The target video can be frame-cut to obtain a plurality of target images, and for each target image, a subtitle in the target image is extracted by OCR (Optical Character Recognition) text recognition, and the subtitles of the plurality of target images constitute the target text data of the target video.

[0042] Alternatively, in the case where the target video has no subtitle, the voice of the target video can be converted into text by voice recognition as the target text data.

[0043] The target text data is text data of the entire target video, and the text data of each target image in the target video can be referred to as subtext data.

[0044] In operation S220, the target text data is divided into paragraphs to obtain at least one text paragraph.

[0045] The target text data is text data of the entire target video, and the target text data is divided into paragraphs, which is semantic segmentation of the target video as a whole.

[0046] The target text data is divided into paragraphs to obtain at least one text paragraph. Each text paragraph corresponds to a video segment, and the text paragraph can include subtext data of multiple target images, and each subtext data can be a sentence.

[0047] In operation S230, a text summary of the text paragraph is generated.

[0048] For each text paragraph, if multiple sentences are included, key sentences and non-key sentences in the multiple sentences can be identified. The key sentences constitute the main content of the text paragraph, and then the main content composed of the key sentences is rewritten, such as changing long sentences into short sentences, deleting adverbs, explanatory words, etc., so that the content is concise, and a text summary with strong summarization is obtained as the main information of the text paragraph.

[0049] In operation S240, a directory of the target video is generated according to the text summaries of the at least one text paragraph.

[0050] Each text summary can be an abstract node, and multiple text summaries can constitute a directory of the target video. The directory of the target video can also be referred to as a content outline of the target video.

[0051] Since the text summary of each text paragraph can be used as the main information of the video segment corresponding to the text paragraph, the multiple text summaries in the directory can constitute the main information of the complete target video.

[0052] Thus, the directory of the target video is generated from the perspective of natural language processing, and the multiple text summaries in the directory can be used as summaries of the main content of multiple segments in the video, respectively, to extract the main information of the target video.

[0053] Embodiments of the present disclosure can extract the main information of the video from the perspective of natural language by obtaining text data of the video, segmenting the text data to obtain multiple paragraphs, generating a summary of each paragraph, and generating a video directory from summaries of the multiple paragraphs. This can help users intuitively and quickly understand the main content of the video and improve the efficiency of users obtaining video information.

[0054] Figure 3 is a schematic diagram of a video directory generation method according to an embodiment of the present disclosure.

[0055] As shown in Figure 3 , the embodiment includes steps of obtaining target text data and generating a video directory.

[0056] In operation S310, a target video is obtained. In a case where the target video contains subtitles, operations S321-S322 are performed, and in a case where the target video does not contain subtitles, operation S323 is performed.

[0057] In operation S321, the target video is frame-cut to obtain a plurality of target images.

[0058] In operation S322, text recognition is performed on the target images.

[0059] In operation S323, speech recognition is performed on the target video.

[0060] In operation S330, the recognition results are preprocessed to obtain target text data.

[0061] For example, in a case where the target video contains subtitles, text recognition is performed on the target images to obtain text recognition results of each target image. The text recognition results are preprocessed (for example, special symbols, emoticons, and the like are removed) to obtain target text data of the target video, and the target text data includes subtext data of each of the target images.

[0062] In a case where the target video does not contain subtitles, speech recognition is performed on the target video to obtain speech recognition results. The speech recognition results are preprocessed to obtain target text data of the target video, and the target text data is divided by frames, and subtext data of each of the target images can also be obtained.

[0063] Next, for the target text data, paragraph division can be performed, for example, paragraph 1, …, paragraph i, …, paragraph m can be obtained, m is an integer greater than 1, and 1

[0064] The paragraph division of the target text data is described in detail below.

[0065] The target text data is divided into paragraphs to obtain at least one text paragraph, including: determining a segment identifier at a connection position of each two adjacent sub-text data, wherein the segment identifier indicates one of segmenting and not segmenting; and performing paragraph division at the connection position where the segment identifier indicates segmenting to obtain at least one text paragraph.

[0066] The division of the paragraph is semantic segmentation of the target text data, and therefore, the division of the paragraph can be implemented by performing semantic segmentation of the target text data by using a natural language processing model.

[0067] For example, whether semantic segmentation occurs at the connection position of each two adjacent sub-text data can be identified by using the natural language processing model, and if the context before and after the connection position is segmented semantically, a paragraph identifier indicating segmenting can be output at the connection position.

[0068] Therefore, the natural language processing model can be trained to classify the paragraphs at the connection position of each two adjacent sub-text data, for example, a classification result of 1 indicates that segmenting is needed at the connection position, and a classification result of 0 indicates that segmenting is not needed at the connection position. The paragraph division can be performed at the connection position where the classification result is 1, thereby obtaining multiple text paragraphs.

[0069] The segmenting can include normal segmenting and prologue segmenting. The division of the paragraph at the connection position where the segment identifier indicates segmenting includes: performing paragraph division at the connection position where the segment identifier indicates normal segmenting to obtain at least one text paragraph.

[0070] For example, the prologue (or beginning) of a video is often an introduction, a prologue, etc., such as “Hello everyone, today I will introduce you to …” Therefore, the text data of the prologue of the video is not the main information of the video content.

[0071] Therefore, the natural language processing model can be trained to output a new segment identifier at the text data corresponding to the position where the prologue ends, indicating that this is a prologue segment. Since the prologue does not contain important information, segmenting can not be needed at this position. Therefore, the natural language processing model can output one of not segmenting (for example, 0), normal segmenting (for example, 1), and prologue segmenting (for example, 2) at the connection position of each two adjacent sub-text data.

[0072] When the target text data processed by the natural language processing model is divided into paragraphs, paragraph division can be performed only at the position indicating normal segmenting (for example, 1), thereby obtaining multiple text paragraphs of the target text data.

[0073] This embodiment uses a natural language processing model to output segmentation identifiers at the connection points of every two adjacent sub-text data. Segmentation is performed based on these segmentation identifiers, which enables the segmentation of video text data from a semantic perspective, facilitating the extraction of the main information from the video.

[0074] The following section provides a detailed explanation of how to generate text summaries.

[0075] Figure 4 This is a schematic diagram of a method for generating text summaries according to an embodiment of the present disclosure.

[0076] Text summarization can be divided into two parts: key sentence identification and content rewriting. Generating a text summary of a text paragraph involves: identifying key sentences from the text paragraph; and rewriting the key sentences to obtain the text summary.

[0077] like Figure 4 As shown, a text paragraph can include multiple statements. For example, statement 1, statement 2, ..., statement N, where N is, for example, an integer greater than 5. Among the N statements, some may be key statements containing crucial information, while others may be explanatory or supplementary non-key statements. Therefore, key and non-key statements can be identified from the text paragraph, and a summary of the text paragraph can be generated based on the key statements.

[0078] For example, a natural language processing model 410 can be used to identify key and non-key sentences in a text paragraph. This natural language processing model 410 can be trained using training text data labeled with key and non-key sentences, enabling the model to label key and non-key sentences in the training text with different identifiers, thus distinguishing between them.

[0079] For example, statements 1, 2, ..., N are input into the natural language processing model 410. The natural language processing model 410 can identify key statements including statements 1, 3, ..., N, and non-key statements including statements 2, 5, ..., M, where M is an integer greater than 5 and less than N.

[0080] Next, the key statements (statement 1, statement 3, ..., statement N) can be rewritten to generate a text summary. Rewriting the key statements to obtain a text summary includes: identifying the keywords in the key statements; and generating a text summary based on the keywords.

[0081] For example, the content of the key sentence can be rewritten using the natural language processing model 420 to obtain a text summary. The natural language processing model 420 can be trained by using training text data labeled with keywords and non-keywords, so that the natural language processing model 420 has the ability to identify keywords and non-keywords. Inputting the key sentence into the natural language processing model 420 can obtain the keywords and non-keywords in the key sentence. Content rewriting includes deleting non-keywords in the key sentence, retaining keywords in the key sentence, and also splicing the keywords and making adaptive adjustments, such as adjusting the order of words, so that the order of words is smooth. After the content of the key sentence is rewritten, the text summary 430 can be obtained.

[0082] For example, the key sentence (sentence 1, sentence 3, …, sentence N) is input into the natural language processing model 420 to obtain a text summary composed of keywords in each key sentence.

[0083] Table 1 is an example of a text paragraph and a text summary according to one embodiment of the present disclosure.

[0084]

[0085] The text paragraph in Table 1 can come from a target video of a food tutorial. For each paragraph, every two adjacent subtext data in the paragraph are separated by a “#” sign. The multiple sentences in the paragraph include key sentences (for example, the first step is to put the shredded tremella into a basin with cold water, soak for more than 2 hours, and avoid using hot water) and non-key sentences (for example, otherwise it will cause the loss of nutrients, and by the way, if it is winter, soak for more hours).

[0086] The text summary corresponding to the paragraph is obtained by rewriting the key sentence. For example, extracting the keywords in the key sentence “the first step is to put the shredded tremella into a basin with cold water, soak for more than 2 hours, and avoid using hot water” to obtain the keywords “tremella”, “put”, “cold water”, “soak for more than 2 hours”, etc. Splice the keywords and make adaptive adjustments to obtain the text summary “tremella is placed in cold water and soaked for more than 2 hours”.

[0087] By key sentence identification and content rewriting, the core content of the text paragraph can be condensed and the text summary of the text paragraph can be summarized.

[0088] After the text summaries are obtained, the text summaries can be composed into a table of contents of the target video. Next, the target video and the table of contents can be associated. Associating the target video and the table of contents includes determining at least one video node in the target video corresponding to each of the at least one text summary, and marking the video node in the target video. Determining the at least one video node in the target video corresponding to each of the at least one text summary includes, for each text summary, determining starting subtext data in a text passage corresponding to the text summary, and determining a position of a target image corresponding to the starting subtext data as the video node corresponding to the text summary.

[0089] For example, for each text summary, a text passage to which the text summary belongs corresponds to a video segment, and the text summary can be a text summary of main content of the video segment. A starting position of the video segment can be the video node corresponding to the text summary.

[0090] If a text passage includes a plurality of subtext data corresponding to a plurality of target images, a target image corresponding to a starting subtext data in the plurality of subtext data can be the starting position of the video segment, i.e., the video node corresponding to the text summary.

[0091] Figure 5 FIG. 1 is a schematic diagram of a target video and a table of contents associated according to one embodiment of the present disclosure.

[0092] As shown in FIG. 1, a target video 510 includes a plurality of video nodes, such as a video node 511, a video node 512, and the like. A table of contents 520 includes a plurality of summary nodes, such as a summary 1, what is XXX, a summary 2, step 1 of XXX, and the like. Figure 5

[0093] The plurality of video nodes in the target video 510 and the plurality of summary nodes in the table of contents 520 correspond to each other. For example, the video node 511 corresponds to the summary node 1, and a video segment from the video node 511 to the video node 512 can correspond to a text passage, and a text summary of the text passage is the summary node 1, i.e., the summary node 1 is a text summary of the video segment between the video node 1 and the video node 2.

[0094] A user can more quickly and intuitively obtain main information of the video content between the video node 511 and the video node 512 through the text summary of the summary node 1. By clicking a summary node of interest in the table of contents, the user can directly jump to a corresponding video node in the target video 510 for playing, so that the user can more quickly obtain content of interest.

[0095] Figure 6 FIG. 2 is a flowchart of a video playing method according to one embodiment of the present disclosure. ​

[0096] As Figure 6 shown, the video playing method 600 includes operation S610 to operation S620.

[0097] At operation S610, in response to obtaining a target video, the target video and a directory of the target video are displayed in association.

[0098] At operation S620, in response to a text summary in the directory being clicked, the video node corresponding to the text summary is jumped to for playing.

[0099] For example, after obtaining the target video, a directory of the target video is displayed below the target video, and the directory includes a plurality of summary nodes, each summary node corresponding to a video node in the target video. Each summary node is a summary of a video segment starting from the corresponding video node, and contains the main information of the video segment.

[0100] In the case that a user clicks any summary node, the target video can automatically jump to the video node corresponding to the clicked summary node to start playing, so that the user can more quickly obtain important information of the video.

[0101] Figure 7 is a flowchart of a training method of a deep learning model according to one embodiment of the present disclosure.

[0102] As Figure 7 shown, the training method 700 of the deep learning model includes operation S710 to operation S730.

[0103] At operation S710, sample splicing data is input into the deep learning model to obtain an output paragraph identifier at each splicing position in the sample splicing data.

[0104] At operation S720, a loss of the deep learning model is determined according to the output paragraph identifier at the splicing position and the label paragraph identifier.

[0105] At operation S730, parameters of the deep learning model are adjusted according to the loss.

[0106] The deep learning model can be a natural language processing model for segmenting text data. For example, a RoBERTa (A Robustly Optimized BERT Pretraining Approach) architecture can be used, and the natural language processing model of the RoBERTa architecture can effectively extract context information.

[0107] The RoBERTa architecture includes two structures for a Mask task and a Next Sentence Prediction task, respectively. The embodiment can delete the structure for the Next Sentence Prediction task and only retain the structure for the Mask task. The Mask task is trained during pre-training.

[0108] The RoBERTa architecture includes a Self-Attention module, and the input length of the Self-Attention module is limited (for example, 128). Considering that the length of the video text data is relatively long, the Self-Attention module cannot meet the input length requirement. Therefore, the Self-Attention module of the RoBERTa architecture is replaced with the Self-Attention module of the Longformer, which can expand the input length of the model. In addition, the hyperparameters of the RoBERTa can be adaptively modified, including the batch size and the learning rate.

[0109] Next, the pre-trained model can be used as the deep learning model of the embodiment to perform the training of the segmentation task. For example, the captions (text data) of each sample image in the sample video can be spliced to obtain sample splicing data. The sample splicing data is labeled with a label paragraph identifier at the splicing position of the captions of each adjacent two sample images. The label paragraph identifier is 0, indicating no segmentation, and the label paragraph identifier is 1, indicating segmentation. The sample splicing data is input into the deep learning model, and the deep learning model classifies the paragraph identifier at the splicing position of the captions of each adjacent two sample images. For example, the classification result includes no segmentation and segmentation, and the output paragraph identifier is 0, indicating no segmentation, and the output paragraph identifier is 1, indicating segmentation.

[0110] Next, the pre-trained model can be used as the deep learning model of the embodiment to perform the training of the segmentation task. For example, the captions (text data) of each sample image in the sample video can be spliced to obtain sample splicing data. The sample splicing data is labeled with a label paragraph identifier at the splicing position of the captions of each adjacent two sample images. The label paragraph identifier is 0, indicating no segmentation, and the label paragraph identifier is 1, indicating segmentation. The sample splicing data is input into the deep learning model, and the deep learning model classifies the paragraph identifier at the splicing position of the captions of each adjacent two sample images. For example, the classification result includes no segmentation and segmentation, and the output paragraph identifier is 0, indicating no segmentation, and the output paragraph identifier is 1, indicating segmentation.

[0111] For example, the beginning of a video is often an introduction, prologue, etc., and thus the subtitle of the beginning of the video is not the main information of the video. Therefore, a label paragraph identifier (e.g., 2) indicating a beginning segment can be labeled at the end position of the subtitle of the beginning of the video in the sample splicing data, and the identifier of the beginning segment indicates that this is a beginning segment and paragraph division can not be performed here. Similarly, the deep learning model performs three classification of paragraph identification at the splicing position of the subtitles of each adjacent two sample images, for example, the classification result includes no segmentation, normal segmentation, and beginning segmentation, the output paragraph identifier is 0 indicating no segmentation, the output paragraph identifier is 1 indicating normal segmentation, and the output paragraph identifier is 2 indicating beginning segmentation.

[0112] For example, an identifier indicating other can be labeled at the position of each word in the sample splicing data, for example, at the position of each word except the splicing position of the subtitles of each adjacent two sample images. Similarly, the deep learning model can also perform four classification of paragraph identification at the position of each word in the sample splicing data, for example, the classification result includes no segmentation, normal segmentation, beginning segmentation, and other, the output paragraph identifier is 0 indicating no segmentation, the output paragraph identifier is 1 indicating normal segmentation, the output paragraph identifier is 2 indicating beginning segmentation, and the output paragraph identifier is 3 indicating other.

[0113] Next, the loss of the deep learning model can be calculated according to the difference (e.g., cross-entropy, mean square error, etc.) between the output paragraph identifier and the label paragraph identifier, the loss is back-propagated to make the parameters of the model gradient descent, and an updated deep learning model is obtained. The above training process is repeated using the updated deep learning model until the model converges, and a trained deep learning model is obtained.

[0114] It should be noted that the loss can be calculated only according to the difference between the output paragraph identifier and the label paragraph identifier at the splicing position of the subtitles of each adjacent two sample images, and the difference at other positions is ignored, so that the model has sentence-level labeling capability, i.e., sentence-level paragraph division capability.

[0115] The embodiment classifies paragraph identification at the splicing position of the text data of each adjacent two sample images, so that the model has sentence-level paragraph division capability, and thus the efficiency and accuracy of text data segmentation can be improved.

[0116] The obtaining of the sample splicing data includes determining sample text data from the subtitles or speech of a sample video, the sample text data including subtext data of each of a plurality of sample images; and splicing the subtext data of each of the plurality of sample images to obtain the sample splicing data.

[0117] Figure 8is a schematic diagram of a training method of a deep learning model according to an embodiment of the present disclosure.

[0118] As shown in Figure 8 , the sample text data 801 is from subtitles or speech of a sample video. The sample video includes a plurality of sample images, and subtext data of each sample image includes a plurality of words. The subtext data is subjected to feature extraction, and features (Tokens) of the plurality of words are obtained.

[0119] For example, the first subtext data is subjected to feature extraction, and Token1, Token2, … are obtained. The second subtext data is subjected to feature extraction, and Token1, Token2, …, j are obtained, where j is an integer greater than 2. Each adjacent two subtext data are connected by a special token (SEP) to obtain sample splicing data 802. j

[0120] For example, the first subtext data and the second subtext data are connected by [SEP]1. The second subtext data and the third subtext data are connected by [SEP]2. In this way, a plurality of splicing features SEP can be obtained, for example, k splicing features SEP can be obtained, which are [SEP]1, [SEP]2, …, [SEP] k , where k is an integer greater than 2.

[0121] The sample splicing data 802 is input into the deep learning model 810, and the context semantic information of each splicing feature SEP is extracted to obtain the output result 803 of each splicing feature. For example, the output result 803 includes output paragraph identifier 1, …, output paragraph identifier k.

[0122] According to the difference between the output paragraph identifier 1, …, output paragraph identifier k and the corresponding label paragraph identifier, the loss 804 of the deep learning model 810 can be calculated. The loss 804 is back propagated to adjust the parameters of the deep learning model 810, and the deep learning model 810 can be optimized. By repeating the above training process, the trained deep learning model 810 can be obtained.

[0123] In this embodiment, the classification of the paragraph identifier is performed at the splicing position of the text data of each adjacent two sample images, so that the model has the ability of paragraph division at the sentence level, and the efficiency and accuracy of the text data segmentation can be improved.

[0124] Figure 9 is a block diagram of a video directory generation apparatus according to an embodiment of the present disclosure.

[0125] As shown in Figure 9 ​As shown, the video directory generation apparatus 900 comprises a text determination module 901, a paragraph division module 902, an abstract generation module 903 and a directory generation module 904.

[0126] The text determination module 901 is configured to determine target text data from a subtitle or a speech of a target video.

[0127] The paragraph division module 902 is configured to divide the target text data into at least one text paragraph.

[0128] The abstract generation module 903 is configured to generate a text abstract of the text paragraph.

[0129] The directory generation module 904 is configured to generate a directory of the target video according to the text abstract of each of the at least one text paragraph.

[0130] The target text data comprises subtext data of each of a plurality of target images. The paragraph division module 902 comprises an identification determination unit and a paragraph division unit.

[0131] The identification determination unit is configured to determine a segment identification at a connection position of each two adjacent subtext data, wherein the segment identification indicates one of a segment and a non-segment.

[0132] The paragraph division unit is configured to divide the target text data into at least one text paragraph at the connection position where the segment identification indicates the segment.

[0133] The segment comprises a normal segment and a first-clip segment. The paragraph division unit is configured to divide the target text data into at least one text paragraph at the connection position where the segment identification indicates the normal segment.

[0134] The text determination module comprises a video division unit, a text extraction unit and a text determination unit.

[0135] The video division unit is configured to divide the target video into a plurality of target images.

[0136] The text extraction unit is configured to extract text content from a subtitle or a speech of each target image, to obtain subtext data.

[0137] The text determination unit is configured to determine target text data of the video according to the subtext data of each of the plurality of target images.

[0138] The video directory generation apparatus 900 further comprises a node determination module and a node marking module.

[0139] The node determination module is configured to determine at least one video node in the target video corresponding to each of the at least one text abstract.

[0140] The node marking module is configured to mark the video node in the target video.

[0141] The node determination module comprises a start position determination unit and a node determination unit.

[0142] The start position determination unit is configured to determine start subtext data in a text passage corresponding to a text summary.

[0143] The node determination unit is configured to determine a position of a target image corresponding to the start subtext data as a video node corresponding to the text summary.

[0144] The summary generation module 903 comprises a key sentence determination unit and a content rewriting unit.

[0145] The key sentence determination unit is configured to determine a key sentence from the text passage.

[0146] The content rewriting unit is configured to perform content rewriting on the key sentence to obtain the text summary.

[0147] The content rewriting unit comprises a keyword determination subunit and a summary generation subunit.

[0148] The keyword determination subunit is configured to determine a keyword in the key sentence.

[0149] The summary generation subunit is configured to generate the text summary according to the keyword.

[0150] Figure 10 FIG. 1 is a block diagram of a video playing device according to an embodiment of the present disclosure.

[0151] As shown in FIG. 1, the video playing device 1000 comprises a display module 1001 and a playing module 1002. Figure 10 The display module 1001 is configured to, in response to obtaining a target video, display the target video and a directory of the target video in association, wherein the directory comprises at least one text summary, the at least one text summary respectively corresponds to at least one video node of the target video, and the directory is determined according to a video directory generation device.

[0152] The playing module 1002 is configured to, in response to a text summary in the directory being clicked, jump to a video node corresponding to the text summary for playing.

[0153]

[0154] FIG. 2 is a block diagram of a deep learning model training device according to an embodiment of the present disclosure. Figure 11 As shown in FIG. 2, the deep learning model training device 1100 comprises a data processing module 1101, a loss calculation module 1102, and an adjustment module 1103.

[0155] Figure 11

[0156] ​​The data processing module 1101 is configured to input sample splicing data into a deep learning model to obtain an output paragraph identifier at each splicing position in the sample splicing data, the splicing position being provided with a label paragraph identifier, and the paragraph identifier indicating one of segmentation and non-segmentation.

[0157] The loss calculation module 1102 is configured to determine a loss of the deep learning model according to the output paragraph identifier at the splicing position and the label paragraph identifier.

[0158] The adjustment module 1103 is configured to adjust parameters of the deep learning model according to the loss.

[0159] The training apparatus 1100 of the deep learning model further includes a sample acquisition module and a sample splicing module.

[0160] The sample acquisition module is configured to determine sample text data from a subtitle or a voice of a sample video, and the sample text data includes subtext data of each sample image.

[0161] The sample splicing module is configured to splice the subtext data of each sample image to obtain sample splicing data.

[0162] According to an embodiment of the present disclosure, the segmentation includes normal segmentation and first-paragraph segmentation, and the paragraph identifier indicates one of non-segmentation, normal segmentation and first-paragraph segmentation.

[0163] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0164] Figure 12 A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0165] As Figure 12As shown, the apparatus 1200 includes a computing unit 1201 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. Various programs and data required for the operation of the apparatus 1200 can also be stored in the RAM 1203. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0166] A plurality of components in the apparatus 1200 are connected to the I / O interface 1205, including an input unit 1206 such as a keyboard, a mouse, etc., an output unit 1207 such as various types of displays, speakers, etc., a storage unit 1208 such as a magnetic disk, an optical disk, etc., and a communication unit 1209 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the apparatus 1200 to exchange information / data with other apparatuses through a computer network such as the Internet and / or various telecommunication networks.

[0167] The computing unit 1201 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1201 performs at least one of the video catalog generation method, the training method of the deep learning model, the video playback method described above. For example, in some embodiments, at least one of the video catalog generation method, the training method of the deep learning model, the video playback method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on the apparatus 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of at least one of the video catalog generation method, the training method of the deep learning model, the video playback method described above can be performed. Alternatively, in other embodiments, the computing unit 1201 can be configured to perform at least one of the video catalog generation method, the training method of the deep learning model, the video playback method by any other appropriate means, e.g., by means of firmware.

[0168] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0169] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0170] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical conductors, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0171] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0172] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0173] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0174] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the spirit and scope of the present disclosure. For example, the steps recited in the disclosure can be executed in parallel, executed in series, or executed in different orders, as long as the desired results of the technology disclosed in the present disclosure are achieved, and the present disclosure is not limited herein.

[0175] The specific embodiments described above have been disclosed by way of example and that other embodiments, individually or in combination, are also within the scope of the disclosure. Thus, it is intended that the disclosure protect all changes and modifications to the disclosure herein contributed by those skilled in the art now known or to be developed in the future, which fall within the scope and spirit of the disclosure.

Claims

1. A method for generating a video directory, comprising: determining target text data from a subtitle or a voice of a target video, the target text data comprising subtext data of each of a plurality of target images; determining a segment identifier at a connection position of each of two adjacent subtext data, wherein the segment identifier represents one of no segment, normal segment, and first-of-clip segment; performing paragraph division only at the connection position where the segment identifier represents normal segment, and not performing paragraph division at the connection position where the segment identifier represents first-of-clip segment, to obtain at least one text paragraph; identifying a key sentence and a non-key sentence from the text paragraph, determining a key word in the key sentence only for the identified key sentence, and deleting a non-key word in the key sentence, and splicing and adjusting the key word to generate a text abstract of the text paragraph; generating a directory of the target video according to the text abstract of each of the at least one text paragraph; determining at least one video node in the target video corresponding to each of the at least one text abstract; and marking the video node in the target video, so that the target video comprises a plurality of video nodes. The determining of the target text data from the subtitle or the voice of the target video comprises: dividing the target video to obtain the plurality of target images; extracting text content from the subtitle or the voice of each target image to obtain the subtext data; and determining the target text data of the video according to the subtext data of each of the plurality of target images. The determining of the at least one video node in the target video corresponding to each of the at least one text abstract comprises, for each text abstract: determining start subtext data in the text paragraph corresponding to the text abstract; and determining a position of a target image corresponding to the start subtext data as a video node corresponding to the text abstract. 4.A method for playing a video, comprising: in response to obtaining a target video, displaying the target video and a directory of the target video in association, wherein the directory comprises at least one text abstract corresponding to at least one video node of the target video, and the directory is determined according to the method of any one of claims 1 to 3; and in response to a text abstract in the directory being clicked, jumping to the video node corresponding to the text abstract for playing. 5.An apparatus for generating a video directory, comprising: a text determining module configured to determine target text data from a subtitle or a voice of a target video, the target text data comprising subtext data of each of a plurality of target images; a segment identifier determining unit configured to determine a segment identifier at a connection position of each of two adjacent subtext data, wherein the segment identifier represents one of no segment, normal segment, and first-of-clip segment; a paragraph dividing unit configured to perform paragraph division only at the connection position where the segment identifier represents normal segment, and not perform paragraph division at the connection position where the segment identifier represents first-of-clip segment, to obtain at least one text paragraph; ​ ​ ​ 2. The method of claim 1, wherein, ​ ​ ​ ​ 3. The method of claim 1, wherein, ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The key sentence determining unit is configured to identify key sentences and non-key sentences from the text passages; The keyword determining sub-unit is configured to determine keywords in the key sentences and delete non-keywords in the key sentences only for the identified key sentences; The content rewriting unit is configured to splice and adjust the keywords to obtain text abstracts; The directory generating module is configured to generate a directory of the target video according to the text abstracts of the at least one text passage respectively; The node determining module is configured to determine at least one video node in the target video corresponding to the at least one text abstract respectively; and The node marking module is configured to mark the video nodes in the target video, so that the target video comprises a plurality of video nodes.

6. The apparatus of claim 5, wherein, The text determining module comprises: The video dividing unit is configured to divide the target video to obtain the plurality of target images; The text extracting unit is configured to extract text content from the subtitles or the voice of each target image to obtain the sub-text data; and The text determining unit is configured to determine the target text data of the video according to the sub-text data of the plurality of target images respectively.

7. The apparatus of claim 5, wherein, The node determining module comprises: The start position determining unit is configured to determine start sub-text data in a text passage corresponding to the text abstract; and The node determining unit is configured to determine a position of a target image corresponding to the start sub-text data as a video node corresponding to the text abstract.

8. A video playing apparatus, comprising: The display module is configured to display the target video and a directory of the target video in association in response to obtaining the target video, wherein the directory comprises at least one text abstract, the at least one text abstract corresponds to at least one video node of the target video respectively, and the directory is determined by the apparatus of any one of claims 5 to 7; and The playing module is configured to jump to a video node corresponding to a text abstract in the directory for playing in response to the text abstract being clicked.

9. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 4.

10. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1 to 4.

11. A computer program product comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program being executed by a processor to implement the method of any one of claims 1 to 4.

Images