Video summary extraction method and system

Abstract

The invention provides a video summary extraction method, which relates to the technical field of data summary extraction. The method comprises the following steps: extracting an audio file of a video to be extracted, dividing the video to be extracted into a plurality of video frames to be extracted; inputting the audio file into a speech-to-text model to obtain a time-series text file of the video to be extracted; determining a non-time-series text file and a text timestamp-text mapping relationship according to the time-series text file; inputting the non-time-series text file into a text summary extraction model to obtain a text summary of the video to be extracted; determining a picture summary of the video to be extracted according to an object recognition model and a plurality of video frames to be extracted; and determining a video summary of the video to be extracted according to a text summary, a picture summary and a text timestamp-text mapping relationship, thereby completing summary extraction of a video without subtitles and scripts.

Description

Technical Field

[0001] The present invention relates to the technical field of data abstraction extraction, and in particular to a video abstraction extraction method and system. Background Art

[0002] Both speech-to-text technology and automatic text summarization technology have undergone great development and have been widely used. Speech recognition technology has evolved from template matching to identify single words, to hidden Markov model recognition of speech fragments, to end-to-end recognition of speech fragments by neural networks. The technology has been continuously improved and has been applied to many fields such as voice assistants, smart homes, and autonomous driving. Automatic text summarization has also been continuously improved, from extractive summaries that extract key sentences from text using keywords, to using neural networks to extract text features and generate new summaries based on text content. It has been used in information retrieval, automatic book tagging and indexing, news information services, and other fields.

[0003] Although speech-to-text and text summarization technologies are relatively mature, both technologies are only widely used in their respective fields. With the development of 5G technology, the capacity and speed of network transmission have made new progress, and video has become the main multimedia content on the network. The emergence of massive videos has made how to quickly select the right video to watch a new demand.

[0004] Existing text-based video summarization technology mainly uses dynamic temporal matching algorithms to match subtitles and scripts, then measures the importance of video frames by analyzing the scripts, and finally edits the video as a summary based on the obtained importance. However, this summary technology can only be used for special videos with subtitles and scripts, and its application scope is relatively limited. Summary of the invention

[0005] The purpose of the present invention is to provide a video abstract extraction method, which can extract abstracts from videos without subtitles and scripts, thereby improving the accuracy of abstract extraction.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A video summary extraction method, comprising:

[0008] Get the video to be extracted;

[0009] Extract the audio file of the video to be extracted, and divide the video to be extracted into multiple video frames to be extracted; each video frame to be extracted corresponds to a video frame timestamp; the video frame timestamp is used to describe the start time, end time and duration of the corresponding video frame to be extracted in the video to be extracted;

[0010] The audio file is input into a speech-to-text model to obtain a time-series text file of the video to be extracted; the speech-to-text model is obtained by training a first deep neural network using an audio file training set; each sentence of text in the time-series text file corresponds to a text timestamp; the text timestamp is used to describe the start time, end time and duration of the corresponding sentence in the video to be extracted;

[0011] Determine a non-time-sequential text file and a text timestamp-text mapping relationship according to the time-sequential text file;

[0012] Inputting the non-time-sequential text file into a text summary extraction model to obtain a text summary of the video to be extracted; the text summary extraction model is obtained by training a second deep neural network using a text file training set;

[0013] Determining a picture summary of the video to be extracted according to an object recognition model and a plurality of the video frames to be extracted; the object recognition model is obtained by training a third deep neural network using a video frame training set;

[0014] A video summary of the video to be extracted is determined according to the text summary, the picture summary and the text timestamp-text mapping relationship.

[0015] Optionally, before obtaining the video to be extracted, the method further includes:

[0016] Acquire an audio file training set; the audio file training set includes a plurality of historical audio files and a plurality of historical text files corresponding to the historical audio files;

[0017] Taking the historical audio file as input and the historical text file corresponding to the historical audio file as output, the first deep neural network is trained to obtain a speech-to-text model.

[0018] Optionally, before obtaining the video to be extracted, the method further includes:

[0019] Acquire a text file training set; the text file training set includes historical text files and text summaries corresponding to the historical text files;

[0020] Taking the historical text file as input and the text summary corresponding to the historical text file as output, the second deep neural network is trained to obtain a text summary extraction model.

[0021] Optionally, before obtaining the video to be extracted, the method further includes:

[0022] Obtain a video frame training set; the video frame training set

[0023] Annotating objects in the video frame training set using a rectangular annotation frame to obtain multiple annotated video frames;

[0024] Taking the multiple annotated video frames as input and the rectangular annotated boxes as output, a third deep neural network is trained to obtain an object recognition model.

[0025] Optionally, determining the picture summary of the video to be extracted according to the object recognition model and the plurality of video frames to be extracted specifically includes:

[0026] Inputting the plurality of video frames to be extracted into an object recognition model to obtain a plurality of annotated video frames to be extracted;

[0027] Determine any annotated video frame to be extracted as the current annotated video frame to be extracted;

[0028] According to the objects and types contained in the currently labeled video frame to be extracted, the formula is used Determine the object weight of each object in the current annotated video frame to be extracted; where W i is the object weight of the i-th object in the video frame to be extracted; W cls is the category weight of the object; Where N is the total number of video frames to be extracted. cls is the total number of video frames to be extracted that contain objects of category cls; i ,y i Represents the coordinates of the center point of the rectangular annotation box, w i ,h i are the width and height of the rectangular annotation box, W and H are the width and height of the video frame to be extracted;

[0029] According to the object weight of each object in the video frame to be extracted, the formula W is used img =α×W i1 +β×W i2 Determine the image weight of the current labeled video frame to be extracted; where W img W represents the image weight of the current annotated video frame to be extracted; i1 , W i2 They represent the object weights of the two objects with larger weights in the current annotated video frame to be extracted; α and β are both hyperparameters;

[0030] Arrange the plurality of video frames to be extracted in descending order according to the object weights to obtain a sequence of video frames to be extracted;

[0031] Determining that the first element in the to-be-extracted video frame sequence is a summary picture;

[0032] Update the video frame sequence to be extracted and return to the step of "determining that the first element in the video frame sequence to be extracted is a summary picture" until the number of elements in the video frame sequence to be extracted is 0, and obtain multiple summary pictures as the picture summaries of the video to be extracted.

[0033] Optionally, the updating of the to-be-extracted video frame sequence specifically includes:

[0034] Determine the similarity between each element except the first element in the video frame sequence to be extracted and the first element;

[0035] The first element in the to-be-extracted video frame sequence and the elements whose similarity is greater than a similarity threshold are deleted to obtain an updated to-be-extracted video frame sequence.

[0036] Optionally, determining the video summary of the video to be extracted according to the text summary, the picture summary and the text timestamp-text mapping relationship specifically includes:

[0037] Splitting the text summary into multiple summary sentence vectors;

[0038] Determining, according to the text timestamp-text mapping relationship, that a plurality of the summary sentence vectors and the text timestamps corresponding to the time series text file are a text timestamp set;

[0039] Determine that the video frame timestamp corresponding to the picture summary is a video frame timestamp set;

[0040] Extracting a plurality of initial summary videos from the video to be extracted according to the intersection of the text timestamp set and the video frame timestamp set;

[0041] The ffmpeg tool is used to splice multiple segments of the initial summary videos to obtain the video summary of the video to be extracted.

[0042] A video summary extraction system, comprising:

[0043] A module for obtaining videos to be extracted, used for obtaining videos to be extracted;

[0044] An audio file extraction module is used to extract the audio file of the video to be extracted, and divide the video to be extracted into multiple video frames to be extracted; each video frame to be extracted corresponds to a video frame timestamp; the video frame timestamp is used to describe the start time, end time and duration of the corresponding video frame to be extracted in the video to be extracted;

[0045] A time-series text file determination module is used to input the audio file into a speech-to-text model to obtain a time-series text file of a video to be extracted; the speech-to-text model is obtained by training a first deep neural network using an audio file training set; each sentence of text in the time-series text file corresponds to a text timestamp; the text timestamp is used to describe the start time, end time and duration of the corresponding sentence in the video to be extracted;

[0046] A text timestamp-text mapping relationship determination module, used to determine a non-time-sequential text file and a text timestamp-text mapping relationship according to the time-sequential text file;

[0047] A text summary determination module, used for inputting the non-time-sequential text file into a text summary extraction model to obtain a text summary of the video to be extracted; the text summary extraction model is obtained by training a second deep neural network using a text file training set;

[0048] A picture summary determination module, used to determine the picture summary of the video to be extracted according to an object recognition model and a plurality of the video frames to be extracted; the object recognition model is obtained by training a third deep neural network using a video frame training set;

[0049] The video summary determination module is used to determine the video summary of the video to be extracted according to the text summary, the picture summary and the text timestamp-text mapping relationship.

[0050] Optionally, the system further includes:

[0051] An audio file training set acquisition module is used to acquire an audio file training set; the audio file training set includes a plurality of historical audio files and historical text files corresponding to the plurality of historical audio files;

[0052] The speech-to-text model determination module is used to train the first deep neural network with the historical audio file as input and the historical text file corresponding to the historical audio file as output to obtain the speech-to-text model.

[0053] Optionally, the system further includes:

[0054] A text file training set acquisition module is used to acquire a text file training set; the text file training set includes historical text files and text summaries corresponding to the historical text files;

[0055] The text summary extraction model determination module is used to take the historical text file as input and the text summary corresponding to the historical text file as output, train the second deep neural network, and obtain the text summary extraction model.

[0056] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0057] The purpose of the present invention is to provide a video summary extraction method, which converts an audio file into a text file by constructing a speech-to-text model, extracts a text summary by using a text summary extraction model, constructs an object recognition model to extract a picture summary, and fuses the text summary and the picture summary to obtain a video summary of a video to be extracted; the method can extract a summary of a video without subtitles and scripts, thereby improving the accuracy of summary extraction; in addition, the present invention can extract a summary of data in text format, audio format and video format, thereby expanding the application scope of summary extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0059] Figure 1 This is a flow chart of a video summary extraction method in an embodiment of the present invention;

[0060] Figure 2 It is a first principle diagram of the video summary extraction method in an embodiment of the present invention;

[0061] Figure 3 This is a second principle diagram of the video summary extraction method in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The purpose of the present invention is to provide a video abstract extraction method, which can extract abstracts from videos without subtitles and scripts, thereby improving the accuracy of abstract extraction.

[0064] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] like Figure 1 The present invention provides a video summary extraction method, comprising:

[0066] Step 101: Obtain the video to be extracted;

[0067] Step 102: extract the audio file of the video to be extracted, and divide the video to be extracted into multiple video frames to be extracted; each video frame to be extracted corresponds to a video frame timestamp; the video frame timestamp is used to describe the start time, end time and duration of the corresponding video frame to be extracted in the video to be extracted;

[0068] Step 103: Input the audio file into the speech-to-text model to obtain a time-series text file of the video to be extracted; the speech-to-text model is obtained by training the first deep neural network using the audio file training set; each sentence in the time-series text file corresponds to a text timestamp; the text timestamp is used to describe the start time, end time and duration of the corresponding sentence in the video to be extracted;

[0069] Step 104: determining a non-time-sequential text file and a text timestamp-text mapping relationship according to the time-sequential text file;

[0070] Step 105: inputting the non-time-sequential text file into a text summary extraction model to obtain a text summary of the video to be extracted; the text summary extraction model is obtained by training the second deep neural network using the text file training set;

[0071] Step 106: determining a picture summary of the video to be extracted according to the object recognition model and the plurality of video frames to be extracted; the object recognition model is obtained by training the third deep neural network using the video frame training set;

[0072] Step 107: Determine the video summary of the video to be extracted according to the text summary, the picture summary and the text timestamp-text mapping relationship.

[0073] Before step 101, the method further includes:

[0074] Obtain an audio file training set; the audio file training set includes multiple historical audio files and historical text files corresponding to the multiple historical audio files;

[0075] With the historical audio files as input and the historical text files corresponding to the historical audio files as output, the first deep neural network is trained to obtain a speech-to-text model.

[0076] Before step 101, the method further includes:

[0077] Obtain a text file training set; the text file training set includes historical text files and text summaries corresponding to the historical text files;

[0078] Taking the historical text files as input and the text summaries corresponding to the historical text files as output, the second deep neural network is trained to obtain a text summary extraction model.

[0079] Before step 101, the method further includes:

[0080] Get video frame training set; video frame training set

[0081] Use rectangular annotation boxes to annotate objects in the video frame training set to obtain multiple annotated video frames;

[0082] With multiple annotated video frames as input and rectangular annotated boxes as output, the third deep neural network is trained to obtain an object recognition model.

[0083] Step 106 specifically includes:

[0084] Inputting multiple video frames to be extracted into the object recognition model to obtain multiple labeled video frames to be extracted;

[0085] Determine any annotated video frame to be extracted as the current annotated video frame to be extracted;

[0086] According to the objects and types contained in the currently labeled video frame to be extracted, the formula is used Determine the object weight of each object in the current annotated video frame to be extracted; where W i is the object weight of the i-th object in the video frame to be extracted; W cls is the category weight of the object; Where N is the total number of video frames to be extracted. cls is the total number of video frames to be extracted that contain objects of category cls; i ,y i Represents the coordinates of the center point of the rectangular annotation box, w i ,h i are the width and height of the rectangular annotation box, W and H are the width and height of the video frame to be extracted;

[0087] According to the object weight of each object in the video frame to be extracted, the formula W is used img =α×W i1 +β×W i2 Determine the image weight of the current labeled video frame to be extracted; where W img W represents the image weight of the current annotated video frame to be extracted; i1 , W i2 They represent the object weights of the two objects with larger weights in the current annotated video frame to be extracted; α and β are both hyperparameters;

[0088] Arrange multiple video frames to be extracted in descending order according to object weights to obtain a sequence of video frames to be extracted;

[0089] Determine that the first element in the video frame sequence to be extracted is a summary picture;

[0090] Update the video frame sequence to be extracted and return to the step of "determining that the first element in the video frame sequence to be extracted is a summary picture" until the number of elements in the video frame sequence to be extracted is 0, and obtain multiple summary pictures as the picture summaries of the video to be extracted.

[0091] Specifically, updating the video frame sequence to be extracted includes:

[0092] Determine the similarity between each element except the first element in the video frame sequence to be extracted and the first element;

[0093] The first element in the video frame sequence to be extracted and the elements whose similarity is greater than the similarity threshold are deleted to obtain an updated video frame sequence to be extracted.

[0094] Step 107 specifically includes:

[0095] Split the text summary into multiple summary sentence vectors;

[0096] According to the text timestamp-text mapping relationship, a plurality of summary sentence vectors and corresponding text timestamps in the time series text file are determined as a text timestamp set;

[0097] Determine the video frame timestamps corresponding to the picture summary as a video frame timestamp set;

[0098] Extracting a plurality of initial summary videos from the video to be extracted according to the intersection of the text timestamp set and the video frame timestamp set;

[0099] Use the ffmpeg tool to splice multiple initial summary videos to obtain the video summary of the video to be extracted.

[0100] Specifically, Figure 2-3 The key information summary extraction method provided by the present invention comprises:

[0101] First, the overall process:

[0102] 1. Enter the Internet multi-channel data file name;

[0103] 2. Determine the specific type of Internet multi-channel data:

[0104] 2.1 Determine whether the data type is text. If so, jump to 6. If not, jump to 2.2.

[0105] 2.2 Determine whether the data type is a picture. If so, jump to 7. If not, jump to 2.3.

[0106] 2.3 Determine whether the data type is video, if so, jump to 3, if not, end the program.

[0107] 3. Use the audio and video separation tool to separate the audio and capture the video frames.

[0108] 4. Determine whether the output result of 3 is audio. If so, jump to 5; if not, jump to 7.

[0109] 5. Input the audio into the speech-to-text model to obtain text content with a timestamp. After saving the mapping relationship between the timestamp and the text, remove the timestamp to obtain the plain text content.

[0110] 6. Input the text into the automatic text summarization model to obtain the text summary.

[0111] 7. Input the image into the image summary model to obtain the image summary and classify the text according to the category of the main object.

[0112] 8. Input the obtained text content and picture summary into the video summary model based on the text summary and picture summary to obtain the video summary.

[0113] Among them, the text summary model is an extractive summary model pre-trained using a public dataset. The summary model can be a traditional model based on word frequency, position and other information, or an end-to-end deep learning model. The text is split into sentences according to punctuation marks such as {'. ', '! ', '? '} that mark the end of the sentence, and an open source word segmentation tool is used to segment the sentences. After removing common words, a dictionary is constructed and the words are vectorized. K keywords are selected based on the frequency, position, part of speech and other information of the words. The vectorized text is used as the input of the summary model; the keywords are also added to the input of the model as auxiliary information. The summary model outputs a text summary.

[0114] Specifically, the image summary model is an object detection model trained using a public dataset: the image is input into the pre-trained model, and the model gives the location and classification of the object. A weight is assigned to each object in the image based on the object's category and its proportion in the image. The weight of the image is the weight of all or part of the objects in the image. Images with high weights and low repetition rates are selected as image summaries.

[0115] Second, specific implementation:

[0116] 1. Train text summarization models, object detection models, and speech recognition models.

[0117] 1.1 Training the text summary model. First, pre-train the automatic text summary model on a general pre-training dataset such as Common Crawl. Common crawl (general web content dataset) is a public dataset obtained by crawling web pages since 2008. The dataset contains corpora in more than 40 languages, and the data can be downloaded directly from the official website. During implementation, the Chinese corpus in CommonCrawl is retrieved as the pre-training dataset. During pre-training, a part of the words are replaced with a probability of 15% (80% of the replaced words are replaced with masks, 15% of the original words are retained, and 15% of the random words are replaced). During pre-training, the training model predicts the replaced words. After pre-training, the parameter weights obtained from pre-training are used as the initial parameter weights of the automatic text summary model, and then fine-tuned using the Chinese automatic text summary dataset. The Chinese summary datasets include THUCNews (Tsinghua Chinese News Dataset), nlpcc2017 Summary Dataset (2017 Natural Language Processing and Chinese Computing Conference Summary Task Dataset), etc. The THUCNews dataset is generated by sorting and filtering data from the Sina News RSS subscription channel between 2005 and 2011. The data includes text and title, and the title can be regarded as a summary. nlpcc2017 is a dataset for the summary task of the 2017 NLPCC competition, which includes text and summary. When fine-tuning after pre-training, the article is used as input and the summary is used as the prediction target. They are input into the model together, and the model parameters obtained by pre-training are updated iteratively. The input of the fine-tuned model is the article (a relatively long text), and the output is the summary generated by the model (a short text containing important information).

[0118] 1.2 Training the target detection model. First, pre-train the model on a larger dataset such as ImageNet (ImageNet dataset) so that the model can better extract image features. The pre-training task is an image classification task, the input is an image, and the output is the category of the image. After the pre-training is completed, the input layer and output layer are discarded, and only the middle network layer used to extract image features is retained for the subsequent training of the target detection model. ImageNet is a large image database with more than 14 million annotated data. Due to copyright issues, ImageNet provides download links and thumbnails of the images. Since the commonly used image classification datasets have only one object per image, it is difficult to classify images with multiple objects. Therefore, the image classification model is not directly used in the implementation, but the target detection model is used to obtain the main object of the image, and then the category of the image is given based on the main object. After pre-training, a new input layer and output layer are built according to the data format of the dedicated dataset for target detection, and the parameters of these two neural network layers are randomly initialized. Special datasets for target detection include COCO-2017 (Common Objects in Context, 2017 version of common object recognition dataset), OpenImageData (open image dataset), etc. After downloading the special dataset from the official website, input these data into the target detection model composed of a new input layer, a pre-trained intermediate layer, and a new output layer, and iterate to optimize the parameters. After training, input a picture into the model, and the model will give the location of each object in the picture and the most likely category of each object.

[0119] 1.3 Training speech recognition model. Train the end-to-end speech recognition model on Chinese speech recognition training sets, such as THCHS-30 (Tsinghua University 30-hour Chinese speech library) and AISHELL (Hill Shell Company speech recognition data set). THCHS-30 is an open source Chinese speech data set released by the Speech and Language Technology Center of Tsinghua University. The speech data was recorded in a quiet office environment through a single carbon microphone, with a total duration of more than 30 hours, a sampling frequency of 16kHz, and a sampling accuracy of 16 bits. AISHELL is a series of speech recognition data sets released by Beijing Hill Shell Technology Co., Ltd., and has now been released to AISHELL-3. The total recording duration of AISHELL-3 is 85 hours, with a total of 88,035 recorded sentences, a recording frequency of 44.1kHz, and a recording accuracy of 16 bits. Input the speech data and text data into the end-to-end speech recognition model for training to obtain a trained speech recognition model. After training, input a piece of audio into the model, and the model will give text content with a timestamp.

[0120] 2. The user inputs the multimedia content file name to the overall control module, and the overall control module determines the type of the input multimedia content. The data type can be determined based on the file suffix entered by the user. If the file name entered by the user is not text, picture, or video, an error message is output and the operation is terminated. If the input is a text file, jump to 5; if the input is a picture file, jump to 6. If the input is a video file, jump to 3 to continue execution.

[0121] 3. Use an open-source audio and video separation tool to separate the audio from the video file and save it as a separate audio file; and use an open-source video cutting tool to capture video frames at a set time interval (5-10ms) and save them as images. The image name is the timestamp of the capture. After the execution is completed, jump to 4 to continue.

[0122] 4. Input the audio file obtained in 3 into the pre-trained speech recognition model to obtain the text data corresponding to the speech data. Each sentence of text has a timestamp (start time, end time, duration). Use HashSet to save the correspondence between text and timestamp, then remove the timestamp to obtain the plain text content.

[0123] 5. Use an open source word segmentation tool to segment the plain text content. Word segmentation is to split the complete sentence into units of words. Use the results of word segmentation to build a dictionary, assign a unique serial number to each different word (the serial number starts from 1), and use the serial number to represent the word. Then vectorize the sentence after word segmentation. Vectorization is to use the serial number of the word instead of the word to represent the sentence. Since the sentence length is inconsistent, a maximum sentence length is set. Sentences with a length less than the maximum length are filled with 0. The vectorized text is used as the input of the trained text summary model. The automatic text summary model gives the most likely text sequence as a text summary based on the input content, and the length of the summary can be set manually. The text summary model uses extractive text summarization, which uses sentences in the original text to form a summary, which facilitates the alignment of the summary document and the video later.

[0124] 6. Input the image into the trained object detection model, and the model gives the location of the object in the image (x i ,y i , w i ,h i ) and the probability distribution of objects in all object categories, the maximum probability c i . Where x i ,y i Indicates the coordinates of the center point of the rectangle that frames the current object, w i ,h i Indicates the width and height of the box.

[0125] 6.1 Classify the images according to the object category. Input each image into the object detection model trained in step 1.2. The model will give the location and category of each object in the image, and then classify the image according to the w of each object. i and h i Calculate the area. Use the object of category C with the largest area i As a picture category.

[0126] 6.2 Select the image with higher weight as the image summary. The image weight is calculated as follows:

[0127] (a) Assign different weights W to objects of different categories cls For example, the weight of the person category is set to 0.8, the weight of the car category is set to 0.5, etc. The weight of each category is assigned according to the frequency. Where N is the number of all pictures, N cls is the number of images containing objects of category cls.

[0128] (b) Calculate the weight W of each object in the image i The importance of each object in the picture is determined by the category, area and position of the picture. where x i ,y i , w i ,h i The definition of is the same as above. W, H are the width and height of the image, W cls Obtained from the previous step.

[0129] (c) Calculate the weights of the images and sort them. i , sort all the objects in the picture, and select the sum of the weights of the first two as the weight W of the picture img , W img =α×W i1 +β×W i2 , where W i1 , W i2 They are the two objects with the largest weights in the current image img. α and β are hyperparameters, and different values ​​can be selected according to the specific input data, and then the best combination can be selected; or they can be weighted according to the weights of the two objects, such as: After obtaining the image weight, according to W img Sort the images in descending order.

[0130] (d) Get the image summary. At the beginning, the image summary list is empty. For the images that have been sorted in descending order by weight, first add the first-ranked image to the summary, then take out the first-ranked image and calculate the similarity with each subsequent image. If the obtained similarity is greater than the threshold, it means that the current image is more similar to the first image and has a lower weight than the first image. Since the first image has been added to the summary, the current image is deleted from the list. Then delete the first image from the sorted image list, and repeat the previous operation for the remaining images in the list until there are no images in the list, and then return to the image summary list.

[0131] Calculate the similarity between any two images.

[0132] Image similarity is calculated using SSIM i (Structural similarity metric), calculates the similarity of images from three aspects: brightness, contrast and structure. According to SSIM similarity, the image summary is obtained according to method (d). The steps of SSIM algorithm to calculate image similarity are as follows:

[0133] Modify the sizes of the two images to be calculated to the same size (manually specified) and convert them into grayscale images.

[0134] The SSIM value in each window is calculated using a sliding window method (the window size is manually specified), and each window on the image is weighted using a Gaussian weighting function with a standard deviation of 1.5.

[0135] For any two input images X and Y, the parts of the images that fall within the current window are recorded as x and y, and the calculation formula for SSIM between x and y is as follows:

[0136] Brightness Similarity: Where C1 is a constant to prevent the denominator from being 0. is the average gray value of x, i is the gray value of the i-th pixel of x, N p is the total number of pixels in x. represents the average gray value of y, y i is the grayscale value of the i-th pixel of y, N q for y The total number of pixels.

[0137] Contrast Similarity: Where C2 is a constant, is the grayscale standard deviation of x, is the grayscale standard deviation of y.

[0138] Structural similarity: Where C3 is a constant.

[0139] SSIM:

[0140] The weighted average of the SSIM of all windows is used as the similarity between the two images. If there are K sliding windows in total, the similarity between image X and image Y is: Where k means calculating the SSIM value in the kth window, X k , Y k Respectively represent the parts of image X and image Y that fall into the kth window. k represents the weight of the k-th window, obtained by a Gaussian function with a standard deviation of 1.5.

[0141] 7. Generate video summary based on text summary and image summary.

[0142] 7.1 Use clustering method on the plain text obtained by converting video to audio to obtain c categories. Clustering uses K-means clustering method, constructs sentence vector representation with words in the sentence, and the distance between two sentences is the cosine similarity of the corresponding vectors. The distance between each sentence and the nearest cluster center is used to determine which category the sentence belongs to. Initially, the cluster center is c randomly specified sentence vectors. Through continuous iteration, the average value of all vectors in the same category is used as the new cluster center. Iteration stops after the specified number of rounds or the minimum distance between categories is greater than the specified threshold.

[0143] 7.2 If the video has a brief text description, calculate the similarity between each category obtained in 7.1 and the text description, and keep the G with the highest similarity. top If there is no text description, keep the category containing the most text. top Categories. Introduction to Computing Text and C top The similarity of each category text in the category adopts Jaccard similarity. Jaccard similarity uses the intersection and union between two sets to express the similarity of two sets. A represents the word segmentation result of the text introduction, B i Represents the words contained in the text of the i-th category.

[0144] 7.3 Split the summary content obtained from the text summary into sentences, and compare each sentence with C top For the current summary sentence, after vectorizing it, we compare it with C top The cosine similarity is calculated by the vectorization result of each sentence in the category. The vectorization method is the same as the vectorization in step 5. top If the cosine similarity of a sentence in a certain category in C is greater than the threshold (set manually), it means that the summary sentence contains information of this category and the sentence is retained.top If the cosine similarity of all categories is less than the threshold, the sentence is discarded.

[0145] 7.4 According to the mapping relationship between text and timestamp, obtain the timestamp of each text summary sentence in the video. In the result obtained by the speech recognition model in step 4, each text sentence has a timestamp (start time, end time, duration). The mapping relationship between the text and its corresponding timestamp is stored in HashSet. Each text sentence is mapped to its own timestamp. Therefore, the timestamp can be found through the text sentence; in addition, another HashSet is used to store the corresponding relationship between the start time and the text. The start time of each text sentence is mapped to the text itself.

[0146] 7.5 For each timestamp of the text summary sentence, check whether the corresponding video frame is in the picture summary obtained from the video frame. If so, keep the current timestamp. The picture obtained from the video frame in step 3, the picture name is the timestamp. According to the timestamp obtained in 7.4, check whether there is a picture named with the corresponding timestamp in the picture summary. If so, keep the timestamp corresponding to the current summary sentence; if not, discard it.

[0147] 7.6 For the timestamps retained in 7.5, find the HashSet mapping the start time and text to obtain the corresponding text sentence. Then, based on the plain text content obtained in step 3, find the n text sentences counting forward and the n text sentences counting backward. For these 2n text sentences, if and C top If the cosine similarity of a category is greater than the threshold or the corresponding video frame is in the picture summary, the timestamp (start time, end time, duration) corresponding to the text sentence is retained.

[0148] 7.7 Based on the timestamp obtained in 7.6, use the ffmpeg tool to extract video segments of the duration from the start time, and finally splice them into a video summary. The specific process is as follows:

[0149] (a) Enter the command ffmpeg -ss{start_time} -i{inputvideo} -t{duration} -c:vcopy -c:a copy{output_video} in the command line, where start_time indicates the start time of the captured video segment, which is the start time of the timestamp corresponding to the text sentence. input_video is the input video, which is the file name entered by the user into the overall control module. duration indicates the duration of the captured segment, which can be the duration corresponding to the text sentence or a manually specified time. output_video indicates the file name where the current video segment is saved.

[0150] (b) Then create a new patch.txt file to save the file name of each segment. Each line in the patch.txt file is: file video_patch_path. Video_patch_path is the storage path of each segment plus the file name.

[0151] (c) Finally, enter the command ffmpeg -f concat -i patch.txt -c copy{video_sum} in the command line. Video_sum is the file name of the video summary, indicating the storage location and name of the video summary.

[0152] In addition, the present invention also provides a video summary extraction system, comprising:

[0153] A module for obtaining videos to be extracted, used for obtaining videos to be extracted;

[0154] The audio file extraction module is used to extract the audio file of the video to be extracted, and divide the video to be extracted into multiple video frames to be extracted; each video frame to be extracted corresponds to a video frame timestamp; the video frame timestamp is used to describe the start time, end time and duration of the corresponding video frame to be extracted in the video to be extracted;

[0155] A time-series text file determination module is used to input the audio file into the speech-to-text model to obtain a time-series text file of the video to be extracted; the speech-to-text model is obtained by training the first deep neural network using the audio file training set; each sentence in the time-series text file corresponds to a text timestamp; the text timestamp is used to describe the start time, end time and duration of the corresponding sentence in the video to be extracted;

[0156] A text timestamp-text mapping relationship determination module, used to determine a non-time-sequential text file and a text timestamp-text mapping relationship based on a time-sequential text file;

[0157] A text summary determination module is used to input a non-time-sequential text file into a text summary extraction model to obtain a text summary of the video to be extracted; the text summary extraction model is obtained by training the second deep neural network using a text file training set;

[0158] A picture summary determination module is used to determine a picture summary of a video to be extracted based on an object recognition model and a plurality of video frames to be extracted; the object recognition model is obtained by training a third deep neural network using a video frame training set;

[0159] The video summary determination module is used to determine the video summary of the video to be extracted according to the text summary, the picture summary and the text timestamp-text mapping relationship.

[0160] An audio file training set acquisition module is used to acquire an audio file training set; the audio file training set includes multiple historical audio files and historical text files corresponding to the multiple historical audio files;

[0161] The speech-to-text model determination module is used to train the first deep neural network with historical audio files as input and historical text files corresponding to the historical audio files as output to obtain a speech-to-text model.

[0162] A text file training set acquisition module is used to acquire a text file training set; the text file training set includes historical text files and text summaries corresponding to the historical text files;

[0163] The text summary extraction model determination module is used to take the historical text file as input and the text summary corresponding to the historical text file as output, train the second deep neural network, and obtain the text summary extraction model.

[0164] Correspondingly, the Internet multi-channel data key information summary system provided by the present invention includes: an overall control module, which determines the data type of the input multi-channel data and determines the input and output of each module. An audio and video conversion module is used to separate audio and video and capture video frames. A speech recognition module converts audio into text. A text summary module automatically generates a summary of a long text. A picture summary module automatically generates a picture summary. A video summary module automatically generates a video summary.

[0165] In summary, the key information summary technology and system for Internet multi-channel data provided by the present invention can determine the input file type according to the file name input by the user, and perform text, picture, and video summaries according to the file type, thereby realizing the key information summary of Internet multi-channel data. Among them, the text summary is realized by the extractive text summary model, the picture summary is realized by the target detection model, and the video summary is realized by separating the audio, intercepting the video frame, and combining the text summary and the picture summary.

[0166] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0167] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A video summary extraction method, characterized in that: The method comprises: Get the video to be extracted; Extract the audio file of the video to be extracted, and divide the video to be extracted into multiple video frames to be extracted; each video frame to be extracted corresponds to a video frame timestamp; the video frame timestamp is used to describe the start time, end time and duration of the corresponding video frame to be extracted in the video to be extracted; The audio file is input into a speech-to-text model to obtain a time-series text file of the video to be extracted; the speech-to-text model is obtained by training a first deep neural network using an audio file training set; each sentence of text in the time-series text file corresponds to a text timestamp; the text timestamp is used to describe the start time, end time and duration of the corresponding sentence in the video to be extracted; Determine a non-time-sequential text file and a text timestamp-text mapping relationship according to the time-sequential text file; Inputting the non-time-sequential text file into a text summary extraction model to obtain a text summary of the video to be extracted; the text summary extraction model is obtained by training a second deep neural network using a text file training set; Determining a picture summary of the video to be extracted according to an object recognition model and a plurality of the video frames to be extracted; the object recognition model is obtained by training a third deep neural network using a video frame training set; Determine a video summary of the video to be extracted according to the text summary, the picture summary and the text timestamp-text mapping relationship; Determining the video summary of the video to be extracted according to the text summary, the picture summary and the text timestamp-text mapping relationship specifically includes: Splitting the text summary into multiple summary sentence vectors; Determining, according to the text timestamp-text mapping relationship, that a plurality of the summary sentence vectors and the text timestamps corresponding to the time series text file are a text timestamp set; Determine that the video frame timestamp corresponding to the picture summary is a video frame timestamp set; Extracting a plurality of initial summary videos from the video to be extracted according to the intersection of the text timestamp set and the video frame timestamp set; The ffmpeg tool is used to splice multiple segments of the initial summary videos to obtain the video summary of the video to be extracted.

2. The video summary extraction method according to claim 1, characterized in that: Before obtaining the video to be extracted, the method further includes: Acquire an audio file training set; the audio file training set includes a plurality of historical audio files and a plurality of historical text files corresponding to the historical audio files; Taking the historical audio file as input and the historical text file corresponding to the historical audio file as output, the first deep neural network is trained to obtain a speech-to-text model.

3. The video summary extraction method according to claim 1, characterized in that: Before obtaining the video to be extracted, the method further includes: Acquire a text file training set; the text file training set includes historical text files and text summaries corresponding to the historical text files; Taking the historical text file as input and the text summary corresponding to the historical text file as output, the second deep neural network is trained to obtain a text summary extraction model.

4. The video summary extraction method according to claim 1, characterized in that: Before obtaining the video to be extracted, the method further includes: Get the video frame training set; Annotating objects in the video frame training set using a rectangular annotation frame to obtain multiple annotated video frames; Taking the multiple annotated video frames as input and the rectangular annotated boxes as output, a third deep neural network is trained to obtain an object recognition model.

5. The video summary extraction method according to claim 1, characterized in that: Determining the picture summary of the video to be extracted according to the object recognition model and the plurality of video frames to be extracted specifically includes: Inputting the plurality of video frames to be extracted into an object recognition model to obtain a plurality of annotated video frames to be extracted; Determine any annotated video frame to be extracted as the current annotated video frame to be extracted; According to the objects and types contained in the currently labeled video frame to be extracted, the formula is used Determine the object weight of each object in the current annotated video frame to be extracted; where W i is the object weight of the i-th object in the video frame to be extracted; W cls is the category weight of the object; Where N is the total number of video frames to be extracted. cls is the total number of video frames to be extracted that contain objects of category cls; i ,y i Represents the coordinates of the center point of the rectangular annotation box, w i ,h i are the width and height of the rectangular annotation box, W and H are the width and height of the video frame to be extracted; According to the object weight of each object in the video frame to be extracted, the formula W is used img =α×W i1 +β×W i2 Determine the image weight of the current labeled video frame to be extracted; where W img W represents the image weight of the current annotated video frame to be extracted; i2 , W i2 They represent the object weights of the two objects with larger weights in the current annotated video frame to be extracted; α and β are both hyperparameters; Arrange the plurality of video frames to be extracted in descending order according to the object weights to obtain a sequence of video frames to be extracted; Determining that the first element in the to-be-extracted video frame sequence is a summary picture; Update the video frame sequence to be extracted and return to the step of "determining that the first element in the video frame sequence to be extracted is a summary picture" until the number of elements in the video frame sequence to be extracted is 0, and obtain multiple summary pictures as the picture summary of the video to be extracted.

6. The video summary extraction method according to claim 5, characterized in that: The updating of the to-be-extracted video frame sequence specifically includes: Determine the similarity between each element except the first element in the video frame sequence to be extracted and the first element; The first element in the to-be-extracted video frame sequence and the elements whose similarity is greater than a similarity threshold are deleted to obtain an updated to-be-extracted video frame sequence.

7. A video summary extraction system, characterized in that: The system comprises: A module for obtaining videos to be extracted, used for obtaining videos to be extracted; An audio file extraction module is used to extract the audio file of the video to be extracted, and divide the video to be extracted into multiple video frames to be extracted; each video frame to be extracted corresponds to a video frame timestamp; the video frame timestamp is used to describe the start time, end time and duration of the corresponding video frame to be extracted in the video to be extracted; A time-series text file determination module is used to input the audio file into a speech-to-text model to obtain a time-series text file of a video to be extracted; the speech-to-text model is obtained by training a first deep neural network using an audio file training set; each sentence of text in the time-series text file corresponds to a text timestamp; the text timestamp is used to describe the start time, end time and duration of the corresponding sentence in the video to be extracted; A text timestamp-text mapping relationship determination module, used to determine a non-time-sequential text file and a text timestamp-text mapping relationship according to the time-sequential text file; A text summary determination module, used for inputting the non-time-sequential text file into a text summary extraction model to obtain a text summary of the video to be extracted; the text summary extraction model is obtained by training a second deep neural network using a text file training set; A picture summary determination module, used to determine the picture summary of the video to be extracted according to an object recognition model and a plurality of the video frames to be extracted; the object recognition model is obtained by training a third deep neural network using a video frame training set; The video summary determination module is used to determine the video summary of the video to be extracted according to the text summary, the picture summary and the text timestamp-text mapping relationship; the video summary determination module is also used to split the text summary into multiple summary sentence vectors; determine that the multiple summary sentence vectors and the corresponding text timestamps in the time-series text file are a text timestamp set according to the text timestamp-text mapping relationship; determine the video frame timestamp corresponding to the picture summary as a video frame timestamp set; intercept multiple initial summary videos in the video to be extracted according to the intersection of the text timestamp set and the video frame timestamp set; use the ffmpeg tool to splice the multiple initial summary videos to obtain the video summary of the video to be extracted.

8. The video summary extraction system according to claim 7, characterized in that: The system further comprises: An audio file training set acquisition module is used to acquire an audio file training set; the audio file training set includes a plurality of historical audio files and historical text files corresponding to the plurality of historical audio files; The speech-to-text model determination module is used to train the first deep neural network with the historical audio file as input and the historical text file corresponding to the historical audio file as output to obtain the speech-to-text model.

9. The video summary extraction system according to claim 7, characterized in that: The system further comprises: A text file training set acquisition module is used to acquire a text file training set; the text file training set includes historical text files and text summaries corresponding to the historical text files; The text summary extraction model determination module is used to take the historical text file as input and the text summary corresponding to the historical text file as output, train the second deep neural network, and obtain the text summary extraction model.

Images