PPT-based video content cutting method and device, electronic equipment and medium

Abstract

The application relates to the field of vision and discloses a PPT-based video content cutting method and device, an electronic device and a storage medium, which can be applied to the field of financial technology and improves the problem of unsynchronized sound and picture during introduction of an insurance product video, wherein the method comprises the following steps: acquiring video frames of a video to be cut, calculating picture similarity between adjacent video frames, and taking adjacent video frames with picture similarity lower than a preset threshold as video segmentation time points; setting an image sliding time window for the video segmentation time points, extracting voice text information in the video to be cut, performing semantic division on the voice text information in the image sliding time window to obtain a division result; determining a target segmentation time point of the video to be cut according to the division result, and performing separation and smoothing on the video to be cut across semantics according to the target segmentation time point to obtain a target video. The application can realize accurate video content cutting of a training video containing PPT content.

Description

Technical Field

[0001] This invention relates to the field of vision, and more particularly to a method, apparatus, electronic device, and readable storage medium for video content cutting based on PPT. Background Technology

[0002] Video content cutting refers to the process of cutting a video based on the cut points of a complete video. For example, in the financial field, when a company needs to introduce an insurance product through a video, it needs to use video editing techniques.

[0003] Currently, there are two common methods for determining video segmentation points: one is based on image information, such as shot transitions or identifying the similarity between image frames, using those with low similarity as segmentation points; the other is based on text information in the video, such as using ASR (Automatic Speech Recognition) to identify the text information of the speech in the video, and then using semantic understanding to segment according to different semantic segments. These two methods are suitable for videos with clear knowledge point structures and videos without speech signals during image transitions. However, for training and explanation videos, the characteristic is that the presenter is constantly speaking and outputting speech information. At the same time, they will also switch PowerPoint slides to display the corresponding explanation content. This often results in the presenter speaking while switching slides, with the corresponding text semantic information spanning two different image contents. If only images are used as segmentation points, a complete sentence from the presenter will be truncated, resulting in incomplete video segmentation. If only text semantics are used as segmentation points, the PowerPoint slides will be turned, resulting in incomplete image information. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and readable storage medium for video content cutting based on PPT, with the aim of achieving accurate cutting of video content in training videos containing PPT content.

[0005] To achieve the above objectives, the present invention provides a video content cutting method based on PowerPoint presentations, the method comprising:

[0006] Obtain video frames from the video to be segmented, calculate the image similarity between adjacent video frames, and take the adjacent video frames with image similarity below a preset threshold as video segmentation time points;

[0007] A sliding time window is set for the video segmentation time point, the speech and text information in the video to be segmented is extracted, and the speech and text information within the sliding time window is semantically segmented to obtain the segmentation result.

[0008] Based on the segmentation results, the target segmentation time point of the video to be segmented is determined, and the video to be segmented across semantic boundaries is separated and smoothed according to the target segmentation time point to obtain the target video.

[0009] Optionally, calculating the image similarity between adjacent video frames includes:

[0010] Adjacent video frames with different verification results are converted to grayscale to obtain grayscale video frames;

[0011] The number of pixel values ​​in the grayscale video frame is counted, and the histogram of the grayscale video frame is calculated based on the number of pixel values.

[0012] The probability distribution of pixels with the same pixel value in the histogram is statistically analyzed, and the image similarity of the grayscale video frames is calculated based on the probability distribution.

[0013] Optionally, extracting the speech and text information from the video to be cut includes:

[0014] Extract the audio data from the video to be cut to obtain a speech sequence stream;

[0015] The speech sequence stream is feature extracted using a two-dimensional attention mechanism layer in a pre-defined speech recognition model to obtain a set of streaming speech feature sequence vectors.

[0016] The streaming speech feature sequence vector set is transformed into a streaming speech sequence vector matrix, and the streaming speech sequence vector matrix is ​​encoded using the coding layer in the deep speech recognition model to obtain a streaming speech sequence correlation matrix.

[0017] The streaming speech sequence association matrix is ​​partially masked using the masking multi-head attention mechanism layer in the preset speech recognition model to obtain the masked streaming speech sequence matrix.

[0018] The residual matrix of the masked streaming speech sequence matrix and the streaming speech sequence correlation matrix is ​​calculated using the residual connection layer in the preset speech recognition model to obtain the residual streaming speech sequence matrix.

[0019] The characters corresponding to the residual streaming speech sequence matrix are calculated using the fully connected layer in the preset speech recognition model to obtain the speech text information.

[0020] Optionally, the step of semantically segmenting the speech and text information within the sliding time window of the image to obtain the segmentation result includes:

[0021] Determine whether the speech-text information within the image sliding time window is continuous speech-text information;

[0022] When the speech and text information within the image sliding time window is not continuous speech and text information, the video to be cut is cut according to the video segmentation time point to obtain the target video;

[0023] When the speech text information within the image sliding time window is continuous speech text information, the continuous speech text information within the image sliding time window and the two speech text information adjacent to the continuous speech text information are respectively processed by word segmentation to obtain continuous word segmented speech text, front-end word segmented speech text and back-end word segmented speech text.

[0024] The first concatenated speech text is obtained by concatenating the front-end segmented speech text with the continuous segmented speech text using preset special characters.

[0025] The continuous segmented speech text is concatenated with the back-end segmented speech text using preset special characters to obtain a second concatenated speech text;

[0026] The first concatenated speech text and the second concatenated speech text are encoded using the encoding layer in the preset language representation model to obtain the first encoded speech text and the second encoded speech text.

[0027] The target values ​​of the first encoded speech text and the second encoded speech text are calculated respectively using the activation function of the fully connected layer in the preset language representation model to obtain the first target value and the second target value.

[0028] The value of the first target value is compared with the value of the second target value to obtain a comparison result. Based on the comparison result, the continuous speech and text information within the sliding time window of the image is divided to obtain a division result.

[0029] Optionally, before calculating the image similarity between adjacent video frames, the method further includes:

[0030] The video frames are semantically segmented using a pre-defined semantic segmentation network to obtain images of people and backgrounds.

[0031] The text information in the background image is extracted, and the text information is verified using a voting algorithm to obtain the verification result.

[0032] Optionally, the step of using a preset semantic segmentation network to perform image semantic segmentation on the video frames to obtain person images and background images includes:

[0033] The video frame is copied twice to obtain a first copied video frame, a second copied video frame, and the original video frame;

[0034] The first downsampling layer in the preset semantic segmentation network is used to downsample the first copied video frame, the second copied video frame and the original video frame at different scales to obtain low-resolution image, medium-resolution image and high-resolution image;

[0035] The low-resolution image, medium-resolution image, and high-resolution image are downsampled at the same scale using the second downsampling layer in the preset semantic segmentation network to obtain the first feature map, the second feature map, and the third feature map.

[0036] The first feature map is upsampled using a cascaded feature fusion layer in a preset semantic segmentation network to obtain a first upsampled feature map. The first upsampled feature map is then summed with each pixel value in the second feature map to obtain a first sum feature map. Finally, a preset activation function is used to perform linear calculation on the first sum feature map to obtain a first target feature map.

[0037] The first target feature map is upsampled using a cascaded feature fusion layer in a preset semantic segmentation network to obtain a second upsampled feature map. The second upsampled feature map is then summed with each pixel value in the third feature map to obtain a second sum feature map. Finally, a preset activation function is used to perform linear calculation on the second sum feature map to obtain a second target feature map.

[0038] The second target feature map is upsampled using the first upsampling layer in the preset semantic segmentation network to obtain an upsampled image;

[0039] The upsampled image is compared with a preset person label image and a preset background label image to obtain a comparison result. The upsampled image is then segmented based on the pixel differences in the comparison result to obtain an upsampled person image and an upsampled background image.

[0040] The upsampled person image and the upsampled background image are upsampled using the second upsampling layer in the preset semantic segmentation network to obtain the person image and the background image.

[0041] Optionally, extracting text information from the background image includes:

[0042] The pixel values ​​in the background image are converted to grayscale to obtain a grayscale background image;

[0043] The pixel values ​​in the grayscale background image are denoised to obtain a denoised background image;

[0044] The pixel values ​​in the denoised background image are binarized to obtain the target background image;

[0045] The target background image is segmented into characters to obtain text blocks;

[0046] The text image is compared with a preset character library to obtain the text information in the background image.

[0047] To address the above problems, the present invention also provides a video content cutting device based on PPT, the device comprising:

[0048] The video segmentation time point acquisition module is used to acquire video frames in the video to be segmented, calculate the image similarity between adjacent video frames, and take the adjacent video frames with image similarity below a preset threshold as video segmentation time points.

[0049] The speech and text information judgment module is used to set an image sliding time window for the video segmentation time point, extract speech and text information in the video to be segmented, perform semantic segmentation on the speech and text information within the image sliding time window, and obtain the segmentation result.

[0050] The video segmentation module is used to determine the target segmentation time point of the video to be segmented based on the segmentation result, and to separate and smooth the video to be segmented across semantics based on the target segmentation time point to obtain the target video.

[0051] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0052] Memory, storing at least one computer program; and

[0053] The processor executes the computer program stored in the memory to implement the PPT-based video content cutting method described above.

[0054] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the PPT-based video content cutting method described above.

[0055] This invention improves the accuracy of video segmentation by acquiring video frames from a video to be segmented, calculating the image similarity between adjacent video frames, and using adjacent video frames with image similarity below a preset threshold as video segmentation time points. The video to be segmented is judged from two dimensions: video background image text content and background image similarity. Furthermore, an image sliding time window is set for each video segmentation time point, and the speech-text information in the video to be segmented is extracted. Semantic segmentation is performed on the speech-text information within the image sliding time window to obtain the segmentation result. Based on the segmentation result, the target segmentation time point of the video to be segmented is determined, and the video to be segmented across semantic boundaries is separated and smoothed according to the target segmentation time point to obtain the target video. This ensures that there is no mismatch between speech and background in the target video, thus completing the video content segmentation of training videos containing PPT content. Therefore, this invention provides a PPT-based video content segmentation method, apparatus, device, and storage medium that can achieve accurate segmentation of video content in training videos containing PPT content. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating a video content cutting method based on PPT provided in an embodiment of the present invention.

[0057] Figures 2 to 3 A detailed implementation flowchart of one step in a PPT-based video content cutting method provided in an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of a PPT-based video content cutting device according to an embodiment of the present invention;

[0059] Figure 5 A schematic diagram of the internal structure of an electronic device for implementing a PPT-based video content cutting method according to an embodiment of the present invention;

[0060] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0061] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0062] This invention provides a method for video content segmentation based on PowerPoint presentations. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the method can be executed by software or hardware installed on a terminal device or a server device; the software may be a blockchain platform. The server may include an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0063] Reference Figure 1 The flowchart shown is a PPT-based video content cutting method according to an embodiment of the present invention. In this embodiment, the PPT-based video content cutting method includes:

[0064] S1. Obtain video frames from the video to be cut, calculate the image similarity between adjacent video frames, and take the adjacent video frames with image similarity below a preset threshold as video segmentation time points.

[0065] In this embodiment of the invention, the video to be segmented can be a training video containing PowerPoint presentations, such as a training video for new employees in the financial industry. The image similarity can be the degree of similarity between the video frames. The preset threshold can be a value determined by the operator based on the results of training with multiple sets of data.

[0066] In an optional embodiment of the present invention, the video to be cut is received by the user through the video receiving port, and the video frames of the video to be cut are extracted one by one to obtain the video frames of the video to be cut. Furthermore, since the video frames need to be input into a preset semantic segmentation network, the video frames need to be processed to obtain input that conforms to the semantic segmentation network.

[0067] In this embodiment of the invention, in order to reduce the difficulty of calculating the image similarity between adjacent video frames, the video frames can be semantically segmented to obtain person images and background images before calculating the image similarity between adjacent video frames, and the similarity of the background images between adjacent video frames can be calculated separately, thereby improving the efficiency of PPT-based video content cutting.

[0068] Specifically, before calculating the image similarity between adjacent video frames, the method further includes:

[0069] The video frames are semantically segmented using a pre-defined semantic segmentation network to obtain images of people and backgrounds.

[0070] The text information in the background image is extracted, and the text information is verified using a voting algorithm to obtain the verification result.

[0071] In this embodiment of the invention, the preset semantic segmentation network can be a trained ICNet (Image Cascade Network) capable of processing high-resolution images. The voting algorithm can be an algorithm for finding the mode in an array.

[0072] In an optional embodiment of the present invention, the video to be cut is received by the user through the video receiving port, and the video frames of the video to be cut are extracted one by one to obtain the video frames of the video to be cut. Furthermore, since the video frames need to be input into a preset semantic segmentation network, the video frames need to be processed to obtain input that conforms to the semantic segmentation network.

[0073] In this embodiment of the invention, the preset semantic segmentation network includes a cascaded feature fusion layer, a first upsampling layer, a second upsampling layer, a first downsampling layer, and a second downsampling layer, wherein the first downsampling layer and the second downsampling layer can perform convolutional pooling on the image; and the first upsampling layer and the second upsampling layer can perform deconvolution operations on the image.

[0074] Furthermore, as an optional embodiment of the present invention, reference is made to... Figure 2 As shown, the step of using a preset semantic segmentation network to perform image semantic segmentation on the video frames to obtain person images and background images includes:

[0075] S11. Copy the video frame twice to obtain the first copied video frame, the second copied video frame, and the original video frame;

[0076] S12. Using the first downsampling layer in the preset semantic segmentation network, the first copied video frame, the second copied video frame and the original video frame are downsampled at different scales to obtain low-resolution images, medium-resolution images and high-resolution images.

[0077] S13. Using the second downsampling layer in the preset semantic segmentation network, the low-resolution image, medium-resolution image and high-resolution image are downsampled at the same scale to obtain the first feature map, the second feature map and the third feature map.

[0078] S14. Upsample the first feature map using the cascaded feature fusion layer in the preset semantic segmentation network to obtain a first upsampled feature map. Sum the first upsampled feature map with each pixel value in the second feature map to obtain a first sum feature map. Then, use a preset activation function to perform linear calculation on the first sum feature map to obtain a first target feature map.

[0079] S15. Upsample the first target feature map using the cascaded feature fusion layer in the preset semantic segmentation network to obtain a second upsampled feature map. Sum the second upsampled feature map with each pixel value in the third feature map to obtain a second sum feature map. Perform linear calculation on the second sum feature map using a preset activation function to obtain a second target feature map.

[0080] S16. Upsample the second target feature map using the first upsampling layer in the preset semantic segmentation network to obtain an upsampled image;

[0081] S17. Compare the upsampled image with the preset person label image and the preset background label image to obtain the comparison result, and segment the upsampled image according to the pixel difference in the comparison result to obtain the upsampled person image and the upsampled background image.

[0082] S18. The upsampled person image and the upsampled background image are upsampled using the second upsampling layer in the preset semantic segmentation network to obtain the person image and the background image.

[0083] In this embodiment of the invention, the preset activation function can be a ReLU activation function. The preset person label image can be a label image containing person features. The preset background label image can be a label image containing PPT document features.

[0084] In this embodiment of the invention, the different scales of downsampling the first copied video frame, the second copied video frame, and the original video frame using a first downsampling layer in a preset semantic segmentation network are typically one-quarter, one-half, and the original scale. The same scale of downsampling the low-resolution image, the medium-resolution image, and the high-resolution image using a second downsampling layer in the preset semantic segmentation network is typically one-eighth. The upsampling of the first feature map using a cascaded feature fusion layer in the preset semantic segmentation network results in an upsampling factor of typically 2 times. The upsampling of the first target feature map using a cascaded feature fusion layer in the preset semantic segmentation network results in an upsampling factor of typically 2 times. The upsampling of the second target feature map using a first upsampling layer in the preset semantic segmentation network results in an upsampling factor of typically 2 times. The upsampling of the upsampled person image and the upsampled background image using a second upsampling layer in the preset semantic segmentation network results in an upsampling factor of typically 4 times.

[0085] In an optional embodiment of the present invention, video frames are copied and input into a waterfall image input channel consisting of three channels to obtain a low-resolution image, a medium-resolution image, and a high-resolution image. Further, the low-resolution image, the medium-resolution image, and the high-resolution image are respectively input into a PSP network with a downsampling rate of 8 to obtain a first feature map, a second feature map, and a third feature map. The first feature map, the second feature map, and the third feature map are respectively input into a CFF (cascade-feature-fusion) module to obtain a second target feature map. The second target feature map is then upsampled by a coefficient of 2 to obtain a feature map of 1 / 4 of the original resolution. At this time, after referencing label guidance of 1 / 4 size, upsampling is performed with a coefficient of 4 to obtain the final result.

[0086] This invention utilizes a pre-defined semantic segmentation network to perform image semantic segmentation on the video frames, obtaining images of people and backgrounds. This provides background and person templates for cutting the video to be cut, ensuring that training videos containing PPT presentations can be accurately segmented. For example, in the fintech industry, if a company's newly launched insurance product requires a segmented product introduction in video format, the pre-defined semantic segmentation network can be used to segment the insurance product presenter and the insurance product introduction in the PPT video before editing, thereby reducing the possibility of audio and background asynchrony in insurance product introduction videos containing PPT presentations.

[0087] Furthermore, in this embodiment of the invention, text information is extracted from the background image to determine whether the background in the background image has changed.

[0088] Specifically, extracting the text information from the background image includes:

[0089] The pixel values ​​in the background image are converted to grayscale to obtain a grayscale background image;

[0090] The pixel values ​​in the grayscale background image are denoised to obtain a denoised background image;

[0091] The pixel values ​​in the denoised background image are binarized to obtain the target background image;

[0092] The target background image is segmented into characters to obtain text blocks;

[0093] The text image is compared with a preset character library to obtain the text information in the background image.

[0094] In this embodiment of the invention, the preset character library can be a database containing various text information, such as Chinese character information and English information.

[0095] In an optional embodiment of the present invention, the computational difficulty is reduced by converting the colored portion of the background image into a grayscale image. Furthermore, to ensure the accuracy of text extraction, the grayscale background image is denoised by changing the pixel values ​​with large differences in the grayscale background image to obtain a denoised background image. Further, the pixel values ​​in the denoised background image are divided into black and white categories to obtain a target background image that is either black or white. The black pixel portion in the target background image is the text portion. Finally, the text portion is compared with a preset character library to obtain the text information corresponding to the text portion.

[0096] This invention utilizes a voting algorithm to verify the text information, obtaining a verification result that ensures the consistency of the text information and provides an accurate background image for subsequent video segmentation of training videos containing PPT presentations.

[0097] Furthermore, as an optional embodiment of the present invention, the step of using a voting algorithm to verify the text information and obtain a verification result includes:

[0098] The text information corresponding to each video frame is compared with the text information corresponding to the other video frames to obtain the comparison result. Based on the comparison result, the text information is divided into groups to obtain text information groups.

[0099] The maximum common string region in the text information group is extracted using the maximum string matching algorithm;

[0100] The text information group is converted into a string region using a string conversion algorithm, and the regions in the string region that are different from the maximum common string region are designated as different string regions.

[0101] Calculate the frequency of each string in different string regions within the text information group, and take the string with the highest frequency as the target string for the different string regions;

[0102] The strings in the largest common string region are normalized and integrated with the target string to obtain the target string region. The content of the target string region is then checked to obtain the verification result.

[0103] In this embodiment of the invention, the maximum string matching algorithm can be the KMP (Knuth Morris Pratt, string search) algorithm.

[0104] In this embodiment of the invention, before verifying the text information using a voting algorithm and obtaining the verification result, in order to ensure the accuracy of the verification result, it is also necessary to compare the text information corresponding to different video frames and classify the text information into groups.

[0105] Furthermore, in this embodiment of the invention, the image similarity between adjacent video frames is calculated, and adjacent video frames with image similarity below a preset threshold are used as video segmentation time points to further verify the accuracy of the video frame grouping and ensure the precision of the PPT-based video content cutting.

[0106] Further, as an optional embodiment of the present invention, calculating the image similarity between adjacent video frames includes:

[0107] Adjacent video frames are converted to grayscale to obtain grayscale video frames;

[0108] The number of pixel values ​​in the grayscale video frame is counted, and the histogram of the grayscale video frame is calculated based on the number of pixel values.

[0109] The probability distribution of pixels with the same pixel value in the histogram is statistically analyzed, and the image similarity of the grayscale video frames is calculated based on the probability distribution.

[0110] In this embodiment of the invention, the histogram may be a statistical report chart composed of a series of vertical stripes or line segments of varying heights.

[0111] In an optional embodiment of the present invention, image similarity is calculated by calculating the histogram pixel distribution probability of adjacent video frames, and the calculation method is relatively simple, thereby improving the calculation speed of image similarity.

[0112] Furthermore, in another optional embodiment of the present invention, when the image similarity is lower than a preset threshold, it indicates that there are many different regions between the adjacent video frames. Therefore, the adjacent video frames with image similarity lower than the preset threshold can be used as video segmentation time points.

[0113] S2. Set an image sliding time window for the video segmentation time point, extract the speech and text information in the video to be segmented, and perform semantic segmentation on the speech and text information within the image sliding time window to obtain the segmentation result. In this embodiment of the invention, the image sliding time window can be a time-sliding window in a series of video frames.

[0114] In an optional embodiment of the present invention, the size of the image sliding time window can be set according to the experience of professionals to improve the efficiency of video content cutting.

[0115] Furthermore, embodiments of the present invention can utilize a preset speech recognition model to extract speech and text information from the video to be segmented. The preset speech recognition model can be a trained deep learning-based speech model, including a two-dimensional attention mechanism layer, an encoding layer, a masked multi-head attention mechanism layer, a residual connection layer, and a fully connected layer.

[0116] This invention utilizes a preset speech recognition model to extract speech and text information from the video to be cut, ensuring the accuracy of the speech dimension of the video to be cut, and providing the cutting points of the speech dimension of the video to be cut, thereby making the video content cutting more precise.

[0117] Furthermore, as an optional embodiment of the present invention, reference is made to... Figure 3 As shown, the extraction of speech and text information from the video to be cut includes:

[0118] S21. Extract the audio data from the video to be cut to obtain a speech sequence stream;

[0119] S22. Use the two-dimensional attention mechanism layer in the preset speech recognition model to extract features from the speech sequence stream to obtain a streaming speech feature sequence vector set;

[0120] S23. The streaming speech feature sequence vector set is converted into a streaming speech sequence vector matrix, and the streaming speech sequence vector matrix is ​​encoded using the coding layer in the deep speech recognition model to obtain a streaming speech sequence correlation matrix.

[0121] S24. The streaming speech sequence association matrix is ​​partially masked using the masking multi-head attention mechanism layer in the preset speech recognition model to obtain the masked streaming speech sequence matrix.

[0122] S25. Calculate the residual matrix of the masked streaming speech sequence matrix and the streaming speech sequence correlation matrix using the residual connection layer in the preset speech recognition model to obtain the residual streaming speech sequence matrix.

[0123] S26. Calculate the corresponding characters of the residual streaming speech sequence matrix using the fully connected layer in the preset speech recognition model to obtain speech text information.

[0124] In this embodiment of the invention, the streaming speech sequence vector matrix can be a matrix formed by concatenating streaming speech feature sequence vector sets. The streaming speech sequence association matrix refers to a matrix containing the semantic features of the preceding and following characters of the streaming speech feature sequence vector set, and the dimensions of the streaming speech sequence vector matrix and the streaming speech sequence association matrix are the same.

[0125] This invention, through its embodiments, extracts audio-text information from the video to be segmented, and determines the video segmentation time point of the video at the text level. For example, in a PPT video introducing an insurance product, by extracting audio-text information from the background PPT document in the introduction PPT video, it is confirmed whether the words spoken by the product presenter in the introduction PPT video are the same as the audio-text information, thereby determining the video segmentation time point of the introduction PPT video.

[0126] In an optional embodiment of the present invention, the main purpose of using the masking multi-head attention mechanism layer is to help the current node obtain the important content that needs to be focused on. The masked content is the text label after the current node in the real label sequence. Moreover, the masking multi-head attention mechanism layer can make the clean part and the noisy part of the input speech features more obvious, thereby masking the noise part, which can more effectively suppress noise and further improve the accuracy of speech recognition.

[0127] This invention embodiment performs semantic segmentation on the speech and text information within the sliding time window of the image to obtain the segmentation result, thereby determining the region to which the continuous speech and text information belongs, and thus more accurately performing video content segmentation on the video to be segmented.

[0128] Furthermore, as an optional embodiment of the present invention, the step of semantically segmenting the speech and text information within the image sliding time window to obtain the segmentation result includes:

[0129] Determine whether the speech-text information within the image sliding time window is continuous speech-text information;

[0130] When the speech and text information within the image sliding time window is not continuous speech and text information, the video to be cut is cut according to the video segmentation time point to obtain the target video;

[0131] When the speech text information within the image sliding time window is continuous speech text information, the continuous speech text information within the image sliding time window and the two speech text information adjacent to the continuous speech text information are respectively processed by word segmentation to obtain continuous word segmented speech text, front-end word segmented speech text and back-end word segmented speech text.

[0132] The first concatenated speech text is obtained by concatenating the front-end segmented speech text with the continuous segmented speech text using preset special characters.

[0133] The continuous segmented speech text is concatenated with the back-end segmented speech text using preset special characters to obtain a second concatenated speech text;

[0134] The first concatenated speech text and the second concatenated speech text are encoded using the encoding layer in the preset language representation model to obtain the first encoded speech text and the second encoded speech text.

[0135] The target values ​​of the first encoded speech text and the second encoded speech text are calculated respectively using the activation function of the fully connected layer in the preset language representation model to obtain the first target value and the second target value.

[0136] The value of the first target value is compared with the value of the second target value to obtain a comparison result. Based on the comparison result, the continuous speech and text information within the sliding time window of the image is divided to obtain a division result.

[0137] In this embodiment of the invention, the continuous speech text can be speech text information with contextual relevance. The preset language representation model can be a trained BERT (Bidirectional Encoder Representations from Transformer) model. The front-end segmented speech text can be the speech text information preceding the position of the continuous speech text information in the video frame. The back-end segmented speech text can be the speech text information following the position of the continuous speech text information in the video frame. The fully connected layer is an artificial neural network constructed from multilayer perceptrons. The activation function can be the sigmoid activation function.

[0138] In an optional embodiment of the present invention, due to the characteristics of training videos containing PPT presentations, situations may easily arise where the background PPT has changed but the instructor is still narrating the previous content, or the instructor is already narrating the next segment but the background PPT has not changed. This results in audio-visual asynchrony, making it impossible to accurately segment the video content of the video containing PPT. Therefore, before segmenting the video content of the video containing PPT, it is necessary to determine whether the audio-text information corresponding to the video frames on both sides of the video segmentation time point is continuous audio-text information.

[0139] Furthermore, in an optional embodiment of the present invention, the text content of the voice information is first identified; secondly, the voice text information is segmented according to the text content to obtain voice text information segments; and finally, according to the number of video frames corresponding to the voice text information segments, it is determined whether the voice text information corresponding to the video frames on both sides of the video segmentation time point is continuous voice text information.

[0140] In an optional embodiment of the present invention, when the audio-visual information corresponding to the video frames on both sides of the video segmentation time point is not continuous audio-visual information, it indicates that there is no situation where the background PPT has changed but the lecturer is still talking about the previous content, or the lecturer is talking about the next part of the content but the background PPT has not changed. That is, the video to be cut containing the PPT explanation does not have the problem of audio-visual asynchrony. Therefore, the target video can be obtained by cutting the video to be cut according to the video segmentation time point.

[0141] In this embodiment of the invention, the comparison of the first target value and the second target value to obtain a comparison result can have the following two possibilities: First, the first target value is less than the second target value, meaning the context of the continuous segmented speech text is more closely connected to the context of the front-end segmented speech text. Therefore, the continuous speech text information corresponding to the continuous segmented speech text can be divided into the speech text information segment corresponding to the front-end segmented speech text. Second, the first target value is greater than the second target value, meaning the context of the continuous segmented speech text is more closely connected to the context of the back-end segmented speech text. Therefore, the continuous speech text information corresponding to the continuous segmented speech text can be divided into the speech text information segment corresponding to the back-end segmented speech text.

[0142] This invention simplifies the calculation of determining the problem to which the continuous speech text information belongs by concatenating the continuous speech text information with the preceding and following speech text information, and then using an activation function to calculate the value after concatenation. This also improves the accuracy of cutting the video to be cut.

[0143] S3. Based on the division results, determine the target segmentation time point of the video to be cut, and perform semantic separation and smoothing on the video to be cut across semantics based on the target segmentation time point to obtain the target video.

[0144] In an optional embodiment of the present invention, when the segmentation result is obtained, the frame number of the video frame corresponding to the end of the continuous voice text information can be used as the target segmentation time point.

[0145] In an optional embodiment of the present invention, the background image of the video frame corresponding to the continuous speech text information is replaced with the background image of the video frame corresponding to the segmentation result. The background image of the corresponding video frame refers to the integration of the string in the largest common string region of the text information group where the text information corresponding to the video frame is located with the target string, thereby ensuring that there will not be a significant change when the background image of the video frame corresponding to the continuous speech text information is connected with the background image of the video frame corresponding to the segmentation result.

[0146] Furthermore, in this embodiment of the invention, since voice information is more difficult to modify than image information, the division point in the division result is selected as the target segmentation time point. Based on the target segmentation time point, the video to be cut, which spans semantics, is separated and smoothed to obtain the target video. This improves the ease of content cutting of the video to be cut and improves the efficiency and accuracy of cutting the content of the video to be cut.

[0147] This invention improves the accuracy of video segmentation by acquiring video frames from a video to be segmented, calculating the image similarity between adjacent video frames, and using adjacent video frames with image similarity below a preset threshold as video segmentation time points. The video to be segmented is judged from two dimensions: video background image text content and background image similarity. Furthermore, an image sliding time window is set for each video segmentation time point, and the speech-text information in the video to be segmented is extracted. Semantic segmentation is performed on the speech-text information within the image sliding time window to obtain the segmentation result. Based on the segmentation result, the target segmentation time point of the video to be segmented is determined, and the video to be segmented across semantic boundaries is separated and smoothed according to the target segmentation time point to obtain the target video. This ensures that there is no mismatch between speech and background in the target video, thus completing the video content segmentation of training videos containing PPT content. Therefore, this invention provides a PPT-based video content segmentation method, apparatus, device, and storage medium that can achieve accurate segmentation of video content in training videos containing PPT content.

[0148] like Figure 4The diagram shown is a functional block diagram of the video content cutting device based on PPT according to the present invention.

[0149] The PPT-based video content cutting device 100 of the present invention can be installed in an electronic device. Depending on the functions implemented, the PPT-based video content cutting device 100 may include a video segmentation time point acquisition module 101, a voice and text information judgment module 102, and a video cutting module 103. The module in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0150] In this embodiment, the functions of each module / unit are as follows:

[0151] The video segmentation time point acquisition module 101 is used to acquire video frames in the video to be segmented, calculate the image similarity between adjacent video frames, and take the adjacent video frames with image similarity lower than a preset threshold as video segmentation time points.

[0152] The speech and text information judgment module 102 is used to set an image sliding time window for the video segmentation time point, extract speech and text information in the video to be segmented, perform semantic segmentation on the speech and text information within the image sliding time window, and obtain the segmentation result.

[0153] The video cutting module 103 is used to determine the target segmentation time point of the video to be cut according to the segmentation result, and to separate and smooth the video to be cut across semantics according to the target segmentation time point to obtain the target video.

[0154] like Figure 5 The diagram shown is a structural schematic of the electronic device that implements the PPT-based video content cutting method of the present invention.

[0155] The electronic device may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a PPT-based video content cutting program.

[0156] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as code for a PowerPoint-based video content cutting program, but also to temporarily store data that has been output or will be output.

[0157] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., a PPT-based video content cutting program) and calls data stored in the memory 11 to perform various functions and process data within the electronic device.

[0158] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The communication bus 12 is configured to enable communication between the memory 11 and at least one processor 10, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus.

[0159] Figure 5 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 5The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0160] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0161] Optionally, the communication interface 13 may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device and other electronic devices.

[0162] Optionally, the communication interface 13 may further include a user interface, which may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0163] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0164] The PPT-based video content cutting program stored in the memory 11 of the electronic device is a combination of multiple computer programs. When run in the processor 10, it can achieve the following:

[0165] Obtain video frames from the video to be segmented, calculate the image similarity between adjacent video frames, and take the adjacent video frames with image similarity below a preset threshold as video segmentation time points;

[0166] A sliding time window is set for the video segmentation time point, the speech and text information in the video to be segmented is extracted, and the speech and text information within the sliding time window is semantically segmented to obtain the segmentation result.

[0167] Based on the segmentation results, the target segmentation time point of the video to be segmented is determined, and the video to be segmented across semantic boundaries is separated and smoothed according to the target segmentation time point to obtain the target video.

[0168] Specifically, the processor 10's implementation method of the above-mentioned computer program can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0169] Furthermore, if the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or volatile. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0170] Embodiments of the present invention may also provide a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0171] Obtain video frames from the video to be segmented, calculate the image similarity between adjacent video frames, and take the adjacent video frames with image similarity below a preset threshold as video segmentation time points;

[0172] A sliding time window is set for the video segmentation time point, the speech and text information in the video to be segmented is extracted, and the speech and text information within the sliding time window is semantically segmented to obtain the segmentation result.

[0173] Based on the segmentation results, the target segmentation time point of the video to be segmented is determined, and the video to be segmented across semantic boundaries is separated and smoothed according to the target segmentation time point to obtain the target video.

[0174] Furthermore, the computer's usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system, applications required for at least one function, etc.; and the data storage area may store data created based on the use of blockchain nodes, etc.

[0175] In the several embodiments provided by this invention, it should be understood that the disclosed electronic devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0176] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0178] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0179] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0180] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0181] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for video content cutting based on PPT, characterized in that, The method includes: Obtain video frames from the video to be segmented, calculate the image similarity between adjacent video frames, and take the adjacent video frames with image similarity below a preset threshold as video segmentation time points; A sliding time window is set for the video segmentation time point, the speech and text information in the video to be segmented is extracted, and the speech and text information within the sliding time window is semantically segmented to obtain the segmentation result. Based on the segmentation results, the target segmentation time point of the video to be segmented is determined, and the video to be segmented across semantics is separated and smoothed according to the target segmentation time point to obtain the target video; The semantic segmentation of the speech and text information within the sliding time window of the image, to obtain the segmentation result, includes: Determine whether the speech-text information within the image sliding time window is continuous speech-text information; When the speech and text information within the image sliding time window is not continuous speech and text information, the video to be cut is cut according to the video segmentation time point to obtain the target video; When the speech text information within the image sliding time window is continuous speech text information, the continuous speech text information within the image sliding time window and the two speech text information adjacent to the continuous speech text information are respectively processed by word segmentation to obtain continuous word segmented speech text, front-end word segmented speech text and back-end word segmented speech text. The front-end segmented speech text is concatenated with the continuous segmented speech text using preset special characters to obtain the first concatenated speech text; The continuous segmented speech text is concatenated with the back-end segmented speech text using preset special characters to obtain a second concatenated speech text; The first concatenated speech text and the second concatenated speech text are encoded using the encoding layer in the preset language representation model to obtain the first encoded speech text and the second encoded speech text. The target values ​​of the first encoded speech text and the second encoded speech text are calculated respectively using the activation function of the fully connected layer in the preset language representation model to obtain the first target value and the second target value. The value of the first target value is compared with the value of the second target value to obtain a comparison result. Based on the comparison result, the continuous speech and text information within the sliding time window of the image is divided to obtain a division result.

2. The video content cutting method based on PPT as described in claim 1, characterized in that, The calculation of image similarity between adjacent video frames includes: Adjacent video frames are converted to grayscale to obtain grayscale video frames; The number of pixel values ​​in the grayscale video frame is counted, and the histogram of the grayscale video frame is calculated based on the number of pixel values. The probability distribution of pixels with the same pixel value in the histogram is statistically analyzed, and the image similarity of the grayscale video frames is calculated based on the probability distribution.

3. The video content cutting method based on PPT as described in claim 1, characterized in that, The extraction of speech and text information from the video to be cut includes: Extract the audio data from the video to be cut to obtain a speech sequence stream; The speech sequence stream is feature extracted using a two-dimensional attention mechanism layer in a pre-defined speech recognition model to obtain a set of streaming speech feature sequence vectors. The streaming speech feature sequence vector set is transformed into a streaming speech sequence vector matrix, and the streaming speech sequence vector matrix is ​​encoded using the coding layer in the speech recognition model to obtain the streaming speech sequence association matrix. The streaming speech sequence association matrix is ​​partially masked using the masking multi-head attention mechanism layer in the preset speech recognition model to obtain the masked streaming speech sequence matrix. The residual matrix of the masked streaming speech sequence matrix and the streaming speech sequence correlation matrix is ​​calculated using the residual connection layer in the preset speech recognition model to obtain the residual streaming speech sequence matrix. The characters corresponding to the residual streaming speech sequence matrix are calculated using the fully connected layer in the preset speech recognition model to obtain the speech text information.

4. The video content cutting method based on PPT as described in claim 1, characterized in that, Before calculating the image similarity between adjacent video frames, the method further includes: The video frames are semantically segmented using a pre-defined semantic segmentation network to obtain images of people and backgrounds. The text information in the background image is extracted, and the text information is verified using a voting algorithm to obtain the verification result.

5. The video content cutting method based on PPT as described in claim 4, characterized in that, The step of performing image semantic segmentation on the video frames using a preset semantic segmentation network to obtain person images and background images includes: The video frame is copied twice to obtain a first copied video frame, a second copied video frame, and the original video frame; The first downsampling layer in the preset semantic segmentation network is used to downsample the first copied video frame, the second copied video frame and the original video frame at different scales to obtain low-resolution image, medium-resolution image and high-resolution image; The low-resolution image, medium-resolution image, and high-resolution image are downsampled at the same scale using the second downsampling layer in the preset semantic segmentation network to obtain the first feature map, the second feature map, and the third feature map. The first feature map is upsampled using a cascaded feature fusion layer in a preset semantic segmentation network to obtain a first upsampled feature map. The first upsampled feature map is then summed with each pixel value in the second feature map to obtain a first sum feature map. Finally, a preset activation function is used to perform linear calculation on the first sum feature map to obtain a first target feature map. The first target feature map is upsampled using a cascaded feature fusion layer in a preset semantic segmentation network to obtain a second upsampled feature map. The second upsampled feature map is then summed with each pixel value in the third feature map to obtain a second sum feature map. Finally, a preset activation function is used to perform linear calculation on the second sum feature map to obtain a second target feature map. The second target feature map is upsampled using the first upsampling layer in the preset semantic segmentation network to obtain an upsampled image; The upsampled image is compared with a preset person label image and a preset background label image to obtain a comparison result. The upsampled image is then segmented based on the pixel differences in the comparison result to obtain an upsampled person image and an upsampled background image. The upsampled person image and the upsampled background image are upsampled using the second upsampling layer in the preset semantic segmentation network to obtain the person image and the background image.

6. The video content cutting method based on PPT as described in claim 4, characterized in that, The step of extracting text information from the background image includes: The pixel values ​​in the background image are converted to grayscale to obtain a grayscale background image; The pixel values ​​in the grayscale background image are denoised to obtain a denoised background image. The pixel values ​​in the denoised background image are binarized to obtain the target background image; The target background image is segmented into characters to obtain text blocks; The text image is compared with a preset character library to obtain the text information in the background image.

7. A PPT-based video content cutting device, used to implement the PPT-based video content cutting method as described in any one of claims 1 to 6, characterized in that, The device includes: The video segmentation time point acquisition module is used to acquire video frames in the video to be segmented, calculate the image similarity between adjacent video frames, and take the adjacent video frames with image similarity below a preset threshold as video segmentation time points. The speech and text information judgment module is used to set an image sliding time window for the video segmentation time point, extract speech and text information in the video to be segmented, perform semantic segmentation on the speech and text information within the image sliding time window, and obtain the segmentation result. The video segmentation module is used to determine the target segmentation time point of the video to be segmented based on the segmentation result, and to separate and smooth the video to be segmented across semantics based on the target segmentation time point to obtain the target video.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the PPT-based video content cutting method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the PPT-based video content cutting method as described in any one of claims 1 to 6.

Images