Short video time axis slicing method and device

Abstract

The embodiment of the invention provides a short video time axis slicing method and device, and the method comprises the steps: receiving a short video file uploaded by a user, and carrying out the mirror splitting of the short video file through a preset computer vision algorithm, and obtaining a plurality of video segments; performing semantic clustering on the original video subtitles and the supplementary video subtitles, determining a time axis of each video clip, and rendering the original video subtitles and the supplementary video subtitles into the corresponding video clips according to the time axes to obtain an updated short video file; according to the invention, short video subtitle processing can be accurately and efficiently carried out.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a short video timeline slicing method and device. Background Art

[0002] With the popularity of mobile Internet and smart devices, short videos have become one of the most popular content forms today. Users can easily create, share and watch various short video content. However, with the development of short video platforms, users have higher requirements for the quality and accessibility of video content. Among them, the accuracy and completeness of video subtitles have become an important factor affecting user experience.

[0003] Traditional short video processing methods usually rely on simple speech recognition technology to generate subtitles. This method works well when processing clear speech content, but often performs poorly when faced with complex scenes, background noise, or no speech content. In addition, traditional methods have difficulty accurately identifying scene transitions and content changes in videos, resulting in the generated subtitles being out of sync with the video content or missing important information.

[0004] To solve these problems, researchers began to explore more advanced video processing technologies. Computer vision algorithms were introduced to achieve more accurate video storyboards, making it possible to structure video content. At the same time, the development of deep learning technology has brought new breakthroughs in video content understanding. Multimodal learning methods can comprehensively utilize the visual, audio, and text information of a video to more comprehensively understand the video content.

[0005] However, existing technologies still face some challenges. First, how to effectively process speechless segments in videos is still a difficult problem. Second, how to accurately synchronize the generated subtitles with the video content also requires further research. In addition, how to improve processing efficiency while ensuring the accuracy of subtitles to meet the needs of large-scale short video processing is also an important issue.

[0006] At present, some researchers are trying to introduce knowledge graph technology into the field of video processing to improve the ability to understand the semantics of video content. Knowledge graphs can provide rich background knowledge and help solve the shortcomings of traditional methods in dealing with complex scenes. At the same time, the attention mechanism has achieved remarkable results in the fields of natural language processing and computer vision, and its application in video processing has also begun to attract attention.

[0007] Although these new technologies have brought new possibilities to short video processing, it is still a challenge to effectively integrate these technologies and build a unified and efficient short video processing framework. In particular, when processing large-scale short video data, how to balance processing accuracy and efficiency, and how to achieve accurate timeline slicing and subtitle generation all require further research and exploration. Summary of the invention

[0008] In response to the problems in the prior art, the present application provides a short video timeline slicing method and device, which can accurately and efficiently process short video subtitles.

[0009] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0010] In a first aspect, the present application provides a short video timeline slicing method, comprising:

[0011] Receive a short video file uploaded by a user, and segment the short video file using a preset computer vision algorithm to obtain multiple video clips;

[0012] Subtitles are extracted from the video clip by a preset speech recognition algorithm to obtain the corresponding original video subtitles, and the video clip to be supplemented that lacks the corresponding video subtitles is determined, the spatiotemporal features of the video clip to be supplemented are extracted by a frame division operation, a low-order feature representation is obtained according to the spatiotemporal features, the extracted low-order feature representation is unified into a feature set and input into a set deep learning multi-layer perceptron to obtain the corresponding high-order feature representation, feature weighted fusion is performed on each of the target high-order feature representations through an attention mechanism to obtain the target feature representation of the video clip to be supplemented, semantic recognition is performed on the target feature representation by a preset knowledge graph inference algorithm, the result of the semantic recognition is input into a set autoregressive generation model, and the supplementary video subtitles corresponding to the video clip to be supplemented are determined according to the output of the autoregressive generation model;

[0013] The original video subtitles and the supplementary video subtitles are semantically clustered to determine the timeline of each of the video segments, and the original video subtitles and the supplementary video subtitles are rendered into the corresponding video segments according to the timeline to obtain an updated short video file.

[0014] Furthermore, the short video file uploaded by the user is received, and the short video file is segmented by a preset computer vision algorithm to obtain multiple video clips, including:

[0015] Receive a short video file uploaded by a user through a preset video upload interface, perform a format check and a security scan on the short video file, and store the short video file that has undergone the format check and the security scan in a distributed storage system on a preset server side;

[0016] Extract low-level visual features of color, texture and shape, and high-level semantic features of objects, faces and actions of each video frame in the short video file, calculate a saliency score of each video frame based on the extracted low-level visual features and high-level semantic features, determine key shot points of the video according to the saliency score, and divide the short video file into multiple video clips according to the determined key shot points.

[0017] Furthermore, the extracting subtitles from the video clips by using a preset speech recognition algorithm to obtain corresponding original video subtitles, and determining the video clips to be supplemented that lack corresponding video subtitles, includes:

[0018] Using a pre-trained speech recognition model to transcribe the audio content in the video clip, extracting corresponding original video subtitles, and performing a rationality check on the extracted original video subtitles;

[0019] The video segments to be supplemented that need to be supplemented with subtitles are determined based on the extracted original video subtitles, video duration, and picture change characteristics.

[0020] Furthermore, the extracting the spatiotemporal features of the video segment to be supplemented by the frame division operation and obtaining a low-order feature representation according to the spatiotemporal features includes:

[0021] Performing a frame division operation on the video segment to be supplemented that is determined to need to be supplemented with subtitles, dividing it into a continuous video frame sequence, and extracting spatiotemporal features including at least one of color, texture, and motion information for each video frame in the video frame sequence;

[0022] The extracted spatiotemporal features are compressed into a compact low-level feature representation, wherein the low-level feature representation includes at least basic visual and dynamic characteristics of video content.

[0023] Furthermore, the extracted low-order feature representations are unified into a feature set and input into a set deep learning multi-layer perceptron to obtain corresponding high-order feature representations, including:

[0024] Integrating the extracted low-level feature representations into a unified feature set, wherein the feature set contains rich visual information of the video clip;

[0025] The feature set is input into a set deep learning multi-layer perceptron model for multi-level feature transformation and combination, so as to obtain a high-order feature representation describing the video content output by the deep learning multi-layer perceptron model.

[0026] Furthermore, the semantic recognition of the target feature representation is performed by a preset knowledge graph inference algorithm, the result of the semantic recognition is input into a set autoregressive generation model, and the supplementary video subtitle corresponding to the to-be-supplemented video segment is determined according to the output of the autoregressive generation model, including:

[0027] Use the pre-built knowledge graph to semantically match and associate the obtained high-order feature representation with the concept entities in the graph to determine the semantic concepts of product functions, attributes, and scenarios corresponding to the high-order feature representation;

[0028] The identified semantic concepts are input as conditions into a pre-trained autoregressive text generation model to obtain supplementary video subtitles that match the semantic concepts and are output by the autoregressive text generation model.

[0029] Furthermore, the semantic clustering of the original video subtitles and the supplementary video subtitles is performed to determine the timeline of each of the video segments, and the original video subtitles and the supplementary video subtitles are rendered into the corresponding video segments according to the timeline to obtain an updated short video file, including:

[0030] Performing semantic clustering analysis on the extracted original video subtitles and the generated supplementary video subtitles, dividing the original video subtitles and the supplementary video subtitles into several time periods based on the semantic relevance of the subtitle contents, and determining the accurate timeline of each of the video clips;

[0031] The original video subtitles and the supplementary video subtitles are rendered into the corresponding video segments based on the determined accurate timeline of the video segments, and an updated short video file is synthesized.

[0032] In a second aspect, the present application provides a short video timeline slicing device, comprising:

[0033] A video segmentation module is used to receive a short video file uploaded by a user, and segment the short video file into multiple video clips using a preset computer vision algorithm;

[0034] A subtitle determination module is used to extract subtitles from the video clips through a preset speech recognition algorithm to obtain corresponding original video subtitles, and determine the video clips to be supplemented that lack corresponding video subtitles, extract the spatiotemporal features of the video clips to be supplemented through a frame division operation, obtain low-order feature representations based on the spatiotemporal features, unify the extracted low-order feature representations into a feature set and input them into a set deep learning multi-layer perceptron to obtain corresponding high-order feature representations, perform feature weighted fusion on each of the target high-order feature representations through an attention mechanism to obtain a target feature representation of the video clip to be supplemented, perform semantic recognition on the target feature representation through a preset knowledge graph inference algorithm, input the result of the semantic recognition into a set autoregressive generation model, and determine the supplementary video subtitles corresponding to the video clip to be supplemented based on the output of the autoregressive generation model;

[0035] The subtitle rendering module is used to perform semantic clustering on the original video subtitles and the supplementary video subtitles, determine the timeline of each video segment, and render the original video subtitles and the supplementary video subtitles into the corresponding video segment according to the timeline to obtain an updated short video file.

[0036] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the short video timeline slicing method when executing the program.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the short video timeline slicing method.

[0038] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the short video timeline slicing method.

[0039] It can be seen from the above technical solution that the present application provides a short video timeline slicing method and device, which receives short video files uploaded by users, divides the short video files into storyboards through a preset computer vision algorithm, and obtains multiple video clips; semantically clusters the original video subtitles and the supplementary video subtitles, determines the timeline of each video clip, and renders the original video subtitles and the supplementary video subtitles into the corresponding video clips according to the timeline to obtain an updated short video file, thereby enabling accurate and efficient short video subtitle processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is one of the flow charts of the short video timeline slicing method in the embodiment of the present application;

[0042] Figure 2 This is the second flow chart of the short video timeline slicing method in the embodiment of the present application;

[0043] Figure 3 This is the third flow chart of the short video timeline slicing method in the embodiment of the present application;

[0044] Figure 4 This is a fourth flow chart of the short video timeline slicing method in an embodiment of the present application;

[0045] Figure 5 This is the fifth flow chart of the short video timeline slicing method in the embodiment of the present application;

[0046] Figure 6 This is the sixth flow chart of the short video timeline slicing method in the embodiment of the present application;

[0047] Figure 7 This is the seventh flow chart of the short video timeline slicing method in the embodiment of the present application;

[0048] Figure 8 This is a structural diagram of a short video timeline slicing device in an embodiment of the present application;

[0049] Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0050] Reference numerals:

[0051] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0054] In view of the problems existing in the prior art, the present application provides a short video timeline slicing method and device, which receives a short video file uploaded by a user, divides the short video file into segments through a preset computer vision algorithm, and obtains multiple video clips; semantically clusters the original video subtitles and the supplementary video subtitles, determines the timeline of each video clip, and renders the original video subtitles and the supplementary video subtitles into the corresponding video clip according to the timeline to obtain an updated short video file, thereby enabling accurate and efficient short video subtitle processing.

[0055] In order to accurately and efficiently process short video subtitles, the present application provides an embodiment of a short video timeline slicing method, see Figure 1 The short video timeline slicing method specifically includes the following contents:

[0056] Step S101: receiving a short video file uploaded by a user, and dividing the short video file into segments using a preset computer vision algorithm to obtain multiple video clips;

[0057] Optionally, in the embodiment, we have established a system for processing short video files uploaded by users. Users upload short videos to the server through the device interface, and the system has an efficient computer vision module to automatically analyze and process the video content. After receiving the video uploaded by the user, the system will perform format detection and decoding to ensure that the file can be processed by the visual algorithm. The video is input into a preset computer vision algorithm, which recognizes the different scene changes in the video and realizes the storyboard segmentation.

[0058] The core principle of the algorithm is to detect scene transitions by analyzing pixel changes and frame differences in video frames. Specifically, the algorithm extracts key frames in the video and performs feature point matching and motion vector analysis on each frame. A deep learning model based on a convolutional neural network (CNN) is used to identify complex scene changes, such as light changes or camera movements. In this way, the video is divided into video clips of multiple independent scenes.

[0059] This technology solves the problem of inaccurate segmentation in traditional video processing methods. Traditional methods usually rely on simple frame differences or fixed time interval segmentation, ignoring the complexity of video content, resulting in inaccurate segmentation. The present invention combines deep learning and computer vision technology to more accurately identify scene transitions and improve segmentation accuracy.

[0060] In practical applications, the system significantly improves video editing efficiency. For example, in short video production or social media platforms, users can quickly upload videos and use the system's automatic storyboarding function to quickly locate and edit video clips. For example, if a user uploads a travel video, the system automatically divides the video into different scenic spots, and the user can selectively edit and share each clip.

[0061] In addition, this technology helps improve the efficiency of video content management and retrieval. By automatically parsing and segmenting video content, the system generates more fine-grained metadata, making it easier for users to quickly retrieve and locate specific scenes in a large number of videos. This technical effect not only improves the user experience, but also provides a basis for further analysis and application of video content.

[0062] For example, in the field of education, teachers upload teaching videos, and the system automatically divides them into different chapters, so students can jump directly to the part they are interested in to learn. In film and television production, editors can find the shots that need to be processed more quickly, greatly saving time and labor costs.

[0063] In addition, the system also supports real-time processing, and users can obtain automatic storyboard results of videos while uploading, which is suitable for content review and segment extraction in live broadcast scenarios. Through efficient algorithm optimization and hardware acceleration, the system can process massive video data in a short time to ensure a smooth user experience.

[0064] In summary, this embodiment demonstrates how to use advanced computer vision algorithms for efficient splitting, solves the technical problems existing in traditional methods, and realizes a more efficient and intelligent short video processing flow, which has important application value in the field of video editing and management.

[0065] Step S102: extract subtitles from the video clips by using a preset speech recognition algorithm to obtain corresponding original video subtitles, and determine the video clips to be supplemented that lack corresponding video subtitles, extract spatiotemporal features of the video clips to be supplemented by a frame division operation, obtain low-order feature representations according to the spatiotemporal features, unify the extracted low-order feature representations into a feature set and input them into a set deep learning multi-layer perceptron to obtain corresponding high-order feature representations, perform feature weighted fusion on each of the target high-order feature representations through an attention mechanism to obtain target feature representations of the video clips to be supplemented, perform semantic recognition on the target feature representations by using a preset knowledge graph inference algorithm, input the result of the semantic recognition into a set autoregressive generation model, and determine the supplementary video subtitles corresponding to the video clips to be supplemented according to the output of the autoregressive generation model;

[0066] Optionally, in this embodiment, the system first receives the video clips uploaded by the user and applies a preset speech recognition algorithm to extract subtitles. The algorithm analyzes the audio signal through a deep neural network model and converts it into a text format. This process uses natural language processing technology to solve the problems of background noise interference and accent differences in traditional speech recognition, thereby ensuring high accuracy of subtitles.

[0067] When subtitle extraction is complete, the system automatically detects the missing segments. For these segments to be supplemented, the system extracts their spatiotemporal features through frame operations. This includes motion information and visual context in each frame. Through frame-by-frame analysis, the system identifies basic elements of video content, such as scene changes and object movement. These spatiotemporal features lay the foundation for subsequent feature representation.

[0068] The extracted low-level feature representations are summarized into a feature set and input into the set deep learning multi-layer perceptron. The multi-layer perceptron further refines and abstracts these low-level features through layers of neural network structures to generate high-level feature representations. The powerful modeling ability of deep learning is reflected in this step, enabling the system to better capture the complex semantic information of video clips through nonlinear mapping.

[0069] In order to improve the effect of feature fusion, the system uses the attention mechanism to perform weighted fusion of each high-level feature representation. The attention mechanism can dynamically adjust the importance of different features, so that the model focuses on the features that are most critical for subtitle generation. This method effectively solves the problem of uneven importance of each feature in multi-feature fusion and greatly improves the overall quality of feature representation.

[0070] Next, the system uses a preset knowledge graph reasoning algorithm to perform semantic recognition on the target feature representation. The knowledge graph provides rich background information and semantic associations, helping the system to more accurately understand the context of the video content. In this process, the system not only recognizes explicit semantic information, but also discovers implicit semantic associations through reasoning. This step significantly improves the semantic accuracy of subtitle generation.

[0071] The results of semantic recognition are input into an autoregressive generative model. The model gradually generates complete subtitle text based on the input semantic information. The autoregressive model ensures the coherence and logic of the generated subtitles through contextual dependencies. Compared with traditional models, this method solves the problem of poor fluency when generating long texts, and the generated subtitles are more natural.

[0072] Through the above steps, the system not only automatically supplements missing subtitles, but also improves the overall quality of video subtitles. This technology greatly reduces the workload of manual editing of subtitles and improves the efficiency of video content production. Specifically, in film and television production and educational videos, the system can quickly generate high-quality subtitles, greatly improving the audience's viewing experience.

[0073] For example, in an educational video application scenario, a teacher uploaded a video about a physics course. The system automatically generated accurate subtitles to help hearing-impaired students better understand the course content. In film and television production, producers can use the system to quickly generate multilingual subtitles to facilitate the global distribution of films.

[0074] In addition, the system's real-time processing capability makes it suitable for content review and segment extraction in live broadcast scenarios. Through efficient algorithm optimization and hardware acceleration, the system is able to process massive amounts of video data in a short period of time, ensuring a smooth user experience. This real-time subtitle generation capability is particularly important for news organizations and social media platforms, which helps to quickly spread information.

[0075] In summary, this embodiment provides an efficient and accurate subtitle generation solution by combining multiple technologies such as speech recognition, deep learning, attention mechanism and knowledge graph reasoning. It not only solves the technical problems in traditional methods, but also creates new application value and innovative significance in the field of video processing. Users and enterprises can better manage and utilize video resources, which promotes the further development and popularization of video technology.

[0076] Step S103: semantically clustering the original video subtitles and the supplementary video subtitles, determining the timeline of each of the video segments, and rendering the original video subtitles and the supplementary video subtitles into the corresponding video segments according to the timeline to obtain an updated short video file.

[0077] The system first performs semantic clustering on the original video subtitles and the supplementary video subtitles. The core of semantic clustering is to group the subtitle text according to semantic similarity to ensure that similar content is reasonably organized on the timeline. This process uses natural language processing technology to convert the subtitle text into a vector representation in a high-dimensional space through a word vector model (such as Word2Vec or BERT). Then, the system uses a clustering algorithm (such as K-means or DBSCAN) to cluster similar subtitles together. This method solves the problem of fragmented subtitle content and inconsistent semantics, making the video content more logically coherent.

[0078] After completing semantic clustering, the system determines the timeline of each video clip based on the clustering results. The timeline is determined based on the actual playback order of the video and the semantic relationship of the subtitles to ensure that the subtitles can accurately correspond to the video content. By adjusting the timeline, the system can effectively coordinate the appearance time of the subtitles with the actual content of the video clip. This step achieves a high degree of consistency between the subtitles and the video content through synchronous analysis of the video and subtitles.

[0079] After the timeline is determined, the system renders the original video subtitles and supplementary video subtitles into the corresponding video clips. The subtitle rendering process involves converting text information into visual subtitle effects and embedding them into the video. During the rendering process, the system considers factors such as the font, size, color, and position of the subtitles to ensure that the subtitles are clearly visible in the video and do not interfere with the main content of the video. In this way, subtitles and video content are seamlessly combined, improving the audience's viewing experience.

[0080] Finally, the system outputs an updated short video file. This video file not only contains accurate subtitle information, but also enhances the coherence and comprehensibility of the video content through semantic clustering and timeline adjustment. This technology greatly reduces the time and cost of manual editing of subtitles and improves the efficiency and quality of video production.

[0081] In practical applications, the system can be used for subtitle processing of movies, TV series and online educational videos. For example, in film production, producers can quickly generate multilingual subtitles for easy release in different countries and regions. In the field of education, teachers can use the system to generate accurate subtitles for video courses to help students better understand complex knowledge points.

[0082] In addition, this technology can also be extended to real-time subtitle generation and automatic translation systems. Through semantic clustering and timeline adjustment, the system can provide real-time, high-quality subtitle services in live broadcasts or online meetings. This capability is of great significance for improving the accessibility and dissemination efficiency of information.

[0083] In summary, this embodiment provides an efficient and intelligent subtitle generation and rendering solution by combining semantic clustering, timeline adjustment and subtitle rendering technology. This solution not only improves the matching degree between subtitles and video content, but also demonstrates its excellent practicality and innovation in various application scenarios.

[0084] From the above description, it can be seen that the short video timeline slicing method provided in the embodiment of the present application can receive short video files uploaded by users, divide the short video files into storyboards through a preset computer vision algorithm, and obtain multiple video clips; perform semantic clustering on the original video subtitles and the supplementary video subtitles, determine the timeline of each video clip, and render the original video subtitles and the supplementary video subtitles into the corresponding video clips according to the timeline to obtain an updated short video file, thereby accurately and efficiently processing short video subtitles.

[0085] In one embodiment of the short video timeline slicing method of the present application, see Figure 2 , and can also include the following:

[0086] Step S201: receiving a short video file uploaded by a user through a preset video upload interface, performing a format check and a security scan on the short video file, and storing the short video file that has undergone the format check and the security scan in a distributed storage system on a preset server side;

[0087] Step S202: extracting low-level visual features of color, texture and shape, and high-level semantic features of objects, faces and actions of each video frame in the short video file, calculating a significance score of each video frame based on the extracted low-level visual features and high-level semantic features, determining key shot points of the video according to the significance score, and dividing the short video file into multiple video clips according to the determined key shot points.

[0088] Optionally, in this embodiment, the system first receives the short video file uploaded by the user through a preset video upload interface. The upload interface is designed as a user-friendly interface that supports multiple video formats to ensure that users can upload smoothly on different devices. After the upload is completed, the system automatically performs a format check to confirm that the video file meets the standard format requirements. This step ensures the compatibility of the video in subsequent processing by parsing the video encoding and metadata.

[0089] Next, the system performs a security scan on the video file to prevent the spread of malware or harmful content. Security scanning includes virus detection and content review, using advanced security algorithms and machine learning technology to identify and isolate potential threats. This process ensures the security of uploaded content and provides double protection for users and systems.

[0090] Once the video file passes the format check and security scan, the system stores it in the distributed storage system on the preset server side. The distributed storage system has high reliability and scalability and can handle large amounts of video data. Through automatic backup and data redundancy, the system ensures safe storage and fast access to video files.

[0091] In step S202, the system performs an in-depth analysis of the short video file. First, the low-level visual features of each video frame are extracted, including color, texture, and shape. These features are extracted through image processing techniques such as edge detection and color histogram, providing basic visual information of the video frame.

[0092] At the same time, the system also extracts high-level semantic features in the video frames, such as objects, faces, and actions. These features are analyzed by deep learning models such as convolutional neural networks and RNNs. The models are trained to accurately identify complex semantic information in videos. This process solves the problem of traditional video analysis that only focuses on low-level features, making video content analysis more comprehensive and accurate.

[0093] Based on the extracted low-level visual features and high-level semantic features, the system calculates a saliency score for each video frame. The saliency score is used to measure the importance of a video frame in the overall video. The calculation method combines the visual attention mechanism to identify the most attractive and informative frames for users. This technology improves the accuracy of key frame selection and makes subsequent video editing more efficient.

[0094] Based on the saliency score, the system determines the key points of the video. Key points refer to frames with important turning points or information nodes in the video, which can represent the main content and flow of the video. By automatically identifying these points, the system can split the video into multiple logically complete video clips.

[0095] The segmentation of video clips not only facilitates video editing and management, but also improves the user's viewing experience. Users can quickly jump to the clips they are interested in to watch, saving time and increasing interactivity. For example, in educational videos, students can jump directly to the key knowledge clips to learn, which improves learning efficiency.

[0096] In summary, this embodiment provides an efficient and intelligent video management solution by describing in detail the entire process of video uploading, processing, storage and analysis. The system uses advanced technical means to achieve secure video storage and content analysis, providing users with an optimized service experience. This technology has broad application prospects in social media, online education, film and television production and other fields.

[0097] In one embodiment of the short video timeline slicing method of the present application, see Figure 3, and can also include the following:

[0098] Step S301: using a pre-trained speech recognition model to transcribe the audio content in the video clip, extracting corresponding original video subtitles, and performing a rationality check on the extracted original video subtitles;

[0099] Step S302: Determine the video segment to be supplemented with subtitles according to the extracted original video subtitles, video duration, and picture change characteristics.

[0100] Optionally, in this embodiment, the system first transcribes the audio content in the video clip using a pre-trained speech recognition model. The speech recognition model is based on deep learning technology and is trained with a large corpus to accurately convert audio signals into text. This process involves feature extraction and pattern matching of the audio signal, and the corresponding original video subtitles are generated through high-precision transcription of the model.

[0101] After the transcription is completed, the system performs a rationality check on the extracted original video subtitles. The rationality check includes grammar correction and content consistency review. The system uses natural language processing technology to automatically identify spelling errors and grammatical problems in subtitles and correct them using context. In addition, the system also compares the subtitle text with the video content to ensure that the subtitles are synchronized with the picture, thereby improving the accuracy and readability of the subtitles.

[0102] In step S302, the system determines the video segments to be supplemented with subtitles based on the extracted original video subtitles, video duration, and picture change characteristics. Video duration and picture change characteristics are key factors. The system identifies segments with missing or incomplete subtitles by analyzing the frame rate and scene transition points of the video.

[0103] The extraction of picture change features relies on computer vision algorithms. By detecting scene switching, character movements, and visual effect changes in the video, the system can identify the logical sections of the video. This process combines image processing technology and time series analysis to ensure accurate positioning of the segments that need to be supplemented with subtitles in complex visual environments.

[0104] After determining the video clips to be supplemented, the system generates a corresponding task list for subsequent subtitle editing and supplementation. This process solves the problem of low efficiency of traditional manual subtitle checking and greatly improves the efficiency and quality of subtitle processing through automated analysis processes.

[0105] This technology has a wide range of application scenarios. For example, in film and television production, the system can quickly identify clips that need to be supplemented with subtitles, helping editors and subtitle editors improve their work efficiency. On online education platforms, videos uploaded by teachers can automatically generate and check subtitles, making course content easier to understand and spread.

[0106] In addition, the system has real-time processing capabilities, which is suitable for occasions that require rapid subtitle generation and proofreading, such as live news broadcasts and online conferences. By automatically identifying and supplementing subtitles, the system provides technical support for the rapid transmission of information and global communication.

[0107] From the technical effect point of view, this embodiment realizes the automation of video subtitle generation and supplementation by combining speech recognition and computer vision technology. This not only reduces labor costs, but also improves the accuracy and consistency of subtitles. The user experience has been significantly improved, and the accessibility and dissemination efficiency of video content have also been enhanced.

[0108] In one embodiment of the short video timeline slicing method of the present application, see Figure 4 , and can also include the following:

[0109] Step S401: performing a frame division operation on the video segment to be supplemented that is determined to need to supplement subtitles, dividing it into a continuous video frame sequence, and extracting spatiotemporal features including at least one of color, texture and motion information for each video frame in the video frame sequence;

[0110] Step S402: compressing the extracted spatiotemporal features into a compact low-order feature representation, wherein the low-order feature representation includes at least basic visual and dynamic characteristics of video content.

[0111] Optionally, in this embodiment, the system first performs a frame operation on the video clip to be supplemented with subtitles. Framing refers to splitting the video into a continuous image sequence, which is achieved through video decoding technology. Each video frame represents a moment of visual information, laying the foundation for subsequent feature extraction. Through an efficient decoding algorithm, the system can quickly and accurately decompose the video into a frame sequence to ensure the order and integrity of the frames.

[0112] Next, the system extracts spatiotemporal features for each video frame in the video frame sequence. Spatiotemporal features include color, texture, and motion information. Color features are extracted by analyzing the color distribution and hue changes in the frame, such as using a color histogram to capture the overall color characteristics of the image. Texture features are obtained by detecting repeated patterns and detail changes in the image. Common methods include gray-level co-occurrence matrix and wavelet transform, which can effectively identify the subtle structure of the picture.

[0113] The extraction of motion information is particularly critical, as it captures the dynamic changes between video frames. Through optical flow or frame difference methods, the system can identify and quantify the movement of objects and backgrounds. This dynamic feature not only reveals the motion pattern of the video content, but also provides an important basis for determining key events in the video.

[0114] After extracting rich spatiotemporal features, the system enters step S402 to compress these features into compact low-order feature representations. The purpose of feature compression is to simplify data, retain the most important information, and reduce computational complexity. The system uses techniques such as principal component analysis (PCA) or autoencoders to reduce the dimensionality of high-dimensional spatiotemporal features. These techniques generate low-order feature representations by finding the main direction of change in the data and removing redundant information.

[0115] The low-level feature representation contains not only the basic visual characteristics of the video content, such as the main tones and texture patterns, but also dynamic characteristics, such as motion direction and speed. These compact features provide the basis for subsequent video analysis and subtitle supplementation. Through compression, the system significantly reduces the amount of data and improves processing efficiency, ensuring that it remains efficient when processing large amounts of video data.

[0116] The application of this technology solves the problem of large data volume and slow processing speed in traditional video analysis. Through effective feature extraction and compression methods, the system can quickly locate important content in the video and provide accurate support for subtitle supplementation. For example, in the playback analysis of sports events, the system can quickly identify key actions and events and provide real-time data for the generation of commentary subtitles.

[0117] In addition, in the field of video editing and production, the system can assist editors in quickly screening out key images and improving work efficiency. In intelligent monitoring, by extracting and analyzing spatiotemporal features, the system can automatically identify abnormal behaviors and generate corresponding warning information to enhance safety.

[0118] In one embodiment of the short video timeline slicing method of the present application, see Figure 5 , and can also include the following:

[0119] Step S501: Integrate the extracted low-level feature representations into a unified feature set, wherein the feature set contains rich visual information of the video clip;

[0120] Step S502: Input the feature set into a set deep learning multi-layer perceptron model for multi-level feature transformation and combination, and obtain a high-order feature representation describing the video content output by the deep learning multi-layer perceptron model.

[0121] Optionally, in this embodiment, the system first integrates the extracted low-level feature representations into a unified feature set. Low-level features include color, texture, and motion information, which are extracted from video frames respectively. These features capture the basic visual elements of the video and provide rich basic information. The integration process is achieved through feature fusion technology to ensure that different types of features can be effectively combined in the same data structure. This integration not only maintains the diversity of features, but also enhances the overall expressiveness of the feature set, laying a solid foundation for subsequent deep learning processing.

[0122] Next, the system inputs the integrated feature set into the set deep learning multi-layer perceptron (MLP) model. MLP is an artificial neural network with a multi-layer structure that can perform complex feature transformations and combinations on input data. Through layer-by-layer transmission and the application of nonlinear activation functions, MLP deeply mines features and generates high-order representations. This process makes full use of the powerful modeling capabilities of deep learning, enabling the system to extract abstract semantic features from rich visual information.

[0123] In the MLP model, neurons in each layer perform weighted summation of input features and introduce nonlinear transformations through activation functions. Such multi-level processing enables the model to capture complex relationships and patterns between features. The system optimizes model parameters through the back-propagation algorithm to ensure that the high-order feature representation output by the model can accurately describe the semantics and structure of the video content.

[0124] This process solves the problem of single feature expression and insufficient information integration in traditional video analysis. Through multi-layer feature transformation of deep learning, the system can identify key patterns and themes in a large amount of video data, improving the accuracy and depth of video content analysis. High-order feature representation not only covers the visual information of the video, but also implies complex semantic information such as dynamic changes and scene transitions.

[0125] In practical applications, this technology can significantly improve the efficiency and effectiveness of video content processing. For example, when automatically generating video summaries, the system can quickly identify important clips in the video and generate concise content descriptions. In intelligent monitoring, the system can analyze video streams in real time, identify abnormal activities, and issue alarms.

[0126] In addition, high-level feature representations of deep learning provide strong support for personalized recommendation systems. By analyzing the characteristics of videos watched by users, the system can more accurately recommend content that matches the user's interests, improving user experience and platform stickiness.

[0127] In summary, this embodiment demonstrates a method for realizing high-level representation of video features through deep learning. The system realizes deep analysis and semantic understanding of video content by integrating low-level features and multi-layer processing of MLP.

[0128] In one embodiment of the short video timeline slicing method of the present application, see Figure 6 , and can also include the following:

[0129] Step S601: Using the pre-built knowledge graph, the obtained high-order feature representation is semantically matched and associated with the concept entities in the graph to determine the semantic concepts of product functions, attributes, and scenarios corresponding to the high-order feature representation;

[0130] Step S602: Input the identified semantic concept as a condition into a pre-trained autoregressive text generation model to obtain a supplementary video subtitle output by the autoregressive text generation model that matches the semantic concept.

[0131] Optionally, in this embodiment, the system first uses a pre-built knowledge graph to perform semantic matching and association reasoning on high-order feature representations. A knowledge graph is a structured semantic network that contains various conceptual entities and their relationships. Through this step, the system matches the high-order features extracted from the video with the conceptual entities in the knowledge graph. The entities in the knowledge graph include product functions, attributes, and various scene information, which enables the system to identify the specific semantic information shown in the video.

[0132] The semantic matching process uses the rich information in the graph. By comparing the semantic similarity of high-level features, the system can identify the product functions, attributes, and scenarios corresponding to the features. This process involves not only simple keyword matching, but also complex semantic reasoning. Associative reasoning helps the system infer hidden semantic information through the relationship chain in the knowledge graph. For example, if the video features show the usage scenario of a certain device, the system can infer the specific functions and related operations of the device through the knowledge graph.

[0133] This process solves the problem of semantic ambiguity in video content understanding. Traditional video analysis methods can only identify low-level features but cannot understand deep semantic information. By combining knowledge graphs, the system can extract clear semantic information from complex features, making video analysis more comprehensive and intelligent.

[0134] After completing semantic matching and reasoning, the system inputs the identified semantic concepts as conditions into the pre-trained autoregressive text generation model in step S602. The autoregressive model is a generative model that generates text in a step-by-step manner. The model first accepts the semantic concept input and then starts to generate the subtitle content that matches it. During the generation process, the autoregressive model uses contextual information to ensure that the generated subtitle sentences are fluent and logically coherent. This model uses a large amount of text data in the training phase, has a strong language generation capability, and can generate a variety of text outputs according to different semantic conditions.

[0135] The core advantage of the autoregressive text generation model lies in its flexibility and accuracy in language generation. By incorporating semantic concepts, the model not only generates accurate subtitles, but also ensures that the subtitles are semantically consistent with the video content. This provides users with a more vivid and accurate viewing experience, especially in teaching and demonstration videos that need to convey information accurately.

[0136] This system solves the problem of lack of semantic depth in traditional subtitle generation. By combining knowledge graphs and autoregressive models, the system can not only generate accurate subtitles, but also ensure the semantic consistency of subtitles with video content. This provides users with a more vivid and accurate viewing experience, especially in teaching and demonstration videos that need to convey information accurately.

[0137] In practical applications, this technology can significantly improve the efficiency and quality of subtitle generation. For example, in a product introduction video, the system can automatically identify product functions and usage scenarios, and generate corresponding captions to help viewers better understand product features. In educational videos, the system can automatically generate supplementary explanations or examples based on classroom content, making the learning process more intuitive and easy to understand.

[0138] In addition, this technology can also be extended to generate subtitles in multiple languages. By combining knowledge graphs in different languages, the system can perform semantic matching and text generation between different languages, provide multilingual support for global users, and expand the audience range of videos. This multilingual support is particularly important for multinational companies and international education projects.

[0139] In summary, this embodiment demonstrates an intelligent subtitle generation method that combines knowledge graph and autoregressive text generation. Through the dual technologies of semantic matching and language generation, the system achieves in-depth understanding of video content and efficient generation of subtitles.

[0140] In one embodiment of the short video timeline slicing method of the present application, see Figure 7 , and can also include the following:

[0141] Step S701: performing semantic clustering analysis on the extracted original video subtitles and the generated supplementary video subtitles, dividing the original video subtitles and the supplementary video subtitles into several time periods based on the semantic relevance of the subtitle contents, and determining the accurate time axis of each of the video clips;

[0142] Step S702: Rendering the original video subtitles and the supplementary video subtitles into the corresponding video segment based on the determined accurate timeline of the video segment to synthesize an updated short video file.

[0143] Optionally, in this embodiment, the system first performs semantic cluster analysis on the extracted original video subtitles and the generated supplementary video subtitles. The purpose of semantic cluster analysis is to divide the subtitles into different time periods based on the semantic relevance of the subtitle content. This process uses natural language processing technology to convert the subtitle text into a vector representation through a word vector model (such as Word2Vec or BERT). The system then uses a clustering algorithm (such as K-means or DBSCAN) to cluster similar subtitles together. In this way, the system can identify the semantic similarities between subtitles and effectively logically group the subtitle content.

[0144] After completing semantic clustering, the system divides the time periods according to the clustering results and determines the accurate timeline of each video clip. This process takes into account the semantic flow of the subtitle content and the actual playback order of the video, ensuring that the subtitles can accurately correspond to the video content. The determination of the timeline achieves a high degree of consistency between the subtitles and the video content by synchronously analyzing the video and subtitles. This technology solves the problem of difficulty in synchronizing subtitles with videos, making the presentation of video content more coherent.

[0145] In step S702, the system renders the original video subtitles and the supplementary video subtitles into the corresponding video clips based on the accurate timeline of the determined video clips. The subtitle rendering process involves converting text information into a visual subtitle effect and embedding it into the video. The system considers factors such as the font, size, color, and position of the subtitles during the rendering process to ensure that the subtitles are clearly visible in the video and do not interfere with the main content of the video. In this way, the subtitles and video content are seamlessly combined, improving the audience's viewing experience.

[0146] The final result of video synthesis is an updated short video file that not only contains accurate subtitle information, but also enhances the coherence and comprehensibility of the video content through semantic clustering and timeline adjustment. This technology greatly reduces the time and cost of manual editing of subtitles and improves the efficiency and quality of video production.

[0147] In practical applications, the system can be used for subtitle processing of movies, TV series and online educational videos. For example, in film production, producers can quickly generate multilingual subtitles for easy release in different countries and regions. In the field of education, teachers can use the system to generate accurate subtitles for video courses to help students better understand complex knowledge points.

[0148] In addition, this technology can also be extended to real-time subtitle generation and automatic translation systems. Through semantic clustering and timeline adjustment, the system can provide real-time, high-quality subtitle services in live broadcasts or online meetings. This capability is of great significance for improving the accessibility and dissemination efficiency of information.

[0149] In summary, this embodiment provides an efficient and intelligent subtitle generation and rendering solution by combining semantic clustering, timeline adjustment and subtitle rendering technology. This solution not only improves the matching degree between subtitles and video content, but also demonstrates its excellent practicality and innovation in various application scenarios. The system optimizes the generation and application of video subtitles through automated processes and promotes the development of multimedia technology.

[0150] In order to accurately and efficiently process short video subtitles, the present application provides an embodiment of a short video timeline slicing device for implementing all or part of the content of the short video timeline slicing method, see Figure 8 The short video timeline slicing device specifically includes the following contents:

[0151] The video segmentation module 10 is used to receive a short video file uploaded by a user, and segment the short video file into multiple video clips by using a preset computer vision algorithm;

[0152] The subtitle determination module 20 is used to extract subtitles from the video clips through a preset speech recognition algorithm to obtain corresponding original video subtitles, and determine the video clips to be supplemented that lack corresponding video subtitles, extract the spatiotemporal features of the video clips to be supplemented through a frame division operation, obtain low-order feature representations based on the spatiotemporal features, unify the extracted low-order feature representations into a feature set and input them into a set deep learning multi-layer perceptron to obtain corresponding high-order feature representations, perform feature weighted fusion on each of the target high-order feature representations through an attention mechanism to obtain target feature representations of the video clips to be supplemented, perform semantic recognition on the target feature representations through a preset knowledge graph inference algorithm, input the result of the semantic recognition into a set autoregressive generation model, and determine the supplementary video subtitles corresponding to the video clips to be supplemented based on the output of the autoregressive generation model;

[0153] The subtitle rendering module 30 is used to perform semantic clustering on the original video subtitles and the supplementary video subtitles, determine the timeline of each of the video segments, and render the original video subtitles and the supplementary video subtitles into the corresponding video segments according to the timeline to obtain an updated short video file.

[0154] From the above description, it can be seen that the short video timeline slicing device provided in the embodiment of the present application can receive short video files uploaded by users, divide the short video files into storyboards through a preset computer vision algorithm, and obtain multiple video clips; perform semantic clustering on the original video subtitles and the supplementary video subtitles, determine the timeline of each video clip, and render the original video subtitles and the supplementary video subtitles into the corresponding video clips according to the timeline to obtain an updated short video file, thereby accurately and efficiently processing short video subtitles.

[0155] From the hardware level, in order to accurately and efficiently process short video subtitles, the present application provides an embodiment of an electronic device for implementing all or part of the content in the short video timeline slicing method, and the electronic device specifically includes the following content:

[0156] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the short video timeline slicing device and the core business system, user terminal and related databases and other related devices; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the short video timeline slicing method and the embodiment of the short video timeline slicing device in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0157] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0158] In practical applications, part of the short video timeline slicing method can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0159] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0160] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0161] In one embodiment, the short video timeline slicing method function can be integrated into the central processing unit 9100.

[0162] The CPU 9100 may be configured to perform the following control:

[0163] Step S101: receiving a short video file uploaded by a user, and dividing the short video file into segments using a preset computer vision algorithm to obtain multiple video clips;

[0164] Step S102: extract subtitles from the video clips by using a preset speech recognition algorithm to obtain corresponding original video subtitles, and determine the video clips to be supplemented that lack corresponding video subtitles, extract spatiotemporal features of the video clips to be supplemented by a frame division operation, obtain low-order feature representations according to the spatiotemporal features, unify the extracted low-order feature representations into a feature set and input them into a set deep learning multi-layer perceptron to obtain corresponding high-order feature representations, perform feature weighted fusion on each of the target high-order feature representations through an attention mechanism to obtain target feature representations of the video clips to be supplemented, perform semantic recognition on the target feature representations by using a preset knowledge graph inference algorithm, input the result of the semantic recognition into a set autoregressive generation model, and determine the supplementary video subtitles corresponding to the video clips to be supplemented according to the output of the autoregressive generation model;

[0165] Step S103: semantically clustering the original video subtitles and the supplementary video subtitles, determining the timeline of each of the video segments, and rendering the original video subtitles and the supplementary video subtitles into the corresponding video segments according to the timeline to obtain an updated short video file.

[0166] From the above description, it can be seen that the electronic device provided in the embodiment of the present application receives a short video file uploaded by a user, divides the short video file into segments through a preset computer vision algorithm, and obtains multiple video clips; semantically clusters the original video subtitles and the supplementary video subtitles, determines the timeline of each video clip, and renders the original video subtitles and the supplementary video subtitles into the corresponding video clip according to the timeline to obtain an updated short video file, thereby being able to accurately and efficiently process short video subtitles.

[0167] In another embodiment, the short video timeline slicing device can be configured separately from the central processing unit 9100. For example, the short video timeline slicing device can be configured as a chip connected to the central processing unit 9100, and the short video timeline slicing method function can be implemented through the control of the central processing unit.

[0168] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0169] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0170] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0171] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0172] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0173] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0174] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0175] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module, etc. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0176] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the short video timeline slicing method in the above-mentioned embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the short video timeline slicing method in the above-mentioned embodiment, where the execution subject is a server or a client, are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0177] Step S101: receiving a short video file uploaded by a user, and dividing the short video file into segments using a preset computer vision algorithm to obtain multiple video clips;

[0178] Step S102: extract subtitles from the video clips by using a preset speech recognition algorithm to obtain corresponding original video subtitles, and determine the video clips to be supplemented that lack corresponding video subtitles, extract spatiotemporal features of the video clips to be supplemented by a frame division operation, obtain low-order feature representations according to the spatiotemporal features, unify the extracted low-order feature representations into a feature set and input them into a set deep learning multi-layer perceptron to obtain corresponding high-order feature representations, perform feature weighted fusion on each of the target high-order feature representations through an attention mechanism to obtain target feature representations of the video clips to be supplemented, perform semantic recognition on the target feature representations by using a preset knowledge graph inference algorithm, input the result of the semantic recognition into a set autoregressive generation model, and determine the supplementary video subtitles corresponding to the video clips to be supplemented according to the output of the autoregressive generation model;

[0179] Step S103: semantically clustering the original video subtitles and the supplementary video subtitles, determining the timeline of each of the video segments, and rendering the original video subtitles and the supplementary video subtitles into the corresponding video segments according to the timeline to obtain an updated short video file.

[0180] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application receives a short video file uploaded by a user, and divides the short video file into multiple video clips through a preset computer vision algorithm; semantically clusters the original video subtitles and the supplementary video subtitles, determines the timeline of each video clip, and renders the original video subtitles and the supplementary video subtitles into the corresponding video clip according to the timeline to obtain an updated short video file, thereby being able to accurately and efficiently process short video subtitles.

[0181] The embodiments of the present application also provide a computer program product capable of implementing all the steps of the short video timeline slicing method in the above embodiments, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the short video timeline slicing method are implemented. For example, the computer program / instruction implements the following steps:

[0182] Step S101: receiving a short video file uploaded by a user, and dividing the short video file into segments using a preset computer vision algorithm to obtain multiple video clips;

[0183] Step S102: extract subtitles from the video clips by using a preset speech recognition algorithm to obtain corresponding original video subtitles, and determine the video clips to be supplemented that lack corresponding video subtitles, extract spatiotemporal features of the video clips to be supplemented by a frame division operation, obtain low-order feature representations according to the spatiotemporal features, unify the extracted low-order feature representations into a feature set and input them into a set deep learning multi-layer perceptron to obtain corresponding high-order feature representations, perform feature weighted fusion on each of the target high-order feature representations through an attention mechanism to obtain target feature representations of the video clips to be supplemented, perform semantic recognition on the target feature representations by using a preset knowledge graph inference algorithm, input the result of the semantic recognition into a set autoregressive generation model, and determine the supplementary video subtitles corresponding to the video clips to be supplemented according to the output of the autoregressive generation model;

[0184] Step S103: semantically clustering the original video subtitles and the supplementary video subtitles, determining the timeline of each of the video segments, and rendering the original video subtitles and the supplementary video subtitles into the corresponding video segments according to the timeline to obtain an updated short video file.

[0185] From the above description, it can be seen that the computer program product provided in the embodiment of the present application receives a short video file uploaded by a user, divides the short video file into segments through a preset computer vision algorithm, and obtains multiple video clips; semantically clusters the original video subtitles and the supplementary video subtitles, determines the timeline of each video clip, and renders the original video subtitles and the supplementary video subtitles into the corresponding video clip according to the timeline to obtain an updated short video file, thereby being able to accurately and efficiently process short video subtitles.

[0186] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0187] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0188] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0189] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0190] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A short video timeline slicing method, characterized in that: The method comprises: Receive a short video file uploaded by a user, and segment the short video file using a preset computer vision algorithm to obtain multiple video clips; Subtitles are extracted from the video clip by a preset speech recognition algorithm to obtain the corresponding original video subtitles, and the video clip to be supplemented that lacks the corresponding video subtitles is determined, the spatiotemporal features of the video clip to be supplemented are extracted by a frame division operation, a low-order feature representation is obtained according to the spatiotemporal features, the extracted low-order feature representation is unified into a feature set and input into a set deep learning multi-layer perceptron to obtain the corresponding high-order feature representation, feature weighted fusion is performed on each of the target high-order feature representations through an attention mechanism to obtain the target feature representation of the video clip to be supplemented, semantic recognition is performed on the target feature representation by a preset knowledge graph inference algorithm, the result of the semantic recognition is input into a set autoregressive generation model, and the supplementary video subtitles corresponding to the video clip to be supplemented are determined according to the output of the autoregressive generation model; The original video subtitles and the supplementary video subtitles are semantically clustered to determine the timeline of each of the video segments, and the original video subtitles and the supplementary video subtitles are rendered into the corresponding video segments according to the timeline to obtain an updated short video file.

2. The short video timeline slicing method according to claim 1, characterized in that: The receiving of the short video file uploaded by the user, and segmenting the short video file by using a preset computer vision algorithm to obtain a plurality of video clips, including: Receive a short video file uploaded by a user through a preset video upload interface, perform a format check and a security scan on the short video file, and store the short video file that has undergone the format check and the security scan in a distributed storage system on a preset server side; Extract low-level visual features of color, texture and shape, and high-level semantic features of objects, faces and actions of each video frame in the short video file, calculate a saliency score of each video frame based on the extracted low-level visual features and high-level semantic features, determine key shot points of the video according to the saliency score, and divide the short video file into multiple video clips according to the determined key shot points.

3. The short video timeline slicing method according to claim 1, characterized in that: The extracting subtitles from the video clips by using a preset speech recognition algorithm to obtain corresponding original video subtitles and determining the video clips to be supplemented that lack corresponding video subtitles, includes: Using a pre-trained speech recognition model to transcribe the audio content in the video clip, extracting corresponding original video subtitles, and performing a rationality check on the extracted original video subtitles; The video segments to be supplemented that need to be supplemented with subtitles are determined based on the extracted original video subtitles, video duration, and picture change characteristics.

4. The short video timeline slicing method according to claim 1, characterized in that: The step of extracting the spatiotemporal features of the video segment to be supplemented by a frame division operation and obtaining a low-order feature representation according to the spatiotemporal features includes: Performing a frame division operation on the video segment to be supplemented that is determined to need to be supplemented with subtitles, dividing it into a continuous video frame sequence, and extracting spatiotemporal features including at least one of color, texture, and motion information for each video frame in the video frame sequence; The extracted spatiotemporal features are compressed into a compact low-level feature representation, wherein the low-level feature representation includes at least basic visual and dynamic characteristics of video content.

5. The short video timeline slicing method according to claim 1, characterized in that: The extracted low-order feature representations are unified into a feature set and input into a set deep learning multi-layer perceptron to obtain corresponding high-order feature representations, including: Integrating the extracted low-level feature representations into a unified feature set, wherein the feature set contains rich visual information of the video clip; The feature set is input into a set deep learning multi-layer perceptron model for multi-level feature transformation and combination, so as to obtain a high-order feature representation describing the video content output by the deep learning multi-layer perceptron model.

6. The short video timeline slicing method according to claim 1, characterized in that: The method of performing semantic recognition on the target feature representation by using a preset knowledge graph inference algorithm, inputting the result of the semantic recognition into a set autoregressive generation model, and determining the supplementary video subtitles corresponding to the to-be-supplemented video segment according to the output of the autoregressive generation model, includes: Use the pre-built knowledge graph to semantically match and associate the obtained high-order feature representation with the concept entities in the graph to determine the semantic concepts of product functions, attributes, and scenarios corresponding to the high-order feature representation; The identified semantic concepts are input as conditions into a pre-trained autoregressive text generation model to obtain supplementary video subtitles that match the semantic concepts and are output by the autoregressive text generation model.

7. The short video timeline slicing method according to claim 1, characterized in that: The semantic clustering of the original video subtitles and the supplementary video subtitles, determining the timeline of each of the video segments, and rendering the original video subtitles and the supplementary video subtitles into the corresponding video segments according to the timeline to obtain an updated short video file includes: Performing semantic clustering analysis on the extracted original video subtitles and the generated supplementary video subtitles, dividing the original video subtitles and the supplementary video subtitles into several time periods based on the semantic relevance of the subtitle contents, and determining the accurate timeline of each of the video clips; The original video subtitles and the supplementary video subtitles are rendered into the corresponding video segments based on the determined accurate timeline of the video segments, and an updated short video file is synthesized.

8. A short video timeline slicing device, characterized in that: The device comprises: A video segmentation module is used to receive a short video file uploaded by a user, and segment the short video file into multiple video clips using a preset computer vision algorithm; A subtitle determination module is used to extract subtitles from the video clips through a preset speech recognition algorithm to obtain corresponding original video subtitles, and determine the video clips to be supplemented that lack corresponding video subtitles, extract the spatiotemporal features of the video clips to be supplemented through a frame division operation, obtain low-order feature representations based on the spatiotemporal features, unify the extracted low-order feature representations into a feature set and input them into a set deep learning multi-layer perceptron to obtain corresponding high-order feature representations, perform feature weighted fusion on each of the target high-order feature representations through an attention mechanism to obtain a target feature representation of the video clip to be supplemented, perform semantic recognition on the target feature representation through a preset knowledge graph inference algorithm, input the result of the semantic recognition into a set autoregressive generation model, and determine the supplementary video subtitles corresponding to the video clip to be supplemented based on the output of the autoregressive generation model; The subtitle rendering module is used to perform semantic clustering on the original video subtitles and the supplementary video subtitles, determine the timeline of each video segment, and render the original video subtitles and the supplementary video subtitles into the corresponding video segment according to the timeline to obtain an updated short video file.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the short video timeline slicing method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the short video timeline slicing method described in any one of claims 1 to 7 are implemented.

Images