Video acceleration processing method

Abstract

The invention discloses a video acceleration processing method, which relates to the technical field of video processing and comprises the following steps of: acquiring image data, audio data and voice texts, extracting corresponding semantic factors, constructing a joint semantic representation vector, calculating burst intensity, generating a semantic burst curve, acquiring user preference and generating a rhythm control strategy. According to the method, video acceleration processing is carried out based on multi-modal semantic burst analysis, the key content can be accurately identified, the rhythm control strategy is generated in combination with user preference, and the user experience is improved. And meanwhile, natural connection of the jump playing segments is realized through transition frame generation, so that the video playing efficiency and continuity are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a video acceleration processing method. BACKGROUND

[0002] With the increasing popularity of long video data in scenarios such as remote education, intelligent conference, security monitoring, and content browsing, users' demand for "efficiently obtaining key information" is increasing. Existing video acceleration technologies generally use fixed speed playback or skip play methods based on shot switching, lacking understanding of video semantic content.

[0003] At present, the Chinese patent with application number CN202111528704.2 discloses an acceleration processing method of a video compression card based on a RISC-V processor, belonging to the technical field of video compression card software compression research. The method performs the following steps: 1) collecting image information; 2) completing AVS format encoding of the image information; 3) using a RISC-V processor and a coprocessor to interactively complete zig-zag scanning and run-length encoding of the image information in step 2); 4) completing AVS format entropy encoding of the image information that has completed zig-zag scanning and run-length encoding in step 3); 5) completing AVS video compression; the coprocessor participates in completing the execution, memory access, and write-back functions of the RISC-V processor through coprocessor instructions.

[0004] The related technology is difficult to dynamically identify key segments based on video semantic content and achieve personalized rhythm control and natural skip play transition. SUMMARY

[0005] The technical problem solved by the present application is that the prior art is difficult to dynamically identify key segments based on video semantic content and achieve personalized rhythm control and natural skip play transition.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] A video acceleration processing method, comprising the following steps:

[0008] Step S1, collecting image data, audio data, and speech text, and extracting corresponding semantic factors;

[0009] Step S2, constructing a joint semantic representation vector, calculating burst intensity, and generating a semantic burst curve;

[0010] Step S3, obtaining user preferences and generating a rhythm control strategy;

[0011] Step S4, performing accelerated playback and key content preservation according to the strategy;

[0012] Step S5, generating skip play transition frames to enhance video continuity;

[0013] Step S6, encapsulating the processed video and generating auxiliary index data.

[0014] Preferably, the step S1 comprises the following sub-steps:

[0015] Step S101, extracting continuous frame image data from the original video stream, and analyzing the dynamic characteristics of the picture based on the inter-frame change, including calculating the pixel change rate of adjacent frames to determine the shot switching rate, tracking the motion path of the picture center point and the main object to measure the picture movement amplitude, applying a target detection model to identify the main content of the video and record its appearance frequency and duration, and constructing an image change sequence;

[0016] Step S102, extracting speech signals from the audio data, analyzing speech speed, tone change and repeated word features, and transcribing the audio data into text to generate a speech text sequence with timestamps;

[0017] Step S103, processing the speech text sequence, performing text normalization, sentence segmentation, semantic word segmentation and keyword extraction operations, and using a syntax analysis tool to identify subject-predicate structures and modification relationships, outputting semantic structure labels and keyword sequences.

[0018] Preferably, the step S2 comprises the following sub-steps:

[0019] Step S201, aligning the image change sequence, speech text sequence, semantic structure label and keyword sequence in the time dimension, synchronizing the continuous frame image data, audio data and speech text sequence based on the unified time axis, constructing a joint semantic representation vector in each time window, the joint semantic representation vector including the degree of visual subject change, speech speed and emotional fluctuation intensity, keyword density and semantic mutation degree;

[0020] Step S202, performing sliding time window analysis on the joint semantic representation vector, calculating the semantic burst intensity score, identifying the burst or turning point semantic section by detecting the peak value and trend mutation, and classifying it into structure nodes and core nodes, outputting the semantic burst curve through the semantic change trajectory obtained by trend modeling.

[0021] Preferably, the step S202 identifies the new topic starting point and event climax segment based on the burst intensity change trend, and correspondingly increases the rhythm weight, and preferentially retains or slows down the playback in rhythm control.

[0022] Preferably, the step S3 comprises the following sub-steps:

[0023] Step S301, obtaining user preference settings for playing rhythm, the preference settings including expected total duration, content type of interest and minimum retention ratio, and converting into structured parameters;

[0024] Step S302, generating rhythm control strategy in combination with semantic burst curve and structured parameters, assigning playing speed and skip mark by analyzing matching conditions of burst intensity and user interested type in each time period within semantic burst curve, and outputting rhythm strategy file.

[0025] Preferably, the step S302 adjusts playing speed and skip range of low-priority fragments and retains key semantic content according to user time budget during generating rhythm control strategy.

[0026] Preferably, the step S4 includes the following sub-steps:

[0027] Step S401, accelerating playing of video segments of non-structure nodes and core nodes, compressing time length by image resampling, retaining key pictures and performing timing reconstruction and aligning speed change and text;

[0028] Step S402, retaining video segments of structure nodes and core nodes, and performing deceleration processing according to user preference.

[0029] Preferably, the step S5 includes the following sub-steps:

[0030] Step S501, extracting key frame images with content continuity features from video segments with skip marks, and performing timing alignment, calibrating color, brightness and motion direction, and establishing picture transition interface;

[0031] Step S502, extracting image features and corresponding semantic information of video segments before and after skip marks based on picture transition interface, and generating transition video frame sequence;

[0032] The construction of transition video frame sequence combines image color distribution, main body edge contour, background texture change and semantic label continuity of video segments before and after skip marks, generates inter-frame transition images in an interpolation manner, and performs time length complementation and tone smoothing processing on audio segments.

[0033] Preferably, the step S502 fuses image change sequence, background dynamic and main body contour when generating virtual transition pictures, applies image interpolation and deformation control methods, and performs edge alignment and inter-frame smoothing processing.

[0034] Preferably, the step S6 includes the following sub-steps:

[0035] Step S601, packaging processed video content, and outputting complete video file, the complete video file including image stream, text track and audio track.

[0036] Step S602, generating auxiliary index data, the auxiliary index data including structure node, core node timestamp, reserved section start and end time, play rate, skip range, chapter number, semantic label and keyword abstract.

[0037] The present application has the beneficial effects that: the present application performs video accelerated processing based on multi-modal semantic burst analysis, can accurately identify key content and generate rhythm control strategy combined with user preference, and realizes natural connection of skip section through transition frame generation, thereby improving the efficiency and continuity of video playing. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A step flowchart of a video accelerated processing method provided for an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the above objectives, features and advantages of the present application more apparent and understandable, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the present application, rather than all the embodiments.

[0040] Embodiments, with reference to Figure 1 , a video accelerated processing method is provided, comprising the following steps:

[0041] Step S1, collecting image data, audio data and speech text, and extracting corresponding semantic factors.

[0042] Step S2, constructing a joint semantic representation vector, calculating burst intensity and generating a semantic burst curve.

[0043] Step S3, obtaining user preference and generating a rhythm control strategy.

[0044] Step S4, performing accelerated playing and key content reservation according to the strategy.

[0045] Step S5, generating skip transition frames to enhance video continuity.

[0046] Step S6, packaging the processed video and generating auxiliary index data.

[0047] Step S1 comprises the following sub-steps:

[0048] Step S101, extract consecutive frame image data from the original video stream and analyze the dynamic characteristics of the picture based on inter-frame change analysis. The dynamic characteristics of the picture include calculating the pixel change rate of adjacent frames to determine the lens switching rate, tracking the motion path of the picture center point and the main object to determine the picture movement amplitude, applying a target detection model to identify the main content of the video and record its appearance frequency and duration, and constructing an image change sequence.

[0049] Step S101 is used to extract and quantify the picture change characteristics between video frames. The image change sequence is constructed by elements such as lens switching rate, main object motion path and object appearance frequency, providing a dynamic basis for visual perception in semantic bursts.

[0050] Step S102, extract speech signals from audio data, analyze speech rate, tone change and repeated word features, and transcribe audio data into text to generate a speech text sequence with timestamps.

[0051] Step S102 is used to extract speech change characteristics such as speech rate and tone from audio data, and transcribe them into a speech text sequence with timestamps, providing acoustic information support for subsequent semantic analysis and multi-modal alignment.

[0052] Step S103, process the speech text sequence, perform text normalization, sentence segmentation, semantic segmentation and keyword extraction operations, and use syntax analysis tools to identify subject-predicate structures and modification relationships, output semantic structure labels and keyword sequences.

[0053] Step S103 is used to linguistically process the speech text sequence, extract keywords and grammatical structures, and output a structured semantic label set as an important semantic factor for constructing joint semantic representation vectors.

[0054] Step S1 is used to extract three types of basic modal data, i.e. image, audio and text, from the video, and construct a multi-modal semantic factor set, providing complete and time-aligned semantic basis data for subsequent semantic burst analysis and rhythm control strategy generation.

[0055] Step S2 includes the following sub-steps:

[0056] Step S201, align the image change sequence, speech text sequence, semantic structure labels and keyword sequence in the time dimension, synchronize the consecutive frame image data, audio data and speech text sequence based on the unified time axis, and construct a joint semantic representation vector in each time window. The joint semantic representation vector includes the degree of visual subject change, the intensity of speech rate and emotional fluctuations, the keyword density and the degree of semantic mutation.

[0057] Step S201 is used to align the image change sequence, the speech text sequence and the semantic label information according to a unified time axis, synchronize the image, the audio and the text features, and construct a joint semantic representation vector containing information such as visual change, speech speed and emotion change and text keyword mutation, so as to provide a unified input data format for subsequent burst detection.

[0058] Step S202 is used to perform sliding time window analysis on the joint semantic representation vector, calculate a semantic burst intensity score, identify a burst or turning point semantic section by detecting a peak value and a trend mutation, classify the semantic section into a structure node and a core node, and output a semantic burst curve through trend modeling.

[0059] Step S202 is used to identify a new topic starting point and an event climax section based on a burst intensity change trend, and correspondingly increase a rhythm weight of the section, so as to preferentially retain or slow down playing in rhythm control.

[0060] Step S202 is used to perform sliding window analysis on the joint semantic representation vector, calculate a semantic burst intensity score in each time period, identify a time section with significant semantic change, and divide the time section into a structure node and a core node, and output a semantic burst curve through trend modeling, which is used as a core basis for content layering and rhythm labeling. When a significant upward trend of the burst intensity is detected, a new topic starting point or an event climax section is identified, and a rhythm weight of the section in the rhythm strategy is correspondingly increased, so as to ensure that the section is preferentially retained or slowed down, thereby ensuring the understandability of the key content and the rhythm continuity.

[0061] Step S2 is used to fuse image, speech and text features, construct a joint semantic representation vector reflecting a multi-modal change trend, identify a semantic burst section based on a time sequence change of the joint semantic representation vector, and generate a semantic burst curve, which provides a structured semantic basis for a rhythm control strategy.

[0062] Step S3 includes the following sub-steps:

[0063] Step S301 is used to obtain a user's preference setting for a playing rhythm, the preference setting including an expected total time length, a content type of interest and a minimum retention ratio, and convert the preference setting into a structured parameter.

[0064] Step S301 is used to collect a user's personalized demand for a playing rhythm, including an expected total time length of a video, a content type of interest and a minimum retention content ratio acceptable by the user, and convert the information into a structured parameter, which provides a constraint condition and a priority basis for strategy generation.

[0065] Step S302 is used to generate a rhythm control strategy in combination with the semantic burst curve and the structured parameter, assign a playing rate and a skip mark by analyzing a matching condition of a burst intensity of each time section in the semantic burst curve and a content type of interest of the user, and output a rhythm strategy file.

[0066] Step S302 is used to adjust the playback rate and skip range of low-priority segments and retain key semantic content according to the user time budget in the process of generating the rhythm control strategy.

[0067] Step S302 is used to integrate the semantic burst curve and structured user preference parameters, identify time periods with high semantic intensity and meet the user's attention type, generate a rhythm control strategy, assign a corresponding playback rate and skip mark to each segment, form a rhythm strategy file for subsequent module calling, and dynamically adjust the playback rate and skip range of low-priority segments in the process of generating the rhythm control strategy, ensure that the final playback time length meets the user budget, and preferentially retain key semantic content when compressed, and ensure that the information core is not missed.

[0068] Step S3 is used to generate a rhythm control strategy according to the user's set playback preference parameters, combine the semantic burst curve, complete intelligent allocation of the playback rate and skip range of each time period, and realize personalized playback rhythm control.

[0069] Step S4 includes the following sub-steps:

[0070] Step S401 is used to accelerate the playback of video segments of non-structure nodes and core nodes, compress the time length through image resampling, retain key pictures and perform timing reconstruction and alignment of speech rate changes and text.

[0071] Step S401 is used to perform acceleration processing on ordinary segments that are not marked as structure nodes or core nodes, specifically by compressing the time length through image resampling, retaining key picture frames and reconstructing the picture order, and aligning speech rate changes and text to avoid content understanding disruption.

[0072] Step S402 is used to retain video segments of structure nodes and core nodes and perform deceleration processing according to user preferences.

[0073] Step S402 is used to maintain complete playback of important semantic segments corresponding to structure nodes and core nodes, and perform moderate deceleration operation according to user preferences, so that the key content is fully displayed and the viewer's absorption ability of key information is enhanced.

[0074] Step S4 is used to perform video acceleration and key segment retention operations according to the rhythm control strategy, compress the overall playback time length through differentiated playback strategies, while ensuring that key content is clearly presented and semantic logic is not destroyed.

[0075] Step S5 includes the following sub-steps:

[0076] Step S501, extract key frame images with content continuity features from the video segment marked by the skip mark, and perform time alignment, calibrate color, brightness and motion direction, and establish picture transition interface.

[0077] Step S501 is used to extract key frame images of two reserved segments before and after the skip from the video segment that is skipped. These key frames have content continuity features. By time alignment of the key frames and calibration of color, brightness and motion direction, a stable transition interface is established to provide image anchor points for subsequent transition frame insertion.

[0078] Step S502, based on the picture transition interface, extracts image features and corresponding semantic information of the video segments before and after the skip mark, and generates a transition video frame sequence.

[0079] The construction of the transition video frame sequence combines the image color distribution, main body edge contour, background texture change and semantic label continuity of the video segments before and after the skip mark, generates inter-frame transition images using the interpolation method, and performs time length filling and tone smoothing processing on the audio segment.

[0080] Step S502 fuses image change sequence, background dynamic and main body contour when generating virtual transition pictures, applies image interpolation and deformation control method, and performs edge alignment and inter-frame smoothing processing.

[0081] Step S502 is used to extract video image features before and after the skip segment based on the picture transition interface and their corresponding semantic information, and generate a transition video frame sequence. The sequence considers the color distribution, main body edge contour, background texture change and semantic label continuity before and after the skip, generates visually coherent frames using the interpolation algorithm, fills and smooths the audio discontinuity caused by the skip, improves the auditory consistency, further fuses the image change sequence, background dynamic features and main body shape contour in the virtual transition frame generation process, applies image interpolation and deformation control method, makes the inserted frames consistent with the original segment in style, structure and dynamics, and realizes video visual natural transition through edge alignment and inter-frame smoothing operation.

[0082] Step S5 is used to construct natural transition image frames and audio streams before and after the skip segment, solve the visual and auditory abruptness problem caused by the skip, and realize the coherent playback experience of the accelerated video.

[0083] Step S6 includes the following sub-steps:

[0084] Step S601, package the processed video content, output the complete video file, and the complete video file includes image stream, text track and audio track.

[0085] Step S601 is used for packaging and outputting the video content after acceleration processing, skip transition compensation and rhythm strategy execution, generating a complete video file containing image stream, synchronization text track and audio track, ensuring synchronization and consistency of multi-modal data, and adapting to multi-terminal playback requirements.

[0086] Step S602 generates auxiliary index data, and the auxiliary index data includes structure nodes, core node timestamps, reserved segment start and end times, playback rates, skip ranges, chapter numbers, semantic labels and keyword abstracts.

[0087] Step S602 is used for generating auxiliary index data corresponding to the processed video content, and the index information includes timestamps of structure nodes and core nodes, reserved segment start and end times, playback rates of each segment, skip segment ranges, chapter numbers, semantic labels and keyword abstracts, which can be used for user quick navigation, system secondary analysis or interface linkage calling.

[0088] Step S6 is used for outputting the complete video file after acceleration processing and the structured auxiliary index data thereof, realizing efficient delivery and navigable management of the video content, and improving user playback experience and system integration capability.

[0089] The present application constructs a semantic burst curve through multi-modal fusion of images, audio and text, can effectively identify key segments such as new topics and climax transitions, automatically generates rhythm control strategies according to user-set time budget and attention types, realizes differentiated playback speed and skip mark allocation, extracts key frames before and after skipping and generates transition frame sequences, realizes natural connection of vision, hearing and subtitles between skip segments, generates auxiliary index data including structure nodes, semantic labels and the like, and is convenient for user navigation, skipping and review. The method is not only suitable for education review and conference recording, but also can be used in content abstract generation, mobile terminal video browsing and the like.

[0090] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (or computer- readable storage media) having computer-usable program code embodied in the medium. The medium can be any available medium or combination thereof that is accessible by a general purpose or special purpose computer. By way of example, such computer-usable storage media can include a volatile memory, such as a random access memory (RAM), a non-volatile memory, such as a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic disk, a flash memory, a compact disk (CD) or a digital versatile disk (DVD). The computer-usable program code can include any suitable set of instructions, statements or Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks

[0091] It should be noted that the above-mentioned embodiments are only used to illustrate but not to limit the technical solutions of the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A method for video acceleration processing, the method comprising: The method comprises the following steps: Step S1, collecting image data, audio data and speech text, and extracting corresponding semantic factors; Step S2, constructing a joint semantic representation vector, calculating burst intensity and generating a semantic burst curve; Step S3, obtaining user preferences and generating a rhythm control strategy; Step S4, performing accelerated playback and key content preservation according to the strategy; Step S5, generating a skip transition frame to enhance video coherence; Step S6, packaging the processed video and generating auxiliary index data.

2. The method of claim 1, wherein, The step S1 comprises the following sub-steps: Step S101, extracting continuous frame image data from the original video stream, and analyzing the picture dynamic characteristics based on the inter-frame change, which includes calculating the pixel change ratio of adjacent frames to determine the lens switching rate, tracking the motion path of the picture center point and the main object to determine the picture movement amplitude, applying a target detection model to identify the main content of the video and record its appearance frequency and duration, and constructing an image change sequence; Step S102, extracting speech signals from the audio data, analyzing speech speed, tone change and repeated word features, and transcribing the audio data into text to generate a speech text sequence with timestamps; Step S103, processing the speech text sequence, performing text normalization, sentence segmentation, semantic word segmentation and keyword extraction operations, and using a syntax analysis tool to identify subject-predicate structures and modification relationships, and outputting semantic structure labels and keyword sequences.

3. The method of claim 2, wherein the video acceleration processing method is characterized by, The step S2 comprises the following sub-steps: Step S201, aligning the image change sequence, speech text sequence, semantic structure labels and keyword sequence in the time dimension, synchronizing the continuous frame image data, audio data and speech text sequence based on the unified time axis, constructing a joint semantic representation vector in each time window, and the joint semantic representation vector includes the degree of visual subject change, the intensity of speech speed and emotional fluctuation, the keyword density and the degree of semantic mutation; Step S202, performing sliding time window analysis on the joint semantic representation vector, calculating the semantic burst intensity score, identifying the burst or turning point semantic section by detecting the peak value and trend mutation, and classifying it into structure nodes and core nodes, and outputting the semantic burst curve through the semantic change trajectory obtained by trend modeling.

4. The method of claim 3, wherein the video acceleration processing method is characterized by, The step S202 identifies the starting point of a new topic and the climax segment of an event based on the trend of burst intensity change, and correspondingly increases the rhythm weight, and preferentially retains or slows down the playback in rhythm control.

5. The method of claim 4, wherein the video acceleration processing method is characterized by, The step S3 comprises the following sub-steps: Step S301, obtaining user preference settings for playback rhythm, the preference settings including expected total length, content type of interest and minimum retention ratio, and converting them into structured parameters; Step S302, generating a rhythm control strategy in combination with the semantic burst curve and the structured parameters, assigning playback speed and skip marks by analyzing the matching of burst intensity and user interested types in each time period of the semantic burst curve, and outputting a rhythm strategy file.

6. The method of claim 5, wherein the video acceleration processing method is characterized by, In the process of generating the rhythm control strategy, the step S302 adjusts the playback speed and skip range of low-priority segments and retains key semantic content according to the user's time budget.

7. The method of claim 6, wherein the video acceleration processing method is characterized by, The step S4 comprises the following sub-steps: Step S401, accelerating the video segments of non-structural nodes and core nodes, compressing the time length by image resampling, retaining key pictures and performing timing reconstruction and aligning the speed changes and text; Step S402, retaining the video segments of structural nodes and core nodes, and performing deceleration processing according to user preferences.

8. The method of claim 7, wherein the video acceleration processing method is characterized by, The step S5 comprises the following sub-steps: Step S501, extracting key frame images with content continuity features from the skip mark video segments, and performing timing alignment, calibrating color, brightness and motion direction, and establishing picture transition interface; Step S502, based on the picture transition interface, extracting image features and corresponding semantic information of the video segments before and after the skip mark, and generating transition video frame sequence; The construction of the transition video frame sequence combines the image color distribution, main body edge contour, background texture change and semantic label continuity of the video segments before and after the skip mark, generates inter-frame transition images by interpolation, and performs time length filling and tone smoothing processing on the audio segments.

9. The method of claim 8, wherein the video acceleration processing method is characterized by, The step S502 fuses image change sequence, background dynamic and main body contour when generating virtual transition pictures, applies image interpolation and deformation control method, and performs edge alignment and inter-frame smoothing processing.

10. The method of claim 9, wherein the video acceleration processing method is characterized by, The step S6 comprises the following sub-steps: Step S601, packaging the processed video content, outputting a complete video file, the complete video file comprising image stream, text track and audio track; Step S602, generating auxiliary index data, the auxiliary index data comprising structural node, core node timestamp, retained segment start and end time, play rate, skip range, chapter number, semantic label and keyword summary.

Images