A video intelligent compression method based on image analysis
Through the intelligent video compression method based on image analysis, customized processing is carried out for different video types, and the problem of distinguishing video content in the existing technology is solved, efficient video compression and quality retention is achieved, suitable for multiple video types, and storage is reduced and bandwidth requirements.
Patent Information
- Application Number
- CN202411173229.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-26
AI Technical Summary
Existing video compression technology cannot effectively distinguish the key information and redundant information in the video content, resulting in the loss of important visual content during the compression process or the retaining of too much invalid data, especially in the compression of live videos and ultra-high-definition videos, it is difficult to balance the image quality and Delay and lack of refined processing for different video types.
Using the intelligent video compression method based on image analysis, through space-time frame processing and video type recognition, multi-modal theme recognition, hierarchical scene structure analysis, and scene complexity evaluation of short videos, animation movies, real scene movies and live videos respectively. It is divided with real-time subject object, realizes semantic-aware adaptive frame rate control, vectorized encoding, adaptive hybrid compression and region bit rate allocation, and adapts to the characteristics of different video types.
It significantly improves compression efficiency and video quality, reduces data redundancy, retains key visual information, is suitable for various video types, reduces bandwidth requirements and storage costs, and meets the high-resolution storage and transmission requirements of ultra-high-definition videos.
Smart Images

Figure CN118714337B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video compression, and in particular to a video intelligent compression method based on image analysis. Background Art
[0002] In the field of video compression technology, with the increasing richness of multimedia content and the limited network bandwidth, how to effectively compress video data to reduce storage space and transmission time while ensuring video quality has become a technical problem that needs to be solved urgently. Traditional video compression methods often adopt a unified encoding strategy, which cannot achieve the optimal compression effect when processing different types of video content. For example, for different types of videos, such as short videos, animated films, live-action films, and live videos, they each have unique content characteristics and visual expressions, requiring more refined compression strategies.
[0003] Existing video compression technologies usually lack in-depth understanding and analysis of video content, and are unable to distinguish between key information and redundant information in the video, resulting in the loss of important visual content or retention of too much invalid data during the compression process. In addition, the lack of distinction between dynamic and static scenes makes it impossible for the compression algorithm to adaptively adjust the compression strategy according to the complexity of the scene, thus failing to achieve efficient data compression while ensuring video quality.
[0004] In the field of live video, due to the real-time and dynamic nature of live content, traditional compression methods are difficult to meet the compression requirements of live video. Live video often needs to achieve low-latency transmission while maintaining high image quality, which puts higher requirements on compression technology. However, existing compression technology often finds it difficult to find a balance between maintaining image quality and reducing latency.
[0005] In addition, with the popularity of ultra-high-definition videos (such as 4K and 8K), the amount of video data has increased dramatically, posing greater challenges to storage and transmission bandwidth. The compression processing of ultra-high-definition videos not only needs to consider the reduction of data volume, but also the high-resolution characteristics of the video to ensure that the compressed video can still present rich details and colors. Summary of the invention
[0006] Based on this, it is necessary for the present invention to provide a video intelligent compression method based on image analysis to solve at least one of the above technical problems.
[0007] To achieve the above object, a video intelligent compression method based on image analysis includes the following steps:
[0008] Step S1: obtaining video data to be compressed; performing spatiotemporal frame processing on the video data to be compressed to obtain a video frame sequence to be compressed; performing video type identification on the video data to be compressed based on the video frame sequence to be compressed to obtain video type data to be compressed, wherein the video type data to be compressed is any one of short video type data, animation film type data, real-life film type data and live video type data;
[0009] Step S2: If the type of video data to be compressed is short video type data, multimodal topic recognition is performed on the video data to be compressed based on the video frame sequence to be compressed to obtain short video topic distribution data; semantic-aware adaptive frame rate control compression is performed on the video data to be compressed according to the short video topic distribution data to obtain first compressed video data;
[0010] Step S3: If the video type data to be compressed is animation film type data, hierarchical scene structure analysis is performed on the video data to be compressed based on the video frame sequence to be compressed to obtain an animation scene structure library; vectorized compression encoding based on semantic relations is performed on the video data to be compressed according to the animation scene structure library to obtain second compressed video data;
[0011] Step S4: if the video type data to be compressed is real-life movie type data, then based on the video frame sequence to be compressed, the scene complexity evaluation is performed on the video data to be compressed to obtain scene complexity distribution data; according to the scene complexity distribution data, the video data to be compressed is adaptively lossy or lossless hybrid compressed to obtain third compressed video data;
[0012] Step S5: If the video type data to be compressed is live video type data, the video data to be compressed is subjected to real-time subject object segmentation and tracking based on the video frame sequence to be compressed to obtain a video subject object mask sequence; the video data to be compressed is subjected to regional bit rate allocation compression according to the video subject object mask sequence to obtain fourth compressed video data.
[0013] The present invention realizes accurate optimization of different video content characteristics through intelligent identification of video types and customized compression, and significantly improves compression efficiency and video quality. Through the application of multimodal theme recognition and semantic perception compression technology, in-depth content understanding and adaptive frame rate control are provided for short video content, which effectively reduces data redundancy, retains key visual information, and enhances the viewing experience. Through hierarchical scene structure analysis and vectorized coding, a scene gallery is constructed for animated films, the compression process is optimized, the amount of data is reduced, and visual coherence and details are maintained. Through scene complexity evaluation and adaptive hybrid compression technology, a dynamically adjusted compression strategy is provided for real-life movies, and the compression rate and video quality are intelligently balanced. Through real-time subject object segmentation and regional bit rate allocation compression technology, it is particularly suitable for live video, improves compression efficiency, and ensures the clarity and real-time of important objects in live broadcast. In addition, the present invention also particularly optimizes the compression of ultra-high-definition video, meeting the storage and transmission requirements of future high-resolution video content. Overall, the compression mechanism of the present invention has a high degree of versatility and adaptability, is applicable to various video types, and effectively reduces bandwidth requirements and storage costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments thereof made with reference to the following drawings:
[0015] Figure 1 A schematic flow chart of the steps of a video intelligent compression method based on image analysis according to an embodiment is shown.
[0016] Figure 2 A detailed flowchart diagram of step S38 of an embodiment is shown.
[0017] Figure 3 A detailed flow chart of step S5 of an embodiment is shown. DETAILED DESCRIPTION
[0018] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.
[0019] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.
[0020] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.
[0021] To achieve this, please refer to Figures 1 to 3 The present invention provides a video intelligent compression method based on image analysis, comprising the following steps:
[0022] Step S1: obtaining video data to be compressed; performing spatiotemporal frame processing on the video data to be compressed to obtain a video frame sequence to be compressed; performing video type identification on the video data to be compressed based on the video frame sequence to be compressed to obtain video type data to be compressed, wherein the video type data to be compressed is any one of short video type data, animation film type data, real-life film type data and live video type data;
[0023] Step S2: If the type of video data to be compressed is short video type data, multimodal topic recognition is performed on the video data to be compressed based on the video frame sequence to be compressed to obtain short video topic distribution data; semantic-aware adaptive frame rate control compression is performed on the video data to be compressed according to the short video topic distribution data to obtain first compressed video data;
[0024] Step S3: If the video type data to be compressed is animation film type data, hierarchical scene structure analysis is performed on the video data to be compressed based on the video frame sequence to be compressed to obtain an animation scene structure library; vectorized compression encoding based on semantic relations is performed on the video data to be compressed according to the animation scene structure library to obtain second compressed video data;
[0025] Step S4: if the video type data to be compressed is real-life movie type data, then based on the video frame sequence to be compressed, the scene complexity evaluation is performed on the video data to be compressed to obtain scene complexity distribution data; according to the scene complexity distribution data, the video data to be compressed is adaptively lossy or lossless hybrid compressed to obtain third compressed video data;
[0026] Step S5: If the video type data to be compressed is live video type data, the video data to be compressed is subjected to real-time subject object segmentation and tracking based on the video frame sequence to be compressed to obtain a video subject object mask sequence; the video data to be compressed is subjected to regional bit rate allocation compression according to the video subject object mask sequence to obtain fourth compressed video data.
[0027] The present invention realizes accurate optimization of different video content characteristics through intelligent identification of video types and customized compression, and significantly improves compression efficiency and video quality. Through the application of multimodal theme recognition and semantic perception compression technology, in-depth content understanding and adaptive frame rate control are provided for short video content, which effectively reduces data redundancy, retains key visual information, and enhances the viewing experience. Through hierarchical scene structure analysis and vectorized coding, a scene gallery is constructed for animated films, the compression process is optimized, the amount of data is reduced, and visual coherence and details are maintained. Through scene complexity evaluation and adaptive hybrid compression technology, a dynamically adjusted compression strategy is provided for real-life movies, and the compression rate and video quality are intelligently balanced. Through real-time subject object segmentation and regional bit rate allocation compression technology, it is particularly suitable for live video, improves compression efficiency, and ensures the clarity and real-time of important objects in live broadcast. In addition, the present invention also particularly optimizes the compression of ultra-high-definition video, meeting the storage and transmission requirements of future high-resolution video content. Overall, the compression mechanism of the present invention has a high degree of versatility and adaptability, is applicable to various video types, and effectively reduces bandwidth requirements and storage costs.
[0028] In this embodiment, the video data to be compressed is first obtained by manual entry or other means, and then the video processing software (such as FFmpeg) is used to perform spatiotemporal frame processing of the video to generate a sequence of video frames to be compressed. Next, a video analysis tool (such as Adobe Premiere Pro or OpenCV) is used to identify the video type of the frame sequence to determine whether the video belongs to the type of short video, animated film, live-action movie or live video. Different compression strategies are implemented according to the video type: short videos are multimodal subject recognition is performed through deep learning models such as Transformer to obtain subject distribution data, and semantic-aware adaptive frame rate control compression is performed to generate first compressed video data; animated films construct scene libraries through hierarchical scene structure analysis, and perform vectorized compression encoding based on semantic relationships to generate second compressed video data; live-action movies evaluate scene complexity and generate distribution data, implement adaptive lossy or lossless hybrid compression, and generate third compressed video data; live videos perform real-time subject object segmentation and tracking, generate mask sequences, and perform regional bit rate allocation compression according to the sequences to generate fourth compressed video data. Preferably, step S1 includes the following steps:
[0029] Step S11: obtaining video data to be compressed;
[0030] Specifically, the video data to be compressed can be obtained by manual entry.
[0031] Step S12: performing multimodal information separation on the video data to be compressed to obtain video audio data to be compressed and pure visual video data to be compressed;
[0032] Specifically, Adobe Premiere Pro or Audacity can be used to process the video and audio tracks of the video to be compressed separately. In this process, the built-in separation algorithm of the software will identify and extract the audio stream, separate it from the video data, and finally obtain the video audio data to be compressed and the pure visual video data to be compressed.
[0033] Step S13: identifying the total duration of the video for the compressed pure visual video data to obtain the total duration data of the video;
[0034] Specifically, you can use a video processing library such as FFmpeg to identify the total duration of the video. The script will traverse the video frames of the pure visual video to be compressed and count the number of frames, and then convert it into the playback duration of the video, usually in seconds or minutes. In addition, you can also use the built-in duration detection function of the video editing software to obtain the total duration data of the video, and finally get the total duration data of the video.
[0035] Step S14: extracting time-series frames from the pure visual video data to be compressed to obtain a video frame sequence to be compressed;
[0036] Specifically, video processing software, such as FFmpeg, can be used to extract time-series frames from a purely visual video to be compressed. By writing a specific FFmpeg command, such as ffmpeg -i input.mp4 -frames:v 1 output_%04d.png, key frames can be extracted from the video file input.mp4 and saved as an image sequence in the format of output_0001.png and output_0002.png. These images represent various moments of the video and constitute the video frame sequence to be compressed.
[0037] Step S15: performing emotional rhythm analysis on the compressed video and audio data to obtain video and audio emotional feature data;
[0038] Specifically, you can use audio editing software such as Audacity to open the video audio to be compressed and export it to a standard audio format, such as WAV or MP3. Then, use the Librosa library to read and analyze the audio file to extract pitch, rhythm, and intensity features. Through these features, you can evaluate the emotional content in the audio, such as happiness, sadness, or neutrality. For example, Librosa can calculate Mel Spectral Coefficients (MFCCs), which can reflect the important characteristics of the audio signal. After the analysis is completed, these feature data are integrated into video audio emotional feature data.
[0039] Step S16: Based on the video frame sequence to be compressed, the video type of the video data to be compressed is identified according to the total video duration data and the video and audio emotional feature data to obtain the video type data to be compressed, wherein the video type data to be compressed is any one of short video type data, animation film type data, real-life film type data and live video type data.
[0040] Specifically, please refer to the sub-steps of step S16 for the detailed implementation process of this embodiment.
[0041] The present invention provides a basis for in-depth analysis and customized compression of video content by accurately separating audio and visual information, and significantly improves the adaptability and efficiency of compression strategies. Resource allocation is further optimized by identifying the total duration of the video, ensuring the rationality and efficiency of the compression process. The time-series frame extraction technology lays the foundation for the selection and compression encoding of key frames, enhancing the fluency and coherence of the compressed video. By introducing emotional rhythm analysis, the compression algorithm can capture and retain emotional expressions in the video, enhancing the ability of semantic understanding and content recognition. Through the intelligent recognition mechanism of video type, combined with the video frame sequence, total duration and audio emotional characteristics, key support is provided for the customization of compression strategies, ensuring the maximization of compression efficiency and video quality. In addition, the present invention has formulated an optimized compression strategy by deeply understanding the video content, including visual information and audio emotions, and achieved efficient compression of the characteristics of different video types, while retaining the original quality and perception of the video to the maximum extent.
[0042] Preferably, step S16 comprises the following steps:
[0043] Step S161: if the total video duration data is greater than a preset video length threshold, then the video frame sequence to be compressed is segmented into time series according to the preset duration threshold to obtain a plurality of groups of video frame subsequences to be compressed;
[0044] Specifically, if the total video length data is greater than a preset video length threshold, an automated tool, such as FFmpeg, can be used to segment the video frame sequence to be compressed according to the time sequence. By setting the parameters of FFmpeg, the start and end time of each video segment can be specified, such as segmenting once every 30 seconds, thereby obtaining several groups of video frame subsequences to be compressed. These subsequences will respectively contain all continuous frames within the specified time period.
[0045] Step S162: capturing the long-term dependency relationship of the video content of each group of video frame subsequences to be compressed, and obtaining semantic feature data of a plurality of video segments;
[0046] Specifically, professional video analysis tools, such as the OpenCV library, can be used to conduct in-depth analysis of each set of video frame subsequences to be compressed. By applying image processing and computer vision techniques, analysis tools can identify and track objects in the video and capture their changes and movements over time. For example, background subtraction methods can be used to separate the foreground and background, and then optical flow algorithms can be used to track the movement of objects. Through these technologies, long-term dependencies in video content can be captured, and this information can be converted into video segment semantic feature data, ultimately obtaining a number of video segment semantic feature data.
[0047] Step S163: performing hierarchical self-attention fusion on the semantic feature data of several video segments to obtain global video semantic feature data;
[0048] Specifically, a deep learning framework, such as TensorFlow or PyTorch, can be used to implement a hierarchical self-attention mechanism. First, the semantic feature data of each video segment is input into a pre-trained deep learning model, such as a Transformer model. The model can handle long-distance dependencies in sequence data through the self-attention mechanism, and can assign different weights to the features of different segments, reflecting their importance in the overall video content. Then, by stacking multiple self-attention layers, the model gradually integrates local features and extracts global video semantic feature data.
[0049] Step S164: if the total video duration data is less than or equal to a preset video length threshold, directly capturing the long-term dependency relationship of the video content on the compressed video frame sequence to obtain global video semantic feature data;
[0050] Specifically, if the total video duration data is less than or equal to a preset video length threshold, a direct analysis method can be used to capture the long-term dependencies of the video content of the video frame sequence to be compressed. Use video analysis software, such as Adobe Premiere Pro or professional computer vision tools, such as OpenCV, to analyze the video frame sequence to be compressed. By applying scene detection algorithms, key visual elements in video frames and their spatial relationships are identified. Subsequently, time series analysis is used to track the changes of these elements over time, thereby capturing the dynamic changes and long-term dependencies of the video content. After the analysis is completed, this information is combined to form global video semantic feature data, which includes but is not limited to scene recognition, object tracking, and motion pattern key information.
[0051] Step S165: performing temporal emotional dynamic modeling on the global video semantic feature data and the video audio emotional feature data to obtain audio emotional semantic feature data;
[0052] Specifically, audio analysis tools such as Praat or Audacity can be used to import the audio track of the video, perform preliminary analysis of the sound waveform, and identify basic audio properties such as pitch, volume, and rhythm. Subsequently, a machine learning framework such as TensorFlow or PyTorch is used to build and train a recurrent neural network or long short-term memory network model specifically for time-series sentiment analysis. The model inputs the previously extracted audio feature data and learns to associate these features with different emotional states through training, such as identifying high pitch and fast rhythm as happiness, and low pitch and slow rhythm as sadness. During training, the model learns to predict different emotional patterns through a large amount of annotated data. In addition, the global video semantic feature data of the video is integrated with the video audio emotional feature data through feature vector splicing or attention mechanism fusion technology to improve the model's prediction accuracy of the emotional state. Finally, the audio emotional semantic feature data generated by the model includes not only emotional classification, but also emotional intensity and change trend.
[0053] Step S166: performing interactive feature enhancement on the video data to be compressed according to the global video semantic feature data and the audio emotional semantic feature data to obtain cross-modal feature data of the video to be compressed;
[0054] Specifically, principal component analysis (PCA) or multidimensional scaling (MDS) methods can be used to project global video semantic feature data and audio emotional semantic feature data into a common feature space to identify the correlation between different modalities. In addition, deep learning models, such as a combination of convolutional neural networks (CNN) and recurrent neural networks (RNN), can be used to learn and extract cross-modal features. In this way, cross-modal feature data of the video to be compressed can be obtained.
[0055] Step S167: Use a preset video type classifier to classify the cross-modal feature data of the video to be compressed into video type data to obtain the video type data to be compressed, wherein the video type data to be compressed is any one of short video type data, animation film type data, real-life film type data and live video type data.
[0056] Specifically, the preset video type classifier can be based on machine learning or deep learning models, such as support vector machines (SVM), random forests, or deep neural networks (DNN). First, a labeled data set is used to train the classifier so that it can recognize different types of video content, such as short videos, animated films, live-action films, and live videos. After the training is completed, the classifier is applied to the cross-modal feature data of the video to be compressed to predict and classify the video type. The classification result will be used as the video type data to be compressed, where the video type data to be compressed is any one of short video type data, animated film type data, live-action film type data, and live video type data.
[0057] The present invention realizes detailed analysis of long video content through refined video segmentation processing, captures key long-term dependencies, and provides precise strategies for video content compression. Through the deep extraction of video segment semantic features, combined with hierarchical self-attention fusion technology, the understanding of global video content is improved, thereby providing rich semantic information for the compression algorithm of video content. The dynamic modeling of audio emotions further enriches the emotional dimension of video content and enhances the intelligence level of compression algorithms. Through the interactive enhancement of cross-modal features, not only the expressiveness of video content is improved, but also the compression algorithm can make more comprehensive use of video and audio data. Through accurate video type classification, it is ensured that various types of video content can obtain the compression processing that best suits their characteristics.
[0058] Preferably, step S2 comprises the following steps:
[0059] Step S21: if the type of video data to be compressed is short video type data, the initial resolution of the video data to be compressed is identified to obtain the initial resolution data of the video; if the initial resolution data of the video is less than or equal to the preset second resolution parameter, the video data to be compressed is directly losslessly compressed to obtain the first compressed video data; otherwise, the next step is executed;
[0060] Specifically, if the video type data to be compressed is short video type data, video processing software, such as MediaInfo, can be used to identify the initial resolution of the video to be compressed. MediaInfo provides a user interface through which the user can view the detailed properties of the video, including its resolution. After obtaining the initial resolution data of the video, it is compared with the preset second resolution parameter. If the initial resolution data of the video meets the conditions for lossless compression, that is, it is less than or equal to the preset second resolution parameter, then a lossless compression method is selected to compress the video data to be compressed to obtain the first compressed video data, such as using HandBrake software and selecting a preset configuration suitable for lossless compression, such as setting the CRF (constant rate factor) of the H.264 video encoder to 0, which indicates lossless compression.
[0061] Step S22: extracting context from the video data to be compressed to obtain context data of the video to be compressed, and performing semantic intent recognition on the context data of the video to be compressed to obtain video semantic intent data;
[0062] Specifically, chat records including text, emoticons, pictures, and voice messages can be collected through the API or export function of the WeChat platform, which form the basis of the video context data to be compressed. Subsequently, natural language processing technology is used to conduct in-depth analysis of text messages, and text mining tools such as NLTK or spaCy are used to identify key phrases, named entities, and emotional tendencies, thereby evaluating the emotional tendencies of the chat content. For non-text content, computer vision technology is applied, such as emoticons using emotional mapping tables to identify emotional colors, and image content uses image recognition technology to extract subject and object information. The results of text and non-text analysis are combined to form a comprehensive understanding of the video context. For example, if a specific topic is frequently mentioned in the chat record and the emotional tendency is positive, the semantic intent of the video is related to celebration or sharing joy. Finally, these analysis results are used as video semantic intent data, including but not limited to key topics, emotional tendencies, and activity types.
[0063] Step S23: grouping the video frame sequences to be compressed at equal intervals according to a preset frame number threshold to obtain a plurality of groups of fixed video frame sequences;
[0064] Specifically, you can use a video processing library, such as FFmpeg or OpenCV, to group the video frame sequence to be compressed into equal intervals according to a preset frame number threshold. The script will calculate the length of the entire video frame sequence and determine the number of frames that each group should contain. Then, the script automatically extracts these frames and saves them as separate image sequences or video clips. For example, if the preset frame number threshold is 30, the script will extract every 30 frames from the video, create multiple groups containing 30 frames, and finally obtain several groups of fixed video frame sequences.
[0065] Step S24: performing video expression theme recognition on each group of fixed video frame sequences to obtain a number of video expression theme data, wherein each video expression theme data corresponds to a group of fixed video frame sequences;
[0066] Specifically, video content analysis tools, such as IBM Watson Media or Google Cloud Video Intelligence API, can be used to accurately identify the theme expressed by each fixed set of video frame sequences. These tools apply deep learning algorithms to perform detailed frame-by-frame analysis of video frames to identify key visual elements, such as scenes, objects, people, and activities. Taking the beach scene as an example, the tool can identify the beach, parasols, crowds, and related activities such as swimming or sunbathing, and further analyze the combination of these elements and the temporal context in the frame sequence, including scene transitions, changes, and audio clues, such as dialogue or background sounds, to infer the overall theme. The identified themes, such as vacation, leisure, sports, or education, will be used to generate a set of video expression theme data for a fixed video frame sequence, which is associated with a corresponding set of fixed video frame sequences. Finally, a number of video expression theme data are obtained, where each video expression theme data corresponds to a set of fixed video frame sequences.
[0067] Step S25: removing irrelevant topics from a plurality of video expression topic data according to the video semantic intent data to obtain a plurality of relevant video expression topic data;
[0068] Specifically, sentiment analysis tools, such as Python's TextBlob library, can be used to perform sentiment tendency analysis on each video expression theme data to identify whether it is consistent with the video semantic intent data. If the video semantic intent data shows a celebratory sentiment, opposite or irrelevant topics, such as sadness or controversy, will be screened out. By comparing the sentiment tendency of each main video expression theme data with the video semantic intent, those inconsistent topics are eliminated. For example, if the identified theme is "sad mourning" and the video semantic intent is celebration, the theme will be deleted. In order to ensure accuracy, manual review can be introduced to review the results of automatic screening based on the video semantic intent and content to avoid erroneous elimination in automatic analysis. Finally, the video expression theme data that is consistent with the video semantic intent emotion is retained, and several related video expression theme data are obtained.
[0069] Step S26: performing spectral clustering and topic fusion on a number of related video expression topic data to obtain short video topic distribution data, wherein the short video topic distribution data includes a number of short video segment topic data;
[0070] Specifically, data science tools, such as Python's Scikit-learn library, can be used to perform spectral clustering analysis on the selected relevant video expression topic data. Spectral clustering can cluster similar topic data together. For example, if multiple video frame sequences are related to outdoor activities, spectral clustering can merge these topic data into a broader "outdoor activities" topic. By setting appropriate clustering parameters, short video topic distribution data can be obtained, which reflects the distribution and importance of different topics in the video.
[0071] Step S27: performing topic relevance aggregation on a plurality of groups of fixed video frame sequences according to the short video topic distribution data to obtain a plurality of topic-aggregated video frame sequences;
[0072] Specifically, we can use the automated script again to aggregate fixed video frame sequences based on the short video topic distribution data. The script will group frame sequences belonging to the same topic together based on the importance and frequency of each topic in the topic distribution data to form topic-aggregated video frame sequences. For example, if the topic of "outdoor activities" accounts for a large proportion in the video, the script will group these frame sequences into one group.
[0073] Step S28: Based on a plurality of topic-aggregated video frame sequences, semantically-aware adaptive frame rate control is performed on the compressed video data according to short video topic distribution data to obtain first compressed video data.
[0074] Specifically, please refer to the sub-steps of step S28 for the detailed implementation process of this embodiment.
[0075] The present invention realizes fast lossless compression of low-resolution videos through intelligent identification of initial resolution, saves computing resources and ensures processing efficiency. Through in-depth analysis of context and semantic intent, video content can be accurately extracted and understood. Through intelligent grouping of frame sequences, video content is systematized, laying a solid foundation for further analysis. Through theme identification and screening, focus is placed on key information in the video, and the data quality and relevance of the compressed video are improved. Through the application of spectral clustering technology, the video expression themes are integrated to optimize the organizational structure of the video content. The coherence and theme integrity of the content during the compression process are ensured through theme relevance aggregation. Through semantic-aware adaptive frame rate control technology, the frame rate is intelligently adjusted according to the importance of the video content, achieving a balance between video fluency and compression efficiency. On the whole, the present invention ensures the quality and perception of the video content while improving the compression efficiency, and achieves dual guarantees of compression efficiency and quality.
[0076] Preferably, step S28 comprises the following steps:
[0077] Step S281: extracting paragraph themes from the short video theme distribution data to obtain first short video paragraph theme data, and extracting corresponding video frame sequences from the first short video paragraph theme data based on a plurality of theme-aggregated video frame sequences to obtain a first theme-aggregated video frame sequence;
[0078] Specifically, video analysis software, such as Adobe Premiere Pro or Final Cut Pro, can be used to process short video topic distribution data. These software allow users to browse video frame sequences and manually or automatically mark video segments corresponding to each theme based on previously determined video expression topic data. For example, if the "beach leisure" theme is identified as the first short video segment topic data, the software will help users find and mark all beach scene segments in the video. After completing the marking, the software will automatically extract the video frame sequences corresponding to these themes to form the first topic aggregation video frame sequence.
[0079] Step S282: traverse each candidate video frame in the first topic-aggregated video frame sequence, and perform topic-frame correlation calculation on each candidate video frame according to the topic data of the first short video segment to obtain a plurality of frame topic correlation score data;
[0080] Specifically, the video content analysis tool IBM Watson Visual Recognition can be used to identify the visual elements of each frame in the first theme aggregated video frame sequence through its deep learning model, including but not limited to people, objects and scenes. Subsequently, the identified visual elements are matched with the keywords of the theme of the first short video paragraph. For example, under the theme of "beach leisure", multiple elements such as sand, sea water, and parasols are searched. Next, a machine learning algorithm, such as logistic regression or decision tree, is used to assign a relevance score to each element based on the frequency, location and significance factors of the element. The relevance scores of all elements are weighted and summed according to their importance to form a comprehensive score of the elements in the video frame. Finally, each candidate video frame will obtain a frame theme relevance score data.
[0081] Step S283: if the frame subject relevance score data is greater than or equal to a preset relevance threshold, the candidate video frame corresponding to the frame subject relevance score data is recorded as a key frame, that is, a key video frame sequence to be compressed is obtained; if the frame subject relevance score data is less than the preset relevance threshold, the candidate video frame corresponding to the frame subject relevance score data is recorded as a normal frame, that is, a normal video frame sequence to be compressed is obtained;
[0082] Specifically, data processing software, such as Microsoft Excel or Python scripts, can be used to filter out frames whose frame subject relevance score data is greater than or equal to a preset relevance threshold, and these frames are considered key frames because they are highly relevant to the subject. These key frames will be marked as key video frame sequences to be compressed. At the same time, frames with scores lower than the preset relevance threshold are considered ordinary frames, which constitute ordinary video frame sequences to be compressed.
[0083] Step S284: generating a region importance map for the corresponding key video frames based on the key video frame sequence to be compressed according to the frame subject relevance score data to obtain a key frame region importance map;
[0084] Specifically, image processing software, such as Adobe After Effects or Nuke, combined with visual analysis tools, such as OpenCV, can be used to process the sequence of key video frames to be compressed. First, the most relevant areas of the subject are determined in each key video frame based on the frame subject relevance score data. Use OpenCV's image segmentation functions, such as segmentation based on skin color or color histogram, to identify and highlight these areas. Then, use this information to create regional importance maps in After Effects or Nuke, which represent the importance of different areas in the frame with different colors or transparency, thereby obtaining key frame regional importance maps.
[0085] Step S285: performing key-common region segmentation on the corresponding key video frames according to the key frame region importance map based on the key video frame sequence to be compressed, to obtain a key-common region segmented video frame sequence, wherein the key-common region segmented video frame sequence includes a plurality of key-common region segmented video frames;
[0086] Specifically, the keyframe region importance map can be further processed using image segmentation techniques. The key regions are separated from the normal regions through OpenCV's mask operation or Nuke's node system. This process involves creating masks based on the keyframe region importance map, which selectively cover the key regions and mark other regions as normal regions. After the segmentation is completed, a sequence of key-normal region segmented video frames is obtained, in which each frame is clearly divided into key regions and normal regions.
[0087] Step S286: compressing and encoding the key area in each key-ordinary area segmented video frame in the key-ordinary area segmented video frame sequence using a preset first resolution parameter, and compressing and encoding the ordinary area in each key-ordinary area segmented video frame in the key-ordinary area segmented video frame sequence and the ordinary video frame sequence to be compressed using a preset second resolution parameter, so as to obtain a first differential compressed video frame sequence, wherein the preset first resolution parameter is greater than the preset second resolution parameter;
[0088] Specifically, video encoding software, such as FFmpeg, can be used to compress and encode the key-ordinary region segmented video frame sequence. First, a preset first resolution parameter is set for the key region. For the ordinary region and the ordinary video frame sequence to be compressed, a preset second resolution parameter is used for compression. The command line tool of FFmpeg can apply different encoding parameters to different regions, and finally generate a first differential compression video frame sequence.
[0089] Step S287: traverse the short video topic distribution data, execute steps S281-S286 for the topic data of each short video segment, that is, obtain a number of differentially compressed video frame sequences, and perform time-series integration on the several differentially compressed video frame sequences to obtain the first compressed video data.
[0090] Specifically, the short video theme distribution data can be traversed, and steps S281 to S286 can be performed on the theme data of each short video segment, and then a video editing software, such as Adobe Premiere Pro, can be used to integrate the generated differential compression video frame sequences. First, according to the short video theme distribution data, the theme of each segment and the corresponding compressed frame sequence are determined. Then, according to the original timing of the video, these frame sequences are recombined to form a coherent video stream. Finally, the first compressed video data is obtained.
[0091] The present invention provides a clear goal for the optimization processing of frame sequences by accurately extracting the distribution of key topics in short videos, ensuring the coherence of video content and the integrity of the theme. By intelligently evaluating the relevance of frames and themes, key frames and ordinary frames are effectively distinguished, allowing the compression algorithm to concentrate resources to optimize the core content of the video. The identification and optimization of key frames ensure the high-quality retention of the main content in the video, and the generation of regional importance maps further highlights the key focus areas of the video. By intelligently segmenting key and ordinary areas and applying differential compression coding, the present invention achieves differentiated processing of areas of different importance, optimizing compression efficiency and video quality. The smoothness of the video and the consistency of the data are maintained through efficient processing of timing integration.
[0092] Preferably, step S3 comprises the following steps:
[0093] Step S31: if the type of video data to be compressed is animation film type data, semantic segmentation and scene graph structure extraction are performed on each animation video frame in the video frame sequence to be compressed to obtain a video scene graph structure library, wherein the video scene graph structure library contains a plurality of scene graph structure objects;
[0094] Specifically, if the video type data to be compressed is animated film type data, professional image and video processing software, such as Adobe After Effects, can be used with deep learning tools such as TensorFlow or PyTorch to perform semantic segmentation of each animated video frame. These tools can identify and distinguish different objects and backgrounds in the frame to achieve semantic segmentation of objects. After segmentation, the tracking and expression functions of After Effects are used to extract the structural information of the scene, such as props and background elements, and integrate this information into the video scene graph structure library, which will contain multiple scene graph structure objects, each of which is associated with a specific scene or action.
[0095] Step S32: modularly encode and annotate each scene graph structure object in the video scene graph structure library to obtain a modular video scene graph structure library;
[0096] Specifically, image processing software Nuke or similar tools can be used to conduct a detailed review of each scene graph structure object in the video scene graph structure library to determine its function and importance in the animation. Then, these objects are modularly encoded using Nuke's node system to create reusable components and structured data representations. At the same time, each modular object is annotated to record its characteristics and role in the animation, thereby obtaining a modular video scene graph structure library.
[0097] Step S33: performing color clustering and encoding on the video frame sequence to be compressed to obtain a color encoding library of the video to be compressed;
[0098] Specifically, color management tools, such as Adobe Color or professional image processing software Photoshop, can be used to perform color analysis on the video frame sequence to be compressed. These tools can identify the main colors and color distribution in the video and realize color clustering. Through analysis, the colors appearing in the video frame are grouped according to similarity to form a color cluster library. Then, color encoding technology, such as color quantization, is used to assign codes to each color cluster, thereby reducing the range of color representation and reducing the complexity of the data. Finally, a color encoding library of the video to be compressed is obtained.
[0099] Step S34: color mapping is performed on each modular scene graph structure object in the modular video scene graph structure library based on the video frame sequence to be compressed and the video color coding library to be compressed, and a structure-color association table is generated to obtain a scene structure object color association library, wherein the scene structure object color association library includes a plurality of structure object-color association tables, and each structure object-color association table stores an association relationship between a minimum structure block number of a scene graph structure object and a corresponding color coding;
[0100] Specifically, image processing software, such as Adobe Photoshop or GIMP, can be used in combination with color management tools, such as Adobe Color, to process each object in the modular video scene graph structure library. The color of a specific scene graph structure object is selected from the video frame sequence to be compressed using Photoshop's color picking tool, and then the color is standardized and encoded in Adobe Color. Subsequently, a structure-color association table is created using a customized database management system or Excel spreadsheet software to record the minimum structural block number of each scene graph structure object and its corresponding color coding. These association tables will be integrated into the scene structure object color association library.
[0101] Step S35: performing structure-color mapping association between the modular video scene graph structure library and the scene structure object color association library to obtain structure object-color table mapping data;
[0102] Specifically, a database management tool such as MySQL or MongoDB can be used to store and manage the modular video scene graph structure library and the scene structure object color association library. Through SQL query or NoSQL query language, the scene graph structure objects are matched and associated with their color codes to generate structure object-color table mapping data.
[0103] Step S36: removing redundant tables from the scene structure object color association library to obtain a simplified scene structure object color association library;
[0104] Specifically, data cleaning tools, such as Python's Pandas library, can be used to analyze and process the scene structure object color association library. Use Pandas' data processing function to identify and delete duplicate or redundant structure-color association tables. This process involves comparing, identifying, and filtering data, and ultimately obtaining a streamlined scene structure object color association library.
[0105] Step S37: deduplicating the structure objects of the modular video scene graph structure library according to the streamlined scene structure object color association library and the structure object-color table mapping data to obtain a non-redundant video scene graph structure library, and assigning and recording an object ID to each scene structure object in the non-redundant video scene graph structure library to obtain a scene graph structure object index library;
[0106] Specifically, professional data management software can be used in combination with programming languages such as Python to process the streamlined scene structure object color association library and structure object-color table mapping data. By writing scripts, each object in the modular video scene graph structure library is checked for uniqueness, and duplicate structure objects are removed to ensure that each object is unique. Then, a unique object ID is assigned to each scene structure object in the non-redundant video scene graph structure library, and an index library is created using a database system to record these object IDs and their associated information, and finally a scene graph structure object index library is obtained.
[0107] Step S38: Based on the video frame sequence to be compressed, the simplified scene structure object color association library and the scene graph structure object index library, the video data to be compressed is vectorized and compressed based on semantic relationships according to the structure object-color table mapping data to obtain second compressed video data.
[0108] Specifically, please refer to the sub-steps of step S38 for the detailed implementation process of this embodiment.
[0109] The present invention accurately captures the key visual elements of animation video frames through efficient semantic segmentation and scene extraction technology, laying a solid foundation for compression coding. Modular coding and annotation not only improve the efficiency of data organization, but also provide convenience for rapid retrieval and processing of different scenes. Through color clustering and coding technology, key colors are effectively identified and encoded, redundancy is reduced, and compression efficiency is improved. Through the establishment of color mapping and structure-color association, the color distribution in the scene is accurately recorded, providing important information for vectorized compression coding. Through redundancy removal and data simplification, the present invention optimizes storage efficiency and processing speed. At the same time, the deduplication of structural objects and the establishment of an index library reduce the amount of data and speed up the retrieval speed. Through vectorized compression coding based on semantic relationships, the key visual features of animated films are more effectively expressed. During the compression process, sophisticated data processing and optimization strategies ensure the high quality of compressed video data, retaining visual details and artistic expression. In summary, the present invention significantly improves the automation and intelligence level of animated film video compression.
[0110] Preferably, step S38 includes the following steps:
[0111] Step S381: performing role instance segmentation on each animation video frame in the video frame sequence to be compressed, and performing role outline extraction and color mapping to obtain a video role instance gallery to be compressed, wherein the video role instance gallery to be compressed contains a plurality of video role instance images to be compressed;
[0112] Specifically, advanced video editing and image processing software, such as Adobe After Effects combined with the machine learning framework TensorFlow, can be used to process each animated video frame in the sequence of video frames to be compressed. First, use the dynamic tracking function of AfterEffects to segment each character instance in the video, and then use TensorFlow's image recognition model to identify and extract the character's outline. Next, use the color picking tool to map the character's color and create a video character instance gallery to be compressed that contains the character's outline and color information. Each instance image in this gallery is associated with a specific character.
[0113] Step S382: De-duplication of role instances in the to-be-compressed video role instance library, and encoding of role identities, to obtain a to-be-compressed video role instance library-identity library;
[0114] Specifically, data management software, such as Python with the Pandas library, can be used to process the image library of video character instances to be compressed. First, Pandas is used to perform data analysis to identify duplicate or similar character instances and delete these duplicates. Then, unique identity codes are assigned to the remaining unique character instances. Finally, a graph of video character instances to be compressed - an identity library is created, in which each character instance corresponds to its identity code.
[0115] Step S383: performing object identification projection on each animation video frame in the video frame sequence to be compressed based on the video role instance graph-identity library and the scene graph structure object index library to obtain an object-annotated video frame sequence, wherein the object-annotated video frame sequence includes a plurality of object-annotated video frames, and all structural objects of each object-annotated video frame are annotated with corresponding identification codes;
[0116] Specifically, image processing tools and database management tools can be used to process the sequence of video frames to be compressed. First, use image processing software Photoshop or GIMP to manually or automatically map the role identity code in the role instance graph-identity library of the video to be compressed to each animation video frame. Then, use a database system, such as MySQL, to store and manage the scene graph structure object index library to ensure that each structure object can be quickly retrieved according to its index. By writing scripts or using professional video processing plug-ins, the role identity code is associated with the index of the scene graph structure object, and the object identification is projected on the video frame to obtain an object-labeled video frame sequence. Each video frame in this sequence is labeled with the identification codes of all structure objects in detail.
[0117] Step S384: performing semantic topological structure encoding on each object-annotated video frame in the object-annotated video frame sequence to obtain a plurality of video frame identification code topological structure data;
[0118] Specifically, we can focus on in-depth analysis and encoding of each frame in the object-annotated video frame sequence. Use professional image processing and analysis software, such as MATLAB or Python's OpenCV library, to identify and encode the spatial relationships and interactions of objects in the video, which constitute the semantic topological structure of the video frame. Through the algorithm, each object in the object-annotated frame is analyzed to determine their relative position, size, and direction, and encode this information into topological structure data, and finally obtain several video frame identification code topological structure data. These data include not only the geometric properties of the objects, but also the semantic relationships between them.
[0119] Step S385: Based on the video frame sequence to be compressed, the video role instance graph to be compressed-identity library, the scene graph structure object index library and the structure object-color table mapping data, the corresponding object-annotated video frame is semantically vectorized and compressed according to the video frame identification code topological structure data to obtain a first semantic vector compressed frame. When the video frame sequence to be compressed is traversed, a number of semantic vector compressed frames are obtained, and the several semantic vector compressed frames are time-sequentially integrated to obtain a second compressed video data.
[0120] Specifically, Adobe After Effects or similar software can be used to perform detailed analysis on each object-labeled video frame based on the video frame identification code topological structure data. Then, Python is used in conjunction with NumPy and SciPy libraries to vectorize the video frame based on the video frame sequence to be compressed, the role instance graph-identity library, the scene graph structure object index library, and the structure object-color table mapping data. In this way, each object in the video frame is converted into a series of vectors that capture the key visual features and semantic information of the object. Next, a compression algorithm, such as singular value decomposition (SVD) or principal component analysis (PCA), is applied to reduce the dimensionality of these vectors. Finally, these compressed vectors are integrated in time sequence to form a first semantic vector compressed frame. When the entire sequence of video frames to be compressed is traversed, several such compressed frames will be obtained, which together constitute the second compressed video data.
[0121] The present invention provides a solid foundation for the identification and compression coding of characters in animation video frames through precise character instance segmentation and mapping, ensuring the accurate extraction of character outlines. Through deduplication and identity coding, data management is optimized, redundancy is reduced, and a unique identifier is provided for the character, realizing effective management and coding of the character instance. Through the precise projection of object identifiers, combined with the role instance graph-identity library and the scene graph structure object index library, an accurate identification code is provided for the annotation of each structural object in the video frame, enhancing the semantic information of the video frame. The relationship between objects is further captured through the effective coding of the semantic topological structure, providing rich semantic content for compression coding. Through the innovative application of vectorized compression coding, combined with the video frame sequence and the structure object index library, the efficient expression of the key visual and semantic features of the animation video is realized. The coherence and integrity of the data are ensured by the temporal integration of compressed data. While improving the compression efficiency, the compression strategy of the present invention realizes the guarantee of video quality through fine semantic and topological structure coding, achieving a dual improvement in efficiency and quality. The compression strategy of the present invention also exhibits strong adaptability and can meet the compression requirements of different animation video contents. Whether it is a complex scene or a variety of characters, it can provide effective compression processing, demonstrating a high level of intelligence and automation.
[0122] Preferably, step S4 comprises the following steps:
[0123] Step S41: if the type of video data to be compressed is real-scene movie type data, then each real-scene video frame in the sequence of video frames to be compressed is subjected to region segmentation processing, and region recognition and identification are performed to obtain a plurality of groups of video frame region segmentation result data, wherein each group of video frame region segmentation result data represents the segmentation of a real-scene video frame, and each group of video frame region segmentation result data contains a plurality of independent regions;
[0124] Specifically, if the video type data to be compressed is real-life movie type data, advanced video editing software, such as Adobe Premiere Pro or DaVinci Resolve, can be used to import the video frame sequence to be compressed. Use the built-in segmentation tools of these software to manually or automatically perform regional segmentation processing on each real-life video frame. For example, in Premiere Pro, you can use graphic tools to draw masks to distinguish different areas in the frame. After segmentation, use the tracking function of After Effects to track moving objects and generate masks for each area. Then, use machine learning models, such as TensorFlow or PyTorch, to automatically identify and mark these areas, such as identifying people, buildings or natural landscapes. In this way, several groups of video frame area segmentation result data are obtained, each group of data describes in detail the segmentation of a real-life video frame, including several independent areas.
[0125] Step S42: performing local texture complexity evaluation on each independent region in a set of video frame region segmentation result data to obtain a plurality of regional texture complexity indicators;
[0126] Specifically, image analysis tools, such as MATLAB's Image Processing Toolbox, can be used to evaluate the local texture complexity of each independent region. First, the image data of each independent region is extracted from a set of video frame region segmentation result data. Then, texture analysis functions in MATLAB, such as graycoprops or localbinarypatterns, are applied to calculate the texture characteristics of each region, such as roughness, contrast, and directionality. These characteristics comprehensively reflect the texture complexity of the region and are quantified as regional texture complexity indicators, and finally several regional texture complexity indicators are obtained.
[0127] Step S43: performing color richness analysis on each independent region in a set of video frame region segmentation result data to obtain a plurality of regional color richness indicators;
[0128] Specifically, color analysis tools, such as Adobe Photoshop's Color Picker tool or the professional image processing library OpenCV, can be used to perform color richness analysis on each independent region based on a set of video frame region segmentation result data. First, select the Color Picker tool in Photoshop and select representative color samples region by region. Then, use OpenCV's color_histogram function to calculate the color histogram of each region to obtain the statistical data of color distribution. These data include information on color diversity, dominant colors, and color balance, which together constitute the regional color richness index. In this way, several regional color richness indices are finally obtained.
[0129] Step S44: performing object diversity measurement on each independent region in a set of video frame region segmentation result data to obtain a plurality of regional object diversity indicators;
[0130] Specifically, the object diversity of each independent region in the corresponding video frame can be measured based on a set of video frame region segmentation result data using the computer vision library OpenCV and the machine learning framework TensorFlow. First, the objects in each region are further subdivided using the image segmentation function of OpenCV, such as the watershed algorithm or connected domain analysis. Then, these objects are classified and identified using pre-trained models in TensorFlow, such as Inception or ResNet. According to the types and numbers of objects identified, the object diversity index of each region is calculated, and finally several regional object diversity indexes are obtained. For example, the Shannon diversity index can be used to quantify the diversity of objects.
[0131] Step S45: performing a video frame scene complexity evaluation on the real-scene video frame corresponding to the corresponding group of video frame region segmentation result data according to a plurality of regional texture complexity indices, a plurality of regional color richness indices and a plurality of regional object diversity indices to obtain first video frame scene complexity data;
[0132] Specifically, several regional texture complexity indices, several regional color richness indices, and several regional object diversity indices can be collected in a comprehensive data table and processed using Excel or more advanced data analysis software. In this data table, each row represents a region, and the columns correspond to different indicators, such as texture roughness and color diversity. Next, a multi-criteria decision method, such as weighted summation or multi-attribute utility theory (MAUT), is used to synthesize these indicators. In this method, each indicator is assigned different weights according to its contribution to the scene complexity. For example, if texture complexity is more important for the judgment of scene complexity, a higher weight is assigned. Then, the indicators of each region are weighted summed to obtain the scene complexity score of the region. Finally, the scores of all regions are summarized, and the scene complexity evaluation result of the entire video frame can be obtained by averaging or accumulating. This result is the scene complexity data of the first video frame.
[0133] Step S46: traverse several groups of video frame region segmentation result data, execute steps S42 to S45 for each group of video frame region segmentation result data, that is, obtain several video frame scene complexity data, and perform time series integration on several video frame scene complexity data to obtain scene complexity distribution data;
[0134] Specifically, several groups of video frame region segmentation result data can be traversed, and steps S42-S45 can be executed for each group of video frame region segmentation result data to obtain several video frame scene complexity data. After completing the evaluation of a single frame, the script will collect the scene complexity data of each frame, and use the timestamp as an index to sort them according to the playback order of the video frames. Using the sorting and grouping functions of Pandas, these data are integrated into a time series, and each time point corresponds to the scene complexity score of a video frame, and finally the scene complexity distribution data is obtained.
[0135] Step S47: performing adaptive lossy or lossless hybrid compression on the video data to be compressed according to the scene complexity distribution data based on the video frame sequence to be compressed, to obtain third compressed video data.
[0136] Specifically, please refer to the sub-steps of step S47 for the detailed implementation process of this embodiment.
[0137] The present invention provides a structured view for real-life movie video frames through precise region segmentation and identification, making the compression processing of video content more efficient. Through the evaluation of local texture complexity and in-depth analysis of color richness, a quantitative visual complexity index is provided for the formulation of compression strategies, and the compression of color data is optimized. Through the effective measurement of object diversity, the types and quantities of objects in the scene are further understood, providing support for maintaining the visual information of the scene. Through the comprehensive evaluation of scene complexity, the complexity of real-life video frames can be accurately evaluated, providing decision support for adaptive compression. By intelligently selecting lossy or lossless compression methods, a balance between compression efficiency and video quality is achieved, and by integrating scene complexity data in a time series, the continuity and overall consistency of the video sequence are maintained. The compression strategy of the present invention not only ensures compression efficiency, but also guarantees the quality of real-life movie videos through refined scene complexity evaluation and adaptive compression methods.
[0138] Preferably, step S47 includes the following steps:
[0139] Step S471: performing multi-scale motion vector analysis on the video frame sequence to be compressed to obtain a video frame inter-frame motion feature atlas;
[0140] Specifically, the video frame sequence to be compressed can be imported into After Effects, and its built-in tracking function can be used to identify and track the moving objects in the frame sequence. Then, the motion vector between two frames is calculated by the calcOpticalFlowPyrLK function of OpenCV, which will generate a video frame motion feature atlas containing the motion information between video frames at different time points.
[0141] Step S472: performing local motion stability evaluation on each real-scene video frame in the to-be-compressed video frame sequence based on the video frame inter-motion feature atlas to obtain a video frame stability score sequence, wherein the video frame stability score sequence includes a plurality of video frame stability scores;
[0142] Specifically, local motion stability assessment can be performed using relevant functions in the OpenCV library based on the inter-frame motion feature atlas. First, the motion vector field in each video frame is analyzed to identify the consistency and predictability of the motion. Then, the stability of the local motion is assessed by calculating the variance and standard deviation of the motion vector. Lower variance and standard deviation indicate higher stability. This assessment is applied to each real-life video frame to generate a video frame stability score sequence containing multiple stability scores.
[0143] Step S473: extracting corresponding index data from the first real-scene video frame in the to-be-compressed video frame sequence based on the video frame stability score sequence and the scene complexity distribution data, to obtain first video frame stability score data and first video frame scene complexity data;
[0144] Specifically, the stability score data and scene complexity data of the first real-life video frame can be extracted by combining the video frame stability score sequence with the scene complexity distribution data, using data analysis software such as Python's Pandas library. First, locate the starting frame of the video according to the time series. Then, extract the stability score corresponding to the frame from the stability score sequence, and extract the scene complexity value of the starting frame from the scene complexity distribution data, and finally obtain the stability score data of the first video frame and the scene complexity data of the first video frame.
[0145] Step S474: if the stability score data of the first video frame is greater than or equal to the scene complexity data of the first video frame, losslessly compress and encode the corresponding real-scene video frame to obtain first compressed frame data;
[0146] Specifically, if the stability score data of the first video frame is greater than or equal to the scene complexity data of the first video frame, a lossless compression encoding scheme may be selected for the corresponding real-scene video frame. In FFmpeg, this may be achieved by setting a higher CRF (Constant Rate Factor) value, such as CRF 0-10, which indicates lossless or near-lossless quality. The video frame is losslessly compressed using FFmpeg to obtain first compressed frame data.
[0147] Step S475: If the stability score data of the first video frame is less than the scene complexity data of the first video frame, lossy compression encoding is performed on the corresponding real-scene video frame to obtain first compressed frame data;
[0148] Specifically, if the stability score data of the first video frame is less than the scene complexity data of the first video frame, lossy compression can be selected for the corresponding real-scene video frame to reduce the amount of data. In FFmpeg, lossy compression can be achieved by setting a lower CRF value, such as CRF 23-28. FFmpeg is used to perform lossy compression on the video frame to obtain the first compressed frame data.
[0149] Step S476: when the sequence of video frames to be compressed is traversed, a number of compressed frame data are obtained, and the several compressed frame data are time-sequentially merged to obtain third compressed video data.
[0150] Specifically, once all the video frames to be compressed are compressed according to the rules of S474 and S475, whether lossless or lossy, video editing software, such as Adobe Premiere Pro, can be used to merge these compressed frame data. First, import all compressed video frame sequences into the timeline of Premiere Pro. Then, arrange these frames in the correct timing according to their order in the original video. Use the automatic sequence function of Premiere Pro to automatically create a new video sequence based on the in and out points of the frames. Finally, output the entire sequence as a coherent video file, use the H.264 encoder in the output module, set the appropriate bit rate control and target bit rate, and finally obtain the third compressed video data.
[0151] The present invention accurately captures the motion characteristics between video frames through multi-scale analysis, providing a detailed data basis for motion stability assessment. Through the effective assessment of local motion stability, a stability score is further generated for each real-scene video frame. These scores are intelligently combined with scene complexity data, so that the present invention can intelligently decide the compression method of the video frame. Whether it is lossless or lossy compression, it is intended to adapt to changes in video content and optimize the compression strategy. The compression strategy selected through adaptive selection automatically determines the most suitable encoding method based on the comparison results of stability and complexity to ensure video quality and improve compression efficiency. After processing all video frames, the present invention ensures the continuity and playback smoothness of video data by time-sequential integration of compressed frame data. The overall compression strategy achieves an optimized balance between compression efficiency and video quality, while significantly improving the intelligence level of video compression, reducing manual intervention, and increasing the degree of automation.
[0152] Preferably, step S5 comprises the following steps:
[0153] Step S51: if the video type data to be compressed is live video type data, the initial resolution of the video data to be compressed is identified to obtain the video initial resolution data; if the video initial resolution data is less than or equal to the preset second resolution parameter, the video data to be compressed is directly losslessly compressed to obtain the first compressed video data; otherwise, the next step is executed;
[0154] Specifically, if the video type data to be compressed is live video type data, video processing software, such as MediaInfo, can be used to identify the initial resolution of the video to be compressed. MediaInfo provides a user interface through which the user can view the detailed properties of the video, including its resolution. After obtaining the initial resolution data of the video, it is compared with the preset second resolution parameter. If the initial resolution data of the video meets the condition of lossless compression, that is, it is less than or equal to the preset second resolution parameter, then a lossless compression method is selected to compress the video data to be compressed to obtain the first compressed video data, otherwise, the next step is executed.
[0155] Step S52: performing live subject object recognition and segmentation on each live video frame in the to-be-compressed video frame sequence to obtain a video subject object mask sequence, wherein the video subject object mask sequence includes a plurality of video subject object masks;
[0156] Specifically, advanced video processing software such as Adobe Premiere Pro can be used in combination with machine learning tools such as TensorFlow's object detection model to identify and segment the main objects in each live video frame in the sequence of video frames to be compressed. In Premiere Pro, its built-in tracking function is used to track moving objects in the video. Then, the tracking data of these objects is used as input to use TensorFlow's object detection model to identify and segment the main objects in the video. After segmentation, a sequence of video main object masks is generated, and each mask corresponds to a main object in the video.
[0157] Step S53: performing live static scene segmentation on the compressed video frame sequence to obtain a live video static scene mask;
[0158] Specifically, the video frame sequence to be compressed can be imported into the OpenCV environment. Then, a background subtraction algorithm, such as the Gaussian Mixture Model (GMM), is applied to distinguish the static background and dynamic foreground in the video. In this way, the areas in the video that have not changed for a long time can be identified and these static scenes can be segmented out. The static scene mask of the live video is obtained by segmentation.
[0159] Step S54: compressing and encoding the video main object mask sequence using a preset first resolution parameter to obtain video main object compressed data, and compressing and encoding the live video static scene mask using a preset second resolution parameter to obtain live video static scene compressed data, wherein the preset first resolution parameter is greater than the preset second resolution parameter;
[0160] Specifically, the mask can be applied using the -vf parameter of FFmpeg according to the video subject object mask sequence, and the compression quality can be set in combination with the -crf parameter and the preset first resolution parameter. For example, if the preset first resolution parameter is 1080p, the command ffmpeg -i input_video.mp4 -vf "mask=mask_sequence.png"-c:v libx264 -crf 20 output_subject_1080p.mp4 is used to compress and encode the video subject object mask sequence to obtain the video subject object compressed data. Then, for the static scene mask, the preset second resolution parameter, such as 720p, is used, and the CRF value is adjusted accordingly to adapt to the lower resolution, and the command ffmpeg -i input_video.mp4 -vf "mask=static_scene_mask.png" -c:v libx264 -crf 25 output_static_scene_720p.mp4 is used for compression encoding to obtain the live video static scene compressed data.
[0161] Step S55: Perform spatiotemporal fusion encoding on the video main object compressed data and the live video static scene compressed data to obtain fourth compressed video data.
[0162] Specifically, the video editing software Adobe Premiere Pro can be used to fuse the video subject object compressed data with the live video static scene compressed data. First, the video subject object compressed data and the live video static scene compressed data are imported into the Premiere Pro project. Then, according to their temporal and spatial relationship in the original live video, the video subject object compressed data is placed in the foreground layer of the video, and the live video static scene compressed data is placed in the background layer. The blending mode and masking tools of Premiere Pro are used to ensure that the two video layers are correctly fused. Finally, the entire fused video sequence is output as the fourth compressed video data.
[0163] The present invention realizes the rapid judgment and application of live video compression strategies through intelligent identification of initial resolution, especially directly adopting lossless compression in low-resolution conditions, effectively saving processing time and computing resources. The precise identification and segmentation of live main objects ensures the priority retention of key video content, while the effective segmentation of static scenes provides customized compression strategies for dynamic subjects and backgrounds. By differentially applying resolution parameters, the main video objects and static scenes are compressed and encoded separately, which realizes the refined processing of visual data and improves the compression efficiency and quality. The spatiotemporal fusion coding technology further ensures the coherence and consistency of the compressed video data. The compression strategy of the present invention not only improves the compression efficiency, but also ensures the viewing quality and fluency of live video through differential compression and fusion coding.
[0164] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.
[0165] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.
Claims
1. A video intelligent compression method based on image analysis, characterized in that: The following steps are involved: Step S1: Obtain video data to be compressed; Performing spatiotemporal frame processing on the video data to be compressed to obtain a sequence of video frames to be compressed; performing video type identification on the video data to be compressed based on the sequence of video frames to be compressed to obtain video type data to be compressed, wherein the video type data to be compressed is any one of short video type data, animation film type data, real-life film type data and live video type data; Step S2: If the type of video data to be compressed is short video type data, multimodal topic recognition is performed on the video data to be compressed based on the video frame sequence to be compressed to obtain short video topic distribution data; semantic-aware adaptive frame rate control compression is performed on the video data to be compressed according to the short video topic distribution data to obtain first compressed video data; Step S3: If the video type data to be compressed is animation film type data, hierarchical scene structure analysis is performed on the video data to be compressed based on the video frame sequence to be compressed to obtain an animation scene structure library; vectorized compression encoding based on semantic relations is performed on the video data to be compressed according to the animation scene structure library to obtain second compressed video data; Step S4: if the video type data to be compressed is real-life movie type data, then based on the video frame sequence to be compressed, the scene complexity evaluation is performed on the video data to be compressed to obtain scene complexity distribution data; according to the scene complexity distribution data, the video data to be compressed is adaptively lossy or lossless hybrid compressed to obtain third compressed video data; Step S5: If the video type data to be compressed is live video type data, then based on the video frame sequence to be compressed, real-time subject object segmentation and tracking are performed on the video data to be compressed to obtain a video subject object mask sequence; regional bit rate allocation compression is performed on the video data to be compressed according to the video subject object mask sequence to obtain fourth compressed video data; Step S2 includes the following steps: Step S21: if the type of video data to be compressed is short video type data, the initial resolution of the video data to be compressed is identified to obtain the initial resolution data of the video; if the initial resolution data of the video is less than or equal to the preset second resolution parameter, the video data to be compressed is directly losslessly compressed to obtain the first compressed video data; otherwise, the next step is executed; Step S22: extracting context from the video data to be compressed to obtain context data of the video to be compressed, and performing semantic intent recognition on the context data of the video to be compressed to obtain video semantic intent data; Step S23: grouping the video frame sequences to be compressed at equal intervals according to a preset frame number threshold to obtain a plurality of groups of fixed video frame sequences; Step S24: performing video expression theme recognition on each group of fixed video frame sequences to obtain a number of video expression theme data, wherein each video expression theme data corresponds to a group of fixed video frame sequences; Step S25: removing irrelevant topics from a plurality of video expression topic data according to the video semantic intent data to obtain a plurality of relevant video expression topic data; Step S26: performing spectral clustering and topic fusion on a number of related video expression topic data to obtain short video topic distribution data, wherein the short video topic distribution data includes a number of short video segment topic data; Step S27: performing topic relevance aggregation on a plurality of groups of fixed video frame sequences according to the short video topic distribution data to obtain a plurality of topic-aggregated video frame sequences; Step S28: performing semantic-aware adaptive frame rate control on the to-be-compressed video data based on a plurality of topic-aggregated video frame sequences according to the short video topic distribution data to obtain first compressed video data; Step S28 includes the following steps: Step S281: extracting paragraph themes from the short video theme distribution data to obtain first short video paragraph theme data, and extracting corresponding video frame sequences from the first short video paragraph theme data based on a plurality of theme-aggregated video frame sequences to obtain a first theme-aggregated video frame sequence; Step S282: traverse each candidate video frame in the first topic-aggregated video frame sequence, and perform topic-frame correlation calculation on each candidate video frame according to the topic data of the first short video segment to obtain a plurality of frame topic correlation score data; Step S283: if the frame subject relevance score data is greater than or equal to a preset relevance threshold, the candidate video frame corresponding to the frame subject relevance score data is recorded as a key frame, that is, a key video frame sequence to be compressed is obtained; if the frame subject relevance score data is less than the preset relevance threshold, the candidate video frame corresponding to the frame subject relevance score data is recorded as a normal frame, that is, a normal video frame sequence to be compressed is obtained; Step S284: generating a region importance map for the corresponding key video frames based on the key video frame sequence to be compressed according to the frame subject relevance score data to obtain a key frame region importance map; Step S285: performing key-common region segmentation on the corresponding key video frames according to the key frame region importance map based on the key video frame sequence to be compressed, to obtain a key-common region segmented video frame sequence, wherein the key-common region segmented video frame sequence includes a plurality of key-common region segmented video frames; Step S286: compressing and encoding the key area in each key-ordinary area segmented video frame in the key-ordinary area segmented video frame sequence using a preset first resolution parameter, and compressing and encoding the ordinary area in each key-ordinary area segmented video frame in the key-ordinary area segmented video frame sequence and the ordinary video frame sequence to be compressed using a preset second resolution parameter, so as to obtain a first differential compressed video frame sequence, wherein the preset first resolution parameter is greater than the preset second resolution parameter; Step S287: traverse the short video topic distribution data, execute steps S281 to S286 for each short video segment topic data, that is, obtain a plurality of differentially compressed video frame sequences, perform time-series integration on the plurality of differentially compressed video frame sequences, and obtain first compressed video data; Step S3 includes the following steps: Step S31: if the type of video data to be compressed is animation film type data, semantic segmentation and scene graph structure extraction are performed on each animation video frame in the video frame sequence to be compressed to obtain a video scene graph structure library, wherein the video scene graph structure library contains a plurality of scene graph structure objects; Step S32: modularly encode and annotate each scene graph structure object in the video scene graph structure library to obtain a modular video scene graph structure library; Step S33: performing color clustering and encoding on the video frame sequence to be compressed to obtain a color encoding library of the video to be compressed; Step S34: color mapping is performed on each modular scene graph structure object in the modular video scene graph structure library based on the video frame sequence to be compressed and the video color coding library to be compressed, and a structure-color association table is generated to obtain a scene structure object color association library, wherein the scene structure object color association library includes a plurality of structure object-color association tables, and each structure object-color association table stores an association relationship between a minimum structure block number of a scene graph structure object and a corresponding color coding; Step S35: performing structure-color mapping association between the modular video scene graph structure library and the scene structure object color association library to obtain structure object-color table mapping data; Step S36: removing redundant tables from the scene structure object color association library to obtain a simplified scene structure object color association library; Step S37: deduplicating the structure objects of the modular video scene graph structure library according to the streamlined scene structure object color association library and the structure object-color table mapping data to obtain a non-redundant video scene graph structure library, and assigning and recording an object ID to each scene structure object in the non-redundant video scene graph structure library to obtain a scene graph structure object index library; Step S38: performing vectorized compression encoding based on semantic relationship on the video data to be compressed based on the video frame sequence to be compressed, the simplified scene structure object color association library and the scene graph structure object index library according to the structure object-color table mapping data to obtain second compressed video data; Step S38 includes the following steps: Step S381: performing role instance segmentation on each animation video frame in the video frame sequence to be compressed, and performing role outline extraction and color mapping to obtain a video role instance gallery to be compressed, wherein the video role instance gallery to be compressed contains a plurality of video role instance images to be compressed; Step S382: De-duplication of role instances in the to-be-compressed video role instance library, and encoding of role identities, to obtain a to-be-compressed video role instance library-identity library; Step S383: performing object identification projection on each animation video frame in the video frame sequence to be compressed based on the video role instance graph-identity library and the scene graph structure object index library to obtain an object-annotated video frame sequence, wherein the object-annotated video frame sequence includes a plurality of object-annotated video frames, and all structural objects of each object-annotated video frame are annotated with corresponding identification codes; Step S384: performing semantic topological structure encoding on each object-annotated video frame in the object-annotated video frame sequence to obtain a plurality of video frame identification code topological structure data; Step S385: Based on the video frame sequence to be compressed, the video role instance graph-identity library to be compressed, the scene graph structure object index library and the structure object-color table mapping data, the corresponding object-labeled video frame is semantically vectorized and compressed according to the video frame identification code topological structure data to obtain a first semantic vector compressed frame. When the video frame sequence to be compressed is traversed, a plurality of semantic vector compressed frames are obtained, and the plurality of semantic vector compressed frames are time-sequentially integrated to obtain a second compressed video data; Step S4 includes the following steps: Step S41: if the type of video data to be compressed is real-scene movie type data, then each real-scene video frame in the sequence of video frames to be compressed is subjected to region segmentation processing, and region recognition and identification are performed to obtain a plurality of groups of video frame region segmentation result data, wherein each group of video frame region segmentation result data represents the segmentation of a real-scene video frame, and each group of video frame region segmentation result data contains a plurality of independent regions; Step S42: performing local texture complexity evaluation on each independent region in a set of video frame region segmentation result data to obtain a plurality of regional texture complexity indicators; Step S43: performing color richness analysis on each independent region in a set of video frame region segmentation result data to obtain a plurality of regional color richness indicators; Step S44: performing object diversity measurement on each independent region in a set of video frame region segmentation result data to obtain a plurality of regional object diversity indicators; Step S45: performing a video frame scene complexity evaluation on the real-scene video frame corresponding to the corresponding group of video frame region segmentation result data according to a plurality of regional texture complexity indices, a plurality of regional color richness indices and a plurality of regional object diversity indices to obtain first video frame scene complexity data; Step S46: traverse several groups of video frame region segmentation result data, execute steps S42 to S45 for each group of video frame region segmentation result data, that is, obtain several video frame scene complexity data, and perform time series integration on several video frame scene complexity data to obtain scene complexity distribution data; Step S47: performing adaptive lossy or lossless hybrid compression on the video data to be compressed based on the video frame sequence to be compressed according to the scene complexity distribution data to obtain third compressed video data; Step S47 includes the following steps: Step S471: performing multi-scale motion vector analysis on the video frame sequence to be compressed to obtain a video frame inter-frame motion feature atlas; Step S472: performing local motion stability evaluation on each real-scene video frame in the to-be-compressed video frame sequence based on the video frame inter-motion feature atlas to obtain a video frame stability score sequence, wherein the video frame stability score sequence includes a plurality of video frame stability scores; Step S473: extracting corresponding index data from the first real-scene video frame in the to-be-compressed video frame sequence based on the video frame stability score sequence and the scene complexity distribution data, to obtain first video frame stability score data and first video frame scene complexity data; Step S474: if the stability score data of the first video frame is greater than or equal to the scene complexity data of the first video frame, losslessly compress and encode the corresponding real-scene video frame to obtain first compressed frame data; Step S475: If the stability score data of the first video frame is less than the scene complexity data of the first video frame, lossy compression encoding is performed on the corresponding real-scene video frame to obtain first compressed frame data; Step S476: after traversing the sequence of video frames to be compressed, a plurality of compressed frame data are obtained, and the plurality of compressed frame data are time-sequentially merged to obtain third compressed video data; Step S5 includes the following steps: Step S51: if the video type data to be compressed is live video type data, the initial resolution of the video data to be compressed is identified to obtain the video initial resolution data; if the video initial resolution data is less than or equal to the preset second resolution parameter, the video data to be compressed is directly losslessly compressed to obtain the first compressed video data; otherwise, the next step is executed; Step S52: performing live subject object recognition and segmentation on each live video frame in the to-be-compressed video frame sequence to obtain a video subject object mask sequence, wherein the video subject object mask sequence includes a plurality of video subject object masks; Step S53: performing live static scene segmentation on the compressed video frame sequence to obtain a live video static scene mask; Step S54: compressing and encoding the video main object mask sequence using a preset first resolution parameter to obtain video main object compressed data, and compressing and encoding the live video static scene mask using a preset second resolution parameter to obtain live video static scene compressed data, wherein the preset first resolution parameter is greater than the preset second resolution parameter; Step S55: Perform spatiotemporal fusion encoding on the video main object compressed data and the live video static scene compressed data to obtain fourth compressed video data.
2. The video intelligent compression method based on image analysis according to claim 1 is characterized in that: Step S1 includes the following steps: Step S11: obtaining video data to be compressed; Step S12: performing multimodal information separation on the video data to be compressed to obtain video audio data to be compressed and pure visual video data to be compressed; Step S13: identifying the total duration of the video for the compressed pure visual video data to obtain the total duration data of the video; Step S14: extracting time-series frames from the pure visual video data to be compressed to obtain a video frame sequence to be compressed; Step S15: performing emotional rhythm analysis on the compressed video and audio data to obtain video and audio emotional feature data; Step S16: Based on the video frame sequence to be compressed, the video type of the video data to be compressed is identified according to the total video duration data and the video and audio emotional feature data to obtain the video type data to be compressed, wherein the video type data to be compressed is any one of short video type data, animation film type data, real-life film type data and live video type data.
3. The video intelligent compression method based on image analysis according to claim 2 is characterized in that: Step S16 includes the following steps: Step S161: if the total video duration data is greater than a preset video length threshold, then the video frame sequence to be compressed is segmented into time series according to the preset duration threshold to obtain a plurality of groups of video frame subsequences to be compressed; Step S162: capturing the long-term dependency relationship of the video content of each group of video frame subsequences to be compressed, and obtaining semantic feature data of a plurality of video segments; Step S163: performing hierarchical self-attention fusion on the semantic feature data of several video segments to obtain global video semantic feature data; Step S164: if the total video duration data is less than or equal to a preset video length threshold, directly capturing the long-term dependency relationship of the video content of the compressed video frame sequence to obtain global video semantic feature data; Step S165: performing temporal emotional dynamic modeling on the global video semantic feature data and the video audio emotional feature data to obtain audio emotional semantic feature data; Step S166: performing interactive feature enhancement on the video data to be compressed according to the global video semantic feature data and the audio emotional semantic feature data to obtain cross-modal feature data of the video to be compressed; Step S167: Use a preset video type classifier to classify the cross-modal feature data of the video to be compressed into video type data to obtain the video type data to be compressed, wherein the video type data to be compressed is any one of short video type data, animation film type data, real-life film type data and live video type data.
