Video stream acquisition method based on deep learning

Abstract

The invention is suitable for the technical field of video acquisition, and particularly relates to a video stream acquisition method based on deep learning, and the method comprises the steps: carrying out the video acquisition through a self-adaptive strategy, and obtaining real-time collected video data; performing parameter adjustment through edge calculation to obtain preprocessed video data; performing key frame selection on the preprocessed video data to obtain key frame data; performing inter-frame difference compression on the key frame data through the optimized variational auto-encoder model, and outputting video frame compression data; and when the original video needs to be recovered, analyzing the video compression data, reconstructing through the pre-trained neural network model to obtain a reconstructed video sequence, and generating a complete video according to the reconstructed video sequence. According to the method, the non-key frame is directly generated in the decoding process by using the deep learning model, so that the compression ratio can be remarkably improved, and the video playing continuity and the image quality, especially the performance in high-dynamic and complex scenes, can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video acquisition, and in particular to a video stream acquisition method based on deep learning. Background Art

[0002] In the field of video compression and transmission technology, with the popularization of high-definition video content and the promotion of ultra-high-definition video formats such as 4K and 8K, the requirements for video compression technology are increasing. The storage and transmission of video content requires a large amount of data bandwidth, so how to reduce the volume of data while ensuring video quality has become a core issue. Currently, video compression technology mainly relies on various standardized codecs (Codecs), such as H.264, HEVC (H.265), VP9, ​​and AV1. These technologies usually use inter-frame and intra-frame compression algorithms to reduce the size of video data.

[0003] Intra-frame compression technology achieves compression by reducing redundant information within a single frame, mainly utilizing the spatial correlation within the image. Inter-frame compression is more complex, relying on the temporal correlation between video frames, and identifying and utilizing these correlations through prediction and motion estimation techniques. In inter-frame compression, key frames (I frames) are used as reference frames, and the entire frame is encoded, while other frames (B frames and P frames) are encoded by pre-setting key frames and encoded frames. Although this method can effectively reduce the amount of data, it still presents challenges in dynamic scenes or scenes with extremely high video quality requirements. Summary of the invention

[0004] The purpose of the present invention is to provide a video stream acquisition method based on deep learning, aiming to solve the problem that the inter-frame compression method adopted in the prior art cannot cope with dynamic scenes and high video quality.

[0005] The present invention is implemented as follows: a video stream acquisition method based on deep learning, the method comprising:

[0006] Video collection is performed through adaptive strategies to obtain real-time collected video data;

[0007] Through edge computing, parameters of real-time collected video data are adjusted to obtain pre-processed video data;

[0008] Performing key frame selection on the preprocessed video data to obtain key frame data;

[0009] The key frame data is compressed between frames through the optimized variational autoencoder model, and the compressed video frame data is output;

[0010] When the original video needs to be restored, the video compression data is parsed and reconstructed through a pre-trained neural network model to obtain a reconstructed video sequence, and a complete video is generated based on the reconstructed video sequence.

[0011] Preferably, in the step of performing video capture through an adaptive strategy to obtain real-time video data, when performing video capture, the video content is detected in real time, and the video capture parameters are adjusted according to the video content, the parameters including at least frame rate and resolution. During the video capture process, the degree of motion and lighting conditions in the picture are captured, the video frames are analyzed through an image processing algorithm, the frame differences and brightness histogram are calculated, the video parameters are adjusted according to preset thresholds and a decision model, and when the lighting changes exceed the preset value range, the exposure time and sensitivity are adjusted, and the white balance parameters are optimized.

[0012] Preferably, the step of adjusting parameters of real-time collected video data through edge computing to obtain pre-processed video data includes: retrieving real-time collected video data, performing data processing through an embedded AI algorithm, the data processing process including motion detection, illumination change adjustment and scene recognition, identifying moving targets in the picture through an optical flow algorithm and a frame difference method, performing picture equalization and gamma correction through a histogram, performing scene classification through a convolutional neural network, and adjusting parameters of real-time collected video data according to the results of motion detection, illumination change adjustment and scene recognition to obtain pre-processed video data.

[0013] Preferably, the step of selecting key frames for the preprocessed video data to obtain key frame data includes: determining the scene switching point by using a scene change detection algorithm and a structural similarity index, calculating a color histogram and a structural similarity index SSIM for each frame of the video, and automatically identifying the key frames according to the calculation results using a preset parameter model;

[0014] The core formula for keyframe selection is expressed by the following formula:

[0015]

[0016] Among them, "arg max" is the abbreviation of "argument of the maximum", which means "the independent variable that makes the function reach the maximum value", K is the selected key frame, S is the set of candidate frames, ΔH(i) is the change in the color histogram of the i-th frame, SSIM(i) is the structural similarity index with the previous frame, w1 and w2 are weight parameters used to adjust the influence of color histogram change and SSIM in key frame selection.

[0017] Preferably, key frame selection is performed on the preprocessed video data, and the steps of obtaining the key frame data include video preprocessing, feature extraction, color histogram analysis, structural similarity analysis, generating a key frame candidate set and key frame selection. The video preprocessing step includes reading the preprocessed video data, inputting an image frame sequence in the preprocessed video data, and standardizing each frame of the image, including resizing and normalizing pixel values. The feature extraction step includes importing the standardized image into a pretrained convolutional neural network, extracting the feature map of the last convolutional layer in the network, obtaining a high-dimensional feature vector, and using principal component analysis to reduce the dimension of the feature vector to 128 dimensions.

[0018] Preferably, the step of performing color histogram analysis includes converting the RGB of each frame of image to HSV color space, calculating the color histogram of HSV space, using 18 bins for H channel, 3 bins for S channel and V channel respectively, and obtaining a 162-dimensional histogram vector.

[0019] Use Bhattacharyya distance to calculate the histogram difference between adjacent frames:

[0020]

[0021] Where i is the index of the current frame, which indicates the specific frame being evaluated, j is the iteration index during the summation process, which is used to traverse each bin of the histogram, n is the total number of bins in the histogram, and p is j represents the value of the jth bin in the histogram of the i-th frame, q j Represents the value of the jth bin in the histogram of the i-1th frame;

[0022] The calculation formula of SSIM index is:

[0023] SSIM(i)=[l(i,i-1)] α ·[c(i,i-1)] β ·[s(i,i-1)] γ ;

[0024] Where l, c and s represent brightness, contrast and structural similarity respectively, and α, β and γ are weight parameters.

[0025] Preferably, the step of generating a key frame candidate set includes setting a color histogram difference threshold τ H and SSIM threshold τ S , if any of the following conditions is met, the current frame is added to the candidate set S:

[0026] a)ΔH(i)>τ H ;

[0027] b) SSIM(i)<τS ;

[0028] Wherein, the character i represents the index of the video frame currently being processed.

[0029] Preferably, for each frame in the candidate set S, the key frame score is calculated:

[0030] Score(i)=w1·ΔH(i)+w2·(1-SSIM(i));

[0031] Select the frame with the highest score as the keyframe:

[0032]

[0033] Preferably, the step of performing inter-frame difference compression on key frame data through an optimized variational autoencoder model and outputting compressed video frame data includes importing a video sequence, preprocessing the video sequence, the preprocessing step includes frame alignment and difference calculation, obtaining an inter-frame difference image, and using the inter-frame difference image as the input of the VAE model. The VAE model consists of an encoder and a decoder. The encoder encodes the difference information into distribution parameters of a latent space, and the decoder reconstructs the difference information from the latent space. The loss function of the optimized VAE model is:

[0034]

[0035] Among them, θ is the parameter set of the decoder, φ is the parameter set of the encoder, x represents the input data, that is, the difference image between frames, z represents the latent variable, generated by the encoder, and represents the representation of the input data in the latent space, q φ (z|x) represents the posterior distribution of the encoder output, which is a conditional probability distribution. Given the distribution of z under input x, the parameters are controlled by φ, p θ (x|z) represents the conditional probability distribution of the decoder output, the probability of reconstructing x given the latent variable z, and the parameters are controlled by θ. p(x) represents the prior distribution of the latent variable z, and β is a hyperparameter used to balance the weight between reconstruction error and KL divergence.

[0036] Preferably, when the original video needs to be restored, the video compression data is parsed and reconstructed through a pre-trained neural network model to obtain a reconstructed video sequence. In the step of generating a complete video according to the reconstructed video sequence, the received compressed data stream is parsed to extract the key frame information and the compressed representation of the difference between frames. The key frames are restored using a standard image decoding algorithm. For each non-key frame, a pre-trained deep neural network model is used to reconstruct the decoded key frames and the corresponding compressed difference information. The deep neural network model adopts an improved U-Net architecture, which includes an encoder-decoder structure and jump connections.

[0037] The deep learning-based video stream acquisition method provided by the present invention utilizes a deep learning model to directly generate non-key frames during the decoding process, which can not only significantly improve the compression ratio, but also improve the continuity and image quality of video playback, especially in high dynamic and complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a deep learning-based video stream acquisition method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0041] Figure 1 A flowchart of a video stream acquisition method based on deep learning provided by an embodiment of the present invention, the method comprising:

[0042] Video collection is performed through an adaptive strategy to obtain real-time collected video data.

[0043] In this step, the adaptive strategy includes adaptive sampling, adaptively adjusting acquisition parameters (such as frame rate and resolution) according to dynamic changes in video content (such as motion and illumination changes, etc.) to ensure the flexibility and adaptability of video acquisition. The adaptive acquisition strategy adaptively adjusts acquisition parameters (such as frame rate and resolution) according to dynamic changes in video content (such as motion and illumination changes, etc.) to ensure the flexibility and adaptability of video acquisition. The strategy can monitor the changes in video content in real time, and dynamically adjust the acquisition parameters according to the detection results to optimize video quality and data volume. The system will adaptively adjust the video acquisition parameters according to the monitored dynamic changes. For example, in a scene with intense motion, the system may increase the frame rate to capture more details, and in a scene with large illumination changes, the system may adjust the exposure and white balance parameters to ensure video quality. The adaptive acquisition strategy provided by the present invention aims to solve the problems of unstable video quality and data redundancy caused by fixed parameters of traditional video acquisition methods. The strategy monitors the dynamic changes of video content in real time, such as the degree of motion and illumination conditions, and dynamically adjusts the acquisition parameters accordingly to optimize video quality and data volume. In the specific implementation process, the system first uses image processing algorithms to analyze video frames and calculate features such as inter-frame differences and brightness histograms. Then, based on preset thresholds and decision models, the system adaptively adjusts video acquisition parameters. For example, when a scene with intense motion is detected, the system will increase the frame rate to capture more details, while possibly reducing the resolution to balance the amount of data; in scenes with large changes in lighting, the system will adjust the exposure time and ISO sensitivity in real time, and optimize the white balance parameters to ensure balanced image brightness and accurate colors. In addition, the system will dynamically adjust the encoding bit rate according to the complexity of the scene to minimize the amount of data while ensuring video quality.

[0044] This adaptive strategy not only improves the flexibility and adaptability of video acquisition, but also maintains stable video quality in various complex environments, while effectively reducing storage and transmission costs. Through intelligent parameter adjustment, this technical means significantly improves the overall performance and efficiency of the video acquisition system.

[0045] The adaptive acquisition strategy can be implemented by following the steps below:

[0046] System initialization: When the video capture device is started, the system loads the preset parameter configuration and decision model. These preset parameters include the default frame rate, resolution, exposure time, ISO sensitivity, white balance setting, encoding bit rate, etc.

[0047] Real-time scene analysis:

[0048] The system uses image processing algorithms to analyze continuous video frames. The specific steps include:

[0049] a) Calculate the difference between frames: Use the optical flow algorithm or block matching algorithm to calculate the motion vector between adjacent frames and quantify the degree of motion of the scene.

[0050] b) Brightness analysis: Calculate the brightness histogram of each frame to evaluate the overall brightness level and distribution.

[0051] c) Color analysis: Analyze the distribution of RGB or HSV color space and evaluate the color balance.

[0052] d) Texture complexity: Use edge detection or frequency domain analysis methods to assess the detail richness of the scene.

[0053] Parameter adaptive adjustment:

[0054] Based on the scenario analysis results, the system dynamically adjusts the following parameters:

[0055] a) Frame rate adjustment: When intense motion is detected (for example, the difference between frames exceeds a preset threshold), the system increases the frame rate from the standard 30fps to 60fps or higher to capture more motion details.

[0056] b) Resolution adjustment: While ensuring video clarity, the system may slightly reduce the resolution in high-motion scenes (such as from 4K to 1080p) to balance the data volume.

[0057] c) Exposure parameter adjustment: Based on the brightness histogram, the system adjusts the exposure time (between 1 / 30 second and 1 / 1000 second) and ISO sensitivity (between 100-3200) in real time to adapt to different lighting conditions.

[0058] d) White balance adjustment: Based on the color analysis results, the system dynamically adjusts the white balance parameters to ensure accurate color reproduction.

[0059] e) Coding bit rate adjustment: Based on the scene complexity and the degree of motion, the system dynamically adjusts the coding bit rate between 2Mbps and 20Mbps.

[0060] Feedback loops and optimization:

[0061] The system continuously monitors the quality of the adjusted video, including objective indicators such as PSNR (peak signal-to-noise ratio) and SSIM (structural similarity). If the adjusted quality is not satisfactory, the system will fine-tune the parameters of the decision model to achieve continuous optimization.

[0062] Through edge computing, parameters of real-time collected video data are adjusted to obtain preprocessed video data.

[0063] In this step, edge computing is implemented through an edge computing module, which includes a high-performance processor and storage unit and is integrated on the video acquisition device, so that the acquisition device can perform preliminary processing of video data locally. Not only does it reduce the demand for data transmission, but it also significantly reduces the occupancy of network bandwidth, thereby reducing transmission delays. By performing preliminary processing on the acquisition device, the system can compress, filter and optimize the video stream before the data is transmitted to the central server, thereby reducing the amount of data transmitted. The edge computing module can process dynamic changes in the video stream in real time, including motion detection, illumination change adjustment, and scene recognition. By performing these processes locally, the system can quickly respond to changes in video content and ensure the stability and consistency of video quality. For example, when a scene with intense motion is detected, the edge computing module can immediately adjust the frame rate and resolution to capture more details; in scenes with large illumination changes, the edge computing module can adjust the exposure and white balance parameters in real time to ensure the clarity and color accuracy of the video.

[0064] The edge computing module is designed to solve the high latency, high bandwidth occupancy and data security problems in traditional cloud processing architectures. The module integrates high-performance processors and storage units on the video acquisition device to achieve local preliminary processing of video data. Its working principle is to disperse complex computing tasks to the edge of the network, that is, the acquisition device end, thereby reducing the burden on the central server. In the specific implementation process, the edge computing module first receives the original video stream and then uses the embedded AI algorithm for real-time analysis and processing. This includes tasks such as motion detection, illumination change adjustment and scene recognition. For example, through the optical flow algorithm and frame difference method, the module can accurately identify moving targets in the picture; using histogram equalization and adaptive gamma correction, the video brightness and contrast can be dynamically adjusted; and convolutional neural network (CNN) is used to achieve fast scene classification. Based on these analysis results, the module automatically optimizes the acquisition parameters, such as increasing the frame rate in high-speed motion scenes, or adjusting the ISO sensitivity in low-light environments. At the same time, the edge computing module also performs preliminary video compression and encoding tasks, such as using the H.265 / HEVC encoding standard, combined with a dynamic bitrate control algorithm, to maximize the compression rate while ensuring video quality.

[0065] In addition, the module also implements data filtering functions, which can filter out key information according to preset rules, significantly reducing the amount of data that needs to be transmitted. Through these local processing, the system not only greatly reduces network bandwidth usage and transmission delay, but also improves data security, because sensitive information can be desensitized on the device side. The introduction of the edge computing module makes the entire video acquisition system more responsive, more intelligent in processing, and more efficient in transmission.

[0066] The edge computing module of the present invention can be implemented by the following steps:

[0067] Hardware configuration: A high-performance embedded processor is integrated into the video acquisition device, with a main frequency of no less than 2.5GHz and equipped with a dedicated AI acceleration chip (such as NPU). It is also equipped with at least 32GB of high-speed LPDDR4X memory and 1TB of eMMC storage. These hardware configurations ensure that the edge computing module has sufficient computing power and storage space.

[0068] Video analysis implementation:

[0069] a) Motion detection: Use the improved Lucas-Kanade optical flow algorithm combined with the frame difference method. Process at least 30 frames per second to identify the speed and direction of the moving target.

[0070] b) Lighting analysis: Adaptive Histogram Equalization (CLAHE) algorithm is used to perform brightness and contrast analysis every 2-3 frames, with a processing time of no more than 10ms / frame.

[0071] c) Scene recognition: Use lightweight CNN models such as MobileNetV3, pre-trained to support more than 50 common scene categories, and control the inference time within 20ms.

[0072] Parameter optimization process:

[0073] Based on the analysis results, the system quickly adjusts the following parameters through a preset decision tree algorithm:

[0074] a) Frame rate: dynamically adjusted within the range of 15-120fps, such as increasing to above 60fps for high-speed motion scenes.

[0075] b) Resolution: Supports dynamic switching from 720p to 4K, adjusted according to scene complexity and network conditions.

[0076] c) Exposure parameters: ISO range 100-12800, exposure time 1 / 10000 second to 1 / 15 second, adjusted in real time according to lighting conditions.

[0077] d) White balance: supports multiple modes such as automatic, sunny, cloudy, fluorescent light, etc., updated once a second.

[0078] Video encoding and compression:

[0079] The H.265 / HEVC encoding standard is used in combination with the x265 encoder. The CRF (Constant Rate Factor) mode is used, and the CRF value is dynamically adjusted between 18-28 to balance quality and compression rate. At the same time, a content-based adaptive bitrate control algorithm is implemented, with a bitrate range from 1Mbps to 20Mbps.

[0080] Data filtering and security processing:

[0081] Implement a rule-based data filtering engine that supports custom filtering rules. For example, you can set the video clip to be transmitted only when a specific target (such as a person or a vehicle) is detected. For sensitive information (such as faces and license plates), real-time fuzzy processing is used, and the processing delay is controlled within 5ms.

[0082] Communication Mechanism:

[0083] It uses the MQTT protocol to communicate with the central server, supports breakpoint resumption and data encryption (using the AES-256 encryption algorithm), sets an adaptive transmission strategy, transmits data in batches when the network conditions are good, and only transmits key information when the network is congested.

[0084] Key frame selection is performed on the preprocessed video data to obtain key frame data.

[0085] In this step, a deep learning compression algorithm is used for video processing, including video framing, key frame selection, deep learning model compression of frame differences, and reconstruction of video sequences, which greatly reduces the amount of video data while ensuring video quality.

[0086] Video framing: Decompose the input video into a series of frames, {F1, F2, …, Fn}.

[0087] Key frame selection: Select a series of key frames, denoted as {K1, K2,…, Km}, where m<<n.

[0088] Deep learning model compresses inter-frame differences: For each key frame and the subsequent non-key frame sequence, a deep learning model is used to compress the inter-frame differences.

[0089] Reconstructing video sequences: In the decoding stage, key frames are first restored, and then non-key frames are reconstructed using the deep learning model and compressed transformation parameters.

[0090] The key frame selection process identifies scene switching points through a scene change detection algorithm. This algorithm can effectively detect scene changes in the video and automatically select key frames to improve the efficiency of video compression and reconstruction.

[0091] Scene change detection algorithms mainly rely on two techniques: color histogram changes and structural similarity index (SSIM). Color histograms are constructed based on the pixel distribution of each color in an image. By comparing the histogram differences between consecutive frames, scene changes can be effectively detected. When there is a significant change in color distribution, such as switching from an indoor scene to an outdoor scene, the color histogram will show a significant difference, and the algorithm can identify this point as a scene switching point. Structural Similarity Index (SSIM) is another important technique that evaluates the similarity between two images by comparing their brightness, contrast, and structure. When the SSIM value between two frames is below a certain preset threshold, it can be considered that there is a significant structural change between the two frames, and thus it can be determined as a scene switching point.

[0092] The scene change detection algorithm first calculates the color histogram and SSIM of each frame of the video, and then analyzes the scene changes based on these data. Through the preset parameter model, the algorithm can automatically identify key frames. The selection of key frames is not only based on scene switching, but also considers the visual quality of the frame and the representativeness of the content. The core formula for key frame selection is expressed by the following formula:

[0093]

[0094] Among them, "arg max" is the abbreviation of "argument of the maximum", which means "the independent variable that makes the function reach the maximum value", K is the selected key frame, S is the set of candidate frames, ΔH(i) is the change in the color histogram of the i-th frame, SSIM(i) is the structural similarity index with the previous frame, w1 and w2 are weight parameters used to adjust the influence of color histogram change and SSIM in key frame selection.

[0095] One of the core technologies of the deep learning-based compression algorithm, video key frame selection technology, aims to solve the problems of inaccurate key frame selection, low efficiency, and inability to adapt to complex scene changes in traditional video compression. This technology uses an innovative scene change detection algorithm to achieve efficient and accurate key frame selection by intelligently identifying scene switching points. Its core working principle is based on the synergy of two key technologies: color histogram change and structural similarity index (SSIM).

[0096] First, the algorithm performs a color histogram analysis on each frame of the input video stream. A multidimensional histogram is constructed by calculating the pixel distribution of each color component in the RGB or HSV color space. Then, the algorithm compares the histogram differences between adjacent frames and quantifies the changes in color distribution using metrics such as chi-square distance or Bhattacharyya distance. When a significant histogram difference is detected, such as a sudden transition from an indoor scene to an outdoor scene, the algorithm marks this as a potential scene switching point. At the same time, the algorithm also calculates the structural similarity index (SSIM) between adjacent frames. SSIM provides a similarity measure in the range of [-1,1] by evaluating the similarity between two frames in terms of brightness, contrast, and structure. The algorithm sets a dynamic threshold τ. When SSIM(i,i-1)<τ, it is considered that there is a significant structural change between the i-th frame and the previous frame, further confirming the possibility of scene switching.

[0097] To further improve the accuracy and robustness of the selection, the algorithm also introduces an adaptive threshold mechanism and time window analysis. The adaptive threshold is dynamically adjusted according to the complexity of the video content to ensure that key frames can be accurately identified in different types of videos. The time window analysis avoids misjudgments due to instantaneous noise or small changes by considering the frame sequence within a certain time range. This technology not only significantly improves the accuracy and efficiency of key frame selection, but also greatly improves the quality and efficiency of video compression. It provides a solid foundation for subsequent inter-frame difference compression, enabling the entire video compression system to achieve higher compression ratios and lower computational complexity while ensuring visual quality.

[0098] The video key frame selection technology based on deep learning of the present invention can be implemented by the following steps:

[0099] System initialization:

[0100] Load the pre-trained deep learning model. Initialize system parameters, including color space selection (HSV space is used by default), histogram comparison method (Bhattacharyya distance is used by default), SSIM calculation parameters, and initial values ​​of weight parameters w1 and w2 (both are 0.5 by default).

[0101] Video preprocessing:

[0102] The input video stream is decoded into a continuous sequence of image frames. Each frame is standardized, including resizing (uniformly adjusted to 224x224 pixels) and normalizing pixel values ​​to meet the input requirements of the deep learning model.

[0103] Feature extraction:

[0104] Use the pre-trained deep learning model to extract features from each frame. The specific steps are as follows:

[0105] a) Input the preprocessed image frame into the convolutional neural network.

[0106] b) Extract the feature map of the last convolutional layer of the network to obtain a high-dimensional feature vector (for example, 2048 dimensions).

[0107] c) Principal component analysis (PCA) is used to reduce the dimension of the feature vector to 128 dimensions to reduce the computational complexity.

[0108] Color histogram analysis:

[0109] a) Convert each frame of image from RGB to HSV color space.

[0110] b) Calculate the color histogram of the HSV space, using 18 bins for the H channel and 3 bins for the S and V channels, to obtain a 162-dimensional histogram vector.

[0111] c) Use Bhattacharyya distance to calculate the histogram difference between adjacent frames:

[0112]

[0113] Where i is the index of the current frame, which indicates the specific frame being evaluated, j is the iteration index during the summation process, which is used to traverse each bin of the histogram, n is the total number of bins in the histogram, and p is j represents the value of the jth bin in the histogram of the i-th frame, q j Represents the value of the jth bin in the histogram of the i-1th frame;

[0114] Structural similarity analysis:

[0115] Calculate the SSIM index between adjacent frames:

[0116] SSIM(i)=[l(i,i-1)] α ·[c(i,i-1)] β ·[s(i,i-1)] γ ;

[0117] Where l, c and s represent brightness, contrast and structural similarity respectively, and α, β and γ are weight parameters (all 1 by default).

[0118] Key frame candidate set generation:

[0119] Set the initial threshold τ H = 0.2 (color histogram difference threshold) and τ S =0.85 (SSIM threshold). If any of the following conditions is met, the current frame is added to the candidate set S:

[0120] a)ΔH(i)>τH ;

[0121] b) SSIM(i)<τ S ;

[0122] Wherein, the character i represents the index of the video frame currently being processed.

[0123] Keyframe selection:

[0124] For each frame in the candidate set S, calculate the key frame score:

[0125] Score(i)=w1·ΔH(i)+w2·(1-SSIM(i));

[0126] Select the frame with the highest score as the keyframe:

[0127]

[0128] The key frame data is compressed between frame differences through an optimized variational autoencoder model to output video frame compressed data.

[0129] In this step, a variational autoencoder is used to learn the change pattern between a key frame and subsequent frames. Through the deep learning model, the system can learn and compress the change pattern between a key frame and subsequent frames, thereby achieving efficient video compression and reconstruction.

[0130] Variational Autoencoder (VAE) is an autoencoder based on a probabilistic generative model, which converts data into distribution parameters of a latent space through an encoder and reconstructs data from this latent space through a decoder. VAE is particularly suitable for processing inter-frame differences because it can not only compress data but also generate missing frames through the continuity of the latent space, thereby optimizing the coherence of the video stream and reducing encoding errors.

[0131] The optimized VAE model formula is:

[0132]

[0133] Among them, θ is the parameter set of the decoder, φ is the parameter set of the encoder, x represents the input data, that is, the difference image between frames, z represents the latent variable, generated by the encoder, and represents the representation of the input data in the latent space, q φ (z|x) represents the posterior distribution of the encoder output, which is a conditional probability distribution. Given the distribution of z under input x, the parameters are controlled by φ, p θ(x|z) represents the conditional probability distribution of the decoder output, the probability of reconstructing x given the latent variable z, and the parameters are controlled by θ, p(x) represents the prior distribution of the latent variable z, and β is a hyperparameter used to balance the weight between the reconstruction error and the KL divergence. In this way, the difference between keyframes and non-keyframes is effectively modeled and compressed, which greatly reduces the amount of data that needs to be transmitted while ensuring the visual quality of the reconstructed video.

[0134] The video inter-frame difference compression technology aims to solve the problems of low compression efficiency, high computational complexity and large visual quality loss faced by traditional video coding methods when processing high-resolution and high-frame-rate videos. This technology is based on an optimized variational autoencoder (VAE) model and uses a deep learning method to efficiently compress the difference information between key frames and non-key frames. Its core working principle is to map the inter-frame difference to a low-dimensional latent space and achieve high-quality difference reconstruction through a probabilistic generative model. The specific implementation process is as follows: First, the system preprocesses the input video sequence, including frame alignment and difference calculation. Then, the calculated inter-frame difference image is used as the input of the VAE model. The VAE model consists of two parts: an encoder and a decoder. The encoder encodes the difference information into the distribution parameters of the latent space, and the decoder reconstructs the difference information from the latent space.

[0135] To further improve compression efficiency, this technology introduces an adaptive quantization mechanism that dynamically adjusts quantization parameters according to the complexity of inter-frame differences, maximizing compression ratio while ensuring visual quality. In addition, a residual learning strategy is adopted to enable the model to focus on learning high-frequency detail information, thereby improving reconstruction quality. During the training process, a curriculum learning strategy is adopted to gradually increase the complexity of training samples, so that the model can better adapt to different types of video content. In order to deal with long-term dependency problems, the model also integrates an attention mechanism to capture the correlation between distant frames. In the inference stage, the system first uses the encoder to compress the inter-frame differences into latent variables, and only needs to transmit these low-dimensional representations and a small amount of key frame information. The receiver uses the decoder and key frame information to reconstruct the complete video sequence. In order to adapt to different network bandwidths and device performance, this technology also supports scalable coding, allowing flexible decoding at different bit rates. Through this deep learning-based inter-frame difference compression method, this technology significantly improves video compression efficiency, greatly reduces data transmission while ensuring high visual quality, and provides an innovative solution for real-time transmission and storage of high-quality video.

[0136] This embodiment describes in detail the specific implementation process of the video frame difference compression technology based on deep learning. This technology uses an optimized variational autoencoder (VAE) model to efficiently compress the difference information between key frames and non-key frames to achieve high-quality video reconstruction.

[0137] The objective to be minimized is the loss function of the variational autoencoder (VAE), which includes the reconstruction loss and the KL divergence term. The goal of VAE is to maximize the marginal likelihood p(x) of the data x.

[0138] Define a variational lower bound (ELBO, Evidence Lower Bound) to approximate the marginal likelihood p(x):

[0139]

[0140] q φ (z|x) is the approximate posterior distribution and the parameters φ are the parameters of the encoder.

[0141] p θ (x|z) is the likelihood function and the parameters θ are the parameters of the decoder.

[0142] D KL (q φ (z|x)||p(z) is the KL divergence, which measures the approximate posterior distribution q φ The difference between (z|x) and the prior distribution p(x).

[0143] Weighted β-VAE loss function:

[0144] In β-VAE, a weight parameter β is introduced to adjust the balance between the reconstruction loss and the KL divergence term:

[0145]

[0146] Here, β is a hyperparameter. When β=1, the formula degenerates into the standard VAE loss function. When β>1, the model pays more attention to the regularity of the latent space. is the reconstruction loss, which measures the model's ability to reconstruct the input data. KL (q φ (z|x)||p(x) is the divergence term, which measures the difference between the approximate posterior distribution and the prior distribution.

[0147] When the original video needs to be restored, the video compression data is parsed and reconstructed through a pre-trained neural network model to obtain a reconstructed video sequence, and a complete video is generated based on the reconstructed video sequence.

[0148] In this step, the specific steps of reconstructing the video sequence include decoding the keyframes and generating each non-keyframe using a deep learning model based on the keyframes and the stored transformation parameters. By combining the detailed image data of the keyframes and the motion vectors processed by the deep learning model, the system can accurately reconstruct the entire video sequence to ensure the continuity and image quality during video playback. The reconstruction of the video sequence aims to solve the problems of image quality degradation, motion incoherence and low computational efficiency faced by traditional video decoding methods when processing high-compression ratio videos. Based on the deep learning model, this technology achieves high-quality and efficient video sequence reconstruction by intelligently fusing keyframe information and compressed inter-frame difference data. Its core working principle includes three main steps: keyframe decoding, non-keyframe generation and sequence optimization. First, the system parses the received compressed data stream to extract the keyframe information and the compressed representation of the inter-frame difference. The keyframe is restored using a standard image decoding algorithm to ensure the high quality of the reference image. Then, for each non-keyframe, the system uses a pre-trained deep neural network model to combine the decoded keyframes and the corresponding compressed difference information for reconstruction. The neural network model adopts an improved U-Net architecture, including an encoder-decoder structure and jump connections, which can effectively capture multi-scale features and detail information. In order to improve the reconstruction quality, the model also integrates an attention mechanism to dynamically adjust the attention to different image areas. In the process of non-key frame generation, the system introduces an adaptive interpolation algorithm to dynamically adjust the interpolation strategy according to the scene complexity and motion amplitude, effectively reducing motion blur and artifacts. In addition, in order to ensure temporal coherence, the technology also adopts a temporal consistency loss function to optimize the visual continuity between adjacent frames during training. In the sequence optimization stage, the system applies motion compensation technology based on optical flow to further refine the inter-frame motion and improve the smoothness of the video. Through this intelligent reconstruction method based on deep learning, the technology significantly improves the quality and efficiency of video decoding, greatly reduces the computational complexity while ensuring high visual quality, and provides an innovative solution for real-time playback and interactive applications of high-compression ratio videos.

[0149] By intelligently fusing key frame information and compressed inter-frame difference data, high-quality and efficient video sequence reconstruction is achieved, solving the problems of image quality degradation, motion incoherence and low computational efficiency faced by traditional video decoding methods when processing high-compression ratio videos.

[0150] Data analysis:

[0151] a) Use a custom binary parser to parse the received compressed data stream.

[0152] b) Extract metadata, including video resolution, frame rate, key frame index, etc.

[0153] c) Separate key frame compressed data and non-key frame difference compressed data.

[0154] Keyframe decoding:

[0155] a) Use the standard JPEG2000 decoding algorithm to restore the key frame:

[0156] b) Applying an inverse discrete wavelet transform (DWT), performing an inverse quantization process, and applying entropy decoding (such as arithmetic decoding).

[0157] c) Post-processing the decoded key frames, including deblocking and sharpening.

[0158] Non-keyframe generation:

[0159] The deep neural network model with improved U-Net architecture is used for non-key frame reconstruction. The model structure includes encoder, decoder and attention module.

[0160] Sequence Optimization:

[0161] Optical flow-based motion compensation techniques are applied to further refine the inter-frame motion.

[0162] Final reasoning:

[0163] a) Decode key frames;

[0164] b) For each non-key frame: reconstruct the initial frame using the deep neural network model, apply an adaptive interpolation algorithm, and perform motion compensation optimization;

[0165] c) Combine the reconstructed frame sequences to generate a complete video.

[0166] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0167] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A video stream acquisition method based on deep learning, characterized in that: The method comprises: Video collection is performed through adaptive strategies to obtain real-time collected video data; Through edge computing, parameters of real-time collected video data are adjusted to obtain pre-processed video data; Performing key frame selection on the preprocessed video data to obtain key frame data; The key frame data is compressed between frames through the optimized variational autoencoder model, and the compressed video frame data is output; When the original video needs to be restored, the video compression data is parsed and reconstructed through a pre-trained neural network model to obtain a reconstructed video sequence, and a complete video is generated based on the reconstructed video sequence.

2. The video stream acquisition method based on deep learning according to claim 1, characterized in that: In the step of performing video acquisition through an adaptive strategy to obtain real-time acquisition of video data, when performing video acquisition, the video content is detected in real time, and the parameters of video acquisition are adjusted according to the video content, and the parameters include at least frame rate and resolution. During the video acquisition process, the degree of motion and lighting conditions in the picture are collected, and the video frames are analyzed by an image processing algorithm, and the frame difference and brightness histogram are calculated. The video parameters are adjusted according to preset thresholds and a decision model. When the lighting change exceeds the preset value range, the exposure time and sensitivity are adjusted, and the white balance parameters are optimized.

3. The video stream acquisition method based on deep learning according to claim 1 is characterized in that: The step of adjusting parameters of real-time collected video data through edge computing to obtain pre-processed video data includes: retrieving real-time collected video data, performing data processing through an embedded AI algorithm, wherein the data processing process includes performing motion detection, adjusting illumination changes, and scene recognition, identifying moving targets in the picture through an optical flow algorithm and a frame difference method, performing picture equalization and gamma correction through a histogram, performing scene classification through a convolutional neural network, and adjusting parameters of real-time collected video data according to the results of motion detection, illumination change adjustment, and scene recognition to obtain pre-processed video data.

4. The video stream acquisition method based on deep learning according to claim 1 is characterized in that: The steps of selecting key frames for the preprocessed video data to obtain key frame data include: determining the scene switching point by using a scene change detection algorithm and a structural similarity index, calculating a color histogram and a structural similarity index SSIM for each frame of the video, and automatically identifying the key frames according to the calculation results using a preset parameter model; The core formula for keyframe selection is expressed by the following formula: Where K is the selected key frame, S is the set of candidate frames, ΔH(i) is the color histogram change of the i-th frame, SSIM(i) is the structural similarity index with the previous frame, and w1 and w2 are weight parameters used to adjust the influence of color histogram change and SSIM in key frame selection.

5. The video stream acquisition method based on deep learning according to claim 4 is characterized in that: Key frame selection is performed on the preprocessed video data. The steps of obtaining the key frame data include video preprocessing, feature extraction, color histogram analysis, structural similarity analysis, generating key frame candidate sets and key frame selection. The video preprocessing step includes reading the preprocessed video data, inputting the image frame sequence in the preprocessed video data, and standardizing each frame of the image, including resizing and pixel value normalization. The feature extraction step includes importing the standardized image into a pretrained convolutional neural network, extracting the feature map of the last convolutional layer in the network, obtaining a high-dimensional feature vector, and using principal component analysis to reduce the dimension of the feature vector to 128 dimensions.

6. The video stream acquisition method based on deep learning according to claim 4 is characterized in that: The steps of color histogram analysis include converting the RGB of each frame image to the HSV color space, calculating the color histogram of the HSV space, using 18 bins for the H channel, and 3 bins for the S channel and the V channel, to obtain a 162-dimensional histogram vector. Use Bhattacharyya distance to calculate the histogram difference between adjacent frames: Where i is the index of the current frame, which indicates the specific frame being evaluated, j is the iteration index during the summation process, which is used to traverse each bin of the histogram, n is the total number of bins in the histogram, and p is j represents the value of the jth bin in the histogram of the i-th frame, q j Represents the value of the jth bin in the histogram of the i-1th frame; The calculation formula of SSIM index is: SSIM(i)=[l(i,i-1)] α ·[c(i,i-1)] β ·[s(i,i-1)] γ ; Where l, c and s represent brightness, contrast and structural similarity respectively, and α, β and γ are weight parameters.

7. The video stream acquisition method based on deep learning according to claim 4 is characterized in that: The step of generating a key frame candidate set includes setting a color histogram difference threshold τ H and SSIM threshold τ S , if any of the following conditions is met, the current frame is added to the candidate set S: a)ΔH(i)>τ H ; b)SSIM(u)<τ S ; Wherein, the character i represents the index of the video frame currently being processed.

8. The video stream acquisition method based on deep learning according to claim 4 is characterized in that: For each frame in the candidate set S, calculate the key frame score: Score(i)=w1·ΔH(i)+w2·(1-SSIM(i)); Select the frame with the highest score as the keyframe:

9. The video stream acquisition method based on deep learning according to claim 1, characterized in that: The step of compressing the inter-frame difference of key frame data through the optimized variational autoencoder model and outputting the compressed video frame data includes importing the video sequence, preprocessing the video sequence, the preprocessing step includes frame alignment and difference calculation, obtaining the inter-frame difference image, and taking the inter-frame difference image as the input of the VAE model. The VAE model consists of an encoder and a decoder. The encoder encodes the difference information into the distribution parameters of the latent space, and the decoder reconstructs the difference information from the latent space. The loss function of the optimized VAE model is: Among them, θ is the parameter set of the decoder, φ is the parameter set of the encoder, x represents the input data, that is, the difference image between frames, z represents the latent variable, generated by the encoder, and represents the representation of the input data in the latent space, q φ (z|x) represents the posterior distribution of the encoder output, which is a conditional probability distribution. Given the distribution of z under input x, the parameters are controlled by φ, p θ (x|z) represents the conditional probability distribution of the decoder output, the probability of reconstructing x given the latent variable z, and the parameters are controlled by θ. p(x) represents the prior distribution of the latent variable z, and β is a hyperparameter used to balance the weight between reconstruction error and KL divergence.

10. The video stream acquisition method based on deep learning according to claim 1, characterized in that: When the original video needs to be restored, the video compression data is parsed and reconstructed through a pre-trained neural network model to obtain a reconstructed video sequence. In the step of generating a complete video based on the reconstructed video sequence, the received compressed data stream is parsed to extract the key frame information and the compressed representation of the frame difference. The key frame is restored using a standard image decoding algorithm. For each non-key frame, a pre-trained deep neural network model is used to reconstruct the decoded key frame and the corresponding compressed difference information. The deep neural network model adopts an improved U-Net architecture, which includes an encoder-decoder structure and jump connections.

Images