State-aware and deep analysis based adaptive video processing push streaming method and system

Abstract

The application discloses a state-aware and deep analysis adaptive video processing push streaming method and system, wherein the method comprises the following steps: collecting and initializing video frames of an input video stream; performing multi-dimensional state-aware analysis on the initialized video frames to construct a video state feature vector; performing adaptive bilateral filtering processing on the video frames to obtain filtered video frames; inputting the filtered video frames into a pre-constructed deep semantic feature extraction network to extract high-level semantic feature mapping of the video frames into semantic attribute parameters and construct a semantic feature vector; fusing the video state feature vector and the semantic feature vector and inputting them into a pre-constructed video quality decision network to generate a picture quality enhancement parameter; performing adaptive picture quality enhancement processing on the video frames; and performing video compression coding on the video frames and sending them to a streaming media server in real time. The application can significantly improve the subjective visual quality, coding efficiency and push streaming stability while ensuring real-time performance.

Description

Technical Field

[0001] This invention relates to the field of computer graphics and multimedia processing technology, and in particular to an adaptive video processing and streaming method and system based on state awareness and depth analysis. Background Technology

[0002] Currently, online video streaming is an important component of live video broadcasting, remote conferencing, online education, smart security, and internet video playback systems. A typical online video streaming system is as follows: Figure 1 The setup typically includes a video sensor array, a master node, a high-definition video processing module, and a user terminal. The video sensor array can consist of multiple high-definition monitoring devices, network cameras, or industrial image acquisition devices, used to acquire video image information from different scenarios. The video sensor array connects to the master node via wired or wireless networks, sending the acquired video data to the master node in real time for unified processing. The master node is responsible for receiving video data, buffering and scheduling, task allocation, and data transmission control, and is responsible for coordinating data interaction between the high-definition video processing module and the streaming module. The high-definition video processing module performs high-definition enhancement processing on the input video frames and pushes the processed video stream data to the user terminal. The user terminal receives the video stream data pushed by the streaming media server and performs video decoding and real-time playback.

[0003] With the increasing demand for video content and the growing complexity of sensor networks, high-definition video processing and real-time streaming technologies have become crucial. Especially in application scenarios such as live video streaming, smart security, telemedicine, online education, industrial monitoring, and intelligent transportation, systems not only need to acquire and transmit massive amounts of video data in real time, but also need to ensure the clarity, stability, and low latency of video images in complex environments.

[0004] Existing technologies primarily focus on the following areas: In video coding and compression, the H.264 / AVC and H.265 / HEVC standards are widely adopted, with the latter significantly reducing bitrate while maintaining the same image quality. To balance coding complexity and real-time performance, hardware-accelerated encoders based on GPUs or ASICs have been developed, utilizing hardware such as GPUs and FPGAs for parallel computing to accelerate video filtering, transcoding, and other processing tasks. Furthermore, video filtering enhancement algorithms, such as deep learning-based noise reduction, super-resolution, and frame interpolation techniques, are used to improve image quality and smoothness.

[0005] However, existing technologies still have significant drawbacks: (1) Lack of video content perception ability Most current video enhancement algorithms primarily process image pixels, such as brightness adjustment, contrast enhancement, sharpening, and noise reduction. These methods typically use fixed or empirical parameters for enhancement, failing to understand the semantic information of the video content and unable to distinguish the relative importance of subject and background areas.

[0006] (2) The enhancement parameters lack an adaptive adjustment mechanism. Existing video enhancement systems typically employ fixed filtering, sharpening, and color correction parameters. When the video scene changes, such as switching from an indoor to an outdoor environment or from a static to a moving scene, these fixed parameters struggle to adapt to varying video quality levels, easily leading to issues like noise retention, loss of detail, blurred edges, or image distortion. Traditional enhancement methods often fail to simultaneously achieve both noise reduction and edge detail preservation, resulting in limited improvements in video quality.

[0007] (3) Lack of multi-dimensional video status comprehensive analysis capability Most existing technologies only optimize a single metric, such as brightness enhancement, noise suppression, or edge enhancement, and lack the ability to jointly analyze multi-dimensional features such as brightness, noise, edges, texture, and motion.

[0008] (4) Insufficient integration of deep learning models with video enhancement processes Although some systems utilize convolutional neural networks to extract features, the extraction results are not further transformed into quality evaluation indicators with clear physical meaning, making it difficult to effectively integrate with traditional video enhancement algorithms and limiting the application effect of deep features in practical video processing systems. Summary of the Invention

[0009] The main objective of this invention is to provide a state-aware and deep-analysis-based adaptive video processing and streaming method and system. By constructing a collaborative processing framework of "multi-dimensional state awareness - adaptive filtering - deep semantic analysis - video quality decision - image quality enhancement - dynamic encoding streaming", the video enhancement process can simultaneously combine video quality state information, video content semantic information and network transmission state information to make comprehensive decisions. This not only improves the video image quality enhancement effect, but also takes into account the transmission efficiency and system stability in real-time streaming scenarios.

[0010] The technical solution adopted in this invention includes: an adaptive video processing and streaming method based on state awareness and depth analysis, comprising: The system acquires video frames from the input video stream and initializes the video frames based on the OpenCL parallel computing framework. It then performs multi-dimensional state-aware analysis on the initialized video frames to extract feature parameters that reflect the current video quality state and construct a video state feature vector. Adaptive bilateral filtering is performed on the video frame based on the video state feature vector. The filtered video frame is obtained by dynamically adjusting the spatial domain parameters and range domain parameters of the bilateral filter. The filtered video frames are input into a pre-built deep semantic feature extraction network to extract high-level semantic features from the video frames. The high-level semantic features are then mapped to semantic attribute parameters using a pre-built semantic attribute mapping network to construct semantic feature vectors. The video state feature vector and semantic feature vector are fused and input into a pre-constructed video quality decision network to generate corresponding image quality enhancement parameters; the video frames are then subjected to adaptive image quality enhancement processing based on the image quality enhancement parameters to obtain enhanced video frames. The enhanced video frames are compressed and encoded, and the encoding parameters and streaming bitrate are dynamically adjusted in combination with real-time network status information. The processed video stream is then sent to the streaming media server in real time.

[0011] According to the above technical solution, based on the OpenCL parallel computing framework, the video frames are initialized, specifically including: using the OpenCL parallel computing framework to initialize the computing environment, schedule device resources, and manage video frame buffers.

[0012] According to the above technical solution, adaptive bilateral filtering is performed on the video frames, specifically including: Multidimensional state-aware analysis is performed on the input video frames to extract state features and construct a state feature vector reflecting the current video quality state. The state features include brightness information, noise information, edge information, texture information, and motion information. Based on the aforementioned state characteristics, the spatial domain parameters and range domain parameters of the bilateral filter are dynamically adjusted to perform adaptive bilateral filtering on the video frame.

[0013] According to the above technical solution, the pre-constructed deep semantic feature extraction network includes the ResNet50 model.

[0014] According to the above technical solution, the method for constructing semantic feature vectors specifically includes: The video frames are input into a pre-constructed deep semantic feature extraction network; the deep semantic feature extraction network uses ResNet50 residual neural network as its backbone structure and extracts high-level semantic feature vectors of the video frames through multi-layer convolution operations and residual learning.

[0015] The method for obtaining the enhanced video frame according to the above technical solution specifically includes: After obtaining high-level semantic features and semantic feature vectors, a pre-constructed video quality decision network is used to fuse and analyze the high-level semantic features and semantic feature vectors, and adaptively generate image quality enhancement parameters based on the current video quality status and content semantic information; the image quality enhancement parameters include brightness enhancement parameters, contrast enhancement parameters, color correction parameters, and detail enhancement parameters; The video frames are enhanced according to the image quality enhancement parameters to obtain enhanced video frames.

[0016] According to the above technical solution, the enhanced video frames are subjected to video compression encoding. Encoding parameters and streaming bitrate are dynamically adjusted based on real-time network status information, and the processed video stream is sent to the streaming media server in real time. Specifically, this includes: Using the FFmpeg tool, the enhanced video frames are encoded into a bitstream of a preset format and then pushed to the specified streaming media server address via the RTMP protocol.

[0017] Another aspect of the present invention provides a computer storage medium storing a computer program executable by a processor, the computer program performing the above-described adaptive video processing streaming method based on state awareness and depth analysis.

[0018] Another aspect of the present invention provides a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described adaptive video processing and streaming method based on state awareness and depth analysis.

[0019] Another aspect of the present invention provides a state-aware and depth-analysis-based adaptive video processing streaming system, comprising: The video feature extraction module is used to acquire video frames from the input video stream and initialize the video frames based on the OpenCL parallel computing framework; it performs multi-dimensional state-aware analysis on the initialized video frames, extracts feature parameters that reflect the current video quality state, and constructs a video state feature vector. The adaptive filtering module is used to perform adaptive bilateral filtering on video frames based on the video state feature vector, and obtain the filtered video frames by dynamically adjusting the spatial domain parameters and range domain parameters of the bilateral filter. The deep semantic feature extraction module is used to input the filtered video frames into a pre-constructed deep semantic feature extraction network, extract the high-level semantic features of the video frames, and use a pre-constructed semantic attribute mapping network to map the high-level semantic features into semantic attribute parameters to construct semantic feature vectors. The image quality enhancement module is used to fuse the video state feature vector and the semantic feature vector, and input them into a pre-constructed video quality decision network to generate corresponding image quality enhancement parameters; and to perform adaptive image quality enhancement processing on the video frame according to the image quality enhancement parameters to obtain the enhanced video frame. The encoding and streaming module is used to compress and encode the enhanced video frames, dynamically adjust the encoding parameters and streaming bitrate based on real-time network status information, and send the processed video stream to the streaming media server in real time.

[0020] The beneficial effects of this invention are: improving processing speed through a parallel computing framework, preserving edges while denoising by using adaptive bilateral filtering, achieving intelligent image quality correction by combining semantic features extracted by deep neural networks with prediction models, and finally achieving high-definition, adaptive enhancement and low-latency transmission of video through compression encoding and real-time streaming.

[0021] Furthermore, this invention builds a video frame processing kernel based on OpenCL, initializes the computing environment and caches video frame data on the device side, providing an efficient underlying data path for all subsequent processing steps and significantly improving the computing speed of the video processing flow.

[0022] Furthermore, this invention constructs a multidimensional state feature vector based on the extracted information, and dynamically adjusts the spatial domain parameters and range domain parameters of the bilateral filter by utilizing the geometric distance between pixels and the grayscale difference. While effectively removing noise, it preserves the important edge and detail information of the image to the greatest extent, avoiding the edge blurring or noise residue problems caused by traditional fixed parameter filtering.

[0023] Furthermore, this invention uses FFmpeg for encoding and pushes the video via the RTMP protocol, ensuring the efficiency and reliability of video encoding, as well as the broad compatibility of the streaming process with most streaming media servers, ensuring that the processed high-definition video can be stably and with low latency distributed and played. Attached Figure Description

[0024] Figure 1 This is a flowchart of the adaptive video processing and streaming method based on state awareness and depth analysis according to an embodiment of the present invention; Figure 2 This is an application scenario diagram of the adaptive video processing and streaming method based on state awareness and depth analysis according to an embodiment of the present invention; Figure 3 This is a flowchart of the adaptive video processing and streaming system module based on state awareness and depth analysis according to an embodiment of the present invention; Figure 4 This is a diagram of the adaptive bilateral filter network structure according to an embodiment of the present invention; Figure 5This is a diagram of the semantic attribute feature extraction network structure according to an embodiment of the present invention; Figure 6 This is a network structure diagram of the image quality enhancement module according to an embodiment of the present invention; Figure 7 This is a network structure diagram of the adaptive streaming module in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0026] Example 1 This embodiment provides an adaptive video processing and streaming method based on state awareness and depth analysis. This method achieves end-to-end optimization from noise suppression and content-aware enhancement to network adaptive streaming by constructing a dual-driven mechanism of multi-dimensional state awareness and deep semantic analysis. The entire solution relies on the OpenCL parallel computing architecture to ensure processing efficiency, while also considering edge-preserving noise reduction, intelligent image quality correction, and low-latency stable streaming. The process is as follows: Figure 1 As shown, it includes the following steps: S1. Capture video frames from the input video stream and initialize the video frames using the OpenCL (Open Computing Language) parallel computing framework. This step establishes the underlying parallel computing environment for subsequent end-to-end video processing.

[0027] Specifically, this step initializes the computing environment, schedules device resources, and manages the video frame cache. By building a video frame processing kernel, the data to be processed is stored locally on the device, adapting to subsequent large-scale parallel reading operations and effectively reducing overall computational latency.

[0028] S2. Perform multi-dimensional state-aware analysis on the initialized video frames, extract feature parameters that reflect the current video quality state, and construct a video state feature vector; then perform adaptive bilateral filtering on the video frames based on the feature vector.

[0029] The feature parameters include, but are not limited to, brightness information, noise information, edge information, texture information, and motion information.

[0030] Specifically, the adaptive bilateral filtering process is achieved by dynamically adjusting the spatial domain parameters (spatial standard deviation) and range domain parameters (range standard deviation) of the bilateral filter. Compared with traditional fixed-parameter filtering methods, this embodiment can adaptively adjust the filtering intensity according to the local features of the video frame (such as noise intensity and edge density): increasing the smoothing intensity in flat areas (high noise) to remove noise, and decreasing the smoothing intensity in edge texture areas (high detail) to preserve edges, thereby filtering out image noise while fully preserving the image edge contours and avoiding the loss of detail caused by over-filtering.

[0031] S3. Input the filtered video frame into the pre-constructed deep semantic feature extraction network to extract the high-level semantic features of the video frame, and use the pre-constructed semantic attribute mapping network to construct a semantic feature vector.

[0032] In this embodiment, the deep semantic feature extraction network preferably adopts the ResNet50 (50-layer residual network) model. The processing includes: adjusting the video frames to a preset size and performing normalization processing, inputting them into the ResNet50 model, extracting features through multi-layer convolution operations and residual learning, and finally compressing them into 2048-dimensional high-dimensional semantic features using global average pooling.

[0033] Furthermore, a semantic attribute mapping network is used to map the above high-dimensional semantic features into specific semantic attribute parameters, thereby constructing a semantic feature vector.

[0034] S4. The video state feature vector and semantic feature vector are fused and input into a pre-constructed video quality decision network to generate corresponding image quality enhancement parameters; the video frames are then subjected to adaptive image quality enhancement processing based on the image quality enhancement parameters to obtain enhanced video frames.

[0035] The image enhancement parameters specifically include brightness enhancement parameters, contrast enhancement parameters, color correction parameters, and detail enhancement parameters.

[0036] S5. Perform video compression encoding on the enhanced video frames, dynamically adjust the encoding parameters and streaming bitrate based on real-time network status information, and send the processed video stream to the streaming media server in real time.

[0037] Specifically, using FFmpeg (an open-source audio and video processing framework), the enhanced video frames are encoded into a bitstream in a preset format (such as H.264 / AVC or H.265 / HEVC). During the streaming phase, network bandwidth and packet loss rate are monitored in real time, the encoding bitrate and GOP structure are dynamically adjusted, and the bitstream is pushed to the designated streaming media server address via RTMP (Real-Time Messaging Protocol) to ensure low latency and high stability during transmission.

[0038] This embodiment also provides a state-aware and depth-analysis-based adaptive video processing and streaming system, which corresponds one-to-one with the above-described method flow, and completes the entire process in a modular and collaborative manner, including: The video feature extraction module is used to acquire video frames from the input video stream and initialize the video frames based on the OpenCL parallel computing framework; it performs multi-dimensional state-aware analysis on the initialized video frames, extracts feature parameters that reflect the current video quality state, and constructs a video state feature vector. The adaptive filtering module is used to perform adaptive bilateral filtering on video frames based on the video state feature vector, and obtain the filtered video frames by dynamically adjusting the spatial domain parameters and range domain parameters of the bilateral filter. The deep semantic feature extraction module is used to input the filtered video frames into a pre-constructed deep semantic feature extraction network, extract the high-level semantic features of the video frames, and use a pre-constructed semantic attribute mapping network to map the high-level semantic features into semantic attribute parameters to construct semantic feature vectors. The image quality enhancement module is used to fuse the video state feature vector and the semantic feature vector, and input them into a pre-constructed video quality decision network to generate corresponding image quality enhancement parameters; and to perform adaptive image quality enhancement processing on the video frame according to the image quality enhancement parameters to obtain the enhanced video frame. The encoding and streaming module is used to compress and encode the enhanced video frames, dynamically adjust the encoding parameters and streaming bitrate based on real-time network status information, and send the processed video stream to the streaming media server in real time.

[0039] Example 2 Based on Example 1, this example provides another adaptive video processing and streaming system based on state awareness and depth analysis, which is used to realize the complete processing flow from video frame input to high-definition streaming. Its core lies in sequentially performing parallel computing initialization, adaptive filtering, depth feature extraction, intelligent image quality correction and real-time encoding streaming, and finally outputting a high-quality video stream.

[0040] Typical application scenarios of the system in this embodiment are as follows: Figure 2 As shown: The video acquisition area captures raw video signals through a wireless sensor network consisting of a video sensor network array, an AP master node, and relay nodes. These signals are transmitted via Ethernet to the high-definition video information processing module in the middle, while simultaneously receiving application demand signals from the push terminal on the left, thus achieving supply and demand linkage.

[0041] The core processing layer is dominated by the high-definition video processing unit, which integrates three major functional modules: frame acquisition and filtering module, video processing kernel function, and editing and streaming frame. It also calls on GPU resources for hardware acceleration to improve computing efficiency.

[0042] Finally, the processed video stream enters the top-level real-time push management module, which is responsible for distributing the data to the leftmost push terminal, supporting various application scenarios such as online video, online live streaming, and video chat, thus forming a closed-loop video service system that integrates efficient acquisition, intelligent processing, and stable distribution.

[0043] Specifically, such as Figure 3 As shown, the overall system architecture is built around seven functional modules.

[0044] The system receives the input video signal, which first enters the multi-dimensional state perception module. As the perception front end of the system, this module performs real-time low-level visual analysis on the input image, accurately extracting five key state feature parameters: brightness (B), noise (N), edge (E), texture (T), and motion (M). These parameters are then combined into an initial state feature vector S=[B,N,E,T,M], providing basic data support for subsequent filtering and decision-making.

[0045] The state feature vector S is fed into the adaptive bilateral filtering module. This module dynamically calculates adaptive spatial domain weights based on local image features, and then generates adaptive range domain weights in parallel. These two weight matrices are jointly applied to the bilateral filtering calculation process, ultimately outputting the filtered image. This design enables the system to accurately preserve object edge contours while eliminating image noise, avoiding detail loss caused by over-smoothing.

[0046] Adaptive bilateral filter network structure as follows Figure 4 As shown, a nighttime city image is used as input, and the underlying visual feature map is analyzed from it through a multi-dimensional state feature extraction module. Specifically, three key indicators are quantified: brightness feature B, which represents the distribution of light and dark, noise intensity N, which represents the purity of the image, and edge density E, which represents the structural complexity.

[0047] Subsequently, in the parameter adaptive calculation stage, the system uses these three characteristic parameters to dynamically calculate the two core standard deviation parameters (σ) required for bilateral filtering through a specific adjustment formula. r and σ s Among them, the stronger the noise, the greater the range parameter σ. r The larger the value, the better for smoothing and noise reduction; the higher the brightness or the denser the edges, the larger the spatial parameter σ. s The smaller the value, the more it limits the filtering range to precisely preserve details in bright areas and object edges.

[0048] Next, the system substitutes these dynamically generated parameters into the adaptive bilateral filtering formula at the top for calculation. This formula is essentially a weighted mean filter, the core of which is determined by spatial weights and range weights: spatial weights are attenuated according to the geometric distance between pixels, while range weights are attenuated according to the grayscale difference between pixels. The combination of the two enables the algorithm to strictly preserve the edges of abrupt changes while smoothing homogeneous regions.

[0049] Finally, the right side shows the final results of the algorithm. Through local magnification and comparison, it can be seen that compared with the traditional bilateral filtering with fixed parameters (which leads to blurred edges), the adaptive method proposed in this paper can effectively eliminate background noise while perfectly preserving the sharpness of high-frequency edges such as window grids, proving that it has achieved a better balance between noise reduction and edge preservation.

[0050] The filtered image enters the deep semantic feature extraction module, which is built on the powerful ResNet50 residual neural network. After deep convolution and residual learning in the ResNet50 backbone network, the image data enters the Global Average Pooling (GAP) layer for feature compression, and is finally transformed into a high-dimensional 2048-dimensional semantic feature vector. The semantic attribute feature extraction network structure of this module is as follows: Figure 5 As shown: The system processing flow is based on a single frame of video (F... f The input is γ, which first enters the backbone feature extraction network. This network is built on the classic ResNet50 structure and goes through five stages from Conv1 to Conv5_x, containing multiple Bottleneck residual modules. The number of channels gradually increases from 64 to 2048, and during this process, convolutions or max pooling operations with a stride of 2 are used to halve the feature map resolution layer by layer from the input H×W to 7×7. After extracting the high-dimensional features, the data is fed into a fully connected sub-network called the semantic attribute mapping network for decision generation. This sub-network contains three hidden layers (feature fusion layer, quality perception layer, and decision mapping layer), which are responsible for a deep understanding and fusion of the underlying visual information. Finally, the network outputs a control parameter vector Z=[γ] containing 6 dimensions. b ,γ c ,g r ,g g ,g b [,s], corresponding to the brightness enhancement coefficient, contrast enhancement coefficient, color gain coefficient of the red, green and blue channels, and sharpening coefficient, respectively. These parameters directly drive the subsequent image or video quality adjustment module, thereby achieving fine-grained adaptive correction for the image content.

[0051] The system feeds the initial state feature vector S and the extracted 2048-dimensional semantic feature vector into the Video Quality Decision Network (VQDN). Before entering the fully connected network, these two sets of features are deeply integrated in the feature fusion layer to form an extended 9-dimensional feature vector F=[B,N,E,T,M,L,D,C,O] containing both low-level visual and high-level semantic features. This decision network is continuously optimized through the training data and supervised learning mechanism below. Its training process relies on a large number of video samples, professional tuning / subjective scoring, and the resulting accurate enhancement parameter labels. After nonlinear mapping and calculation by the fully connected network, the network finally outputs a set of refined enhancement parameter outputs D. p Specifically, this includes: brightness enhancement (y b ), contrast enhancement (y c ), color correction parameters (r) g ,g g ,b g ) and sharpening enhancement (s).

[0052] Next is the image quality reconstruction unit. The image quality enhancement module, acting as the execution terminal, receives the aforementioned enhancement parameters D. p The module sequentially performs operations such as brightness enhancement, contrast enhancement, color correction, and sharpening enhancement on the image, ultimately outputting an enhanced image with significantly improved visual effects. The structure of this module is as follows: Figure 6 As shown: The input frame undergoes independent fine-tuning processing in four dimensions: In the brightness enhancement module, the brightness coefficient γ is used... b The input is linearly scaled; in the contrast enhancement module, a principle similar to histogram stretching is used, utilizing the enhancement coefficient γ. c and offset μ c Adjust the dynamic range of the image; in the color correction module, apply independent gain coefficients (g) to the three color channels: red (R), green (G), and blue (B). r ,g g ,g b To achieve color balance, the sharpening enhancement module calculates a Gaussian blurred reference image G(I′) of the input image and performs a weighted difference with the original image (introducing a sharpening coefficient s) to extract and enhance high-frequency edge details of the image. The enhanced image (I′) generated by these four parallel processing branches... b ,I c ,I rgb ,I s The outputs are then fed into the core weighted fusion module, which adaptively assigns optimal weights to the outputs of each branch using a specific dynamic weighting control strategy, and fuses them into a single enhanced feature map, ultimately outputting a fully enhanced video frame I. enhThe magnified comparison image in the lower right corner visually verifies the effectiveness of the system: the processed image not only significantly improves global contrast and color saturation, but also makes details such as road texture and building edges clearer and sharper compared to the original image (I′).

[0053] The encoding and adaptive streaming module is responsible for the final encapsulation and transmission tasks. This module first uses video encoding (H.264) technology to efficiently compress the enhanced image, then executes an adaptive bitrate adjustment strategy to cope with complex and changing network environments, and finally pushes the processed video stream to the adaptive bitrate streaming cloud server through distribution and streaming steps, completing the closed-loop processing of the entire system. The structure of this module is as follows: Figure 7 As shown: The original audio and video source is initially set to 1080P resolution and 60fps, and then enters the network status feature perception and extraction module. This module monitors and extracts the available bandwidth (B) of the current network in real time. t ), delay (T) t ) and packet loss rate (P t Combine them into the network state vector S t These multidimensional network features, along with video content features, are input into the central core computational region. In this region, the system first transforms the original network parameters into a standardized network quality index Q using nonlinear mapping functions (Sigmoid and exponential transformation). n Simultaneously, a content quality index Q is generated based on the complexity of the video content. v Subsequently, these two factors are input into the QoE index Q. QoE The calculation module performs weighted fusion to form the final Quality of Experience (QoE) metric. Based on this QoE metric, the adaptive streaming bitrate formula at the top is Rt=Rmax. Q QoE k Triggered, the output bitrate is dynamically adjusted. Finally, the optimized stream is pushed to the client. Compared to stuttering under low bandwidth or blurry images without adaptive filtering, this mechanism can adjust in real time according to network fluctuations, always maintaining the optimal user experience with smooth and clear visuals.

[0054] This embodiment also provides an adaptive video processing and streaming method based on state awareness and depth analysis, the specific steps of which are as follows: T1. To fully utilize the parallel computing capabilities of hardware such as GPU / CPU / FPGA to accelerate subsequent processing, perform the initial setup of the OpenCL environment.

[0055] like Figure 3As shown, the processing flow of this step includes: First, the raw video stream is sent to the central OpenCL video stream management module. The OpenCL video stream management module is the core processor of the system. Driven by the parallel computing of the underlying CPU and GPU, it executes a series of processing steps sequentially. First, it performs OpenCL initialization, then uses an adaptive bilateral filtering algorithm to optimize the image. Simultaneously, the feature extraction and regression model prediction unit analyzes the image content and dynamically generates optimization parameters for subsequent color enhancement steps. The video frames after image enhancement are then sent to the video encoding module for compression. The final high-definition video stream is pushed to the user terminal in real time, completing the entire chain from acquisition and intelligent processing to distribution. Specifically, it is divided into sub-steps: T1.1 Obtain available OpenCL platforms and devices (such as GPUs), and create the corresponding computing contexts and command queues.

[0056] T1.2 Write the video frame processing kernel function algorithm code to be executed on the device, and create and compile OpenCL program objects based on the above environment to generate machine code that can be executed in parallel by the GPU.

[0057] T1.3. Create an OpenCL buffer from the input video frame data and transfer it to the GPU's device memory for caching. This step pre-loads the data into high-speed memory, preparing it for subsequent parallel pixel processing.

[0058] This invention uses the OpenCL heterogeneous parallel computing framework to complete the construction of the video frame processing kernel. Specifically, it implements: (1) initialization of heterogeneous computing devices; (2) GPU / OpenCL context construction; (3) video frame buffer construction; (4) parallel task scheduling; and (5) video data device-side mapping.

[0059] T2. Perform adaptive bilateral filtering on the video frames transmitted to the device to effectively remove noise while preserving important image edges and details. Specific implementation: Video image state analysis: First, extract brightness information, noise intensity, and edge density from the video image. Output a decision vector.

[0060] Construct a decision model and define video image state information. , The grayscale value is the coordinate system. For neighboring pixels, Center pixel.

[0061] Brightness characteristics , Represents the number of pixels.

[0062] Noise intensity N= .

[0063] Edge density ,in Number of pixels at the edge; This represents the total number of pixels.

[0064] definition .

[0065] This invention suppresses noise in video frames while preserving image edges and texture details, providing high-quality input images for subsequent deep feature extraction. It achieves adaptive noise reduction by dynamically adjusting filtering parameters based on image features output from the multi-dimensional state-aware module.

[0066] Define spatial weights: Range weight

[0067] Filter parameter adjustment coefficients: ,

[0068] Filtering output results

[0069] T2.1 Perform image resolution transformation on the input video frames and uniformly match them to the preset size.

[0070] T2.2. Traverse the neighborhood of the currently processed pixel and calculate the local mean and local variance of the pixels within that neighborhood. The local mean is used to estimate the expected value of the pixel, while the variance is used to adjust the range weights subsequently.

[0071] T2.3 uses the absolute difference between the channel values ​​of a pixel and its neighboring pixels to dynamically determine the range standard deviation parameter, enabling the filter to dynamically adjust according to local image features (such as edges and textures).

[0072] T2.4 Calculation of Spatial and Range Weights: Based on the dynamically determined range standard deviation parameter, spatial weights (based on pixel distance) and range weights (based on pixel value similarity) are calculated using a Gaussian function, and both are used as weighting factors.

[0073] T2.5. Based on the calculated spatial weight and range weight, perform weighted calculation and update on the channel values ​​of the currently processed pixel.

[0074] T2.6 Normalize the updated channel data and output the filtered video frame with adaptive denoising and edge preservation.

[0075] T3. Input the filtered, clear video frames into the pre-trained ResNet50 deep neural network model to extract high-level semantic features that can characterize the complex content of the image.

[0076] Specifically, the overall structure of the ResNet50 deep neural network model presents a logical progression from left to right and from macro to micro, including... 1. Input preprocessing: processing video frames Uniform size adjustment And then normalize it. c represents the channel.

[0077] 2. Conv1 Feature Extraction Layer: This layer uses a 7×7 convolution with 64 kernels and a stride of 2 stages. It mainly extracts brightness variations, edge contours, color variations, and basic textures. These are the original pixel values. The channel mean. The standard deviation of the channel. , Represents the k-th convolutional kernel

[0078] 3. Pooling layer This layer uses 3×3 MaxPooling with a stride of 2, resulting in an output size of 56×56×64. This reduces the spatial resolution of the feature map, decreases computational complexity, improves the robustness of features to local displacements, and preserves significant response regions. 4. Conv2_x residual blocks This layer contains 3 Bottleneck residuals, with an output of 56×56×256. The main learning stages are local edge combination structures, basic shape information, and simple texture patterns.

[0079] 5. Conv3_x stage: Local semantic formation This layer contains four residual blocks, with an output of 28×28×512. It establishes semantic relationships in local regions, extracts mesoscale structural information, and begins to form object component-level representations.

[0080] 6. Conv4_x stage: Target-level semantic extraction Output 14×14×1024, perceive the complete target structure, extract target-level semantic expression, and enhance the ability to model regional consistency.

[0081] 7. Conv5_x stage: Scene-level semantic understanding, output 7×7×2048, model global scene semantics, capture cross-regional contextual relationships, and provide high-level semantic representation capabilities.

[0082] 8. Global Average Pooling Global average pooling is used to compress the feature map, reducing spatial features to channel-level descriptions, eliminating spatial redundancy, enhancing feature stability, and preventing overfitting.

[0083] For the k-th feature map

[0084] Finally obtained

[0085] ResNet50 constructs an efficient and deep semantic feature extraction network by stacking standardized Bottleneck modules and utilizing shortcut connections to alleviate gradient vanishing.

[0086] T3.1 Adjust the video frame rate to 224. The value is 224, and preprocessing operations such as mean normalization and centering are performed.

[0087] T3.2. Input the preprocessed frames into the ResNet50 network. First, they pass through the STAGE0 convolutional layer (performing 7... 7. Convolutional operations (converting pixel values ​​into feature representations) and pooling layers (reducing image size and computational cost).

[0088] T3.3 Frame data is processed sequentially through a deep network consisting of multiple residual blocks (STAGE1 to STAGE4). Each residual block is connected via skip connections to ensure that the original characteristics of the image are preserved when processing deep features, effectively extracting high-level features and avoiding the gradient vanishing problem.

[0089] T3.4 After the final convolutional layer, global average pooling is performed on the feature map to reduce it to a 2048-dimensional feature vector that highly condenses the semantic information of the image, which serves as the deep feature representation of the video frame.

[0090] T3.5 introduces a deep semantic feature extraction network using a three-layer MLP fully connected network to extract 2048-dimensional semantic features F. 2048 Perform layer-by-layer mapping: First layer : Perform high-dimensional feature fusion to extract basic semantic combination information.

[0091] Second floor Introducing quality perception capabilities enhances the expression of visual quality-related features.

[0092] Output layer Generates a 6-dimensional control parameter vector for subsequent image / video quality adjustment modules.

[0093] Define three hidden layers: the first layer is a feature fusion layer, and the second layer is a quality-aware layer.

[0094]

[0095]

[0096] Will

[0097] [ ]= Sigmoid ),in The coefficient matrix, This is the brightness enhancement factor. This is the contrast enhancement factor. , where are the gain coefficients for the three channels, and s is the sharpening coefficient.

[0098] T4. By using the predicted brightness correction coefficient and contrast correction coefficient, the overall brightness and contrast of the image are adjusted, making the image with insufficient brightness brighter and the image with low contrast clearer. At the same time, by adjusting the gain of the three RGB channels, the saturation and brightness of each color are precisely controlled, significantly improving the visual effect and color performance of the image.

[0099] The specific methods for enhancing image quality include: Brightness enhancement:

[0100] Contrast enhancement:

[0101] Color correction:

[0102] Enhanced Sharpening: G( ): Gaussian blur operator.

[0103] ,definition To enhance the image quality of video frames through weighted fusion.

[0104] T5. The video frames, enhanced by the above steps, are efficiently compressed using the H.264 protocol. Parameters such as available bandwidth, network latency, packet loss rate, and bitrate are adaptively adjusted based on real-time network conditions. The compressed frames are then pushed out in real-time for playback on the terminal. T5.1. Call the FFmpeg tool, set the push stream address (such as the RTMP server address), the encoding format to libx264, the frame rate to 60 frames per second, and other parameters, and initialize the push stream pipeline.

[0105] T5.2. The enhanced video frame data is passed to the input pipe of FFmpeg in the form of a byte stream for compression and encoding, and then pushed to the specified streaming media server in real time via the network (such as the RTMP protocol).

[0106] T5.3 After the streaming task is completed, release all the computing and network resources occupied by the video capture module, FFmpeg pipeline and related memory in an orderly manner.

[0107] The adaptive adjustment process of streaming parameters specifically includes: the enhanced video frame FeF_eFe enters the encoding and streaming module; the current network state is monitored in real time, and a network state vector is constructed. Introducing nonlinear mapping modeling improves robustness and enhances content quality. With network quality It enables dual-drive control, allowing dynamic adjustment of the video encoding bitrate, thereby improving video transmission quality and stability under limited bandwidth conditions.

[0108] Introducing mapping

[0109]

[0110]

[0111] Define network quality index , H is the coefficient matrix. Available bandwidth. Network latency Packet loss rate .

[0112]

[0113] Final confirmed bitrate

[0114] It includes information such as available bandwidth, network latency, and packet loss rate, and dynamically adjusts the video encoding bitrate based on network conditions.

[0115] In summary, this invention provides a state-aware and depth-analysis-based adaptive video processing and streaming method and system, which can enhance video quality in real time and ensure smooth streaming, solving the speed and adaptability problems of high-definition video processing in complex environments.

[0116] The various modules or mechanisms of the system are mainly used to implement the various steps of the above method embodiments, and will not be described in detail here.

[0117] It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.

[0118] The order of the steps in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0119] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A state-aware and depth-analysis-based adaptive video processing and streaming method, characterized in that, include: The system acquires video frames from the input video stream and performs initialization processing on the video frames based on the OpenCL parallel computing framework. Perform multi-dimensional state-aware analysis on the initialized video frames, extract feature parameters that reflect the current video quality state, and construct a video state feature vector. Adaptive bilateral filtering is performed on the video frame based on the video state feature vector. The filtered video frame is obtained by dynamically adjusting the spatial domain parameters and range domain parameters of the bilateral filter. The filtered video frames are input into a pre-constructed deep semantic feature extraction network to extract high-level semantic features of the video frames, and then used to construct semantic feature vectors using a pre-constructed semantic attribute mapping network. The video state feature vector and semantic feature vector are fused and input into a pre-constructed video quality decision network to generate corresponding image quality enhancement parameters; the video frames are then subjected to adaptive image quality enhancement processing based on the image quality enhancement parameters to obtain enhanced video frames. The enhanced video frames are compressed and encoded, and the encoding parameters and streaming bitrate are dynamically adjusted in combination with real-time network status information. The processed video stream is then sent to the streaming media server in real time.

2. The adaptive video processing and streaming method based on state awareness and depth analysis according to claim 1, characterized in that, Based on the OpenCL parallel computing framework, video frames are initialized, specifically including: initializing the computing environment, scheduling device resources, and managing video frame buffers using the OpenCL parallel computing framework.

3. The adaptive video processing and streaming method based on state awareness and depth analysis according to claim 1, characterized in that, Adaptive bilateral filtering is performed on the video frames, specifically including: Multidimensional state-aware analysis is performed on the input video frames to extract state features and construct a state feature vector reflecting the current video quality state. The state features include brightness information, noise information, edge information, texture information, and motion information. Based on the aforementioned state characteristics, the spatial domain parameters and range domain parameters of the bilateral filter are dynamically adjusted; the adaptive bilateral filtering process performs weighted mean filtering by jointly applying spatial weights based on spatial distance and range weights based on range differences; wherein the spatial weights decrease as the geometric distance between pixels increases, and the range weights decrease as the grayscale difference between pixels increases. Adaptive bilateral filtering is performed on the video frames using the adjusted spatial domain parameters and range domain parameters.

4. The adaptive video processing and streaming method based on state awareness and depth analysis according to claim 1, characterized in that, The pre-built deep semantic feature extraction network includes the ResNet50 model.

5. The adaptive video processing and streaming method based on state awareness and depth analysis according to claim 4, characterized in that, Methods for constructing semantic feature vectors include: The video frames are input into a pre-constructed deep semantic feature extraction network; the deep semantic feature extraction network uses ResNet50 residual neural network as its backbone structure and extracts high-level semantic feature vectors of the video frames through multi-layer convolution operations and residual learning.

6. The adaptive video processing and streaming method based on state awareness and depth analysis according to claim 1, characterized in that, Methods for obtaining enhanced video frames include: After obtaining high-level semantic features and semantic feature vectors, a pre-constructed video quality decision network is used to fuse and analyze the high-level semantic features and semantic feature vectors, and adaptively generate image quality enhancement parameters based on the current video quality status and content semantic information; the image quality enhancement parameters include brightness enhancement parameters, contrast enhancement parameters, color correction parameters, and detail enhancement parameters; The video frames are enhanced according to the image quality enhancement parameters to obtain enhanced video frames.

7. The adaptive video processing and streaming method based on state awareness and depth analysis according to claim 1, characterized in that, The enhanced video frames are compressed and encoded, and the encoding parameters and streaming bitrate are dynamically adjusted based on real-time network status information. The processed video stream is then sent to the streaming media server in real time. This process includes: Using the FFmpeg tool, the enhanced video frames are encoded into a bitstream of a preset format and then pushed to the specified streaming media server address via the RTMP protocol.

8. A computer storage medium, characterized in that, It contains a computer program that can be executed by a processor, which performs the state-aware and depth-analysis-based adaptive video processing streaming method according to any one of claims 1-7.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the state-aware and depth-analysis-based adaptive video processing streaming method according to any one of claims 1-7.

10. A state-aware and depth-analysis-based adaptive video processing and streaming system, characterized in that, include: The video feature extraction module is used to acquire video frames from the input video stream and perform initialization processing on the video frames based on the OpenCL parallel computing framework. Perform multi-dimensional state-aware analysis on the initialized video frames, extract feature parameters that reflect the current video quality state, and construct a video state feature vector. The adaptive filtering module is used to perform adaptive bilateral filtering on video frames based on the video state feature vector, and obtain the filtered video frames by dynamically adjusting the spatial domain parameters and range domain parameters of the bilateral filter. The deep semantic feature extraction module is used to input the filtered video frames into a pre-constructed deep semantic feature extraction network to extract high-level semantic features of the video frames and construct semantic feature vectors. The image quality enhancement module is used to fuse the video state feature vector and semantic feature vector, and input them into a pre-constructed video quality decision network to generate corresponding image quality enhancement parameters; and to perform adaptive image quality enhancement processing on the video frame according to the image quality enhancement parameters to obtain the enhanced video frame. The encoding and streaming module is used to compress and encode the enhanced video frames, dynamically adjust the encoding parameters and streaming bitrate based on real-time network status information, and send the processed video stream to the streaming media server in real time.

Images