Intelligent video coding optimization system and method based on computer vision

Abstract

The invention relates to the technical field of video coding, in particular to an intelligent video coding optimization system and method based on computer vision, and the system comprises a computer vision video content analysis module; identifying an object, a behavior and a face in the video; the video classification module based on the neural network is in communication connection with the computer video content analysis module; based on the video feature data, classifying videos by using a deep neural network; the video coding parameter optimization module is in communication connection with the video classification module based on the neural network; adaptively optimizing video coding parameters based on the video classification result and the video feature data; the video coding quality evaluation module is in communication connection with the video coding parameter optimization module; evaluating the video quality of coding based on the optimized coding parameters; and the evaluation result is fed back to the video coding parameter optimization module for further optimizing the coding parameter, so that the understanding depth of the video content is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video coding, and in particular to an intelligent video coding optimization system and method based on computer vision. Background Art

[0002] With the rapid development of video technology and the continuous improvement of network bandwidth, the demand for high-quality video content is growing. However, the massive nature of video data poses a huge challenge to storage and transmission systems. Therefore, efficient video coding technology has become a research hotspot. Traditional video coding methods mainly rely on preset coding parameters and fixed coding strategies, which are difficult to adapt to the special needs of different types of video content. Although some adaptive coding methods have emerged in recent years, they often only focus on simple statistical features and cannot fully understand and utilize the semantic information of the video.

[0003] The closest existing technologies usually adopt content-based adaptive coding methods. These methods dynamically adjust coding parameters by analyzing features such as texture complexity and motion intensity of the video. However, this approach has several key problems: First, they have a relatively shallow understanding of the video content and cannot capture high-level semantic information, resulting in poor coding results in some complex scenarios. Second, existing methods usually adopt fixed parameter adjustment strategies, which lack flexibility and are difficult to cope with diverse video content. Furthermore, these methods often simplify video quality assessment into a single objective indicator (such as PSNR), ignoring the subjective factors of human eye perception. Finally, the existing technology lacks special considerations when dealing with important visual elements such as faces, which may lead to quality degradation in key areas. Summary of the invention

[0004] The present invention aims to solve the above technical problems and provide an intelligent video coding optimization system and method based on computer vision. The system realizes comprehensive understanding of video content and precise coding optimization by deeply combining computer vision technology and deep learning algorithms.

[0005] The present invention proposes an intelligent video coding optimization system based on computer vision, comprising:

[0006] Computer vision video content analysis module for:

[0007] Perform content analysis on the input video to obtain the spatial domain parameters and temporal domain parameters of the video;

[0008] Identify objects, actions, and faces in videos;

[0009] The video classification module based on the neural network is connected to the computer video content analysis module for:

[0010] Receiving video feature data sent by the computer vision video content analysis module;

[0011] Based on the video feature data, classify the video using a deep neural network;

[0012] The video encoding parameter optimization module is communicatively connected with the neural network-based video classification module and is used to:

[0013] Receiving the video classification result sent by the neural network-based video classification module;

[0014] Adaptively optimizing video encoding parameters based on the video classification results and video feature data;

[0015] A video coding quality assessment module is communicatively connected with the video coding parameter optimization module and is used to:

[0016] Receiving the optimized encoding parameters sent by the video encoding parameter optimization module;

[0017] evaluating the quality of a video encoded based on the optimized encoding parameters;

[0018] The evaluation results are fed back to the video encoding parameter optimization module for further optimizing the encoding parameters.

[0019] Preferably, the computer vision video content analysis module comprises:

[0020] An object recognition unit, used to identify specific objects in the video;

[0021] Behavior recognition unit, used to analyze action behaviors in the video;

[0022] Face detection unit, used to detect and analyze faces in videos;

[0023] A scene classification unit, used to classify video scenes;

[0024] Among them, the analysis results of the object recognition unit, behavior recognition unit, face detection unit and scene classification unit together constitute the video feature data.

[0025] Preferably, the neural network-based video classification module comprises:

[0026] A feature extraction unit, used for extracting deep features from the video feature data using a convolutional neural network;

[0027] A deep belief network unit, for classifying videos based on the deep features;

[0028] The classification result fusion unit is used to fuse the classification result of the deep belief network unit with the analysis result of the computer vision video content analysis module to obtain the final video classification result.

[0029] Preferably, the video encoding parameter optimization module comprises:

[0030] A bit rate adjustment unit, used to adaptively adjust the encoding bit rate according to the video classification result and content characteristics;

[0031] A frame rate optimization unit, used to dynamically adjust the frame rate based on the dynamic characteristics of the video;

[0032] A macroblock parameter adjustment unit, used for adaptively adjusting the size and number of macroblocks for intra-frame coding and inter-frame coding;

[0033] A quantization parameter setting unit, used to intelligently set the quantization parameter according to the complexity of the video content;

[0034] Among them, each unit works together to generate an optimized encoding parameter set.

[0035] Preferably, the video coding quality assessment module comprises:

[0036] Objective evaluation unit, used to calculate the peak signal-to-noise ratio PSNR and structural similarity SSIM objective quality indicators;

[0037] Subjective evaluation unit, used to simulate the human visual system for subjective quality evaluation;

[0038] A deep learning evaluation unit, used to predict video quality scores using a trained deep neural network;

[0039] The evaluation result fusion unit is used to weightedly fuse the results of objective evaluation, subjective evaluation and deep learning evaluation to obtain a comprehensive quality score.

[0040] As a preference, it also includes:

[0041] The iterative optimization control module is communicatively connected with the video encoding parameter optimization module and the video encoding quality assessment module, and is used to:

[0042] Receiving an evaluation result of the video coding quality evaluation module;

[0043] The video encoding parameter optimization module is controlled to perform multiple rounds of iterative optimization until a preset quality target or an upper limit of the number of iterations is reached.

[0044] Preferably, the computer vision video content analysis module also includes:

[0045] Multimodal feature fusion unit, used for:

[0046] Fusion of spatial domain features, including color, texture, edge, and temporal domain features, including motion vectors and optical flow;

[0047] Generate comprehensive feature representation to improve the accuracy of video content understanding.

[0048] Preferably, the video encoding parameter optimization module further includes:

[0049] Face area optimization unit, used for:

[0050] Receiving face region information detected by the computer vision video content analysis module;

[0051] Use higher encoding quality parameters for the face area to ensure the clarity of facial details.

[0052] As a preference, it also includes:

[0053] A coding efficiency evaluation module is communicatively connected with the video coding parameter optimization module and the video coding quality evaluation module, and is used to:

[0054] Calculate the ratio of encoding bit rate to video quality to evaluate encoding efficiency;

[0055] The coding efficiency evaluation result is fed back to the video coding parameter optimization module to optimize the bit rate while ensuring the quality.

[0056] The computer vision-based intelligent video coding optimization method based on the system comprises the following steps:

[0057] S1, using computer vision technology to analyze the content of the input video and obtain the spatial domain parameters and temporal domain parameters of the video;

[0058] S2, based on the analysis results of step S1, classify the video using a deep neural network;

[0059] S3, adaptively optimize video encoding parameters, including bit rate, frame rate, macroblock parameters and quantization parameters, based on video classification results and content features;

[0060] S4, encoding the video using the optimized encoding parameters;

[0061] S5, evaluates the quality of the encoded video, including objective evaluation, subjective evaluation and deep learning-based evaluation;

[0062] S6, based on the quality evaluation result, further adjusting the encoding parameters through iterative optimization until the preset quality target or the upper limit of the number of iterations is reached;

[0063] S7, outputs the final optimized encoding parameters and encoded video.

[0064] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0065] First, the system of the present invention greatly improves the depth of understanding of video content by introducing advanced computer vision technology. Unlike traditional methods that rely only on simple statistical features, this system can identify objects, behaviors, and scenes in the video and capture high-level semantic information. This deep understanding of content provides a solid foundation for subsequent encoding optimization, enabling the system to make more intelligent encoding decisions based on the actual content of the video. For example, for sports videos containing fast movements, the system can automatically increase the search range and accuracy of motion estimation; and for landscape documentaries dominated by static scenes, more use can be made of intra-frame prediction technology to improve compression efficiency.

[0066] Secondly, the present invention adopts a video classification strategy based on deep neural networks, combined with multimodal feature fusion technology, to achieve accurate classification of video content. This classification result not only guides the selection of encoding parameters, but also provides valuable prior knowledge for the system. For example, for videos classified as news reports, the system will automatically increase the encoding quality of the face area; while for cartoons, it will pay more attention to the fidelity of colors. This content-aware encoding strategy significantly improves the encoding effect of different types of videos.

[0067] Furthermore, the present invention innovatively introduces a coding parameter optimization method based on reinforcement learning. This method models the coding process as a Markov decision process and learns the optimal parameter adjustment strategy by continuously interacting with the environment. Compared with the traditional fixed rule method, the reinforcement learning method has stronger adaptability and learning ability. It can dynamically adjust various coding parameters, such as quantization parameters, frame rate, macroblock size, etc., according to different video content characteristics and quality requirements. This adaptive optimization strategy not only improves coding efficiency, but also better balances video quality and bit rate.

[0068] In addition, the present invention adopts a multi-dimensional video quality assessment method, which comprehensively considers objective indicators, subjective models and deep learning assessment results. This comprehensive quality assessment strategy can more accurately reflect the actual viewing experience and avoid the one-sidedness that may be caused by relying on a single indicator. In particular, by introducing a quality assessment model based on deep learning, the system can learn the complex characteristics of human eye perception, so as to make decisions that are more in line with the human visual system during the encoding optimization process.

[0069] Finally, the system of the present invention pays special attention to the processing of important visual elements such as faces. By designing a special face area optimization unit, the system can use higher encoding quality parameters for the detected face area and achieve gradual quality adjustment from the center of the face to the outside. This fine-grained optimization strategy not only ensures the clarity of key areas, but also creates a natural and soft visual effect, greatly improving the viewing experience, and is particularly suitable for application scenarios with people as the main focus, such as video conferencing and online education.

[0070] In general, the present invention constructs an intelligent and adaptive video coding optimization system by cleverly combining computer vision, deep learning and video coding technology. The system can deeply understand the video content, accurately classify the video type, dynamically optimize the coding parameters, comprehensively evaluate the video quality, and perform special processing on important visual elements. These innovations not only significantly improve the efficiency and quality of video coding, but also provide customized coding schemes for different types of video content. In the context of the increasing richness of current video applications and the continuous improvement of user experience requirements, the present invention provides a new direction for the advancement of video coding technology, which has important theoretical significance and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a logic block diagram of the entire system of the present invention.

[0072] Figure 2 It is a logic block diagram of the computer vision video content analysis module of the present invention.

[0073] Figure 3 It is a logic block diagram of the neural network-based video classification module of the present invention.

[0074] Figure 4 It is a logic block diagram of the video encoding parameter optimization module of the present invention.

[0075] Figure 5 It is a logic block diagram of the video encoding quality assessment module of the present invention.

[0076] Figure 6 It is a logic block diagram of the coding efficiency evaluation module of the present invention. DETAILED DESCRIPTION

[0077] See also Figure 1-6 The present invention provides an intelligent video coding optimization system and method based on computer vision. The system realizes intelligent and adaptive optimization of the video coding process by deeply combining computer vision technology and deep learning algorithms, which can significantly improve the efficiency and quality of video coding.

[0078] The system of the present invention comprises a computer vision video content analysis module 1, a neural network-based video classification module 2, a video encoding parameter optimization module 3, and a video encoding quality assessment module 4. These modules are connected through communication to form a closed-loop optimization system, and work together to achieve intelligent video encoding optimization.

[0079] Specifically, the computer vision video content analysis module 1 is used to perform content analysis on the input video to obtain the spatial domain parameters and temporal domain parameters of the video. Preferably, the module uses a deep convolutional neural network (DCNN) to analyze the video frame and extract spatial features including color, texture, edge, and temporal features such as motion vector and optical flow. For example, a pre-trained model such as VGG16 or ResNet50 can be used as a feature extractor to extract high-level semantic features of the video frame. At the same time, the module is also responsible for identifying objects, behaviors and faces in the video. In one embodiment of the present invention, the Faster R-CNN algorithm can be used for object recognition, the I3D network can be used for behavior recognition, and the MTCNN algorithm can be used for face detection. The combination of these algorithms enables the system to fully understand the video content and provide rich semantic information for subsequent encoding optimization.

[0080] The neural network-based video classification module 2 is connected to the computer vision video content analysis module 1. The module receives the video feature data sent by the computer vision video content analysis module 1 and classifies the video using a deep neural network. Preferably, this module uses a long short-term memory network (LSTM) combined with an attention mechanism to make full use of the temporal information of the video. Specifically, the extracted frame-level features can be input into the LSTM network, and then the features of different time steps are weighted by the time attention mechanism, and finally the classification result of the video is obtained by a fully connected layer and a softmax function. In one embodiment of the present invention, the video can be divided into categories such as sports, news, movies, animations, etc., and each category corresponds to a different encoding strategy.

[0081] The video coding parameter optimization module 3 is connected to the video classification module 2 based on the neural network. The module receives the video classification result and adaptively optimizes the video coding parameters based on the result and the video feature data. Preferably, this module adopts a reinforcement learning algorithm to achieve adaptive optimization of parameters. Specifically, the coding parameters (such as bit rate, quantization parameter, frame rate, etc.) can be used as the action space, the weighted sum of the coding quality and bit rate can be used as the reward function, and then the deep Q network (DQN) can be used to learn the optimal parameter selection strategy. In one embodiment of the present invention, the reward function can be expressed as:

[0082] R = α·PSNR-β·BitRate,

[0083] Among them, PSNR is the peak signal-to-noise ratio, BitRate is the bit rate, and α and β are weight coefficients. By adjusting the values ​​of α and β, a balance can be achieved between video quality and file size. For example, for sports videos with high dynamic range, α can be set to 0.7 and β to ensure picture quality; while for news videos with mainly static scenes, α can be set to 0.4 and β to reduce the bit rate.

[0084] The video coding quality assessment module 4 is in communication connection with the video coding parameter optimization module 3. This module is responsible for assessing the quality of the video encoded based on the optimized parameters, and feeding back the assessment results to the video coding parameter optimization module 3 for further optimizing the coding parameters. Preferably, this module adopts a multi-index fusion method for quality assessment. Specifically, objective indicators such as PSNR, SSIM (structural similarity), VQM (video quality metric) can be calculated simultaneously, and combined with the output of a quality assessment model based on deep learning (such as VSFA), the final quality score is obtained by weighted averaging. In one embodiment of the present invention, the quality score can be expressed as:

[0085] Score=w1·PSNR+w2·SSIM+w3·(1-VQM)+w4·VSFA,

[0086] Among them, w1, w2, w3, w4 are weight coefficients and satisfy These weights can be adjusted according to different application scenarios. For example, for medical imaging videos that require high fidelity, the weight of PSNR can be increased; while for entertainment videos, the weight of VSFA can be increased to better reflect the subjective feelings of the human eye.

[0087] The computer vision video content analysis module 1 includes an object recognition unit 11, a behavior recognition unit 12, a face detection unit 13 and a scene classification unit 14. The analysis results of these units together constitute the video feature data, which provides a comprehensive content understanding basis for subsequent video classification and encoding optimization.

[0088] The object recognition unit 11 uses advanced target detection algorithms, such as Faster R-CNN or YOLO v3, to accurately identify various objects in the video. For example, for a traffic monitoring video, the unit can identify objects such as vehicles, pedestrians, and traffic signs. In one embodiment of the present invention, the confidence threshold of object recognition is set to 0.75, which is a balance between accuracy and recall.

[0089] The behavior recognition unit 12 uses a 3D convolutional neural network or a temporal behavior recognition model (such as I3D, SlowF astnetworks) to analyze the action behavior in the video. For example, in a sports video, the unit can recognize running, jumping, pitching and other actions. The present invention preferably uses a 16-frame time window for behavior recognition, which can capture the complete process of most behaviors without causing excessive computational complexity.

[0090] The face detection unit 13 uses algorithms such as MTCNN or RetinaFace to perform face detection and key point positioning. In news or interview videos, the role of this unit is particularly important, and it can accurately locate the face area and provide a basis for subsequent encoding optimization. In one embodiment of the present invention, the minimum size of face detection is set to 24x24 pixels, which is a compromise between detection speed and accuracy.

[0091] The scene classification unit 14 is responsible for classifying the overall scene of the video, such as indoor, outdoor, urban, natural, etc. This unit can be implemented by using a global average pooling (GAP) layer combined with a backbone network such as ResNet. Scene information is of great guiding significance for the selection of encoding strategies. For example, for complex natural scenes, a higher bit rate may be required to retain details.

[0092] The output results of these units are fused to form a multi-dimensional feature vector, which is passed to subsequent modules as video feature data. Feature fusion can be achieved by simple concatenation operations or by using more complex attention mechanisms or graph convolutional networks (GCNs) to capture the relationship between different features.

[0093] The neural network-based video classification module 2 includes a feature extraction unit 21, a deep belief network unit 22, and a classification result fusion unit 23. These three units work together to achieve accurate classification of the video.

[0094] The feature extraction unit 21 uses a convolutional neural network to extract deep features from the video feature data. The present invention preferably uses a 3D convolutional network, such as C3D or I3D, to capture information in both spatial and temporal dimensions. Specifically, the video can be segmented into several time segments (such as 16 frames), and then features are extracted for each segment, and finally a video-level feature representation is obtained through a pooling operation in the temporal dimension.

[0095] The deep belief network unit 22 classifies the video based on the extracted deep features. The deep belief network (DBN) is a generative model composed of multiple layers of restricted Boltzmann machines (RBMs). In one embodiment of the present invention, the structure of the DBN can be designed as follows: input layer (feature dimension) - 1000 - 500 - 100 - output layer (number of categories). Each layer of RBM is pre-trained using the contrastive divergence algorithm, and then the back propagation algorithm is used to fine-tune the entire network.

[0096] The classification result fusion unit 23 fuses the classification result of the deep belief network unit 22 with the analysis result of the computer vision video content analysis module 1 to obtain the final video classification result. The fusion strategy can adopt weighted voting or a method based on Dempster-Shafer evidence theory. For example, for each category c, its confidence after fusion can be calculated:

[0097] Conf(c)=w1·P DBN (c)+w2·P CV (c),

[0098] Among them, P DBN (c) is the output probability of the deep belief network, P C V(c) is the confidence of the category in the computer vision analysis result. w1 and w2 are weight coefficients and satisfy w1+w2=1. In one embodiment of the present invention, w1=0.6 and w2=0.4 can be set to balance the generalization ability of the deep learning model and the interpretability of the computer vision analysis.

[0099] Through this multi-module collaborative design, the system of the present invention can fully understand and accurately classify the input video content, laying a solid foundation for subsequent coding parameter optimization. The modular structure of the system also allows each component to be independently optimized and upgraded, with good scalability and adaptability. In the system of the present invention, the video coding parameter optimization module 3 is the core component for realizing intelligent coding. The module includes a bit rate adjustment unit 31, a frame rate optimization unit 32, a macroblock parameter adjustment unit 33 and a quantization parameter setting unit 34. These units work together to jointly generate an optimized coding parameter set to maximize the video coding efficiency.

[0100] The bit rate adjustment unit 31 is responsible for adaptively adjusting the encoding bit rate according to the video classification results and content characteristics. In a preferred embodiment of the present invention, the unit adopts a content-based bit rate control algorithm. Specifically, for scenes with high complexity (such as sports videos with intense movements), the system will allocate a higher bit rate; while for static scenes (such as news broadcasts), the bit rate will be reduced to save bandwidth. Preferably, the bit rate adjustment adopts the following formula:

[0101] R=Rbase ·(1+α·C+β·M),

[0102] Among them, R is the adjusted bit rate, R base is the basic bit rate, C is the scene complexity factor, M is the motion intensity factor, and α and β are weight coefficients. In one embodiment of the present invention, α and β can be set to 0.3 and 0.5 respectively to balance the encoding requirements of static content and dynamic content.

[0103] The frame rate optimization unit 32 dynamically adjusts the frame rate based on the dynamic characteristics of the video. The system of the present invention determines the optimal frame rate by analyzing the differences between consecutive frames. Preferably, the structural similarity index (SSIM) is used to measure the difference between frames. When the SSIM value of consecutive frames is higher than a preset threshold (e.g., 0.98), it is possible to consider reducing the frame rate to reduce the encoding of redundant information. On the contrary, for scenes with drastic changes, the system will maintain a higher frame rate to ensure the smoothness of the picture. This adaptive frame rate adjustment strategy can significantly reduce the bit rate while ensuring visual quality.

[0104] The macroblock parameter adjustment unit 33 is responsible for adaptively adjusting the size and number of macroblocks for intra-frame coding and inter-frame coding. In a preferred embodiment of the present invention, the unit adopts a content-based adaptive block strategy. For areas with complex textures, the system tends to use smaller macroblocks (such as 8x8 or 4x4) to improve coding accuracy; while for flat areas, larger macroblocks (such as 32x32 or 64x64) are used to improve compression efficiency. Preferably, the present invention adopts the following criteria to determine the macroblock size:

[0105]

[0106] Wherein, S is the macroblock size, Var is the regional variance, T1, T2 and T3 are preset thresholds. In one embodiment of the present invention, these thresholds can be set to 50, 100 and 200 respectively, which are empirical values ​​obtained based on a large amount of experimental data.

[0107] The quantization parameter setting unit 34 intelligently sets the quantization parameter according to the complexity of the video content. The system of the present invention adopts a perception-based quantization strategy, that is, a larger quantization step size is used in areas that are not sensitive to the human eye, and a smaller quantization step size is used in important areas (such as faces, text, etc.). Preferably, the setting of the quantization parameter QP follows the following formula:

[0108] QP=QP base -ΔQP·(w1·I face +w2·I text +w3·I motion ),

[0109] Among them, QP baseis the basic quantization parameter, ΔQP is the adjustment step size, I f ace, I text and I m otion are indicator functions of face area, text area and motion area respectively, w1, w2 and w3 are weight coefficients. In one embodiment of the present invention, ΔQP=4, w1=0.5, w2=0.3, w3=0.2 can be set. This set of parameters achieves a good balance between subjective visual quality and coding efficiency.

[0110] The video coding quality assessment module 4 of the present invention comprises an objective assessment unit 41, a subjective assessment unit 42, a deep learning assessment unit 43 and an assessment result fusion unit 44. This diversified assessment strategy ensures the comprehensiveness and reliability of the coding quality assessment.

[0111] The objective evaluation unit 41 is responsible for calculating objective quality indicators such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). In a preferred embodiment of the present invention, the calculation of PSNR adopts the following formula:

[0112]

[0113] Among them, MAX I is the maximum possible value of a pixel (255 for an 8-bit image), and MSE is the mean square error. The calculation of SSIM is based on the comparison of brightness, contrast, and structure:

[0114] SSIM(x,y)=[l(x,y)] α [c(x,y)] β ·[s(x,y)] γ ,

[0115] Wherein, l, c and s represent brightness, contrast and structure comparison functions respectively, and α, β and γ are weight coefficients. In one embodiment of the present invention, these weights can all be set to 1 to equally consider the influence of the three aspects.

[0116] The subjective assessment unit 42 is intended to simulate the human visual system for subjective quality assessment. The system of the present invention uses a perception-based video quality assessment model (VQM) to achieve this goal. VQM takes into account the sensitivity of the human eye to different types of distortion, including blur, blockiness, color distortion, etc. Preferably, the calculation of VQM is based on the following formula:

[0117] VQM=a1·SI loss +a2 HV loss +a3·CHROMA spread +a4 CT ati +a5·CHROMA extreme ,

[0118] Among them, each item represents spatial information loss, horizontal / vertical edge loss, chromaticity diffusion, consistency of temporal information, and extreme chromaticity change, and a1 to a5 are corresponding weight coefficients. In one embodiment of the present invention, these weights can be obtained by training with large-scale subjective test data.

[0119] The deep learning evaluation unit 43 uses the trained deep neural network to predict the video quality score. In a preferred embodiment of the present invention, an end-to-end reference-free video quality evaluation network, such as VSFA (Video Speed-Quality Assessment), is used. The network can directly extract features from compressed videos and output quality scores without the need for the original video as a reference. The core idea of ​​the VSFA network is to capture the spatiotemporal features of the video by combining a spatial feature extractor (such as ResNet) and a temporal aggregation model (such as Gated Recurrent Unit, GRU).

[0120] The evaluation result fusion unit 44 is responsible for weighted fusion of the results of objective evaluation, subjective evaluation and deep learning evaluation to obtain a comprehensive quality score. The present invention adopts an adaptive weighting strategy to dynamically adjust the weights of each evaluation method according to the characteristics of the video content. Specifically, the calculation formula of the fusion score S is as follows:

[0121] S=w1·PSNR norm +w2·SSIM+w3·(1-VQM norm )+w4·VSFA score

[0122] Among them, PSNR norm and VQM norm are the normalized PSNR and VQM values, w1 to w4 are weight coefficients, and satisfy ∑ i =1 4 w i = 1. Preferably, these weights can be dynamically adjusted according to video features through a machine learning algorithm (such as random forest) to adapt to different types of video content.

[0123] The system of the present invention further comprises an iterative optimization control module 5. The module is in communication connection with the video encoding parameter optimization module 3 and the video encoding quality evaluation module 4, and is responsible for controlling the iterative optimization process of the entire system.

[0124] In a preferred embodiment of the present invention, the generation optimization control module 5 adopts a strategy based on reinforcement learning to guide the parameter optimization process. Specifically, the module models the coding parameter optimization problem as a Markov decision process (MDP). The state space is composed of the current coding parameters and video features, and the action space includes the adjustment operations of each coding parameter. The reward function is designed as a weighted combination of coding quality and bit rate:

[0125] R = α·Q-β·log(BitRate),

[0126] Wherein, Q is the quality score, BitRate is the bit rate, and α and β are weight coefficients. In one embodiment of the present invention, α=1 and β=0.5 can be set. This set of parameters achieves a good balance between quality and bit rate.

[0127] The iterative optimization control module 5 uses the deep Q learning (DQN) algorithm to learn the optimal parameter adjustment strategy. The input of the DQN network is the current state, and the output is the Q value of each possible action. In each iteration, the system selects an action according to the ε-greedy strategy, performs parameter adjustment, and then observes the new state and reward. These experiences are stored in the experience replay buffer for subsequent network training.

[0128] Preferably, the system of the present invention adopts Double DQN to alleviate the problem of over-estimation of Q value. Specifically, two networks with the same structure but different parameters are maintained: the main network is used to select actions, and the target network is used to evaluate action values. The parameters of the target network are regularly copied from the main network to stabilize the learning process.

[0129] The termination condition of the iterative optimization process can be set to reach a preset quality target or an upper limit of the number of iterations. In one embodiment of the present invention, the quality target can be set to PSNR reaching 38dB or SSI M reaching 0.95, and the upper limit of the number of iterations is set to 50 times. These thresholds are based on empirical values ​​obtained from a large number of experiments, and can obtain satisfactory coding effects in most cases.

[0130] The computer vision video content analysis module 1 of the present invention further includes a multimodal feature fusion unit 15. The introduction of this unit greatly improves the system's ability to understand video content and lays a solid foundation for subsequent encoding optimization.

[0131] In a preferred embodiment of the present invention, the multimodal feature fusion unit 15 uses a method combining an attention mechanism and a graph convolutional network (GCN) to achieve efficient feature fusion. Specifically, the unit first represents the spatial domain features (including color, texture, edge, etc.) and the temporal domain features (including motion vectors, optical flow, etc.) as a feature map:

[0132] G=(V,E),

[0133] Among them, V is a set of nodes, each node represents a feature, and E is a set of edges, which represents the relationship between features. Then, the node features are updated through multi-layer graph convolution operations:

[0134]

[0135] Among them, H (l) is the node feature matrix of the lth layer, A is the adjacency matrix, D is the degree matrix, W (l) is a learnable weight matrix, and σ is an activation function. In order to further improve the effect of feature fusion, the present invention introduces a multi-head self-attention mechanism. For each attention head i, the attention weight is calculated

[0136]

[0137] Among them, h i and h j is the node feature, W i and W j is the linear transformation matrix, and a is the learnable attention vector. The final fusion feature is obtained by weighted combination of the outputs of multiple attention heads:

[0138]

[0139] Wherein, K is the number of attention heads. In one embodiment of the present invention, K can be set to 8, which is a good balance between expressive power and computational complexity.

[0140] Through this fusion method, the system of the present invention can make full use of the multimodal information of the video to generate a comprehensive and expressive feature representation. This not only improves the accuracy of subsequent video classification, but also provides a richer basis for the precise tuning of encoding parameters. For example, the system can dynamically adjust the search range of inter-frame prediction based on the motion information in the fused features, or adaptively adjust the quantization parameters based on the texture complexity information in the fused features. This multimodal feature fusion strategy enables the system of the present invention to adapt to different types of video content more intelligently, thereby achieving more efficient encoding optimization.

[0141] The video encoding parameter optimization module 3 of the present invention further includes a face region optimization unit 35. The introduction of this unit reflects the fine-grained understanding and processing capabilities of the system of the present invention for video content, especially the special attention paid to the important visual element of face.

[0142] In a preferred embodiment of the present invention, the face region optimization unit 35 first receives the face region information detected by the computer vision video content analysis module 1. This information usually includes the position, size, posture, etc. of the face. Subsequently, the unit will use a higher encoding quality parameter for the face region to ensure the clarity of the face details. Specifically, the present invention adopts an adaptive quality enhancement strategy:

[0143] QP face =QP base -ΔQP·f(I face ,S face ),

[0144] Among them, QP face is the quantization parameter of the face area, QP base is the basic quantization parameter, ΔQP is the adjustment step size, f is the adjustment function, I face is the face importance score, S face is the face size. Preferably, the adjustment function f can be designed as:

[0145] f(I face ,S fsace )=α·I face +β·log(S face ),

[0146] Wherein, α and β are weight coefficients. In one embodiment of the present invention, ΔQP=6, α=0.7, and β=0.3 may be set. This set of parameters can ensure the clarity of the face while avoiding excessive increase in the encoding bit rate.

[0147] In addition, the face area optimization unit 35 also performs special processing on the area around the face. The present invention adopts a gradual quality adjustment strategy, that is, the encoding quality is gradually reduced from the center of the face to the outside. This strategy can not only ensure the high quality of the face area, but also create a soft transition effect visually. Specifically, for the pixel d away from the center of the face, its quantization parameter can be expressed as:

[0148] QP(d)=QP face +(QP base -QP face )·(1-e -λd ),

[0149] Among them, λ is the attenuation coefficient, which controls the speed of quality transition. In a preferred embodiment of the present invention, λ can be dynamically adjusted according to the resolution of the video and the size of the face to adapt to different scene requirements. Through this detailed face area optimization strategy, the system of the present invention can effectively improve the viewer's perception experience of the characters in the video while ensuring the overall coding efficiency. This is particularly important for application scenarios dominated by characters, such as video conferencing, online education, and news reporting.

[0150] The system of the present invention further comprises a coding efficiency evaluation module 6. This module is in communication with the video coding parameter optimization module 3 and the video coding quality evaluation module 4, and is responsible for evaluating the overall efficiency of the coding process and providing additional feedback for parameter optimization.

[0151] In a preferred embodiment of the present invention, the coding efficiency evaluation module 6 adopts a comprehensive evaluation index, namely, the coding efficiency factor (CEF). The calculation formula of CEF is as follows:

[0152]

[0153] Among them, Q is the video quality score (PSNR, SSIM or the comprehensive quality score mentioned above can be used), R is the bit rate, T is the encoding time, and γ and δ are weight indexes. This indicator takes into account the three aspects of video quality, compression efficiency and encoding speed at the same time, and can fully reflect the efficiency of the encoding process.

[0154] Preferably, the values ​​of γ and δ can be adjusted according to the specific application scenario. For example, for real-time streaming applications, the value of δ can be increased to pay more attention to the encoding speed; while for offline video storage, the value of δ can be reduced to pay more attention to the compression efficiency. In one embodiment of the present invention, γ=0.8 and δ=0.2 can be set. This set of parameters can achieve a good balance in most application scenarios.

[0155] The coding efficiency evaluation module 6 will feed back the calculated CEF value to the video coding parameter optimization module 3. In the parameter optimization process, in addition to considering the video quality, CEF will also be taken as an important optimization target. Specifically, CEF can be incorporated into the reward function of reinforcement learning:

[0156] R=w1·Q+w2·CEF,

[0157] Wherein, w1 and w2 are weight coefficients. In this way, the system of the present invention can continuously improve the overall coding efficiency while ensuring the video quality.

[0158] In addition, the coding efficiency evaluation module 6 also maintains a historical CEF database to record the best CEF values ​​of different types of video content. These historical data can be used to quickly initialize coding parameters and accelerate the optimization process. For example, when the system encounters video content similar to historical data, the historical best parameters can be directly used as the starting point, thereby greatly reducing the time for iterative optimization.

[0159] The present invention also provides an intelligent video coding optimization method based on computer vision. The method includes seven main steps, which fully embodies the workflow and core idea of ​​the system of the present invention.

[0160] Step S1 uses computer vision technology to perform content analysis on the input video to obtain the spatial domain parameters and temporal domain parameters of the video. In a preferred embodiment of the present invention, this step uses a deep convolutional neural network (DCNN) to extract high-level semantic features of video frames. Specifically, a pre-trained ResNet-101 model can be used as a feature extractor, and then a feature vector of fixed dimension is obtained by global average pooling (GAP). For temporal domain features, the present invention uses a 3D convolutional network (such as C3D or I3D) to capture the dynamic information of the video. These two features are fused through the multimodal feature fusion unit described above to obtain a comprehensive video content representation.

[0161] Step S2 uses a deep neural network to classify the video based on the analysis results of step S1. The present invention adopts an innovative two-stage classification strategy in this step. First, a long short-term memory network (LSTM) is used to model the video sequence to capture long-term dependencies. Then, the output features of the LSTM are fed into a multi-task learning framework to simultaneously perform video category classification and scene label prediction. This multi-task learning strategy not only improves the accuracy of classification, but also provides richer semantic information for subsequent coding optimization.

[0162] Step S3 adaptively optimizes the video encoding parameters according to the video classification results and content features. As mentioned above, the present invention adopts a method based on reinforcement learning to achieve adaptive optimization of parameters. In this step, the system dynamically adjusts the reward function in the reinforcement learning algorithm according to the category and content features of the video. For example, for action film-type videos, the bit rate weight can be increased to ensure the clarity of fast-moving scenes; for documentary-type videos, the SSIM weight can be increased to better preserve texture details.

[0163] Step S4 encodes the video using the optimized encoding parameters. In a preferred embodiment of the present invention, the encoding process adopts the latest video coding standards, such as H.265 / HEVC or AV1. In order to further improve the coding efficiency, the present invention also introduces an adaptive coding tool selection mechanism. Specifically, the system dynamically selects the most suitable intra-frame prediction mode, motion estimation algorithm, and deblocking filter parameters according to the characteristics of the video content. This fine-grained tool selection strategy can give full play to the potential of the coding standard and achieve more efficient compression.

[0164] Step S5 evaluates the quality of the encoded video. As mentioned above, the present invention adopts a fusion strategy of multiple evaluation methods, including objective evaluation, subjective evaluation model and deep learning-based evaluation. In this step, the system also considers the special needs of different types of video content. For example, for sports event videos, more attention will be paid to the clarity of dynamic targets; while for landscape documentaries, more attention will be paid to color restoration and texture retention. Through this content-aware quality assessment strategy, the system of the present invention can more accurately reflect the audience's real viewing experience.

[0165] Step S6 further adjusts the encoding parameters through iterative optimization based on the quality assessment results. In this step, the present invention adopts an innovative adaptive learning rate strategy. Specifically, the system dynamically adjusts the step size of parameter updates according to the current optimization state. In the early stage of optimization, a larger learning rate is used to quickly explore the parameter space; as the optimization proceeds, the learning rate is gradually reduced to achieve fine adjustment. This strategy can greatly improve the optimization efficiency while ensuring convergence.

[0166] Step S7 outputs the final optimized encoding parameters and the encoded video. Before outputting the final result, the system of the present invention will also perform an overall performance evaluation to ensure that the optimized encoding parameters have reached the expected goals in all aspects (such as video quality, bit rate, encoding speed, etc.). If it is found that the performance of a certain aspect does not meet the requirements, the system will start a fast fine-tuning process to locally optimize the relevant parameters.

[0167] Through this series of carefully designed steps, the method of the present invention can make full use of computer vision and deep learning technology to achieve intelligent and adaptive optimization of the video encoding process. The method can not only significantly improve the efficiency and quality of video encoding, but also has strong versatility and scalability, and can adapt to various types of video content and application scenarios.

[0168] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. Intelligent video coding optimization system based on computer vision, characterized by ,include: Computer vision video content analysis module for: Perform content analysis on the input video to obtain the spatial domain parameters and temporal domain parameters of the video; Identify objects, actions, and faces in videos; The video classification module based on the neural network is connected to the computer video content analysis module for: Receiving video feature data sent by the computer vision video content analysis module; Based on the video feature data, classify the video using a deep neural network; The video encoding parameter optimization module is communicatively connected with the neural network-based video classification module and is used to: Receiving the video classification result sent by the neural network-based video classification module; Adaptively optimizing video encoding parameters based on the video classification results and video feature data; A video coding quality assessment module is communicatively connected with the video coding parameter optimization module and is used to: Receiving the optimized encoding parameters sent by the video encoding parameter optimization module; evaluating the quality of a video encoded based on the optimized encoding parameters; The evaluation results are fed back to the video encoding parameter optimization module for further optimizing the encoding parameters.

2. The system according to claim 1, characterized in that , the computer vision video content analysis module includes: An object recognition unit, used to identify specific objects in the video; Behavior recognition unit, used to analyze action behaviors in the video; Face detection unit, used to detect and analyze faces in videos; A scene classification unit, used to classify video scenes; Among them, the analysis results of the object recognition unit, behavior recognition unit, face detection unit and scene classification unit together constitute the video feature data.

3. The system according to claim 1, characterized in that ,The neural network based video classification module includes: A feature extraction unit, used for extracting deep features from the video feature data using a convolutional neural network; A deep belief network unit, for classifying videos based on the deep features; The classification result fusion unit is used to fuse the classification result of the deep belief network unit with the analysis result of the computer vision video content analysis module to obtain the final video classification result.

4. The system according to claim 1, characterized in that , the video encoding parameter optimization module includes: A bit rate adjustment unit, used to adaptively adjust the encoding bit rate according to the video classification result and content characteristics; A frame rate optimization unit, used to dynamically adjust the frame rate based on the dynamic characteristics of the video; A macroblock parameter adjustment unit, used for adaptively adjusting the size and number of macroblocks for intra-frame coding and inter-frame coding; A quantization parameter setting unit, used to intelligently set the quantization parameter according to the complexity of the video content; Among them, each unit works together to generate an optimized encoding parameter set.

5. The system according to claim 1, characterized in that , the video coding quality assessment module includes: Objective evaluation unit, used to calculate the peak signal-to-noise ratio PSNR and structural similarity SSIM objective quality indicators; Subjective evaluation unit, used to simulate the human visual system for subjective quality evaluation; A deep learning evaluation unit, used to predict video quality scores using a trained deep neural network; The evaluation result fusion unit is used to weightedly fuse the results of objective evaluation, subjective evaluation and deep learning evaluation to obtain a comprehensive quality score.

6. The system according to claim 1, characterized in that , also includes: The iterative optimization control module is communicatively connected with the video encoding parameter optimization module and the video encoding quality assessment module, and is used to: Receiving an evaluation result of the video coding quality evaluation module; The video encoding parameter optimization module is controlled to perform multiple rounds of iterative optimization until a preset quality target or an upper limit of the number of iterations is reached.

7. The system according to claim 1, characterized in that , the computer vision video content analysis module also includes: Multimodal feature fusion unit, used for: Fusion of spatial domain features, including color, texture, edge, and temporal domain features, including motion vectors and optical flow; Generate comprehensive feature representation to improve the accuracy of video content understanding.

8. The system according to claim 1, characterized in that , the video encoding parameter optimization module also includes: Face area optimization unit, used for: Receiving face region information detected by the computer vision video content analysis module; Use higher encoding quality parameters for the face area to ensure the clarity of facial details.

9. The system according to claim 1, characterized in that , also includes: A coding efficiency evaluation module is communicatively connected with the video coding parameter optimization module and the video coding quality evaluation module, and is used to: Calculate the ratio of encoding bit rate to video quality to evaluate encoding efficiency; The coding efficiency evaluation result is fed back to the video coding parameter optimization module to optimize the bit rate while ensuring the quality.

10. A computer vision-based intelligent video coding optimization method based on the system according to any one of claims 1 to 9, characterized in that , including the following steps: S1, using computer vision technology to analyze the content of the input video and obtain the spatial domain parameters and temporal domain parameters of the video; S2, based on the analysis results of step S1, classify the video using a deep neural network; S3, adaptively optimize video encoding parameters, including bit rate, frame rate, macroblock parameters and quantization parameters, based on video classification results and content features; S4, encoding the video using the optimized encoding parameters; S5, evaluates the quality of the encoded video, including objective evaluation, subjective evaluation and deep learning-based evaluation; S6, based on the quality evaluation result, further adjusting the encoding parameters through iterative optimization until the preset quality target or the upper limit of the number of iterations is reached; S7, outputs the final optimized encoding parameters and encoded video.

Images