Video content adaptive optimization system based on deep learning

Abstract

The invention relates to the technical field of video optimization, and discloses a video content adaptive optimization system based on deep learning, which comprises a video extraction module, a video decoding module, a coding adjustment module, a video backtracking module and a video evaluation module, the video extraction module is configured to obtain video feature data according to the motion complexity feature vector, the scene switching feature vector and the texture feature vector, and the video decoding module is configured to decode the video feature data based on an initial deep learning model to obtain video adjustment parameters. The coding adjustment module is configured to obtain a first output video, the video backtracking module is configured to obtain a target deep learning model and obtain a second output video based on a first output video adjustment parameter, and the video evaluation module is configured to judge whether the video quality of the second output video reaches the standard or not; and if not, obtaining a third output video according to the historical input video adjustment parameters. The method provided by the invention has the characteristics of good adaptability and optimization for different input videos.

Description

Technical Field

[0001] The present invention relates to the field of video optimization technology, and in particular to a video content adaptive optimization system based on deep learning. Background Art

[0002] With the increasing popularity of video content, video coding technology plays a vital role in various video applications. Currently, mainstream video coding technologies use fixed encoding parameters to process video content. This fixed encoding method cannot be optimized for different input video types, resulting in unsatisfactory video quality. Furthermore, when adjusting the encoding to accommodate different scenarios, frame rates, and network conditions, the output video quality is affected not only by the original video content itself but also by factors such as the transmission network bandwidth. This results in inflexible encoding adjustments, leading to video distortion, image freezes, and delays.

[0003] Therefore, how to provide a video content adaptive optimization system based on deep learning is a technical problem that technicians in this field urgently need to solve. Summary of the Invention

[0004] In view of this, the present invention proposes a video content adaptive optimization system based on deep learning, which aims to solve the problem that different types of input videos cannot be dynamically adjusted for encoding and the output video has poor video quality.

[0005] The present invention proposes a video content adaptive optimization system based on deep learning, comprising: Video extraction module, video decoding module, encoding adjustment module, video backtracking module and video evaluation module; The video extraction module is configured to analyze an input video using a motion vector to obtain a motion complexity feature vector, identify scene changes in the input video using a scene detection algorithm to obtain a scene change feature vector, analyze texture details to obtain a texture feature vector, and obtain video feature data based on the motion complexity feature vector, the scene change feature vector, and the texture feature vector. The video decoding module is configured to establish an initial deep learning model, and decode the plurality of video feature data based on the initial deep learning model to obtain video adjustment parameters; The encoding adjustment module is configured to adjust the video adjustment parameter according to the network status of the input video, the scene of the input video and the frame rate of the input video to obtain a first output video; The video tracing module is configured to extract edge image features of the first output video, calculate edge feature values ​​of the first output video, and determine whether the video quality of the first output video meets the requirements based on the edge feature values; If the target is not met, the initial deep learning model is optimized using a gradient descent algorithm to obtain a target deep learning model, the first output video is processed to obtain first video feature data, the first video feature data is substituted into the target deep learning model to obtain first output video adjustment parameters, and a second output video is obtained based on the first output video adjustment parameters; The video evaluation module is configured to demarcate the second output video into a grayscale frame area, extract the video frame of the grayscale frame area, determine the grayscale image brightness value of each pixel based on the video frame, determine the contrast based on all the grayscale image brightness values, and judge whether the video quality of the second output video meets the standards based on the contrast. If not, adjust the parameters based on the historical input video to obtain the third output video.

[0006] Furthermore, the input video is analyzed using motion vectors to obtain a motion complexity feature vector, a scene detection algorithm is used to identify scene switching of the input video to obtain a scene switching feature vector, and texture detail analysis is used to obtain a texture feature vector, including: The video extraction module extracts video pixels of the input video and calculates the motion complexity feature vector based on the video pixels. The motion complexity feature vector is obtained by the following formula: ; in, represents the motion complexity feature vector, Indicates the total number of video pixels. represents the optical flow vector of the i-th video pixel, μ represents the mean of the optical flow vector; The scene detection algorithm obtains a binary variable feature by comparing the scene differences between different frames of the input video. If there is a scene difference, the binary variable feature is equal to 1, and if there is no scene difference, the binary variable feature value is equal to 0, thereby obtaining a scene switching feature vector; The texture detail analysis captures local texture features of the input video by generating binary values ​​from the grayscale values ​​of the video pixels, and obtains a texture feature vector.

[0007] Furthermore, when deriving video feature data according to the motion complexity feature vector, the scene switching feature vector, and the texture feature vector, the method includes: The video feature data is obtained by the following formula: ; in, Represents video feature data, represents the motion complexity feature vector, represents the scene switching feature vector, represents the texture feature vector, represents the weight of the motion complexity feature vector, represents the weight of the scene switching feature vector, represents the weight of the texture feature vector, and .

[0008] Furthermore, the video decoding module is configured to establish an initial deep learning model, and when decoding the plurality of video feature data based on the initial deep learning model to obtain the video adjustment parameters, the method includes: The video decoding module collects a plurality of the video feature data and performs standard processing to obtain standard data, and the standard data is obtained by the following formula: ; in, Indicates standard data, Represents video feature data, Represents the mean of multiple video feature data, Represents the standard deviation of multiple video feature data; The standard data is divided into a training set and a test set. The initial deep learning network model adopts a neural network model. Cross-validation combined with grid search is used to find model parameters of the neural network model, and a neural network model is established. The training set is used to fit the neural network model. The test set is substituted into the neural network model and the accuracy of the video adjustment parameters is calculated. When the accuracy of the video adjustment parameters reaches a preset accuracy threshold, the video decoding module decodes the standard data based on the neural network model to obtain the video adjustment parameters. The video adjustment parameters include a corrected bit rate, a corrected quantization level, and a corrected frame rate.

[0009] Furthermore, the video decoding module decodes the standard data based on the neural network model to obtain video adjustment parameters, and when the video adjustment parameters include a modified bit rate, a modified quantization level, and a modified frame rate, the following steps are included: When the video decoding module obtains the modified bit rate, the modified quantization level, and the modified frame rate through decoding by the neural network model, the neural network model outputs continuous bit rate values, continuous quantization level values, and continuous frame rate values. The continuous bit rate values, continuous quantization level values, and continuous frame rate values ​​are rounded to obtain the modified bit rate, the modified quantization level, and the modified frame rate. The rounding calculation is obtained by the following formula: ; ; ; in, Indicates the rounded bit rate, Indicates the continuous value of bit rate, Indicates the rounding quantization level, Represents a continuous value of the quantization level, Indicates the integer frame rate. represents a continuous value of the frame rate, the rounded bit rate is the modified bit rate, the rounded quantization level is the modified quantization level, and the rounded frame rate is the modified frame rate.

[0010] Furthermore, when the encoding adjustment module is configured to adjust the video adjustment parameters according to the network status of the input video, the scene of the input video, and the frame rate of the input video to obtain the first output video, the encoding adjustment module includes: The encoding adjustment module determines whether to adjust the modified bit rate based on the network status of the input video and obtains a first report; determines whether to adjust the modified quantization level based on the scene of the input video and obtains a second report; determines whether to adjust the modified frame rate based on the frame rate of the input video and obtains a third report; and outputs the first output video based on the first report, the second report and the third report.

[0011] Furthermore, the video tracing module is configured to extract edge image features of the first output video, calculate edge feature values ​​of the first output video, and determine whether the video quality of the first output video meets the standard based on the edge feature values, including: The video tracing module extracts the horizontal gradient and the vertical gradient of the first output video, and the edge feature value is obtained by the following formula:

[0012] in, represents the edge eigenvalue, represents the horizontal gradient, represents the vertical gradient, and represents the weight coefficient, and ; when If the value is greater than or equal to a preset characteristic value, it is determined that the video quality of the first output video meets the standard; when If the value is less than a preset characteristic value, it is determined that the video quality of the first output video does not meet the standard.

[0013] Furthermore, if the target is not met, the initial deep learning model is optimized using a gradient descent algorithm to obtain a target deep learning model, including: The video retracing module substitutes the test set into the neural network model to obtain a predicted value of the video adjustment parameter, and determines a cross entropy loss based on the predicted value and the combined value of the video adjustment parameter. The cross entropy loss is obtained by the following formula:

[0014]

[0015] in, Represents the stapled value, represents the i-th predicted value, represents the cross entropy loss, represents the rounded bit rate, represents the rounding quantization level, represents the rounded frame rate; The target deep learning model is obtained by controlling the learning of the neural network model according to the cross entropy loss.

[0016] Furthermore, the video evaluation module is configured to demarcate the second output video into a grayscale frame region, extract a video frame from the grayscale frame region, determine a grayscale image brightness value of each pixel based on the video frame, determine a contrast ratio based on all grayscale image brightness values, and determine whether the video quality of the second output video meets the quality standards based on the contrast ratio, including: The video evaluation module converts the second output video into a grayscale image, divides the grayscale image into a 2x2 matrix area, sets the matrix area as a grayscale frame area, counts each pixel of the video frame and records the number of all pixels, determines the contrast based on the variance of the grayscale image brightness values ​​of all pixels, and sets a contrast threshold; When the contrast is greater than the contrast threshold, it is determined that the video quality of the second output video is qualified; When the contrast is less than or equal to the contrast threshold, it is determined that the video quality of the second output video is unqualified.

[0017] Furthermore, if the standard is not met, the third output video is obtained by adjusting the parameters according to the historical input video, including: The video evaluation module counts the video adjustment parameters of each input video and establishes a video adjustment parameter set. When the video quality of the second output video is unqualified, the second output video adjustment parameters are determined based on the average of the adjustment parameters of the historical input videos in the video adjustment parameter set, and the third output video is obtained based on the second output video adjustment parameters.

[0018] Compared with the prior art, the beneficial effects of the present invention are: by analyzing the motion complexity, scene switching and texture details of the input video to generate corresponding feature vectors, and decoding and dynamic adjustment based on the deep learning model, adaptive optimization of different types of input videos is achieved. Whether in complex motion states, rapid scene switching and detailed texture content, the system can ensure that the video quality is improved by taking dynamic adjustment encoding, and optimize the deep learning model through the gradient descent algorithm to improve the encoding accuracy, so as to adapt to the needs of different network environments and playback devices, ensure the smooth playback of the first output video, the second output video and the third output video and reduce the impact of network bandwidth limitations on the video quality, and through evaluation of the grayscale image brightness value and contrast, it can ensure that the video quality of the first output video, the second output video and the third output video meets the expected standard, thereby providing efficient video quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings: Figure 1 This is a functional block diagram of a deep learning-based video content adaptive optimization system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, unless there is a conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0021] See Figure 1As shown, this embodiment provides a video content adaptive optimization system based on deep learning, including: a video extraction module, a video decoding module, an encoding adjustment module, a video backtracking module and a video evaluation module. The video extraction module is configured to use motion vector analysis to input video to obtain a motion complexity feature vector, use a scene detection algorithm to identify the scene switching of the input video to obtain a scene switching feature vector, use texture detail analysis to obtain a texture feature vector, and obtain video feature data based on the motion complexity feature vector, the scene switching feature vector and the texture feature vector. The video decoding module is configured to establish an initial deep learning model, decode multiple video feature data based on the initial deep learning model to obtain video adjustment parameters, the encoding adjustment module is configured to adjust the video adjustment parameters according to the network status of the input video, the scene of the input video and the frame rate of the input video to obtain a first output video, and the video backtracking module is configured to obtain a first output video. It is configured to extract edge image features of the first output video, calculate edge feature values ​​of the first output video, and determine whether the video quality of the first output video meets the standards based on the edge feature values. If it does not meet the standards, the initial deep learning model is optimized using a gradient descent algorithm to obtain a target deep learning model, the first output video is processed to obtain first video feature data, and the first video feature data is substituted into the target deep learning model to obtain first output video adjustment parameters, and the second output video is obtained based on the first output video adjustment parameters. The video evaluation module is configured to demarcate the second output video into a grayscale frame area, extract the video frame of the grayscale frame area, determine the grayscale image brightness value of each pixel point based on the video frame, determine the contrast based on all the grayscale image brightness values, and determine whether the video quality of the second output video meets the standards based on the contrast. If it does not meet the standards, the third output video is obtained based on the historical input video adjustment parameters.

[0022] Specifically, the video extraction module is responsible for real-time monitoring and analysis of the input video. By analyzing the motion vector of the input video, the resulting motion complexity feature vector reflects the direction of movement of objects in the input video. The scene detection algorithm is used to identify the scene switching of the input video to obtain the scene switching feature vector, which helps to optimize the encoding strategy of subsequent scene switching and reduce the visual quality loss caused by scene mutation. The texture detail technology is used to analyze the texture complexity of the input video screen and obtain the texture feature vector. The motion complexity feature vector, scene switching feature vector and texture feature vector constitute a comprehensive description of the input video content, providing accurate video feature data for the subsequent optimization of the deep learning model. The video decoding module is responsible for establishing the initial deep learning model, parsing the video feature data and obtaining the video adjustment parameters. Based on the training set, test set and deep learning algorithm, an initial deep learning model that can identify the multi-dimensional features of the input video is established. The initial deep learning model can provide customized video adjustment parameters for different input videos. The encoding adjustment module is responsible for dynamically optimizing the video adjustment parameters and adjusting the video adjustment parameters according to the network status, the picture scene of the input video and the frame rate of the input video to generate a preliminarily optimized first output video. The video backtracking module analyzes the first output video to ensure that the first output video meets the video quality. It uses edge image feature extraction technology to reflect the video quality of the first output video. If the quality of the first output video does not meet the standard, the initial deep learning model is optimized by the gradient descent algorithm to generate a new target deep learning model. The target deep learning model improves the video processing capability. The first output video adjustment parameters are obtained through deep processing of the target deep learning model and the second output video is generated, realizing feedback optimization and improving the video quality of the second output video. The video evaluation module is responsible for comprehensively evaluating the second output video. It extracts the grayscale frame area in the second output video, calculates the grayscale image brightness value of the pixel points in the grayscale frame area to determine the contrast, and evaluates the video quality of the second output video. If it still does not meet the standard, the historical input video adjustment parameters are referred to to generate the third output video, ensuring the video quality of the third output video.

[0023] It can be understood that, combined with the analysis of deep learning models and multi-dimensional feature vectors, the system can derive video adjustment parameters based on different types of input videos, dynamically adjust the input video, and perform corresponding video quality detection, thereby improving the accuracy and adaptability of optimizing the input video content and ensuring that the video quality of the first output video, the second output video, and the third output video meet the standards.

[0024] In some embodiments of the present application, when using motion vector analysis to analyze an input video to obtain a motion complexity feature vector, using a scene detection algorithm to identify scene changes in the input video to obtain a scene change feature vector, and using texture detail analysis to obtain a texture feature vector, the following steps are included: The video extraction module extracts the video pixels of the input video and calculates the motion complexity feature vector based on the video pixels. The motion complexity feature vector is obtained by the following formula: ; in, represents the motion complexity feature vector, Indicates the total number of video pixels. Represents the optical flow vector of the i-th video pixel, μ represents the mean of the optical flow vector. The scene detection algorithm derives binary variable features by comparing the scene differences between different frames of the input video. If there is a scene difference, the binary variable feature is equal to 1, and if there is no scene difference, the binary variable feature value is equal to 0, thereby obtaining the scene switching feature vector. Texture detail analysis captures the local texture features of the input video by generating binary values ​​from the grayscale values ​​of the video pixels, and obtains the texture feature vector.

[0025] It can be understood that the motion complexity feature vector reflects the moving direction of the object in the input video. The scene detection algorithm is used to identify the scene switching of the input video to obtain the scene switching feature vector. The scene detection algorithm obtains the binary variable feature by comparing the scene differences between different frames of the input video. The scene differences include the shape and number of objects in the input video. When the shape and number of objects in the input video change, the binary variable feature is equal to 1. When the shape and number of objects in the input video do not change, the binary variable feature value is equal to 0, and the scene switching feature vector is obtained. For example: the input video time is 5 seconds and 20 frames per second, then the input video will have 100 frames. If scene switching is detected in the 1st frame, the 3rd frame and the 100th frame, the scene switching feature vector is Texture detail analysis captures the local texture features of the input video by converting the grayscale values ​​of the video pixels into binary values. For example, a video pixel in the input video is in a 3x3 pixel matrix, and the pixel matrix is ​​as follows:

[0026] The grayscale value of the video pixel at the center of the pixel matrix is ​​70. The grayscale value of the video pixel at the center of the matrix is ​​compared with the 8 grayscale values ​​around it in clockwise order from the upper left corner. If the grayscale value is greater than or equal to 70, the output is 1, otherwise it is 0. The binary number obtained is 00001110, which is converted to decimal to get 14. 14 is its local texture feature. Assuming there are 3 video pixels, the decimal values ​​are 35, 20, and 16, then the texture feature vector is By comprehensively analyzing the motion complexity, scene switching, and texture details of the input video, a data foundation is laid for the subsequent calculation of video feature data.

[0027] In some embodiments of the present application, when video feature data is derived based on the motion complexity feature vector, the scene switching feature vector, and the texture feature vector, the video feature data is derived using the following formula: ; in, Represents video feature data, represents the motion complexity feature vector, represents the scene switching feature vector, represents the texture feature vector, represents the weight of the motion complexity feature vector, represents the weight of the scene switching feature vector, represents the weight of the texture feature vector, and .

[0028] It is understandable that the video feature data comprehensively considers the motion complexity, scene changes and texture features of the input video. When different types of feature vectors are fused, their corresponding weights can reasonably distribute the weights of the feature vectors, thereby improving the adaptability and accuracy of the adaptive optimization system and providing accurate data support for subsequent decoding.

[0029] In some embodiments of the present application, the video decoding module is configured to establish an initial deep learning model, and decode multiple video feature data based on the initial deep learning model to obtain video adjustment parameters, including: the video decoding module collects multiple video feature data and performs standard processing to obtain standard data, and the standard data is obtained by the following formula: ; in, Indicates standard data, Represents video feature data, Represents the mean of multiple video feature data, Represents the standard deviation of multiple video feature data, divides the standard data into a training set and a test set, adopts a neural network model as the initial deep learning network model, adopts cross-validation combined with grid search to find the model parameters of the neural network model, establishes the neural network model, uses the training set to fit the neural network model, substitutes the test set into the neural network model and calculates the accuracy of the video adjustment parameters, when the accuracy of the video adjustment parameters reaches the preset accuracy threshold, the video decoding module decodes the standard data based on the neural network model to obtain the video adjustment parameters, and the video adjustment parameters include the corrected bit rate, the corrected quantization level and the corrected frame rate.

[0030] Specifically, standardization improves the uniformity of the standard data distribution and prevents numerical errors during the calculation process. If the video feature data spans a large range, data overflow or precision loss may occur. Standardization constrains the resulting standard data to a limited range, reducing the risk of data redundancy and precision loss. During the video decoding process, to improve the accuracy of video adjustment parameters, a deep learning neural network model is used to process the standard data. The standard data is divided into a training set and a test set. The neural network model is initialized, and cross-validation and grid search techniques are used to collect hyperparameters for the neural network model. These parameters are then adjusted to improve decoding accuracy. During training, the neural network model is fitted using the training set. The test set is then substituted into the trained neural network model for validation, and the performance of the neural network model is evaluated by calculating the accuracy of the video adjustment parameters. If the video adjustment parameter accuracy is greater than or equal to a preset accuracy of 70%, the neural network model is considered to have good decoding prediction capabilities. Otherwise, cross-validation and grid search techniques are used to collect hyperparameters for the neural network model and optimize the parameters until the video adjustment parameter accuracy is greater than or equal to the preset accuracy.

[0031] It is understood that when the neural network model reaches the preset accuracy, it is used to decode the standard data and output the predicted modified bit rate, modified quantization level, and modified frame rate. This improves the system's ability to adaptively optimize the input video, ensures smooth playback of subsequent output videos, and improves the video quality of the output videos.

[0032] In some embodiments of the present application, the video decoding module decodes the standard data based on the neural network model to obtain the video adjustment parameters, and the video adjustment parameters include the modified bit rate, the modified quantization level, and the modified frame rate, including: when the video decoding module decodes the modified bit rate, the modified quantization level, and the modified frame rate through the neural network model, the neural network model outputs continuous bit rate values, continuous quantization level values, and continuous frame rate values, and the continuous bit rate values, the continuous quantization level values, and the continuous frame rate values ​​are rounded to obtain the modified bit rate, the modified quantization level, and the modified frame rate, and the rounding calculation is obtained by the following formula: ; ; ; in, Indicates the rounded bit rate, Indicates the continuous value of bit rate, Indicates the rounding quantization level, Represents a continuous value of the quantization level, Indicates the integer frame rate. Indicates the continuous value of frame rate. The rounded bit rate is the corrected bit rate, the rounded quantization level is the corrected quantization level, and the rounded frame rate is the corrected frame rate.

[0033] It's understandable that the bit rate, quantization level, and frame rate obtained by the video decoding module through neural network model decoding are continuous values. However, the bit rate increases in fixed steps, the quantization level is also discrete, and the frame rate is an integer number of frames per second. Effective optimization of the input video requires discrete values. Rounding ensures that the modified bit rate, quantization level, and frame rate meet the discrete limits, improving the optimization of the input video and ensuring the video quality and playback smoothness of the output video.

[0034] In some embodiments of the present application, when the encoding adjustment module is configured to adjust the video adjustment parameters according to the network status of the input video, the scene of the input video and the frame rate of the input video to obtain the first output video, it includes: the encoding adjustment module determines whether to adjust the corrected bit rate according to the network status of the input video to obtain a first report, determines whether to adjust the corrected quantization level according to the scene of the input video to obtain a second report, determines whether to adjust the corrected frame rate according to the frame rate of the input video to obtain a third report, and outputs the first output video according to the first report, the second report and the third report.

[0035] Specifically, the encoding adjustment module has the ability to dynamically adjust the modified bit rate, modified quantization level, and modified frame rate. When the network delay of the input video transmitted to the system is less than 30 milliseconds, it is considered that the network status of the input video is good, and the integer value of the modified bit rate is increased to further improve the video quality of the output video. When the network delay of the input video transmitted to the system is greater than or equal to 30 milliseconds, it is considered that the network status of the input video is poor, and the integer value of the modified bit rate is maintained. While adapting to enhance the video quality of the output video, the network burden is reduced, and a first report is obtained based on the result of maintaining the modified bit rate or increasing the integer value of the modified bit rate. When the scene of the input video is a moving scene, the integer value of the modified quantization level is reduced to retain details. When the scene of the input video is a static scene, the integer value of the modified quantization level is increased to improve the compression rate of the input video. A second report is obtained based on the result of reducing the integer value of the modified quantization level or increasing the integer value of the modified quantization level. When the frame rate of the input video is greater than or equal to 60FPS, the frame rate of the input video is considered to be good, and the integer value of the corrected frame rate is maintained. When the frame rate of the input video is less than 60FPS, the frame rate of the input video is considered to be poor, and the integer value of the corrected frame rate is increased. The third report is obtained based on the result of maintaining the integer value of the corrected frame rate or increasing the integer value of the corrected frame rate.

[0036] It is understood that the encoding adjustment module adjusts the bit rate, quantization level, and frame rate of the input video based on the first report, the second report, and the third report to produce the first output video. By monitoring the video quality and network status of the input video, the modified bit rate, modified quantization level, and modified frame rate can be predicted and adjusted in real time, thereby changing the bit rate, quantization level, and frame rate of the input video to optimize the video quality of the input video and produce the output video. This not only ensures the video quality of the output video, but also effectively utilizes network resources.

[0037] In some embodiments of the present application, the video tracing module is configured to extract edge image features of the first output video, calculate edge feature values ​​of the first output video, and determine whether the video quality of the first output video meets the standard based on the edge feature values, including: the video tracing module extracts the horizontal gradient and vertical gradient of the first output video, and the edge feature values ​​are obtained by the following formula:

[0038] in, represents the edge eigenvalue, represents the horizontal gradient, represents the vertical gradient, and represents the weight coefficient, and ,when is greater than or equal to the preset characteristic value, it is determined that the video quality of the first output video meets the standard. If the value is less than the preset characteristic value, it is determined that the video quality of the first output video does not meet the standard.

[0039] It can be understood that the video retracing module is used to evaluate the video quality of the first output video. Each frame image of the first output video has its corresponding horizontal gradient matrix. The horizontal gradient of the first output video is a time series of the horizontal gradient matrices of multiple frames. Through edge image feature technology, the horizontal gradient of the first output video can be extracted. The vertical gradient extraction is similar. By extracting the horizontal gradient and the vertical gradient, the edge feature value is calculated, and the video quality evaluation of the first output video is realized. The preset feature value is 30. When the edge feature value is less than 30, it is considered that the video quality of the first output video does not meet the standard. Otherwise, it is judged to meet the standard. The integrity and clarity of the first output video image are quantified by edge image feature technology, providing an intuitive quality evaluation standard without the need for human intervention, reducing subjective errors caused by human factors, and flexibly adjusting the weight coefficient and preset threshold. It can adapt to the quality requirements of different input videos for the output video, enhancing the applicability and flexibility of the system.

[0040] In some embodiments of the present application, if the target is not met, the initial deep learning model is optimized using a gradient descent algorithm to obtain a target deep learning model, including: the video backtracking module substitutes the test set into the neural network model to obtain a predicted value of the video adjustment parameter, and determines the cross entropy loss based on the combined value of the predicted value and the video adjustment parameter. The cross entropy loss is obtained by the following formula:

[0041]

[0042] in, Represents the stapled value, represents the i-th predicted value, represents the cross entropy loss, Indicates the rounded bit rate, Indicates the rounding quantization level, Indicates the integer frame rate, and the target deep learning model is obtained by controlling the learning of the neural network model according to the cross entropy loss.

[0043] Specifically, if the video quality of the first output video does not meet the standard, the video retracing module inputs the test set into the neural network model to generate predicted values ​​for the video adjustment parameters. The test set consists of samples from the standard data. The neural network model processes these samples to produce predicted values ​​for the video adjustment parameters for each input video. To accurately evaluate the model's performance, the predicted values ​​are compared with the bounded values, which integrate each of the video adjustment parameters. By comparing the predicted and bounded values, the cross-entropy loss is calculated. Cross-entropy loss is a loss function in deep learning that reflects the difference between the predicted and bounded values. After calculating the cross-entropy loss, the system uses the gradient descent algorithm to optimize the neural network model. Gradient descent is an optimization method that iteratively updates model parameters to reduce cross-entropy loss. At each iteration, the gradient descent algorithm calculates the gradient of the loss function with respect to the neural network model parameters. The gradient descent algorithm adjusts the neural network model parameters using the following formula.

[0044] ; in, Represents the current model parameters of the neural network model, represents the learning rate, Represents the gradient of the loss function with respect to the current model parameters.

[0045] It is understandable that through multiple iterations, the parameters of the current neural network model will be gradually optimized. When the accuracy of the predicted video adjustment parameters is greater than or equal to the preset accuracy of 80%, it is considered that the target deep learning model is obtained. The first output video is used to obtain the first video feature data, and its processing method is consistent with the processing method of the input video, which will not be repeated here. The obtained first video feature data is substituted into the target deep learning model to obtain the first output video adjustment parameters. The first output video adjustment parameters further adjust the bit rate, quantization level and frame rate of the first output video to obtain the second output video. The adjustment process is consistent with the process of obtaining the first output video from the input video, which will not be repeated here. The second output video obtained by processing the first output video further improves the video quality, meets the video quality requirements of different types of input videos, and improves the system's optimization processing capabilities and adaptability to the input video.

[0046] In some embodiments of the present application, the video evaluation module is configured to demarcate the second output video into a grayscale frame area, extract the video frame of the grayscale frame area, determine the grayscale image brightness value of each pixel based on the video frame, determine the contrast based on all the grayscale image brightness values, and judge whether the video quality of the second output video meets the standard based on the contrast, including: the video evaluation module converts the second output video into a grayscale image, divides a 2x2 matrix area on the grayscale image, sets the matrix area as the grayscale frame area, counts each pixel of the video frame and records the number of all pixels, determines the contrast based on the variance of the grayscale image brightness values ​​of all pixels, sets a contrast threshold, when the contrast is greater than the contrast threshold, the video quality of the second output video is determined to be qualified, and when the contrast is less than or equal to the contrast threshold, the video quality of the second output video is determined to be unqualified.

[0047] Specifically, the video evaluation module converts the second output video into a grayscale image. The grayscale image converts the color information of each pixel of the second output video frame into a brightness value. By converting it into a grayscale image, color interference can be removed and the focus can be placed on brightness changes. A 2x2 matrix area is divided on the grayscale image and set as the grayscale frame area. The matrix area contains a certain number of pixels. The grayscale image brightness value of each pixel in the matrix area is counted. The grayscale image brightness value reflects the brightness distribution. A higher grayscale image brightness value indicates that the video image has better contrast and clarity. The contrast is calculated based on the variance of the grayscale image brightness values ​​of all pixels. A lower contrast indicates that the video image has a certain degree of blur or distortion. A contrast threshold is set. When the calculated contrast is greater than the set contrast threshold, the system will determine that the video quality of the second output video is qualified. If the contrast is less than or equal to the contrast threshold, the video quality of the second output video is determined to be unqualified.

[0048] It is understood that the contrast threshold is a manually set threshold for image detection and can be adjusted appropriately. By extracting the grayscale image and calculating the contrast, the consistency and objectivity of the evaluation are guaranteed, ensuring the video quality of the second output video.

[0049] In some embodiments of the present application, if the standard is not met, a third output video is obtained based on the historical input video adjustment parameters, including: a video evaluation module counts the video adjustment parameters of each input video and establishes a video adjustment parameter set, when the video quality of the second output video is unqualified, the adjustment parameter mean of the historical input video in the video adjustment parameter set is determined as the second output video adjustment parameter, and the third output video is obtained based on the second output video adjustment parameter.

[0050] Specifically, the video evaluation module will count the video adjustment parameters of each input video and aggregate these parameters to form a video adjustment parameter set. Different types of input videos will obtain different video adjustment parameters. The video evaluation module records and stores the video adjustment parameters to provide a reference basis for the subsequent video quality evaluation and optimization of the second output video. When the second output video is judged to have unqualified video quality, the video evaluation module will extract the adjustment parameters of all historical input videos from the video adjustment parameter set, calculate the mean of the adjustment parameters of the historical input videos and set it as the adjustment parameters of the second output video. The mean reflects the average adjustment level of multiple input videos under specific conditions, so it can avoid the deviation of the adjustment parameters of a single output video and provide a stable and reliable reference value for adjusting the second output video. The system adjusts the parameters of the second output video, including bit rate, quantization level and frame rate, according to the mean of the adjustment parameters of the historical input videos to generate a third output video with good video quality.

[0051] It can be understood that by introducing the mean of the adjustment parameters of historical input videos, the system can accurately optimize the video quality of the second output video, meet the requirements of different types of input videos for different video qualities, avoid blindly adjusting the video quality of the second output video, and improve the scientificity and rigor of system optimization, so that the system can maintain good adaptability while meeting the requirements of optimized video.

[0052] In summary, the beneficial effects of the present invention are: by analyzing the motion complexity, scene switching and texture details of the input video to generate corresponding feature vectors, and decoding and dynamic adjustment based on the deep learning model, adaptive optimization of different types of input videos is achieved. Whether in complex motion states, rapid scene switching and detailed texture content, the system can ensure that the video quality is improved by taking dynamic adjustment encoding, and optimize the deep learning model through the gradient descent algorithm to improve the encoding accuracy, so as to adapt to the needs of different network environments and playback devices, ensure the smooth playback of the first output video, the second output video and the third output video and reduce the impact of network bandwidth limitations on the video quality, and through evaluation of the grayscale image brightness value and contrast, it can ensure that the video quality of the first output video, the second output video and the third output video meets the expected standard, thereby providing efficient video quality.

[0053] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0054] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0055] These computer program instructions may also be stored in a computer-readable storage device that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable storage device produce an article of manufacture comprising an instruction device that implements the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0056] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A video content adaptive optimization system based on deep learning, characterized in that: include: Video extraction module, video decoding module, encoding adjustment module, video backtracking module and video evaluation module; The video extraction module is configured to use motion vector analysis to obtain a motion complexity feature vector from the input video, use a scene detection algorithm to identify scene switching of the input video to obtain a scene switching feature vector, use texture detail analysis to obtain a texture feature vector, and obtain video feature data based on the motion complexity feature vector, the scene switching feature vector and the texture feature vector; The video decoding module is configured to establish an initial deep learning model, and decode the plurality of video feature data based on the initial deep learning model to obtain video adjustment parameters; The encoding adjustment module is configured to adjust the video adjustment parameter according to the network status of the input video, the scene of the input video and the frame rate of the input video to obtain a first output video; The video tracing module is configured to extract edge image features of the first output video, calculate edge feature values ​​of the first output video, and determine whether the video quality of the first output video meets the standard according to the edge feature values; If the target is not met, the initial deep learning model is optimized by a gradient descent algorithm to obtain a target deep learning model, the first output video is processed to obtain first video feature data, and the first video feature data is substituted into the target deep learning model to obtain a first output video adjustment parameter, and a second output video is obtained based on the first output video adjustment parameter; The video evaluation module is configured to delimit the second output video into a grayscale frame area, extract the video frame of the grayscale frame area, determine the grayscale image brightness value of each pixel based on the video frame, determine the contrast based on all the grayscale image brightness values, and determine whether the video quality of the second output video meets the standards based on the contrast. If not, adjust the parameters based on the historical input video to obtain the third output video.

2. The video content adaptive optimization system based on deep learning according to claim 1, characterized in that: When the motion complexity feature vector is obtained by analyzing the input video using a motion vector, the scene switching of the input video is identified using a scene detection algorithm to obtain a scene switching feature vector, and the texture feature vector is obtained by analyzing the texture details, the method includes: The video extraction module extracts video pixels of the input video, and calculates the motion complexity feature vector according to the video pixels. The motion complexity feature vector is obtained by the following formula: ； in, represents the motion complexity feature vector, Indicates the total number of video pixels. represents the optical flow vector of the i-th video pixel, μ represents the mean of the optical flow vector; The scene detection algorithm obtains a binary variable feature by comparing the scene differences between different frames of the input video, and if there is a scene difference, the binary variable feature is equal to 1, and if there is no scene difference, the binary variable feature value is equal to 0, thereby obtaining a scene switching feature vector; The texture detail analysis captures the local texture features of the input video by generating binary values ​​from the grayscale values ​​of the video pixels, and obtains a texture feature vector.

3. The video content adaptive optimization system based on deep learning according to claim 2, characterized in that: When video feature data is derived according to the motion complexity feature vector, the scene switching feature vector and the texture feature vector, the method includes: The video feature data is obtained by the following formula: ； in, Represents video feature data, represents the motion complexity feature vector, represents the scene switching feature vector, represents the texture feature vector, represents the weight of the motion complexity feature vector, represents the weight of the scene switching feature vector, represents the weight of the texture feature vector, and .

4. The video content adaptive optimization system based on deep learning according to claim 3 is characterized in that: The video decoding module is configured to establish an initial deep learning model, and when decoding the plurality of video feature data based on the initial deep learning model to obtain the video adjustment parameter, it includes: The video decoding module collects a plurality of the video feature data and performs standard processing to obtain standard data, and the standard data is obtained by the following formula: ； in, Indicates standard data, Represents video feature data, Represents the mean of multiple video feature data, Represents the standard deviation of multiple video feature data; The standard data is divided into a training set and a test set. The initial deep learning network model adopts a neural network model. Cross-validation combined with grid search is used to find model parameters of the neural network model, and a neural network model is established. The training set is used to fit the neural network model. The test set is substituted into the neural network model and the accuracy of the video adjustment parameters is calculated. When the accuracy of the video adjustment parameters reaches a preset accuracy threshold, the video decoding module decodes the standard data based on the neural network model to obtain the video adjustment parameters, and the video adjustment parameters include a corrected bit rate, a corrected quantization level, and a corrected frame rate.

5. The video content adaptive optimization system based on deep learning according to claim 4 is characterized in that: The video decoding module decodes the standard data based on the neural network model to obtain the video adjustment parameters, and when the video adjustment parameters include a modified bit rate, a modified quantization level, and a modified frame rate, the video decoding module comprises: When the video decoding module obtains the modified bit rate, the modified quantization level and the modified frame rate by decoding the neural network model, the neural network model outputs continuous values ​​of bit rate, continuous values ​​of quantization level and continuous values ​​of frame rate, and the continuous values ​​of bit rate, quantization level and frame rate are rounded to obtain the modified bit rate, the modified quantization level and the modified frame rate, and the rounding calculation is obtained by the following formula: ； ； ； in, Indicates the rounded bit rate, Indicates the continuous value of bit rate, Indicates the rounding quantization level, Represents a continuous value of the quantization level, Indicates the frame rate. represents a continuous value of the frame rate, the rounded bit rate is the modified bit rate, the rounded quantization level is the modified quantization level, and the rounded frame rate is the modified frame rate.

6. The video content adaptive optimization system based on deep learning according to claim 5, characterized in that: When the encoding adjustment module is configured to adjust the video adjustment parameter according to the network status of the input video, the scene of the input video and the frame rate of the input video to obtain the first output video, it includes: The encoding adjustment module determines whether to adjust the modified bit rate according to the network status of the input video to obtain a first report, determines whether to adjust the modified quantization level according to the scene of the input video to obtain a second report, determines whether to adjust the modified frame rate according to the frame rate of the input video to obtain a third report, and outputs the first report, the second report and the third report as a first output video.

7. The video content adaptive optimization system based on deep learning according to claim 6, characterized in that: The video tracing module is configured to extract edge image features of the first output video, calculate edge feature values ​​of the first output video, and determine whether the video quality of the first output video meets the standard according to the edge feature values, including: The video tracing module extracts the horizontal gradient and the vertical gradient of the first output video, and the edge feature value is obtained by the following formula: in, represents the edge eigenvalue, represents the horizontal gradient, represents the vertical gradient, and represents the weight coefficient, and ; when is greater than or equal to a preset characteristic value, it is determined that the video quality of the first output video meets the standard; when If the value is less than a preset characteristic value, it is determined that the video quality of the first output video does not meet the standard.

8. The video content adaptive optimization system based on deep learning according to claim 7, characterized in that: If the target is not met, the initial deep learning model is optimized using a gradient descent algorithm to obtain a target deep learning model, including: The video tracing module substitutes the test set into the neural network model to obtain a predicted value of the video adjustment parameter, and determines a cross entropy loss according to the predicted value and the combined value of the video adjustment parameter. The cross entropy loss is obtained by the following formula: in, Represents the stapled value, represents the i-th predicted value, represents the cross entropy loss, represents the rounded bit rate, represents the rounding quantization level, represents the rounded frame rate; The target deep learning model is obtained by controlling the learning of the neural network model according to the cross entropy loss.

9. The video content adaptive optimization system based on deep learning according to claim 8, characterized in that: The video evaluation module is configured to delimit the second output video into a grayscale frame area, extract a video frame of the grayscale frame area, determine a grayscale image brightness value of each pixel according to the video frame, determine a contrast according to all grayscale image brightness values, and determine whether the video quality of the second output video meets the standard according to the contrast, including: The video evaluation module converts the second output video into a grayscale image, divides a 2x2 matrix area on the grayscale image, sets the matrix area as a grayscale frame area, counts each pixel of the video frame and records the number of all pixels, determines the contrast according to the variance of the grayscale image brightness values ​​of all pixels, and sets a contrast threshold; When the contrast is greater than the contrast threshold, it is determined that the video quality of the second output video is qualified; When the contrast is less than or equal to the contrast threshold, it is determined that the video quality of the second output video is unqualified.

10. The video content adaptive optimization system based on deep learning according to claim 9, characterized in that: If the standard is not met, the parameters are adjusted according to the historical input video to obtain the third output video, including: The video evaluation module counts the video adjustment parameters of each input video and establishes a video adjustment parameter set. When the video quality of the second output video is unqualified, the second output video adjustment parameters are determined according to the average of the adjustment parameters of the historical input videos in the video adjustment parameter set, and the third output video is obtained according to the second output video adjustment parameters.

Images