Video quality evaluation method and system based on deep learning

Abstract

The invention relates to the technical field of machine learning, and provides a video quality evaluation method and system based on deep learning. Carrying out average pooling on the original video sequence to generate a first dynamic parameter, carrying out attention-based convolution processing on the original video sequence based on the first dynamic parameter, and generating a spatial-temporal feature consisting of a spatial feature and a time feature; performing high-order tensor decomposition on the spatial-temporal characteristics to generate degradation factors; extracting audio features from the original video sequence through a pre-trained audio network, and fusing the audio features and the degeneration factor based on a cross-modal attention mechanism to generate a video quality score; determining a difference parameter based on the video quality score and the manual annotation score, and generating a model parameter based on the difference parameter; and generating an evaluation score of the original video sequence through an evaluation model output by the model parameters. By means of the scheme, the quality score which is objective and close to subjective perception can be output, and efficient and accurate video quality evaluation is achieved.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and more specifically, to a video quality evaluation method and system based on deep learning. Background Technology

[0002] With the widespread adoption of 5G, video has become a core information carrier. However, video is susceptible to distortion during acquisition, compression, and transmission due to physical environmental interference, such as network jitter, hardware limitations leading to blurring, stuttering, or screen tearing, directly impacting user experience. This is especially true in underground coal mine applications, where video acquisition is difficult and has a low success rate, making it impossible to extract complete information from limited video resources.

[0003] Traditional methods rely on manual annotation or simple indicators, which are difficult to accurately quantify complex distortions and have a low accuracy rate in video quality evaluation. Therefore, videos captured by non-professionals or in complex environments require more intelligent quality assessment methods. Summary of the Invention

[0004] This application provides a video quality evaluation method and system based on deep learning, which can at least partially solve the problem of low accuracy in video quality evaluation.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to one aspect of this application, a deep learning-based video quality assessment method is provided, comprising: acquiring an original video sequence; performing information compression processing on the original video sequence to generate a first dynamic parameter; performing attention-based convolution processing on the original video sequence based on the first dynamic parameter to generate spatiotemporal features composed of spatial and temporal features; performing high-order tensor decomposition on the spatiotemporal features to generate a degradation factor; extracting audio features from the original video sequence through a pre-trained audio network; fusing the audio features and the degradation factor based on a cross-modal attention mechanism to generate a video quality score; determining a difference parameter based on the video quality score and a manually labeled score; generating model parameters based on the difference parameter; and generating an evaluation score for the original video sequence through an evaluation model output by the model parameters.

[0007] In this application, based on the aforementioned scheme, the step of performing information compression processing on the original video sequence to generate a first dynamic parameter, and performing attention-based convolution processing on the original video sequence based on the first dynamic parameter to generate spatiotemporal features composed of spatial and temporal features, includes: performing information compression processing on video frames in the original video sequence to generate the first dynamic parameter, and extracting spatial features from the video frames based on the first dynamic parameter; compressing the spatial features to generate a first vector, and extracting temporal features from the video frames based on the first vector.

[0008] In this application, based on the aforementioned scheme, the step of performing information compression processing on video frames in the original video sequence to generate a first dynamic parameter, and extracting spatial features from the video frames based on the first dynamic parameter, includes: performing information compression processing and multi-layer sensing processing on video frames in the original video sequence to generate a first dynamic parameter a. i for:

[0009]

[0010] Where GAP(·) represents information compression processing, MLP1(·) and MLP2(·) represent multilayer sensing processing, v t Let t represent the video frame, i and j represent the identifiers of the spatial convolution kernels, and K represent the number of spatial convolution kernels; based on the first dynamic parameters and the spatial convolution kernels in the spatial dimension, spatial features F are extracted from the video frame. s (t) is:

[0011]

[0012] Among them, Conv si This represents the i-th spatial convolution kernel, which only modulates the video frame v in the spatial dimension. t Perform the operation to extract local spatial information.

[0013] In this application, based on the aforementioned scheme, the step of compressing the spatial features to generate a first vector and extracting temporal features from the video frame based on the first vector includes: compressing the spatial features to generate a temporal feature vector as the first vector; inputting the first vector into a preset number of multilayer perceptrons for multilayer perceptron processing to generate temporal weights; and extracting temporal features from the video frame in the temporal dimension based on the temporal weights and a preset number of temporal convolution kernels.

[0014] In this application, based on the aforementioned scheme, the step of performing high-order tensor decomposition on the spatiotemporal features to generate a degradation factor includes: obtaining a target value corresponding to the spatiotemporal features at a preset position; and performing high-order tensor decomposition on the spatiotemporal features based on the target value to generate a degradation factor.

[0015] In this application, based on the aforementioned scheme, the step of extracting audio features from the original video sequence using a pre-trained audio network, fusing the audio features and the degradation factor based on a cross-modal attention mechanism, and generating a video quality score includes: extracting audio features from the original video sequence using a pre-trained audio network; determining an attention factor based on a learnable projection matrix, and enhancing the degradation factor using the attention factor and the audio features to output an enhanced degradation factor; and calculating the video quality score based on the enhanced degradation factor.

[0016] In this application, based on the aforementioned scheme, the step of determining the difference parameter based on the video quality score and the manually labeled score, and generating model parameters based on the difference parameter, includes: comparing the video quality score and the manually labeled score to determine the difference parameter; and adjusting the model parameters using the difference parameter.

[0017] According to one aspect of this application, a deep learning-based video quality assessment system is provided, comprising:

[0018] The acquisition unit is used to acquire the original video sequence, perform information compression processing on the original video sequence to generate a first dynamic parameter, and perform attention-based convolution processing on the original video sequence based on the first dynamic parameter to generate spatiotemporal features composed of spatial features and temporal features.

[0019] The decomposition unit is used to perform high-order tensor decomposition on the spatiotemporal features to generate degradation factors;

[0020] The score unit is used to extract audio features from the original video sequence through a pre-trained audio network, and fuse the audio features and the degradation factor based on a cross-modal attention mechanism to generate a video quality score;

[0021] The parameter unit is used to determine the difference parameters based on the video quality score and the manually labeled score, and to generate model parameters based on the difference parameters.

[0022] The evaluation unit is used to generate an evaluation score for the original video sequence using the evaluation model output by the model parameters.

[0023] According to one aspect of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the deep learning-based video quality evaluation method as described in the above embodiments.

[0024] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the deep learning-based video quality evaluation method as described in the above embodiments.

[0025] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the deep learning-based video quality assessment method provided in the various alternative implementations described above.

[0026] In this application's technical solution, the original video sequence is acquired, information compression processing is performed on the original video sequence to generate a first dynamic parameter, attention-based convolution processing is performed on the original video sequence based on the first dynamic parameter to generate spatiotemporal features composed of spatial and temporal features; high-order tensor decomposition is performed on the spatiotemporal features to generate a degradation factor; audio features are extracted from the original video sequence through a pre-trained audio network, and the audio features and the degradation factor are fused based on a cross-modal attention mechanism to generate a video quality score; difference parameters are determined based on the video quality score and manually labeled scores, and model parameters are generated based on the difference parameters; an evaluation score for the original video sequence is generated through an evaluation model output by the model parameters. By generating spatiotemporal features to balance computational efficiency and focus on key regions; decoupling degradation factors through high-order tensor decomposition to quantify the independent impact of different degradation types and improve the interpretability of mixed degradation; fusing audio features and degradation factors through cross-modal attention to compensate for the limitations of single-modality and enhance the detection of issues such as audio-visual asynchrony; and finally, dynamically adjusting model parameters through difference parameters and suppressing annotation noise by combining uncertainty modeling to output a quality score that is both objective and close to subjective perception, thus achieving efficient and accurate video quality evaluation.

[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0029] Figure 1 The flowchart illustrating a deep learning-based video quality evaluation method in one embodiment of this application is shown.

[0030] Figure 2 A flowchart illustrating the extraction of time features is shown in one embodiment of this application.

[0031] Figure 3 The illustration shows a schematic diagram of a deep learning-based video quality evaluation system in one embodiment of this application.

[0032] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0033] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0034] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0037] The implementation details of the technical solution of this application are described below:

[0038] Figure 1A flowchart of a deep learning-based video quality assessment method according to an embodiment of this application is shown. (Refer to...) Figure 1 As shown, this deep learning-based video quality assessment method includes at least steps S110 to S150, which are detailed below:

[0039] S110, acquire the original video sequence, perform information compression processing on the original video sequence to generate a first dynamic parameter, and perform attention-based convolution processing on the original video sequence based on the first dynamic parameter to generate spatiotemporal features composed of spatial features and temporal features.

[0040] In this embodiment, the original video sequence is first acquired, then decomposed into a continuous frame sequence by a video decoder, and preprocessing operations such as normalization and size alignment are performed. Subsequently, average pooling is applied to the preprocessed video frames, and the average pixel value of local regions is calculated using a sliding window to generate a first dynamic parameter characterizing the global statistical properties of the video. This parameter can be understood as a summary feature of the video content, used to guide the weight allocation of the subsequent attention mechanism. A spatial-temporal attention map is constructed based on this dynamic parameter, and weighted convolution processing is performed on the video frames. The attention mechanism adjusts the attention region of the convolution kernel according to the dynamic parameter, enabling the model to focus on the motion trajectory or texture details of key objects. Finally, a three-dimensional convolutional layer fuses spatial features (such as object edges and color distribution) and temporal features (such as motion vectors and inter-frame variations), outputting a joint feature representation containing spatiotemporal dimensions, providing structured input for subsequent quality assessment.

[0041] In one embodiment of this application, the original video sequence is subjected to information compression processing to generate a first dynamic parameter. Based on the first dynamic parameter, the original video sequence is subjected to attention-based convolution processing to generate spatiotemporal features composed of spatial features and temporal features. This includes: performing information compression processing on video frames in the original video sequence to generate the first dynamic parameter, and extracting spatial features from the video frames based on the first dynamic parameter; compressing the spatial features to generate a first vector, and extracting temporal features from the video frames based on the first vector.

[0042] In one embodiment of this application, information compression processing is performed on video frames in the original video sequence to generate first dynamic parameters, and spatial features are extracted from the video frames based on the first dynamic parameters. This includes: performing information compression processing and multi-layer perceptron processing on video frames in the original video sequence to generate first dynamic parameters; and extracting spatial features from the video frames in the spatial dimension based on the first dynamic parameters and spatial convolution kernels.

[0043] like Figure 2As shown, in one embodiment of this application, the spatial features are compressed to generate a first vector, and temporal features are extracted from the video frame based on the first vector, including:

[0044] S210, The spatial features are compressed to generate a time feature vector as the first vector;

[0045] S220, the first vector is input into a preset number of multilayer perceptrons for multilayer perceptron processing to generate time weights;

[0046] S230, based on the time weights and a preset number of time convolution kernels, extract time features from the video frames in the time dimension.

[0047] In existing technologies, traditional methods for video quality assessment often combine spatiotemporal features. However, this approach has significant drawbacks because the spatiotemporal correlation of quality degradation factors (such as blur and jitter) in the video is disrupted, making it impossible to accurately capture the true state of video quality. This module aims to achieve spatiotemporal decoupling through separable convolution and attention mechanisms to better analyze video quality.

[0048] In one embodiment of this application, the original video sequence is subjected to information compression and multilayer perceptron processing to generate a first dynamic parameter. Global information compression is performed on the video frames in the original video sequence to obtain a compressed second vector. This second vector is then input into two multilayer perceptrons for multilayer perceptron processing. The first dynamic parameter 'a' of each spatial convolution kernel is calculated using an exponential function and a normalization operation. i for:

[0049]

[0050] Where GAP(·) represents information compression processing, MLP1(·) and MLP2(·) represent multilayer sensing processing, v t Let t represent the video frame, i and j represent the identifiers of the spatial convolution kernels, and K represent the number of spatial convolution kernels.

[0051] In one embodiment of this application, a spatial convolution kernel is used to extract spatial local information from video frames in the spatial dimension. The spatial local information is then multiplied by the corresponding dynamic weights and summed to obtain the spatial features F of the t-th frame through an activation function. s (t) is:

[0052]

[0053] Among them, Conv si This represents the i-th spatial convolution kernel, which only modulates the video frame v in the spatial dimension.t Perform the operation to extract local spatial information.

[0054] Next, the extracted spatial features F s (t) is compressed to obtain a compressed temporal feature vector for the t-th frame, which is used as the first vector. Then, the first vector is input into a predetermined number of multilayer perceptrons for multilayer sensing processing. The temporal weight β of each temporal convolution kernel is calculated using an exponential function and normalization operations. k for:

[0055]

[0056] Among them, GMP(F s (t) represents the spatial feature F s (t) is the first vector obtained by compression; MLP k (·) and MLP j (·) indicates multi-layer sensing processing.

[0057] Then, a predetermined number of temporal convolution kernels are used to perform convolution operations on a continuous frame sequence centered at frame t. The convolution results are then multiplied by the corresponding temporal weights and summed to obtain the temporal feature F of frame t. t (t) is:

[0058]

[0059] Where k and L represent the identifier and number of temporal convolution kernels, respectively, Conv tk This represents the k-th temporal convolution kernel, which operates on 2k+1 consecutive frames centered at frame t (i.e., [v... t-k ,…,v t+k The operation is performed to extract motion information in the time dimension.

[0060] In this embodiment, the spatiotemporal feature decoupling module can extract independent features of the video in both spatial and temporal dimensions. Spatial feature extraction focuses on local texture and structural information within video frames. Based on dynamic parameters and spatial convolution kernels, it extracts local texture, edges, and other information in the spatial dimension. The attention mechanism allows the model to focus on salient regions, such as object outlines, reducing redundant computation. Temporal feature extraction, on the other hand, focuses on the motion continuity between frames. It generates temporal vectors by compressing spatial features and combines them with temporal convolution kernels to capture motion information between frames, such as rapid motion or scene transitions, improving sensitivity to temporal degradation.

[0061] The decoupling method in this embodiment avoids the problem of mutual interference between spatiotemporal features in traditional hybrid processing methods, enabling the model to more accurately capture quality degradation factors in the video, providing more accurate feature representations for subsequent video quality evaluation, thereby improving the accuracy and reliability of video quality evaluation.

[0062] S120, perform high-order tensor decomposition on the spatiotemporal features to generate a degradation factor.

[0063] In this embodiment, a spatiotemporal feature tensor containing spatial features (such as object edges and texture details) and temporal features (such as motion trajectories and inter-frame variations) is input into the decomposition module. High-order tensor decomposition methods such as Tucker decomposition are used to decompose it into an interactive combination of a core degradation pattern and multiple low-dimensional degradation factors. For example, through third-order tensor decomposition, the spatiotemporal features can be decomposed into the product of spatial degradation factors, temporal degradation factors, and degradation type interaction factors. Each factor matrix corresponds to a degradation type, such as blurring, compression artifacts, or jitter. Finally, a degradation factor tensor incorporating multi-dimensional degradation information is generated by combining a nonlinear activation function and a noise term. By decoupling complex degradation patterns, the model can more accurately quantify the independent impact of different degradation types on quality, while reducing computational complexity and improving generalization ability for mixed degradation scenarios.

[0064] In one embodiment of this application, the spatiotemporal features are subjected to high-order tensor decomposition to generate a degradation factor, including:

[0065] Obtain the target value corresponding to the spatiotemporal feature at the preset location;

[0066] Based on the target value, the spatiotemporal features are subjected to high-order tensor decomposition to generate a degradation factor.

[0067] In one embodiment of this application, the main objective of the quality degradation factor modeling module is to further process the decoupled spatiotemporal features to model the impact of different degradation types on quality. First, the decoupled spatiotemporal features F are received. s and F t These two features represent information in the spatial and temporal dimensions, respectively, and are data with specific structures and meanings obtained through preprocessing.

[0068] Obtain the target value corresponding to the spatiotemporal feature at a preset location, and perform high-order tensor decomposition on the spatiotemporal feature based on the target value to generate a degradation factor D. i,j,k,m for:

[0069]

[0070] Among them, F s (i,p) represents the spatial feature F sAt the target value at position (i,p), F t (j,q) represents the time feature F t At the target value at position (i,p), γ p,q,m ε represents the interaction weights of the learnable degradation type m in the spatial dimension p and the temporal dimension q, where P and Q represent the number of features in the spatial and temporal dimensions, respectively. i,j,k,m This represents the Gaussian noise term.

[0071] In this embodiment of the application, spatiotemporal features are mapped to a degenerate type space through tensor product, where each element D... i,j,k,m This represents the intensity of the spatiotemporal location (i,j) under degradation type m, providing a fine-grained basis for subsequent quality prediction. Through higher-order tensor decomposition, the interactions between degradation factors can be modeled more meticulously, fully considering the complex relationships between different degradation types in the spatial and temporal dimensions.

[0072] The above process, through tensor decomposition and the introduction of interaction weights, can explain the contributions of different degradation types on different dimensions, enhancing the interpretability of the model. By considering the interactions of different degradation types, the impact of various degradation factors on the final quality can be captured more accurately, generating degradation factors that accurately reflect the influence of different degradation types, thereby improving the accuracy of quality assessment.

[0073] S130, audio features are extracted from the original video sequence through a pre-trained audio network, and the audio features and the degradation factor are fused based on a cross-modal attention mechanism to generate a video quality score.

[0074] In one embodiment of this application, a pre-trained audio network is first used to extract audio features from the original video sequence. These features include spectrograms, Mel-frequency cepstral coefficients, or rhythmic patterns, etc., to quantify the rhythm, pitch, and background noise characteristics of the audio. Then, a cross-modal attention mechanism is used to fuse the audio features with previously generated degradation factors, such as blurring, compression artifacts, and other visual degradation patterns. The correlation weights between the audio and visual degradation factors are dynamically calculated to highlight modal information that significantly impacts quality. Finally, the fused multimodal features are input into a regression model, and a comprehensive video quality score is generated through nonlinear transformation. This score simultaneously reflects the combined impact of audio interference and visual degradation on overall quality, achieving a quality assessment that more closely resembles human subjective perception.

[0075] In one embodiment of this application, audio features are extracted from the original video sequence using a pre-trained audio network, and the audio features and the degradation factor are fused based on a cross-modal attention mechanism to generate a video quality score, including:

[0076] Audio features are extracted from the original video sequence using a pre-trained audio network;

[0077] Attention factors are determined based on a learnable projection matrix, and the degradation factor is enhanced using the attention factors and the audio features to output the enhanced degradation factor.

[0078] The video quality score is calculated based on the enhancement degradation factor.

[0079] In one embodiment of this application, the multimodal interaction quality prediction module aims to comprehensively consider degradation factors and audio features to predict video quality scores, overcoming the limitation of traditional methods that ignore the impact of audio on video quality perception. Audio features are extracted from the original video sequence through a pre-trained audio network.

[0080] In one embodiment of this application, the attention factor Attn(A,F) is determined based on the learnable projection matrix as follows:

[0081]

[0082] Among them, W q W k D represents the learnable projection matrix. k Here, A represents the attention dimension, F represents the audio features in the original video sequence, and softmax represents the normalization process.

[0083] In one embodiment of this application, the degradation factor is audio-enhanced using the attention factor and the audio features, and an enhanced degradation factor D′ is output. i,j,k,m for:

[0084] D′ i,j,k,m =D i,j,k,m +δ m ·Attn[A,F s (i)]·Attn[A,F t (j)]

[0085] Where, δ m Attn[A,F] represents the audio sensitivity coefficient for degradation type m. s (i)]、Attn[A,F t [j] represents the cross-modal attention function, and represents the audio feature A and the spatial feature F, respectively. s At position i, audio feature A and time feature F t Attention calculation result at position j.

[0086] In one embodiment of this application, the video quality score Q is calculated based on the enhancement degradation factor as follows:

[0087]

[0088] Where σ represents the Sigmoid function, ω m The global weights represent the degradation type m, ReLU represents the linear rectified function, and θ n,m The coefficients represent the nonlinear coefficients associated with the degradation type m and the polynomial order n, where N represents the polynomial order. This formula yields a comprehensive quality score by performing a multi-dimensional weighted summation and nonlinear transformation on the enhanced degradation factors.

[0089] In real-world scenarios, video quality is often influenced by a combination of factors, including visual degradation and audio issues. This module can handle this multimodal information, enabling the model to better adapt to complex real-world application scenarios. The technical solution in this application, through a cross-modal attention mechanism, fuses audio features with degradation factors. The multimodal interaction quality prediction module effectively integrates degradation factors and audio features, allowing the model to comprehensively consider audiovisual information, calculate accurate video quality scores, and more accurately predict video quality, providing reliable prediction results for video quality assessment.

[0090] S140, determine the difference parameters based on the video quality score and the manually labeled score, and generate model parameters based on the difference parameters.

[0091] In this technical solution, based on the predicted video quality score and the manually labeled score, the difference parameter between the two is first calculated. The absolute difference between the median of the predicted score and the labeled score can be used to measure individual bias, and the global difference distribution can be statistically analyzed. Then, a dynamic weight adjustment strategy is used to assign lower weights to samples with larger differences, suppressing the interference of abnormal labels on model training. Simultaneously, uncertainty modeling is introduced, using a multilayer perceptron to quantify the credibility of the prediction results, incorporating uncertainty as part of the optimization objective, making the model's predictions for ambiguous samples more conservative. Finally, combining dynamic weights and uncertainty loss, the model parameters are iteratively updated through backpropagation, ensuring that the optimized parameters can accurately fit the effective labeled data and possess noise resistance, thereby improving the generalization and robustness of the video quality evaluation model.

[0092] In one embodiment of this application, determining a difference parameter based on the video quality score and the manually labeled score, and generating model parameters based on the difference parameter, includes:

[0093] The video quality score and the manually labeled score are compared to determine the difference parameters;

[0094] The model parameters are adjusted using the difference parameters.

[0095] In one embodiment of this application, the process begins by receiving two sets of core data: one is the video quality score Q predicted by the preceding module, and the other is the manually labeled true quality score Q′. These data form the basis for model optimization. The video quality score Q reflects the algorithm's assessment of video quality, while the manually labeled true quality score Q′ represents the standard of human subjective evaluation. By comparing these two sets of data, the difference parameters between the predicted results and the manually labeled data are identified, providing a basis for subsequent optimization.

[0096] Optionally, the difference between the median of the video quality score and the true quality score can be used as the difference parameter.

[0097] To mitigate the impact of potential noise from manual annotations, such as mislabeling or subjective bias, on the model, a dynamic weight adjustment strategy is introduced. A weight value is calculated for each sample, inversely proportional to the difference parameter. The larger the difference parameter, indicating a more likely anomalous annotation, the smaller the weight. This dynamic weighting significantly suppresses the contribution of anomalous annotations to the total loss, allowing the model to focus more on highly consistent annotated samples. The predicted quality score and the manually annotated score serve as inputs, forming the basis for model optimization. By comparing the predicted and actual annotated values, the module identifies the differences, providing a basis for subsequent adjustments. Based on the difference between the median predicted score and the median manually annotated score for each sample, the weight of anomalous annotations (such as mislabeling or subjective bias) is automatically reduced, ensuring the model focuses more on highly consistent annotated samples and avoids interference from noisy data.

[0098] Simultaneously, uncertainty modeling is introduced. By combining spatiotemporal and audio features through a multilayer perceptron, the uncertainty of the model's predictions is calculated. This uncertainty is used to adjust the optimization objective, making the model more cautious when facing ambiguous or contradictory labels and preventing overfitting. By simultaneously optimizing the dynamic weight loss and the uncertainty objective, the adjusted model parameters are output. This process not only improves the model's robustness to label noise but also enhances its generalization ability on unseen data, ensuring more reliable prediction results.

[0099] S150, the evaluation score of the original video sequence is generated by the evaluation model output by the model parameters.

[0100] In one embodiment of this application, the evaluation model can perform end-to-end quality assessment of the original video sequence through optimized model parameters: First, the spatiotemporal features, degradation factors, and audio features of the video are input into the model. The model performs weighted fusion of multimodal features based on the learned parameters. The dynamic weighting mechanism reduces the influence of abnormal features, and uncertainty modeling improves the accuracy of judging blurred regions. Then, the fused features are mapped to a quality score through a nonlinear regression layer. This score comprehensively reflects the video's visual degradation (such as blurring and compression artifacts), audio interference (such as noise and audio-visual asynchrony), and cross-modal consistency, and finally outputs an objective evaluation score that is close to human subjective perception.

[0101] Optionally, after generating the evaluation score of the video, a target video with an evaluation score greater than a set threshold is obtained, and image frames are extracted from the target video for subsequent image recognition, image extraction, and pattern recognition processes.

[0102] In one embodiment of this application, an original video sequence is acquired, information compression processing is performed on the original video sequence to generate a first dynamic parameter, attention-based convolution processing is performed on the original video sequence based on the first dynamic parameter to generate spatiotemporal features composed of spatial and temporal features, high-order tensor decomposition is performed on the spatiotemporal features to generate a degradation factor, audio features are extracted from the original video sequence through a pre-trained audio network, the audio features and the degradation factor are fused based on a cross-modal attention mechanism to generate a video quality score, difference parameters are determined based on the video quality score and manually labeled scores, model parameters are generated based on the difference parameters, and an evaluation score for the original video sequence is generated through an evaluation model output by the model parameters. By generating spatiotemporal features to balance computational efficiency and focus on key regions; decoupling degradation factors through high-order tensor decomposition to quantify the independent impact of different degradation types and improve the explanatory power of mixed degradation; fusing audio features and degradation factors through cross-modal attention to compensate for the limitations of single-modal approaches and enhance the detection of issues such as audio-visual asynchrony; and finally, dynamically adjusting model parameters through difference parameters and suppressing annotation noise by combining uncertainty modeling, the system outputs a quality score that is both objective and close to subjective perception through AI-based reinforcement learning and machine learning, achieving efficient and accurate video quality evaluation.

[0103] The following describes embodiments of the deep learning-based video quality assessment system of this application, which can be used to execute the deep learning-based video quality assessment method in the above embodiments of this application. It is understood that the deep learning-based video quality assessment system can be a computer program (including program code) running on a computer device; for example, the deep learning-based video quality assessment system is an application software. This deep learning-based video quality assessment system can be used to execute the corresponding steps in the method provided in the embodiments of this application. For details not disclosed in the embodiments of the deep learning-based video quality assessment system of this application, please refer to the embodiments of the deep learning-based video quality assessment method of this application described above.

[0104] Figure 3 A block diagram of a deep learning-based video quality assessment system according to an embodiment of this application is shown.

[0105] Reference Figure 3 As shown, a deep learning-based video quality evaluation system according to an embodiment of this application includes:

[0106] The acquisition unit 310 is used to acquire the original video sequence, perform information compression processing on the original video sequence to generate a first dynamic parameter, and perform attention-based convolution processing on the original video sequence based on the first dynamic parameter to generate spatiotemporal features composed of spatial features and temporal features.

[0107] Decomposition unit 320 is used to perform high-order tensor decomposition on the spatiotemporal features to generate degradation factors;

[0108] The scoring unit 330 is used to extract audio features from the original video sequence through a pre-trained audio network, and fuse the audio features and the degradation factor based on a cross-modal attention mechanism to generate a video quality score.

[0109] Parameter unit 340 is used to determine difference parameters based on the video quality score and the manually labeled score, and to generate model parameters based on the difference parameters;

[0110] Evaluation unit 350 is used to generate an evaluation score for the original video sequence through the evaluation model output by the model parameters.

[0111] In this application, based on the aforementioned scheme, the step of performing information compression processing on the original video sequence to generate a first dynamic parameter, and performing attention-based convolution processing on the original video sequence based on the first dynamic parameter to generate spatiotemporal features composed of spatial and temporal features, includes: performing information compression processing on video frames in the original video sequence to generate the first dynamic parameter, and extracting spatial features from the video frames based on the first dynamic parameter; compressing the spatial features to generate a first vector, and extracting temporal features from the video frames based on the first vector.

[0112] In this application, based on the aforementioned scheme, the step of performing information compression processing on video frames in the original video sequence to generate a first dynamic parameter, and extracting spatial features from the video frames based on the first dynamic parameter, includes: performing information compression processing and multi-layer sensing processing on video frames in the original video sequence to generate a first dynamic parameter a. i for:

[0113]

[0114] Where GAP(·) represents information compression processing, MLP1(·) and MLP2(·) represent multilayer sensing processing, v t Let t represent the video frame, i and j represent the identifiers of the spatial convolution kernels, and K represent the number of spatial convolution kernels; based on the first dynamic parameters and the spatial convolution kernels in the spatial dimension, spatial features F are extracted from the video frame. s (t) is:

[0115]

[0116] Among them, Conv si This represents the i-th spatial convolution kernel, which only modulates the video frame v in the spatial dimension. t Perform the operation to extract local spatial information.

[0117] In this application, based on the aforementioned scheme, the step of compressing the spatial features to generate a first vector and extracting temporal features from the video frame based on the first vector includes: compressing the spatial features to generate a temporal feature vector as the first vector; inputting the first vector into a preset number of multilayer perceptrons for multilayer perceptron processing to generate temporal weights; and extracting temporal features from the video frame in the temporal dimension based on the temporal weights and a preset number of temporal convolution kernels.

[0118] In this application, based on the aforementioned scheme, the step of performing high-order tensor decomposition on the spatiotemporal features to generate a degradation factor includes: obtaining a target value corresponding to the spatiotemporal features at a preset position; and performing high-order tensor decomposition on the spatiotemporal features based on the target value to generate a degradation factor.

[0119] In this application, based on the aforementioned scheme, the step of extracting audio features from the original video sequence using a pre-trained audio network, fusing the audio features and the degradation factor based on a cross-modal attention mechanism, and generating a video quality score includes: extracting audio features from the original video sequence using a pre-trained audio network; determining an attention factor based on a learnable projection matrix, and enhancing the degradation factor using the attention factor and the audio features to output an enhanced degradation factor; and calculating the video quality score based on the enhanced degradation factor.

[0120] In this application, based on the aforementioned scheme, the step of determining the difference parameter based on the video quality score and the manually labeled score, and generating model parameters based on the difference parameter, includes: comparing the video quality score and the manually labeled score to determine the difference parameter; and adjusting the model parameters using the difference parameter.

[0121] In one embodiment of this application, an original video sequence is acquired, information compression processing is performed on the original video sequence to generate a first dynamic parameter, attention-based convolution processing is performed on the original video sequence based on the first dynamic parameter to generate spatiotemporal features composed of spatial and temporal features, high-order tensor decomposition is performed on the spatiotemporal features to generate a degradation factor, audio features are extracted from the original video sequence through a pre-trained audio network, the audio features and the degradation factor are fused based on a cross-modal attention mechanism to generate a video quality score, difference parameters are determined based on the video quality score and manually labeled scores, model parameters are generated based on the difference parameters, and an evaluation score for the original video sequence is generated through an evaluation model output by the model parameters. By generating spatiotemporal features to balance computational efficiency and focus on key regions; decoupling degradation factors through high-order tensor decomposition to quantify the independent impact of different degradation types and improve the interpretability of mixed degradation; fusing audio features and degradation factors through cross-modal attention to compensate for the limitations of single-modality and enhance the detection of issues such as audio-visual asynchrony; and finally, dynamically adjusting model parameters through difference parameters and suppressing annotation noise by combining uncertainty modeling to output a quality score that is both objective and close to subjective perception, thus achieving efficient and accurate video quality evaluation.

[0122] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0123] It should be noted that the computer system of the electronic device in this embodiment is only an example and should not impose any limitations on the function and scope of use of the embodiments of this application.

[0124] In this embodiment, the computer system includes a central processing unit 401, which can perform various appropriate actions and processes based on a program stored in read-only memory 402 or a program loaded from storage section 408 into random access memory 403, such as executing the deep learning-based video quality evaluation method described in the above embodiment. The random access memory 403 also stores various programs and data required for system operation. The central processing unit 401, read-only memory 402, and random access memory 403 are interconnected via bus 404. Input / output interface 405 is also connected to bus 404.

[0125] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0126] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit 401, it performs various functions defined in the system of this application.

[0127] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0129] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0130] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0131] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the deep learning-based video quality evaluation method described in the above embodiments.

[0132] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0133] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0134] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0135] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A video quality evaluation method based on deep learning, characterized in that, include: The original video sequence is obtained, and information compression processing is performed on the original video sequence to generate a first dynamic parameter. Based on the first dynamic parameter, attention-based convolution processing is performed on the original video sequence to generate spatiotemporal features composed of spatial features and temporal features. The spatiotemporal features are subjected to high-order tensor decomposition to generate degradation factors; Audio features are extracted from the original video sequence through a pre-trained audio network, and the audio features and the degradation factor are fused based on a cross-modal attention mechanism to generate a video quality score. Based on the video quality score and the manually labeled score, the difference parameters are determined, and model parameters are generated based on the difference parameters. The evaluation score of the original video sequence is generated by the evaluation model output by the model parameters.

2. The video quality evaluation method based on deep learning according to claim 1, characterized in that, The original video sequence is compressed to generate a first dynamic parameter. Based on the first dynamic parameter, the original video sequence is subjected to attention-based convolution to generate spatiotemporal features composed of spatial and temporal features, including: The video frames in the original video sequence are compressed to generate a first dynamic parameter, and spatial features are extracted from the video frames based on the first dynamic parameter. The spatial features are compressed to generate a first vector, and temporal features are extracted from the video frame based on the first vector.

3. The video quality evaluation method based on deep learning according to claim 2, characterized in that, The process involves compressing video frames in the original video sequence to generate first dynamic parameters, and extracting spatial features from the video frames based on these first dynamic parameters, including: The video frames in the original video sequence are subjected to information compression and multi-layer sensing processing to generate the first dynamic parameter a. i for: Where GAP(·) represents information compression processing, MLP1(·) and MLP2(·) represent multilayer sensing processing, v t Let i represent the video frame t, i and j represent the identifiers of the spatial convolution kernels, and K represent the number of spatial convolution kernels. Based on the first dynamic parameters and the spatial convolution kernel, spatial features F are extracted from the video frames in the spatial dimension. s (t) is: Among them, Conv si This represents the i-th spatial convolution kernel, which only modulates the video frame v in the spatial dimension. t Perform the operation to extract local spatial information.

4. The video quality evaluation method based on deep learning according to claim 3, characterized in that, The spatial features are compressed to generate a first vector, and temporal features are extracted from the video frame based on the first vector, including: The spatial features are compressed to generate a time feature vector as the first vector; The first vector is input into a preset number of multilayer perceptrons for multilayer sensing processing to generate time weights; Based on the time weights and a preset number of time convolution kernels, temporal features are extracted from the video frames in the time dimension.

5. The video quality evaluation method based on deep learning according to claim 1, characterized in that, The spatiotemporal features are subjected to higher-order tensor decomposition to generate degradation factors, including: Obtain the target value corresponding to the spatiotemporal feature at the preset location; Based on the target value, the spatiotemporal features are subjected to high-order tensor decomposition to generate a degradation factor.

6. The video quality evaluation method based on deep learning according to claim 1, characterized in that, Audio features are extracted from the original video sequence using a pre-trained audio network. These audio features and the degradation factor are then fused using a cross-modal attention mechanism to generate a video quality score, including: Audio features are extracted from the original video sequence using a pre-trained audio network; Attention factors are determined based on a learnable projection matrix, and the degradation factor is enhanced using the attention factors and the audio features to output the enhanced degradation factor. The video quality score is calculated based on the enhancement degradation factor.

7. The video quality evaluation method based on deep learning according to claim 1, characterized in that, Based on the video quality score and the manually labeled score, difference parameters are determined, and model parameters are generated based on the difference parameters, including: The video quality score and the manually labeled score are compared to determine the difference parameters; The model parameters are adjusted using the difference parameters.

8. A video quality evaluation system based on deep learning, characterized in that, include: The acquisition unit is used to acquire the original video sequence, perform information compression processing on the original video sequence to generate a first dynamic parameter, and perform attention-based convolution processing on the original video sequence based on the first dynamic parameter to generate spatiotemporal features composed of spatial features and temporal features. The decomposition unit is used to perform high-order tensor decomposition on the spatiotemporal features to generate degradation factors; The score unit is used to extract audio features from the original video sequence through a pre-trained audio network, and fuse the audio features and the degradation factor based on a cross-modal attention mechanism to generate a video quality score; The parameter unit is used to determine the difference parameters based on the video quality score and the manually labeled score, and to generate model parameters based on the difference parameters. The evaluation unit is used to generate an evaluation score for the original video sequence using the evaluation model output by the model parameters.

9. The deep learning-based video quality evaluation system according to claim 8, characterized in that, The original video sequence is compressed to generate a first dynamic parameter. Based on the first dynamic parameter, the original video sequence is subjected to attention-based convolution to generate spatiotemporal features composed of spatial and temporal features, including: The video frames in the original video sequence are compressed to generate a first dynamic parameter, and spatial features are extracted from the video frames based on the first dynamic parameter. The spatial features are compressed to generate a first vector, and temporal features are extracted from the video frame based on the first vector.

10. The deep learning-based video quality evaluation system according to claim 9, characterized in that, The process involves compressing video frames in the original video sequence to generate first dynamic parameters, and extracting spatial features from the video frames based on these first dynamic parameters, including: The video frames in the original video sequence are subjected to information compression and multi-layer sensing processing to generate the first dynamic parameter a. i for: Where GAP(·) represents information compression processing, MLP1(·) and MLP2(·) represent multilayer sensing processing, v t Let i represent the video frame t, i and j represent the identifiers of the spatial convolution kernels, and K represent the number of spatial convolution kernels. Based on the first dynamic parameters and the spatial convolution kernel, spatial features F are extracted from the video frames in the spatial dimension. s (t) is: Among them, Conv si This represents the i-th spatial convolution kernel, which only modulates the video frame v in the spatial dimension. t Perform the operation to extract local spatial information.

Images