Training method, evaluation method, system, device and medium of no-reference video quality evaluation model

Abstract

The application discloses a training method, a recognition method, a system, equipment and a medium of a no-reference video quality evaluation model, wherein the training method comprises the following steps: acquiring a labeled video dataset; performing mapping extraction processing on the video dataset to obtain the space-time features of the video; inputting the space-time features into an initialized STGRU module for feature fusion processing; mapping the fused space-time features to a video quality score through a regression module; training the model by using an optimization algorithm and a loss function to obtain a trained no-reference video quality evaluation model. The training method deeply extracts the features of the video from two key dimensions of space and time, can not only improve the space-time feature extraction capability of the no-reference video quality evaluation model on the video to be evaluated, but also improve the evaluation accuracy of the user-generated video, and meanwhile, the architecture design of the no-reference video quality evaluation model is relatively universal. The application can be widely applied to the technical field of video quality evaluation.

Description

Technical Field

[0001] This invention relates to the field of video quality assessment technology, and in particular to a training method, assessment method, system, device and medium for a no-reference video quality assessment model. Background Technology

[0002] In today's era of booming internet technology and social media, video has gradually become a crucial element of people's daily lives. The rapid development of smartphones and portable cameras has led to an ever-increasing number of videos created by users of all kinds every day, which are continuously uploaded to popular social media websites or applications such as YouTube and TikTok, and this growth trend continues.

[0003] Therefore, the importance of constructing a robust and powerful objective video quality assessment (VQA) method for these natural videos without original references, namely the no-reference VQA (NR-VQA) method for natural videos, is becoming increasingly prominent.

[0004] Traditional video quality assessment methods primarily rely on subjective evaluation, depending on human observers' subjective perceptions of video quality. However, this approach has several limitations. Firstly, it is costly, requiring significant human and material resources to organize and implement subjective evaluation experiments. Secondly, the subjective evaluation process is extremely time-consuming, making it difficult to meet the practical need for rapid video quality assessment. Furthermore, because different observers have different subjective standards, the evaluation results are highly subjective and lack a unified objective standard.

[0005] To overcome the aforementioned problems, researchers have focused on developing objective video quality assessment methods. Objective video quality assessment aims to use computer algorithms and models to automatically predict video quality, thereby providing fast, accurate, and repeatable assessment results. These methods are based on in-depth analysis of video content and features, covering multiple dimensions such as spatial, temporal, frequency, and perceptual features. Common objective assessment methods include structured methods, no-reference methods, and full-reference methods. No-reference methods assess quality solely based on information from the video itself, while full-reference methods require comparative analysis using reference videos or the original video.

[0006] With the continuous advancement of computer vision and machine learning technologies, numerous novel objective video quality assessment methods have emerged in recent years. These methods, leveraging advanced technologies such as deep learning and convolutional neural networks (CNNs), learn and model from massive amounts of video data, achieving more accurate and efficient video quality prediction. However, despite these progresses, existing technologies still have many problems that need to be addressed and optimized to further improve the accuracy, reliability, and practicality of video quality assessment and meet the ever-growing demands of video applications. Summary of the Invention

[0007] In order to at least partially solve one of the technical problems existing in the prior art, the purpose of this invention is to provide a training method, evaluation method, system, device and medium for a no-reference video quality assessment model.

[0008] The first technical solution adopted in this invention is:

[0009] A method for training a no-reference video quality assessment model includes the following steps:

[0010] Obtain the labeled video dataset;

[0011] The video dataset is mapped and extracted to obtain the spatiotemporal features of the video.

[0012] The spatiotemporal features are input into the initialized STGRU module for feature fusion processing;

[0013] The fused spatiotemporal features are mapped to video quality scores using a regression module;

[0014] The model is trained using optimization algorithms (such as Adam) and loss functions (such as mean squared error) to obtain a trained no-reference video quality assessment model.

[0015] Furthermore, the spatiotemporal features refer to the feature information in the spatial and temporal dimensions contained in the video, including but not limited to the spatial features of video frames (such as color, texture, shape, etc.) and the temporal features of video segments (such as motion changes, timing information, etc.).

[0016] The mapping and extraction process of the video dataset to obtain the spatiotemporal features of the video includes:

[0017] Spatial features are extracted from the video dataset by extracting a keyframe every few frames from the video, saving the keyframes to a specified folder, and processing the keyframes using the Swin Transformer model to obtain spatial features.

[0018] Temporal features are extracted from the video dataset by using a Slow Fast model to process video segments in the dataset and obtain temporal features.

[0019] Furthermore, the Swin Transformer, as a visual model based on the Transformer architecture, can capture the global and local spatial features of an image through a self-attention mechanism; these features include, but are not limited to, color, texture, shape, and spatial structure.

[0020] The Slow Fast model is a deep learning model for video understanding that includes two paths: a slow path for capturing long-term temporal information in the video and a fast path for capturing short-term temporal information. This design enables the model to understand both fast motion and slow changes in the video simultaneously.

[0021] Spatial and temporal features are combined to form a complete spatiotemporal feature set.

[0022] Further, the step of inputting the spatiotemporal features into the initialized STGRU module for feature fusion processing includes:

[0023] The spatiotemporal features are segmented into different gating signals (such as t_z, t_r, t_t, t_s and s_z, s_r, s_t, s_s), and the gating signals are used to obtain intermediate features through gating mechanisms and convolution operations.

[0024] The spatiotemporal features and the intermediate features are weighted and fused to obtain the updated hidden states T_new and S_new;

[0025] The hidden states T_new and S_new are concatenated to obtain the fused output features.

[0026] Furthermore, the temporal and spatial features are used as inputs for fusion processing, including:

[0027] Perform convolution operations on the inputs T_t and S_t, and then segment them to obtain different gated signals;

[0028] The updated hidden state is calculated using gating signals;

[0029] The input is mapped using a linear layer and combined with the updated hidden state;

[0030] The updated T and S are concatenated as the output, and the hidden state and gating signal are returned.

[0031] Furthermore, the step of mapping the fused spatiotemporal features to video quality scores via a regression module includes:

[0032] The fused spatiotemporal features are mapped to the predicted quality score through two fully connected layers;

[0033] The ReLU activation function is used to increase the non-linearity of the model, which helps the model learn more complex feature representations.

[0034] Furthermore, the step of training the model using a loss function includes:

[0035] Calculate the loss weights of the fused spatiotemporal features as input to the initialized quality regression module;

[0036] The parameters of the initialized no-reference video quality assessment model are updated based on the loss weights.

[0037] The second technical solution adopted in this invention is:

[0038] A no-reference video quality assessment method includes the following steps:

[0039] Obtain the user-generated video to be evaluated;

[0040] The user-generated video is input into the trained no-reference video quality assessment model for evaluation, and the evaluation result is obtained.

[0041] The no-reference video quality assessment model is trained using the method described above.

[0042] Furthermore, the spatiotemporal features refer to the feature information in the spatial and temporal dimensions contained in the video, including but not limited to the spatial features of video frames (such as color, texture, shape, etc.) and the temporal features of video segments (such as motion changes, timing information, etc.).

[0043] The mapping and extraction process of the video dataset to obtain the spatiotemporal features of the video includes:

[0044] Spatial features are extracted from the video dataset by extracting a keyframe every few frames from the video, saving the keyframes to a specified folder, and processing the keyframes using the Swin Transformer model to obtain spatial features.

[0045] Temporal features are extracted from the video dataset by using a Slow Fast model to process video segments in the dataset and obtain temporal features.

[0046] Furthermore, the Swin Transformer, as a visual model based on the Transformer architecture, can capture the global and local spatial features of an image through a self-attention mechanism; these features include, but are not limited to, color, texture, shape, and spatial structure.

[0047] The Slow Fast model is a deep learning model for video understanding that includes two paths: a slow path for capturing long-term temporal information in the video and a fast path for capturing short-term temporal information. This design enables the model to understand both fast motion and slow changes in the video simultaneously.

[0048] Spatial and temporal features are combined to form a complete spatiotemporal feature set.

[0049] Further, the step of inputting the spatiotemporal features into the initialized STGRU module for feature fusion processing includes:

[0050] The spatiotemporal features are segmented into different gating signals (such as t_z, t_r, t_t, t_s and s_z, s_r, s_t, s_s), and the gating signals are used to obtain intermediate features through gating mechanisms and convolution operations.

[0051] The spatiotemporal features and the intermediate features are weighted and fused to obtain the updated hidden states T_new and S_new;

[0052] The hidden states T_new and S_new are concatenated to obtain the fused output features.

[0053] Furthermore, the temporal and spatial features are used as inputs for fusion processing, including:

[0054] Perform convolution operations on the inputs T_t and S_t, and then segment them to obtain different gated signals;

[0055] The updated hidden state is calculated using gating signals;

[0056] The input is mapped using a linear layer and combined with the updated hidden state;

[0057] The updated T and S are concatenated as the output, and the hidden state and gating signal are returned.

[0058] Furthermore, the step of mapping the fused spatiotemporal features to video quality scores via a regression module includes:

[0059] The fused spatiotemporal features are mapped to the predicted quality score through two fully connected layers;

[0060] The ReLU activation function is used to increase the non-linearity of the model, which helps the model learn more complex feature representations.

[0061] Furthermore, the step of training the model using a loss function includes:

[0062] Calculate the loss weights of the fused spatiotemporal features as input to the initialized quality regression module;

[0063] The parameters of the initialized no-reference video quality assessment model are updated based on the loss weights.

[0064] The third technical solution adopted in this invention is:

[0065] A training system for a no-reference video quality assessment model, comprising:

[0066] The module retrieves the labeled video dataset;

[0067] The extraction module performs mapping and extraction processing on the video dataset to obtain the spatiotemporal features of the video;

[0068] The fusion module inputs the spatiotemporal features into the initialized STGRU module for feature fusion processing;

[0069] The regression module maps the fused spatiotemporal features to the video quality score.

[0070] The training module uses a loss function to train the model, resulting in a trained no-reference video quality assessment model.

[0071] The fourth technical solution adopted in this invention is:

[0072] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a training method for a no-reference video quality assessment model and / or a no-reference video quality assessment method as described above.

[0073] The fifth technical solution adopted in this invention is:

[0074] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a training method for a no-reference video quality assessment model and / or a no-reference video quality assessment method as described above.

[0075] The sixth technical solution adopted in this invention is:

[0076] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the training method for a no-reference video quality assessment model and / or a no-reference video quality assessment method described above.

[0077] The beneficial effects of this invention are as follows: This invention extracts video features from two key dimensions, space and time, which not only improves the ability of the no-reference video quality assessment model to extract spatiotemporal features of the video to be assessed, but also improves the accuracy of the assessment of user-generated videos. At the same time, the architecture design of the no-reference video quality assessment model is relatively universal and is not limited to specific types of video content or scenarios. This universality brings broader application prospects and higher practical value to the field of video quality assessment. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 A flowchart illustrating a training method for a no-reference video quality assessment model provided in this application embodiment;

[0080] Figure 2 This is a structural diagram of the no-reference video quality assessment model provided in the embodiments of this application;

[0081] Figure 3 A schematic diagram of the structure of a training system for a no-reference video quality assessment model provided in this application embodiment;

[0082] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0083] Figure 5 This is a schematic diagram of the STGRU module workflow provided in an embodiment of this application. Detailed Implementation

[0084] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0085] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0086] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0087] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0088] Terminology Explanation:

[0089] The Swin Transformer model is an innovative vision model that successfully solves the problems faced by traditional Transformers in computer vision tasks through hierarchical multi-scale architecture, shift window strategy, and local self-attention mechanism. It achieves efficient image processing capabilities and reduces the consumption of computing resources.

[0090] The Slow Fast model, proposed by Facebook Research, is a two-stream model framework for human behavior recognition. This model consists of two branches: a spatial branch and a temporal branch. The spatial branch operates at a lower frame rate, primarily capturing image features; the temporal branch operates at a higher frame rate, primarily capturing motion features. This design balances efficiency and accuracy.

[0091] Currently, traditional video quality assessment algorithms primarily focus on spatial features of the video, such as pixel brightness, color, and texture, during feature extraction. However, they neglect temporal features, such as motion information and frame rate changes. This results in incomplete feature extraction, impacting the accuracy of the assessment results. Existing methods often struggle to effectively extract high-level semantic information during feature extraction, such as the video's theme and content coherence, leading to discrepancies between the assessment results and human subjective perception. Furthermore, video shooting and transmission environments are complex and variable; different lighting conditions, shooting equipment, and network environments all affect video quality. Existing no-reference video quality assessment methods often fail to adapt well to such complex video environments, ultimately resulting in inaccurate assessment results.

[0092] In view of this, embodiments of the present invention provide a training method for a no-reference video quality assessment model. This training method deeply extracts video features from the two key dimensions of space and time, which not only improves the no-reference video quality assessment model's ability to extract spatiotemporal features of the video to be assessed, but also enhances the accuracy of assessments of user-generated videos. Furthermore, the architecture of the no-reference video quality assessment model has strong versatility and is not limited to specific types of video content or scenarios. This versatility brings broader application prospects and higher practical value to the field of video quality assessment.

[0093] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0094] Reference Figure 1 In this embodiment of the application, a training method for a no-reference video quality assessment model includes:

[0095] Step 110: Obtain the labeled video dataset.

[0096] In this application embodiment, there are various ways to obtain video datasets. Firstly, various camera devices can be used to capture various types of video content, covering landscapes, human activities, and object movement, under different environmental conditions (such as varying lighting and weather conditions) and shooting settings (including different resolutions, frame rates, and encoding parameters). Secondly, numerous video sharing websites contain a large number of user-uploaded videos with diverse themes and styles. When collecting video data from these platforms through legal channels, it is essential to pay attention to copyright issues and the representativeness of the data. Furthermore, during model training, publicly available datasets published by research institutions, organizations, or companies for video quality assessment research can also be selected. These datasets are often carefully organized and labeled, possessing a certain scale and strong standardization.

[0097] When labeling video data, video samples should be presented to observers in a random order to avoid the influence of the order effect on the evaluation results. After watching each video, observers rate it according to a given evaluation scale. Evaluation scales come in various forms, such as the commonly used Mean Opinion Score (MOS) scale, which typically uses integer scores from 1 (very poor quality) to 5 (very good quality), or a continuous quality rating scale that allows observers to give specific scores within a certain range. For the processed scores from multiple observers, a common method for statistical calculation is to calculate the average as the final subjective quality score for the video. If individual differences between observers need to be considered, methods such as weighted averaging can be used, assigning different weights to the scores based on relevant characteristics of the observers (such as experience level, visual acuity, etc.), and then calculating a weighted average to obtain a more reasonable subjective quality score. Simultaneously, statistical measures such as the standard deviation of the scores can also be calculated to reflect the dispersion and consistency of the scores.

[0098] Step 120: Perform mapping and extraction processing on the video dataset to obtain the spatiotemporal features of the video;

[0099] Reference Figure 2 In some embodiments, step 120, mapping and extracting the video dataset to obtain the spatiotemporal features of the video, includes:

[0100] Step 121: Extract spatial features from the video dataset. Extract a keyframe every 8 frames from the video and save the keyframes to a designated folder. Use the SwinTransformer model to process the keyframes and extract spatial features. This method of extracting keyframes every 8 frames effectively reduces computation while capturing the spatial feature information of the video quite well. Furthermore, a similar processing method can be used for different types of video datasets, demonstrating strong adaptability.

[0101] Step 122: Extract temporal features from the video dataset. The SlowFast model is used to process video segments in the dataset and extract temporal features. Due to its unique design architecture, the SlowFast model can adapt well to videos of different lengths and content. Regardless of the video's length or the complexity or simplicity of its content, the model can extract effective temporal features, providing crucial temporal dimension information for subsequent video quality assessment.

[0102] In this embodiment, for each video in the video dataset, keyframes are extracted every 8 frames, and a pre-trained SwinTransformer model is used to process each keyframe. The SwinTransformer model requires a fixed-size image as input, so the keyframes need to be resized to the model's required dimensions, such as 224×224 pixels. The model extracts features from each keyframe through its multi-layered self-attention mechanism and convolutional operations. Specifically, it focuses on features such as color distribution, texture details, object shapes, and spatial relationships within the image. After processing by the SwinTransformer, each keyframe yields a corresponding spatial feature vector. For each set of video keyframes, the extracted spatial features are subjected to average pooling to obtain the spatial feature representation of the video.

[0103] For each video in the video dataset, it is divided into multiple video segments, which are then processed using a pre-trained SlowFast model. The SlowFast model includes a slow path and a fast path, processing the video segments at different frame rates to capture long-term and short-term temporal information in the video. After processing by the SlowFast model, each video segment yields a corresponding temporal feature vector. For multiple video segments within each video, these temporal feature vectors are concatenated and further processed to obtain the temporal feature representation of the video.

[0104] Step 130: Input the spatiotemporal features into the initialized STGRU module for feature fusion processing;

[0105] Reference Figure 2 In some embodiments, step 130, inputting the spatiotemporal features into the initialized STGRU module for feature fusion processing, includes:

[0106] Step 131: Segment the spatiotemporal features into different gating signals (such as t_z, t_r, t_t, t_s and s_z, s_r, s_t, s_s), and obtain intermediate features by gating the gating signals through gating mechanisms and convolution operations;

[0107] Step 132: Perform weighted fusion processing on the spatiotemporal features and the intermediate features to obtain the updated hidden states T_new and S_new;

[0108] Step 133: Concatenate T_new and S_new to obtain the fused output features.

[0109] In this embodiment, the STGRU module, through the fusion processing of spatiotemporal features, can better capture spatiotemporal dynamic changes and complex patterns in videos. It combines gating mechanisms, convolution operations, and linear mapping, enabling the model to adaptively select important feature information and ignore irrelevant or noisy information, thus improving the quality of feature fusion. The fused features can more accurately reflect the quality characteristics of the video, providing more valuable information input for subsequent evaluation and classification tasks, helping to improve the model's generalization ability, enabling it to adapt to different types of video data, and improving the accuracy and reliability of no-reference video quality assessment.

[0110] For example, refer to Figure 5 The temporal and spatial features are processed through a 3×3 convolutional layer to obtain temporal and spatial convolutional outputs. This convolutional processing helps extract the spatiotemporal correlation of features. The spatiotemporal convolutional outputs are divided into four parts, corresponding to the update gate (z), reset gate (r), candidate hidden state (t), and hidden state (s) of the GRU unit. The update gate and reset gate are calculated by adding the corresponding temporal and spatial convolutional outputs and applying the sigmoid activation function. This allows the model to control the flow of information and enhances the capture of important features. The temporal and spatial candidate hidden states are obtained by multiplying the temporal and spatial features with the reset gate output and then applying the tanh activation function. This step provides the basis for updating the hidden states. The original temporal and spatial features are mapped to dimensions matching the hidden states through a linear layer. Combining the update gate and the hidden state of the previous time step, the updated hidden state of the current time step is calculated. The updated temporal and spatial hidden states are concatenated along the feature dimensions to form the final fused output features. This fusion feature can more comprehensively reflect the spatiotemporal characteristics of video.

[0111] Step 140: Map the fused spatiotemporal features to the video quality score using the regression module;

[0112] In some embodiments, step 140, mapping the spatiotemporal fusion features into a regression module to obtain a video quality score, includes:

[0113] Step 141: Map the fused spatiotemporal features to the predicted quality scores using two fully connected layers. The parameters of these layers are learned during training to minimize the difference between the predicted and true scores.

[0114] Step 142: Use the ReLU activation function to increase the non-linearity of the model, helping the model learn more complex feature representations;

[0115] Step 143: Video quality scores are typically in the range of 0 to 100. Using the torch.clamp method ensures that the output value is within this range, which helps generate a realistic and interpretable quality score.

[0116] In this embodiment, after obtaining the fused spatiotemporal features, they are input into a first fully connected layer with weights W1 and bias b1. Then, a ReLU activation function is applied to non-linearly transform the output of the first layer, increasing the model's expressive power. To map the non-linearly transformed features to the range of video quality scores, a second fully connected layer is designed. The output of the ReLU activation function is input into the second fully connected layer with weights W2 and bias b2. Since video quality scores are typically in the range of 0 to 100, a clamp operation is used to limit the output to this range.

[0117] Step 150: Train the model using an optimization algorithm (such as Adam) and a loss function (such as L1RankLoss) to obtain a trained no-reference video quality assessment model, including:

[0118] Step 151: Calculate the loss weights of the fused features input to the initialized quality regression module;

[0119] Step 152: Update the parameters of the initialized no-reference video quality assessment model according to the loss weights.

[0120] In this embodiment, the program first determines whether to use a CPU or GPU for training, then loads a predefined model architecture and deploys it to the selected device. If pre-trained weights are provided, the model loads these weights, but excludes the weights of the head layer so that these weights can be updated during training. If specified, the parameters of all layers in the model except the head layer are frozen to prevent them from being updated during training. Next, the parameters in the model that require gradient updates are determined, and these parameters are optimized using the AdamW optimizer. A learning rate scheduler is set to adjust the learning rate according to a predetermined interval and decay rate during training, and the loss function L1RankLoss is initialized. For each training epoch, the model performs the following operations:

[0121] Set to training mode;

[0122] Iterate through the training dataset, perform forward and backward propagation for each batch, calculate the loss, and update the model parameters;

[0123] Print training loss and computation time;

[0124] Adjust the learning rate;

[0125] Validate on the test dataset and calculate performance metrics such as Spearman Rank Correlation Coefficient (SRCC), Kendall Rank Correlation Coefficient (KRCC), Pearson Rank Correlation Coefficient (PLCC), and Root Mean Square Error (RMSE).

[0126] If the performance metrics for the current period are better than the previous best performance, then save the model's state.

[0127] It should be noted that before inputting the video dataset into the no-reference video quality assessment model for training, a certain number of user-generated video validation sets can be obtained by data partitioning. The partition ratio between the user-generated video validation set and the user-generated video set can be set according to actual needs, such as 7:3, 6:4, 8:2, etc. This application will not elaborate further on this point.

[0128] This application also provides a method for identifying a no-reference video quality assessment model, including:

[0129] Step 160: Obtain the user-generated video to be evaluated;

[0130] Step 170: Input the user-generated video into the trained no-reference video quality assessment model as described in the first aspect for evaluation, and obtain the evaluation result.

[0131] In this embodiment, user-generated videos can be acquired using various camera devices or downloaded from social media platforms through legal means. The acquired user-generated videos are then input into a trained no-reference video quality assessment model for evaluation, and the evaluation results output by the trained no-reference video quality assessment model can be obtained.

[0132] The training system for a no-reference video quality assessment model according to an embodiment of this application is described in detail below with reference to the accompanying drawings.

[0133] Reference Figure 3 The no-reference video quality assessment model proposed in this application includes:

[0134] Module 101 is used to acquire the labeled video dataset;

[0135] Extraction module 102 is used to perform mapping extraction processing on the video dataset to obtain the spatiotemporal features of the video;

[0136] The fusion module 103 is used to input the spatiotemporal features into the initialized STGRU module for feature fusion processing;

[0137] Regression module 104 is used to map the fused spatiotemporal features to video quality scores;

[0138] The training module 105 is used to input the fused output features into the initialized quality regression module for training, so as to obtain a trained no-reference video quality assessment model.

[0139] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0140] Reference Figure 4 This application also provides an electronic device, including:

[0141] At least one processor 201;

[0142] At least one memory 202 is used to store at least one program;

[0143] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the method embodiment described above.

[0144] Similarly, it can be understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0145] This application also provides a computer-readable storage medium storing a program executable by a processor 201, which, when executed by the processor 201, is used to implement the above-described method embodiments.

[0146] Similarly, the content of the above method embodiments is applicable to the present computer-readable storage medium embodiments. The specific functions implemented by the present computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0147] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0148] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0149] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0151] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0152] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0153] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0154] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0155] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A training method for a no-reference video quality assessment model, characterized in that, Includes the following steps: Obtain the labeled video dataset; The video dataset is mapped and extracted to obtain the spatiotemporal features of the video. The spatiotemporal features are input into the initialized STGRU module for feature fusion processing; The fused spatiotemporal features are mapped to video quality scores using a regression module; The model is trained using a loss function to obtain a trained no-reference video quality assessment model. The step of inputting the spatiotemporal features into the initialized STGRU module for feature fusion processing includes: The spatiotemporal features are segmented into different gated signals, and the gated signals are used to obtain intermediate features through a gating mechanism and convolution operation. The spatiotemporal features and the intermediate features are weighted and fused to obtain the updated hidden states T_new and S_new; The hidden states T_new and S_new are concatenated to obtain the fused output features.

2. The training method for a no-reference video quality assessment model according to claim 1, characterized in that, The spatiotemporal features refer to the spatial and temporal features contained in the video, including the spatial features of video frames and the temporal features of video segments; The mapping and extraction process of the video dataset to obtain the spatiotemporal features of the video includes: Spatial features are extracted from the video dataset by extracting a keyframe every few frames from the video, saving the keyframes to a specified folder, and processing the keyframes using the Swin Transformer model to obtain spatial features. Temporal features are extracted from the video dataset by using a Slow Fast model to process video segments in the dataset and obtain temporal features.

3. The training method for a no-reference video quality assessment model according to claim 2, characterized in that, The Swin Transformer model, as a visual model based on the Transformer architecture, can capture global and local spatial features of an image through a self-attention mechanism; these features include color, texture, shape, and spatial structure. The Slow Fast model is a deep learning model for video understanding that includes two paths: a slow path for capturing long-term temporal information in the video and a fast path for capturing short-term temporal information. This design enables the model to understand both fast motion and slow changes in the video simultaneously. Spatial and temporal features are combined to form a complete spatiotemporal feature set.

4. The training method for a no-reference video quality assessment model according to claim 1, characterized in that, The process of mapping the fused spatiotemporal features to video quality scores via a regression module includes: The fused spatiotemporal features are mapped to the predicted quality score through two fully connected layers; The ReLU activation function is used to increase the non-linearity of the model, which helps the model learn more complex feature representations.

5. The training method for a no-reference video quality assessment model according to claim 1, characterized in that, The training of the model using a loss function includes: Calculate the loss weights of the fused spatiotemporal features as input to the initialized quality regression module; The parameters of the initialized no-reference video quality assessment model are updated based on the loss weights.

6. A no-reference video quality assessment method, characterized in that, Includes the following steps: Obtain the user-generated video to be evaluated; The user-generated video is input into the trained no-reference video quality assessment model for evaluation, and the evaluation result is obtained. The no-reference video quality assessment model is trained using the method described in any one of claims 1-5.

7. A training system for a no-reference video quality assessment model, characterized in that, include: The module retrieves the labeled video dataset; The extraction module performs mapping and extraction processing on the video dataset to obtain the spatiotemporal features of the video; The fusion module inputs the spatiotemporal features into the initialized STGRU module for feature fusion processing; The regression module maps the fused spatiotemporal features to the video quality score. The training module uses a loss function to train the model and obtain a trained no-reference video quality assessment model. The step of inputting the spatiotemporal features into the initialized STGRU module for feature fusion processing includes: The spatiotemporal features are segmented into different gated signals, and the gated signals are used to obtain intermediate features through a gating mechanism and convolution operation. The spatiotemporal features and the intermediate features are weighted and fused to obtain the updated hidden states T_new and S_new; The hidden states T_new and S_new are concatenated to obtain the fused output features.

8. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method as described in any one of claims 1 to 6.

Images