No-reference video quality evaluation model training method, evaluation method, system, device and medium

Abstract

The invention discloses a training method, an identification method, a system, equipment and a medium for a no-reference video quality evaluation model, and the training method comprises the steps: obtaining a marked video data set; performing mapping extraction processing on the video data set to obtain spatio-temporal characteristics of the video; inputting the spatial-temporal features into an initialized STGRU module for feature fusion processing; mapping the fused spatial-temporal features to video quality scores through a regression module; and training the model by using an optimization algorithm and a loss function to obtain a trained non-reference video quality evaluation model. According to the training method, the features of the video are deeply extracted from the two key dimensions of space and time, the space-time feature extraction capability of the non-reference video quality evaluation model on the to-be-evaluated video can be improved, the evaluation accuracy of the video generated by the user can be improved, and meanwhile, the architecture design of the non-reference video quality evaluation model is relatively universal. The method can be widely applied to the technical field of video quality evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of video quality assessment, and in particular to a training method, an assessment method, a system, a device and a medium for a reference-free video quality assessment model. Background Art

[0002] In today's era of booming Internet industry technology and self-media, video has gradually become a vital element in people's daily lives. The rapid development of smartphones and portable cameras has led to an increasing number of videos being created by various users every day and continuously uploaded to popular social media websites or applications such as YouTube and TikTok, and this growth trend is still continuing.

[0003] In view of this, it is increasingly important to build a robust and powerful objective video quality assessment (VQA) method for these natural videos without original references, namely the no-reference VQA (NR-VQA) method for natural videos.

[0004] Traditional video quality assessment methods mainly focus on subjective evaluation, that is, relying on the subjective feelings of human observers on video quality to make judgments. However, this method has many limitations. On the one hand, it is costly and requires a lot of manpower and material resources to organize and implement subjective evaluation experiments. On the other hand, the subjective evaluation process is extremely time-consuming and difficult to meet the actual needs of quickly evaluating video quality. In addition, due to the differences in subjective standards of different observers, the evaluation results are highly subjective and lack unified objective standards.

[0005] To overcome the above problems, researchers have focused on the development of objective video quality assessment methods. Objective video quality assessment aims to use computer algorithms and models to automatically predict video quality, thereby providing fast, accurate and repeatable assessment results. Such methods are based on in-depth analysis of video content and features, covering multiple dimensions such as spatial domain, temporal domain, frequency domain and perceptual features. Common objective assessment methods include structured methods, no-reference methods and full-reference methods. Among them, the no-reference method only relies on the information of the video to be evaluated to perform quality assessment, while the full-reference method requires the use of reference videos or original videos for comparative analysis.

[0006] With the continuous advancement of computer vision and machine learning technologies, many novel objective video quality assessment methods have emerged in recent years. These methods use advanced technologies such as deep learning and convolutional neural networks (CNN) to learn and model from massive video data, achieving more accurate and efficient video quality prediction. However, despite certain progress, there are still many problems in the existing technology that need to be solved and optimized to further improve the accuracy, reliability and practicality of video quality assessment and meet the growing demand for video applications. Summary of the invention

[0007] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a training method, an evaluation method, a system, a device and a medium for a no-reference video quality evaluation model.

[0008] The first technical solution adopted by the present invention is:

[0009] A method for training a no-reference video quality assessment model comprises the following steps:

[0010] Get the labeled video dataset;

[0011] Performing mapping and extraction processing on the video data set to obtain spatiotemporal features of the video;

[0012] Input the spatiotemporal features into the initialized STGRU module for feature fusion processing;

[0013] The fused spatiotemporal features are mapped to video quality scores through a regression module;

[0014] The model is trained using an optimization algorithm (such as Adam) and a loss function (such as mean square error) to obtain a trained no-reference video quality assessment model.

[0015] Furthermore, the spatiotemporal features refer to feature information in the spatial and temporal dimensions contained in the video, including but not limited to the spatial features of the video frames (such as color, texture, shape, etc.) and the temporal features of the video clips (such as motion changes, timing information, etc.);

[0016] The mapping and extraction processing is performed on the video data set to obtain the spatiotemporal features of the video, including:

[0017] Extracting spatial features from the video data set, extracting a key frame from the video every multiple frames, saving the key frame to a specified folder, and processing the key frame using a Swin Transformer model to obtain spatial features;

[0018] Temporal features are extracted from the video data set, and the video clips in the video data set are processed using a Slow Fast model to obtain temporal features.

[0019] Furthermore, the Swin Transformer, as a visual model based on the Transformer architecture, can capture the global spatial features and local spatial features of an image through a self-attention mechanism; these features include but are not limited to color, texture, shape, and spatial structure;

[0020] The Slow Fast model is a deep learning model for video understanding, which contains two paths: a slow path for capturing long-term timing information in the video, and a fast path for capturing short-term timing information; this design enables the model to understand both fast actions and slow changes in the video;

[0021] The spatial features and temporal features are combined to form a complete spatiotemporal feature set.

[0022] Furthermore, the step of inputting the spatiotemporal features into the initialized STGRU module for feature fusion processing includes:

[0023] The spatiotemporal features are divided into different gating signals (such as t_z, t_r, t_t, t_s and s_z, s_r, s_t, s_s), and the gating signals are subjected to gating mechanisms and convolution operations to obtain intermediate features;

[0024] Performing weighted fusion processing on the spatiotemporal features and the intermediate features to obtain updated hidden states T_new and S_new;

[0025] The hidden states T_new and S_new are concatenated to obtain fused output features.

[0026] Furthermore, the temporal and spatial features are taken as inputs for fusion processing, including:

[0027] Perform convolution operation on the input T_t and S_t, and segment them to obtain different gating signals;

[0028] Calculate the updated hidden state through the gating signal;

[0029] Use a linear layer to map the input and combine it with the updated hidden state;

[0030] The updated T and S are concatenated as output, and the hidden state and gating signal are returned.

[0031] Furthermore, mapping the fused spatiotemporal features to the video quality score through the regression module includes:

[0032] The fused spatiotemporal features are mapped to the predicted quality scores through two fully connected layers;

[0033] The ReLU activation function is used to increase the nonlinearity of the model to help the model learn more complex feature representations.

[0034] Furthermore, the adopting loss function to train the model includes:

[0035] Calculate the loss weight of the fused spatiotemporal features input into the initialized quality regression module;

[0036] According to the loss weight, the parameters of the initialized no-reference video quality assessment model are updated.

[0037] The second technical solution adopted by the present invention is:

[0038] A no-reference video quality assessment method comprises the following steps:

[0039] Obtaining user-generated videos to be evaluated;

[0040] Inputting the user-generated video into a trained no-reference video quality assessment model for assessment to obtain an assessment result;

[0041] The no-reference video quality assessment model is trained using the method described above.

[0042] Furthermore, the spatiotemporal features refer to feature information in the spatial and temporal dimensions contained in the video, including but not limited to the spatial features of the video frames (such as color, texture, shape, etc.) and the temporal features of the video clips (such as motion changes, timing information, etc.);

[0043] The mapping and extraction processing is performed on the video data set to obtain the spatiotemporal features of the video, including:

[0044] Extracting spatial features from the video data set, extracting a key frame from the video every multiple frames, saving the key frame to a specified folder, and processing the key frame using a Swin Transformer model to obtain spatial features;

[0045] Temporal features are extracted from the video data set, and the video clips in the video data set are processed using a Slow Fast model to obtain temporal features.

[0046] Furthermore, the Swin Transformer, as a visual model based on the Transformer architecture, can capture the global spatial features and local spatial features of an image through a self-attention mechanism; these features include but are not limited to color, texture, shape, and spatial structure;

[0047] The Slow Fast model is a deep learning model for video understanding, which contains two paths: a slow path for capturing long-term timing information in the video, and a fast path for capturing short-term timing information; this design enables the model to understand both fast actions and slow changes in the video;

[0048] The spatial features and temporal features are combined to form a complete spatiotemporal feature set.

[0049] Furthermore, the step of inputting the spatiotemporal features into the initialized STGRU module for feature fusion processing includes:

[0050] The spatiotemporal features are divided into different gating signals (such as t_z, t_r, t_t, t_s and s_z, s_r, s_t, s_s), and the gating signals are subjected to gating mechanisms and convolution operations to obtain intermediate features;

[0051] Performing weighted fusion processing on the spatiotemporal features and the intermediate features to obtain updated hidden states T_new and S_new;

[0052] The hidden states T_new and S_new are concatenated to obtain fused output features.

[0053] Furthermore, the temporal and spatial features are taken as inputs for fusion processing, including:

[0054] Perform convolution operation on the input T_t and S_t, and segment them to obtain different gating signals;

[0055] Calculate the updated hidden state through the gating signal;

[0056] Use a linear layer to map the input and combine it with the updated hidden state;

[0057] The updated T and S are concatenated as output, and the hidden state and gating signal are returned.

[0058] Furthermore, mapping the fused spatiotemporal features to the video quality score through the regression module includes:

[0059] The fused spatiotemporal features are mapped to the predicted quality scores through two fully connected layers;

[0060] The ReLU activation function is used to increase the nonlinearity of the model to help the model learn more complex feature representations.

[0061] Furthermore, the adopting loss function to train the model includes:

[0062] Calculate the loss weight of the fused spatiotemporal features input into the initialized quality regression module;

[0063] According to the loss weight, the parameters of the initialized no-reference video quality assessment model are updated.

[0064] The third technical solution adopted by the present invention is:

[0065] A training system for a no-reference video quality assessment model, comprising:

[0066] Acquisition module, obtains the labeled video dataset;

[0067] An extraction module performs mapping and extraction processing on the video data set to obtain the spatiotemporal features of the video;

[0068] A fusion module inputs the spatiotemporal features into the initialized STGRU module for feature fusion processing;

[0069] Regression module, through which the fused spatiotemporal features are mapped to video quality scores;

[0070] The training module uses the loss function to train the model and obtains the trained no-reference video quality assessment model.

[0071] The fourth technical solution adopted by the present invention is:

[0072] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a training method for a no-reference video quality assessment model and / or a no-reference video quality assessment method as described above.

[0073] The fifth technical solution adopted by the present invention is:

[0074] A computer-readable storage medium, wherein at least one instruction, at least one program, code set or instruction set is stored in the storage medium, wherein the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement a training method for a no-reference video quality assessment model and / or a no-reference video quality assessment method as described above.

[0075] The sixth technical solution adopted by the present invention is:

[0076] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned training method of a no-reference video quality assessment model and / or a no-reference video quality assessment method.

[0077] The beneficial effects of the present invention are as follows: the present invention deeply extracts the features of the video from two key dimensions, space and time, which can not only improve the spatiotemporal feature extraction capability of the no-reference video quality assessment model for the video to be assessed, but also improve the assessment accuracy of user-generated videos. At the same time, the architectural design of the no-reference video quality assessment model is relatively general and is not limited to a specific type of video content or scene. This generality brings broader application prospects and higher practical value to the field of video quality assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0079] Figure 1 A flowchart of a method for training a no-reference video quality assessment model provided in an embodiment of the present application;

[0080] Figure 2 is a structural diagram of a no-reference video quality assessment model provided in an embodiment of the present application;

[0081] Figure 3 A schematic diagram of the structure of a training system for a reference-free video quality assessment model provided in an embodiment of the present application;

[0082] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0083] Figure 5 A schematic diagram of the STGRU module workflow provided for an embodiment of the present application. DETAILED DESCRIPTION

[0084] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0085] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0086] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0087] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0088] Terminology explanation:

[0089] Swin Transformer model: is an innovative visual model that successfully solves the problems faced by traditional Transformer in computer vision tasks through designs such as hierarchical multi-scale architecture, shift window strategy and local self-attention mechanism, achieves efficient image processing capabilities, and reduces the consumption of computing resources.

[0090] Slow Fast model: It is a two-stream model framework proposed by Facebook Research for human behavior recognition. The model consists of two branches: spatial branch and temporal branch. The spatial branch runs at a lower frame rate and is mainly used to capture image features; the temporal branch runs at a higher frame rate and is mainly used to capture action features. This design takes into account both efficiency and accuracy.

[0091] At present, traditional video quality assessment algorithms focus on the spatial features of videos, such as pixel brightness, color, and texture, when extracting features, but ignore the features in the temporal dimension, such as motion information and frame rate changes. This leads to incomplete feature extraction, which in turn affects the accuracy of the assessment results. In the process of feature extraction, existing methods often find it difficult to effectively obtain high-level semantic information, such as the theme of the video and the coherence of the content, which makes the assessment results different from human subjective feelings. In addition, the video shooting and transmission environment is complex and changeable, and different lighting conditions, shooting equipment, and network environment will affect the video quality. Existing no-reference video quality assessment methods often cannot adapt well to such complex video environments, which ultimately leads to inaccurate assessment results.

[0092] In view of this, an embodiment of the present invention provides a training method for a no-reference video quality assessment model. The training method deeply extracts the features of the video from the two key dimensions of space and time, which can not only improve the spatiotemporal feature extraction capability of the no-reference video quality assessment model for the video to be assessed, but also improve the assessment accuracy of user-generated videos. At the same time, the architecture design of the no-reference video quality assessment model has strong versatility and is not limited to a specific type of video content or scene. This versatility brings broader application prospects and higher practical value to the field of video quality assessment.

[0093] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0094] Reference Figure 1 In an embodiment of the present application, a training method for a no-reference video quality assessment model includes:

[0095] Step 110: Obtain a labeled video dataset.

[0096] In the embodiments of the present application, there are many ways to obtain video data sets. First, various video equipment can be used to shoot various types of video content under different environmental conditions (such as different lighting conditions, weather conditions, etc.) and different shooting settings (including different resolutions, frame rates, encoding parameters, etc.), covering various types such as scenery, human activities, and object movement. Second, there are a large number of user-uploaded videos on many video sharing websites, and their themes and styles are all-encompassing. When collecting video data from these platforms through legal channels, be sure to pay attention to copyright issues and the representativeness of the data. In addition, in the process of training the model, some public data sets for video quality assessment research released by scientific research institutions, organizations or companies can also be selected. This type of data set is often carefully organized and labeled, has a certain scale and is highly standardized.

[0097] When annotating video data, video samples need to be presented to observers in a random order to avoid the influence of order effects on the evaluation results. After watching each video, the observers score it according to a given evaluation scale. Evaluation scales come in various forms, such as the commonly used Mean Opinion Score (MOS) scale, which is usually an integer score from 1 (extremely poor quality) to 5 (excellent quality), or a continuous quality rating scale that allows observers to give specific scores within a specific range. For the scores of multiple observers after processing, when performing statistical calculations, the common method is to calculate the average value as the final subjective quality score of the video. If individual differences between observers need to be considered, methods such as weighted averaging can be used to assign different weights to their scores according to their relevant characteristics (such as experience level, vision status, etc.), and then calculate the weighted average to obtain a more reasonable subjective quality score. At the same time, statistics such as the standard deviation of the scores can also be calculated to reflect the degree of dispersion and consistency of the scores.

[0098] Step 120: performing mapping and extraction processing on the video data set to obtain spatiotemporal features of the video;

[0099] Reference Figure 2 In some embodiments, the step 120 of performing mapping and extraction processing on the video data set to obtain the spatiotemporal features of the video includes:

[0100] Step 121, extract spatial features from the video data set, extract a key frame from every 8 frames of the video, save the key frame to a specified folder, and use the SwinTransformer model to process the key frame to extract spatial features. This method of extracting key frames every 8 frames can effectively reduce the amount of calculation while capturing the spatial feature information of the video more effectively. At the same time, similar processing methods can be used for different types of video data sets, which has strong adaptability. ;

[0101] Step 122: extract temporal features from the video dataset, and use the SlowFast model to process the video clips in the video dataset to extract temporal features; the SlowFast model, with its unique design architecture, can adapt well to videos of different lengths and contents. Regardless of whether the video is long or short, and whether the content is complex or relatively simple, the model can extract effective temporal features from it, providing important temporal dimension information for subsequent video quality assessment.

[0102] In an embodiment of the present application, for each video in the video data set, key frames are extracted every 8 frames, and each key frame is processed using a pre-trained SwinTransformer model. The input requirement of the SwinTransformer model is an image of a fixed size, so the key frames need to be adjusted to the size required by the model, such as 224×224 pixels. The model extracts features from each key frame through its multi-layer self-attention mechanism and convolution operations. Specifically, it focuses on features such as color distribution, texture details, object shapes, and spatial relationships between them in the image. After processing by SwinTransformer, each key frame will obtain a corresponding spatial feature vector. For each set of video key frames, the extracted spatial features are average pooled to obtain a spatial feature representation of the video;

[0103] For each video in the video dataset, it is divided into multiple video clips, and these video clips are processed using the pre-trained SlowFast model. The SlowFast model contains a slow path and a fast path, which process video clips at different frame rates to capture long-term and short-term timing information in the video. After being processed by the SlowFast model, each video clip will obtain a corresponding time feature vector. For multiple video clips of each video, these time feature vectors are spliced ​​and further processed to obtain the time feature representation of the video.

[0104] Step 130, inputting the spatiotemporal features into the initialized STGRU module for feature fusion processing;

[0105] Reference Figure 2 In some embodiments, the step 130 of inputting the spatiotemporal features into the initialized STGRU module for feature fusion processing includes:

[0106] Step 131, dividing the spatiotemporal features into different gating signals (such as t_z, t_r, t_t, t_s and s_z, s_r, s_t, s_s), and obtaining intermediate features through a gating mechanism and a convolution operation on the gating signals;

[0107] Step 132: performing weighted fusion processing on the spatiotemporal features and the intermediate features to obtain updated hidden states T_new and S_new;

[0108] Step 133: concatenate the T_new and S_new to obtain fused output features.

[0109] In the embodiment of the present application, the STGRU module can better capture the spatiotemporal dynamic changes and complex patterns in the video by fusing spatiotemporal features. It combines the gating mechanism, convolution operation and linear mapping, so that the model can adaptively select important feature information, ignore irrelevant or noise information, and improve the quality of feature fusion. The fused features can more accurately reflect the quality characteristics of the video, provide more valuable information input for subsequent evaluation and classification tasks, help improve the generalization ability of the model, enable it to adapt to different types of video data, and improve the accuracy and reliability of reference-free video quality assessment.

[0110] For example, refer to Figure 5 , the temporal features and spatial features are processed through a 3×3 convolution layer to obtain temporal convolution output and spatial convolution output. This convolution processing helps to extract the temporal and spatial correlation of features. The temporal and spatial convolution outputs are divided into four parts, corresponding to the update gate (z), reset gate (r), candidate hidden state (t) and hidden state (s) of the GRU unit. The calculation of the update gate and reset gate is achieved by adding the corresponding temporal convolution output and spatial convolution output and applying the sigmoid activation function. This allows the model to control the flow of information and enhance the capture of important features. The temporal candidate hidden state and the spatial candidate hidden state are obtained by multiplying the temporal features and spatial features with the reset gate output and then applying the tanh activation function. This step provides the basis for the update of the hidden state. The original temporal features and spatial features are mapped to the dimensions matching the hidden state through a linear layer. Combined with the update gate and the hidden state of the previous time step, the updated hidden state of the current time step is calculated. The updated temporal hidden state and spatial hidden state are concatenated in the feature dimension to form the final fusion output feature. This fusion feature can more comprehensively reflect the spatiotemporal characteristics of the video.

[0111] Step 140: Mapping the fused spatiotemporal features to a video quality score through a regression module;

[0112] In some embodiments, the step 140 of inputting the spatiotemporal fusion features into a regression module to map and obtain a video quality score includes:

[0113] Step 141, map the fused spatiotemporal features to predicted quality scores through two fully connected layers. The parameters of these layers are learned during the training process to minimize the difference between the predicted scores and the true scores;

[0114] Step 142: Use the ReLU activation function to increase the nonlinearity of the model to help the model learn more complex feature representations;

[0115] Step 143, video quality scores are usually in the range of 0 to 100. Using the torch.clamp method to ensure that the output value is within this range helps to generate realistic and interpretable quality scores.

[0116] In the embodiment of the present application, after obtaining the fused spatiotemporal features, they are input into the first fully connected layer, and their weight is W 1 , with a bias of b 1 , and then apply the ReLU activation function to perform nonlinear transformation on the output of the first layer to increase the expressive power of the model. In order to map the features after nonlinear transformation to the range of video quality score, a second fully connected layer is designed. The output of the ReLU activation function is input to the second fully connected layer, and its weight is W 2 , with a bias of b 2 Since video quality scores are usually in the range of 0 to 100, a clamp operation is used to limit the output to this range.

[0117] Step 150: train the model using an optimization algorithm (such as Adam) and a loss function (such as L1RankLoss) to obtain a trained no-reference video quality assessment model, including:

[0118] Step 151, calculating the loss weight of the fusion feature input to the initialized quality regression module;

[0119] Step 152: Update the parameters of the initialized no-reference video quality assessment model according to the loss weight.

[0120] In an embodiment of the present application, the program first determines whether to use the CPU or GPU for training, then loads the predefined model architecture and deploys it to the selected device. If pre-trained weights are provided, the model will load these weights, but will exclude the weights of the head layer so that these weights can be updated during training. If specified, the parameters of all layers in the model except the head will be frozen to prevent them from being updated during training. Secondly, determine the parameters in the model that require gradient updates and optimize them using the AdamW optimizer. Set the learning rate scheduler to adjust the learning rate according to a predetermined interval and decay rate during training, and initialize the loss function L1RankLoss. For each training cycle (epoch), the model performs the following operations:

[0121] Set to training mode;

[0122] Iterate over the training dataset, perform forward and backward propagation for each batch, calculate the loss, and update the model parameters;

[0123] Print training loss and computation time;

[0124] Adjust the learning rate;

[0125] Validation was performed on the test dataset and performance metrics such as Spearman rank correlation coefficient (SRCC), Kendall rank correlation coefficient (KRCC), Pearson rank correlation coefficient (PLCC) and root mean square error (RMSE) were calculated.

[0126] If the performance metric for the current cycle is better than the previous best performance, the state of the model is saved.

[0127] It should be noted that before inputting the video data set into the no-reference video quality assessment model for training, a certain number of user-generated video verification sets can be obtained by data division. The division ratio between the user-generated video verification set and the user-generated video set can be set according to actual needs, for example, it can be set to 7:3, 6:4, 8:2, etc., and this application will not go into details here.

[0128] The present application also provides a method for identifying a no-reference video quality assessment model, including:

[0129] Step 160: Obtain the user-generated video to be evaluated;

[0130] Step 170: input the user-generated video into the trained no-reference video quality assessment model as described in the first aspect for assessment to obtain an assessment result.

[0131] In an embodiment of the present application, user-generated videos can be acquired using various camera devices or downloaded from social platforms through legal channels. The acquired user-generated videos are input into a trained no-reference video quality assessment model for evaluation, and the evaluation results output by the trained no-reference video quality assessment model can be obtained.

[0132] The following describes in detail a training system for a no-reference video quality assessment model proposed according to an embodiment of the present application with reference to the accompanying drawings.

[0133] Reference Figure 3 , the no-reference video quality assessment model proposed in the embodiment of the present application includes:

[0134] An acquisition module 101 is used to acquire a labeled video data set;

[0135] An extraction module 102 is used to perform mapping extraction processing on the video data set to obtain the spatiotemporal features of the video;

[0136] A fusion module 103 is used to input the spatiotemporal features into the initialized STGRU module for feature fusion processing;

[0137] A regression module 104, used to map the fused spatiotemporal features to a video quality score;

[0138] The training module 105 is used to input the fused output features into an initialized quality regression module for training to obtain a trained no-reference video quality assessment model.

[0139] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] Reference Figure 4 , the embodiment of the present application further provides an electronic device, including:

[0141] at least one processor 201;

[0142] At least one memory 202, used to store at least one program;

[0143] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above method embodiment.

[0144] Similarly, it can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0145] The embodiment of the present application further provides a computer-readable storage medium, in which a program executable by the processor 201 is stored. The program executable by the processor 201 is used to implement the above-mentioned method embodiment when executed by the processor 201.

[0146] Similarly, the contents of the above method embodiments are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0147] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0148] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0149] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0150] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0151] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0152] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0153] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0154] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0155] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A training method for a no-reference video quality assessment model, characterized in that: The following steps are involved: Get the labeled video dataset; Performing mapping and extraction processing on the video data set to obtain spatiotemporal features of the video; Input the spatiotemporal features into the initialized STGRU module for feature fusion processing; The fused spatiotemporal features are mapped to video quality scores through a regression module; The loss function is used to train the model to obtain a trained no-reference video quality assessment model.

2. The training method of a no-reference video quality assessment model according to claim 1, characterized in that: The spatiotemporal features refer to feature information in the spatial and temporal dimensions contained in the video, including the spatial features of the video frame and the temporal features of the video segment; The mapping and extraction processing is performed on the video data set to obtain the spatiotemporal features of the video, including: Extracting spatial features from the video data set, extracting a key frame from the video every multiple frames, saving the key frame to a specified folder, and processing the key frame using a Swin Transformer model to obtain spatial features; Temporal features are extracted from the video data set, and the video clips in the video data set are processed using a Slow Fast model to obtain temporal features.

3. The training method of a no-reference video quality assessment model according to claim 2, characterized in that: As a visual model based on the Transformer architecture, the SwinTransformer can capture the global and local spatial features of an image through a self-attention mechanism; these features include color, texture, shape, and spatial structure; The Slow Fast model is a deep learning model for video understanding, which contains two paths: a slow path for capturing long-term timing information in the video, and a fast path for capturing short-term timing information; this design enables the model to understand both fast actions and slow changes in the video; The spatial features and temporal features are combined to form a complete spatiotemporal feature set.

4. The training method of a no-reference video quality assessment model according to claim 1, characterized in that: The step of inputting the spatiotemporal features into the initialized STGRU module for feature fusion processing includes: The spatiotemporal features are divided into different gating signals, and the gated signals are subjected to gating mechanisms and convolution operations to obtain intermediate features; Performing weighted fusion processing on the spatiotemporal features and the intermediate features to obtain updated hidden states T_new and S_new; The hidden states T_new and S_new are concatenated to obtain fused output features.

5. The training method of a no-reference video quality assessment model according to claim 1, characterized in that: The fused spatiotemporal features are mapped to the video quality score through the regression module, including: The fused spatiotemporal features are mapped to the predicted quality scores through two fully connected layers; The ReLU activation function is used to increase the nonlinearity of the model to help the model learn more complex feature representations.

6. The training method of a no-reference video quality assessment model according to claim 1, characterized in that: The use of a loss function to train the model includes: Calculate the loss weight of the fused spatiotemporal features input into the initialized quality regression module; According to the loss weight, the parameters of the initialized no-reference video quality assessment model are updated.

7. A method for assessing video quality without reference, characterized in that: The following steps are involved: Obtaining user-generated videos to be evaluated; Inputting the user-generated video into a trained no-reference video quality assessment model for assessment to obtain an assessment result; Wherein, the no-reference video quality assessment model is trained using the method described in any one of claims 1-6.

8. A training system for a no-reference video quality assessment model, characterized in that: include: Acquisition module, obtains the labeled video dataset; An extraction module performs mapping and extraction processing on the video data set to obtain the spatiotemporal features of the video; A fusion module inputs the spatiotemporal features into the initialized STGRU module for feature fusion processing; Regression module, through which the fused spatiotemporal features are mapped to video quality scores; The training module uses the loss function to train the model and obtains the trained no-reference video quality assessment model.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

Images