Video quality evaluation method and video quality evaluation model training method, electronic device, storage medium and program product

Abstract

This application provides a video quality assessment method, a video quality assessment model training method, an electronic device, a storage medium, and a program product. The video quality assessment method includes: acquiring a video frame sequence to be assessed; extracting spatiotemporal features from at least a portion of the video frames in the video frame sequence that have a temporal relationship to obtain a corresponding spatiotemporal feature sequence; performing feature extraction based on a first temporal receptive field on the spatiotemporal feature sequence to obtain local quality change features corresponding to each spatiotemporal feature; and performing feature extraction based on a second temporal receptive field on the spatiotemporal feature sequence to obtain global quality change features corresponding to each spatiotemporal feature; adaptively fusing the local quality change features and the global quality change features to obtain a quality change feature sequence; and determining the quality assessment result of the video frame sequence to be assessed based on the quality change feature sequence, so as to make the quality assessment result more accurate.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to video quality assessment methods and video quality assessment model training methods, electronic devices, computer storage media, and computer program products. Background Technology

[0002] With the popularization of ultra-high-definition video, live streaming, short video and other applications, users have increasingly higher requirements for video viewing experience. Therefore, it is becoming more and more important to identify distortion phenomena that are prone to occur in video transmission and compression, such as blurring, blockiness, stuttering and bitrate abrupt changes.

[0003] Referenceless VQA, used in relevant video quality assessment (VQA) schemes, can predict the quality of a video based solely on the video itself, without the need for an original, distortion-free video as a reference. However, because referenceless VQA schemes struggle to reproduce the complex human perception of dynamic quality, they rely heavily on manual review, increasing the cost and reducing the efficiency of video assessment. Summary of the Invention

[0004] In view of the above, embodiments of this application provide a video quality assessment method, a video quality assessment model training method, an electronic device, a storage medium, and a program product to at least partially solve the above problems.

[0005] According to a first aspect of the embodiments of this application, a video quality assessment method is provided, comprising: acquiring a video frame sequence to be assessed, and extracting spatiotemporal features from at least a portion of the video frames in the video frame sequence that have a temporal relationship to obtain a corresponding spatiotemporal feature sequence; performing feature extraction based on a first temporal receptive field on the spatiotemporal feature sequence to obtain local quality change features corresponding to each spatiotemporal feature; and performing feature extraction based on a second temporal receptive field on the spatiotemporal feature sequence to obtain global quality change features corresponding to each spatiotemporal feature, wherein the first temporal receptive field is smaller than the second temporal receptive field; adaptively fusing the local quality change features and the global quality change features to obtain a quality change feature sequence, and determining the quality assessment result of the video frame sequence to be assessed based on the quality change feature sequence.

[0006] According to a second aspect of the embodiments of this application, a video quality assessment model training method is provided, comprising: acquiring training samples, wherein the training samples include video frame sequence samples and video quality labels corresponding to the video frame sequence samples; inputting the video frame sequence samples into a video quality assessment model to be trained, so as to obtain local quality change features and global quality change features corresponding to the video frame sequence samples through the video quality assessment model, and adaptively fusing the local quality change features and global quality change features, predicting the quality assessment result of the video frame sequence samples based on the adaptive fusion result; and training the video quality assessment model based on the difference between the predicted quality assessment result and the video quality labels.

[0007] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first or second aspect.

[0008] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the operation corresponding to the method described in the first or second aspect.

[0009] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including computer instructions that instruct a computing device to perform an operation corresponding to the method described in the first or second aspect.

[0010] According to the video quality assessment scheme provided in this application, a video frame sequence to be assessed can be obtained, and spatiotemporal features can be extracted from at least a portion of the video frames in the video frame sequence that have a temporal relationship to obtain the corresponding spatiotemporal feature sequence. Then, feature extraction based on a first temporal receptive field is performed on the spatiotemporal feature sequence to accurately assess the distortion details of local segments, simulating human short-term memory to obtain local quality change features corresponding to each spatiotemporal feature; and feature extraction based on a second temporal receptive field is performed on the spatiotemporal feature sequence to capture the global quality change trend of the entire video, simulating human long-term memory to obtain global quality change features corresponding to each spatiotemporal feature. To enable the video quality assessment model to simulate both human long and short-term memory, the first temporal receptive field should be smaller than the second temporal receptive field. Next, the local quality change features and global quality change features can be adaptively fused to intelligently weigh the contributions of local and global information to the final quality score, achieving adaptive fusion of local and global quality change features to obtain a quality change feature sequence. Ultimately, the quality assessment results of the video frame sequence to be evaluated can be determined based on the quality change feature sequence. This simulates the complex cognitive process of the human visual system under the combined effect of short-term and long-term memory when assessing video quality. As a result, the final quality assessment results can reflect both local distortion details and the global quality evolution trend and content correlation. This reduces the reliance on manual review and the cost of video quality assessment, while improving the efficiency and accuracy of video quality assessment. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0012] Figure 1 This is a schematic diagram of an exemplary system to which the embodiments of this application are applicable; Figure 2 This is a flowchart illustrating the steps of a video quality assessment method according to an embodiment of this application. Figure 3 This is a flowchart of a video quality assessment model training method according to an embodiment of this application; Figure 4 This is a structural diagram of a video quality assessment model according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating a video quality assessment scenario according to an embodiment of this application; Figure 6This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0013] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0014] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0015] Figure 1 An exemplary system for a video quality assessment method applicable to embodiments of this application is shown. For example... Figure 1 As shown, the system 100 may include a cloud server 102, a communication network 104, and / or one or more user devices 106. Figure 1 The example in the text shows multiple user devices.

[0016] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, the cloud server 102 can be used to perform quality assessment on the acquired sequence of video frames to be evaluated, such as through a video quality assessment model. In other embodiments, the cloud server 102 may also be deployed with a video quality assessment model, which can optionally be used to train the video quality assessment model. In some embodiments, during the video quality assessment process, the cloud server 102 can be used to acquire the video frame sequence to be assessed, and extract spatiotemporal features from at least some of the video frames in the video frame sequence that have a temporal relationship to obtain the corresponding spatiotemporal feature sequence; perform feature extraction based on a first temporal receptive field on the spatiotemporal feature sequence to obtain local quality change features corresponding to each spatiotemporal feature; and perform feature extraction based on a second temporal receptive field on the spatiotemporal feature sequence to obtain global quality change features corresponding to each spatiotemporal feature, wherein the first temporal receptive field is smaller than the second temporal receptive field; adaptively fuse the local quality change features and the global quality change features to obtain a quality change feature sequence, and determine the quality assessment result of the video frame sequence to be assessed based on the quality change feature sequence.

[0017] As an optional example, in some embodiments, the cloud server 102 can be used to obtain training samples during the training of the video quality assessment model. The training samples include video frame sequence samples and corresponding video quality labels. The video frame sequence samples are input into the video quality assessment model to be trained, so as to obtain the local quality change features and global quality change features corresponding to the video frame sequence samples through the video quality assessment model. Adaptive fusion is performed based on the local quality change features and global quality change features. The quality assessment result of the video frame sequence samples is predicted based on the result of adaptive fusion. The video quality assessment model is trained based on the difference between the predicted quality assessment result and the video quality label.

[0018] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user equipment 106 can connect to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud server 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user equipment 106 and the cloud server 102, such as a network link, a dial-up link, a wireless link, a hardwired link, any other suitable communication link, or any suitable combination of such links.

[0019] User device 106 may include one or more user devices adapted to interact with a user, acquire user-uploaded videos, and transmit them to cloud server 102 so that cloud server 102 can generate a sequence of video frames to be evaluated or training samples based on the acquired videos. For example, in some embodiments, user device 106 may include mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, game consoles, media players, vehicle entertainment systems, and / or any other suitable type of user device.

[0020] Based on the above system, this application provides a video quality assessment method and a training scheme for a video quality assessment model applicable to scenarios with no reference video quality assessment. The following is an illustration through several embodiments.

[0021] Reference Figure 2 The flowchart illustrates a video quality assessment method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps: Step S101: Obtain the video frame sequence to be evaluated, and extract spatiotemporal features from at least some of the video frames in the video frame sequence that have temporal relationships to obtain the corresponding spatiotemporal feature sequence.

[0022] In this embodiment, the video frame sequence to be evaluated can be any sequence of frames in a video. Video quality evaluation can be performed on all frames of this video frame sequence. However, in an alternative approach, to improve the efficiency of video quality evaluation, sampling can be used to obtain video frames with temporal relationships from the video frame sequence. For example, by fixing the frame rate, such as 2fps, the obtained video frame sequence can be sampled in chronological order to obtain a new sequence (hereinafter referred to as video frame sequence V for ease of description), such as a sequence containing 64 frames, in which the video frames also retain the original temporal relationship.

[0023] Based on this, spatiotemporal feature extraction (including temporal feature extraction and spatial feature extraction) can be performed on the video frame sequence V to capture the joint feature representation of the spatial layout and dynamic changes in time of the video frames, and obtain the corresponding spatiotemporal feature sequence. The spatiotemporal feature sequence carries the content information of each video frame and also implies the quality-related information of the video frame.

[0024] For example, before performing spatiotemporal feature extraction, the video frame sequence V can be preprocessed to conform to the initial input tensor dimension of the video quality assessment model. For instance, the initial input tensor dimension can be [B, T, C, H, W], such as [1, 64, 3, 224, 224]. Here, B represents the batch size of data input to the model each time; T represents the time dimension, i.e., the number of frames or time steps (the number of consecutive frames); C represents the number of channels, such as 3 channels for RGB; H represents the video frame height; and W represents the video frame width.

[0025] In one example, spatial features can be extracted from each video frame in the video frame sequence V using a pre-trained visual backbone network (such as Swing Transformer or ViT; these are just examples in this embodiment, and the specific feature extraction network used will be whatever is feasible, and will not be listed here). This yields the spatial features corresponding to each video frame. However, to obtain richer information and provide a better foundation for subsequent processing, this example takes the spatial features output by the visual backbone network before global average pooling. Combined with the temporal series information of each video frame, a spatiotemporal feature sequence corresponding to the video frame sequence V can be obtained, denoted as the feature sequence. Its dimension is T×D, where T represents the number of video frames in the video frame sequence V, and D represents the feature dimension.

[0026] Optionally, the spatiotemporal feature sequence can be represented as:

[0027] in, Indicates the feature extraction operator, This represents the number of video frames in the video frame sequence V. Indicates the first Frame spatiotemporal features.

[0028] In an optional embodiment, the video quality assessment method of this application can be implemented through a video quality assessment model, which may include at least a spatiotemporal feature extraction layer, a prediction output layer, and parallel first branches, second branches, and adaptive fusion branches. The aforementioned visual backbone network can serve as the spatiotemporal feature extraction layer of this video quality assessment model, enabling the extraction of spatiotemporal features from at least a portion of video frames in a video frame sequence that have a temporal relationship, thereby obtaining the corresponding spatiotemporal feature sequence. The parallel first branch, second branch, and adaptive fusion branch are connected after the spatiotemporal feature extraction layer, while the prediction output layer is connected after the aforementioned three parallel parts. The functions of each part of this video quality assessment model will be explained in conjunction with the steps of this embodiment.

[0029] Step S103: Perform feature extraction based on the first time receptive field on the spatiotemporal feature sequence to obtain the local quality change features corresponding to each spatiotemporal feature; and perform feature extraction based on the second time receptive field on the spatiotemporal feature sequence to obtain the global quality change features corresponding to each spatiotemporal feature, wherein the first time receptive field is smaller than the second time receptive field.

[0030] In the embodiments of this application, local quality change features can assess the quality of local videos, such as blockiness, image blurring, color shift, etc., to obtain local "detailed viewing features", while global quality change features can capture the overall quality of the video and reflect the global quality change trend of the video, such as brightness fluctuation, sharpness fluctuation, etc., to obtain the video "rough viewing features".

[0031] In this embodiment, different temporal receptive fields are set, namely a first temporal receptive field and a second temporal receptive field, to extract corresponding local quality change features and global quality change features, respectively. The temporal receptive field characterizes the range of input sequences that a single neuron can perceive in the time dimension, determining how much historical information the neuron "can see." The larger the temporal receptive field, the longer the temporal dependencies the model can capture. Based on this, in this embodiment, the first temporal receptive field is smaller than the second temporal receptive field.

[0032] For example, each time point can be extracted based on the first receptive field. Corresponding local mass change characteristics The local quality change feature sequence was obtained. Additionally, each time point can be extracted based on the second time receptive field. Corresponding global quality change characteristics The global quality change feature sequence is obtained. If the above video quality assessment model is used, this step can be implemented by the first branch and the second branch. The first branch can perform feature extraction on the spatiotemporal feature sequence based on the first time receptive field to obtain the local quality change features corresponding to each spatiotemporal feature; the second branch can perform feature extraction on the spatiotemporal feature sequence based on the second time receptive field to obtain the global quality change features corresponding to each spatiotemporal feature.

[0033] Step S105: Adaptively fuse local quality change features and global quality change features to obtain a quality change feature sequence, and determine the quality assessment result of the video frame sequence to be evaluated based on the quality change feature sequence.

[0034] In this embodiment, adaptive fusion based on local and global quality change features is a fusion strategy that dynamically adjusts the local and global quality change features. Through adaptive fusion, the final quality change feature sequence can more accurately reflect the spatiotemporal dynamic changes in video quality. For example, adaptive fusion can be implemented using methods such as dynamic weight allocation, feature selection and gating, and nonlinear fusion.

[0035] In one optional embodiment of this application, a fusion weight corresponding to each spatiotemporal feature in the obtained spatiotemporal feature sequence can be obtained. The fusion weight indicates the degree of influence of the video frame corresponding to each spatiotemporal feature on the global quality change indicated by the global quality change feature. Then, based on the fusion weight, the local quality change features corresponding to each spatiotemporal feature are weighted and fused with the global quality change feature to obtain a quality change feature sequence. This method can obtain the quality change feature sequence more efficiently, and is more flexible and has greater generalization ability.

[0036] For example, assume that the fusion weights corresponding to the spatiotemporal feature sequence at each time point are predicted. The characteristic sequence of the local quality change mentioned above was obtained. and global quality change feature sequence Corresponding fusion weight sequence The fusion weight sequence Here, the weights are merged. Used to indicate at each time node The degree of influence of video frames corresponding to each spatiotemporal feature on the global quality change indicated by the global quality change feature suggests whether the local quality change feature or the global quality change feature should be trusted more.

[0037] After determining the fusion weight sequence Then, based on this fusion weight sequence For each time node The corresponding local quality change features and global quality change features are adaptively fused to obtain the results for each time point. Corresponding mass change characteristics And based on the characteristics of quality change Determine the characteristic sequence of quality change ,in, .

[0038] Next, we can base our analysis on the quality change characteristic sequence. Further processing (including but not limited to processing by a video quality assessment model, or performing temporal analysis, or statistical modeling) is required to determine the video quality assessment result, such as a quality score, for the video frame sequence to be evaluated. wait.

[0039] If the above video quality assessment model is adopted, this step can use the adaptive fusion branch to adaptively fuse local quality change features and global quality change features to obtain a quality change feature sequence; and the prediction output layer can predict and output the video quality assessment result based on the quality change feature sequence.

[0040] As described above, in this embodiment, a video frame sequence to be evaluated can be obtained, and spatiotemporal features can be extracted from at least a portion of the video frames in the sequence that have a temporal relationship to obtain a corresponding spatiotemporal feature sequence. Then, feature extraction based on a first temporal receptive field is performed on the spatiotemporal feature sequence to accurately evaluate the distortion details of local segments, simulating human short-term memory to obtain local quality change features corresponding to each spatiotemporal feature. Furthermore, feature extraction based on a second temporal receptive field is performed on the spatiotemporal feature sequence to capture the global quality change trend of the entire video, simulating human long-term memory to obtain global quality change features corresponding to each spatiotemporal feature. To enable the video quality evaluation model to simulate both human long and short-term memory, the first temporal receptive field should be smaller than the second temporal receptive field. Next, adaptive fusion of local and global quality change features can be performed to intelligently weigh the contributions of local and global information to the final quality score, achieving adaptive fusion of local and global quality change features to obtain a quality change feature sequence. Ultimately, the quality assessment results of the video frame sequence to be evaluated can be determined based on the quality change feature sequence. This simulates the complex cognitive process of the human visual system under the combined effect of short-term and long-term memory when assessing video quality. As a result, the final quality assessment results can reflect both local distortion details and the global quality evolution trend and content correlation. This reduces the reliance on manual review and the cost of video quality assessment, while improving the efficiency and accuracy of video quality assessment.

[0041] In some optional implementations, step S103 above, which involves extracting features from the spatiotemporal feature sequence based on the first time receptive field to obtain the local quality change features corresponding to each spatiotemporal feature, may include: performing feature extraction from the spatiotemporal feature sequence based on the first time receptive field, aggregating adjacent features of each spatiotemporal feature in the spatiotemporal feature sequence, and obtaining the local quality change features corresponding to each spatiotemporal feature.

[0042] In this embodiment of the application, adjacent feature aggregation is used to aggregate the time nodes corresponding to each spatiotemporal feature in the spatiotemporal feature sequence. Aggregate local contexts, such as for each time node. The spatiotemporal features corresponding to the nearest 3-5 frames are aggregated. It should be understood that the 3-5 frames are only an example, and the aggregation range of the receptive field at the first moment can be set according to actual usage requirements.

[0043] In one alternative approach, during the aggregation of adjacent spatiotemporal features, multiple stacked convolutional layers can be used to extract features from the spatiotemporal feature sequence based on the first time receptive field, thereby aggregating adjacent features in the spatiotemporal feature sequence and obtaining the local quality change features corresponding to each spatiotemporal feature.

[0044] For example, the convolutional layer can be a one-dimensional convolutional layer. This one-dimensional convolutional layer With a smaller kernel size and stride, it can effectively extract features and aggregate adjacent features, while also having a smaller scale and better flexibility.

[0045] It should be understood that the kernel size... The smaller the setting, the more refined the analysis of local distortion details, such as... However, considering that the distortion details that spatiotemporal features can reveal within 3 frames are extremely short-term, such as single-frame noise, the aggregation depth can be increased by stacking multiple convolutional layers to increase the types of short-term quality changes that can be identified without diluting local details.

[0046] For example, if the number of stacked convolutional layers is 3, then the first layer is used for recognition. Very short-term quality changes in frames, such as t The second layer is used to capture the quality abrupt changes between frames and two adjacent frames, building upon the first layer. Short-term details of the frame, such as Frame noise and The fuzzy correlation of frames, the third layer is used to capture, based on the second layer. Short-term details of the frame, such as Frame blocking and Frame stuttering, etc.

[0047] Taking the first branch of the aforementioned video quality assessment model as an example, this first branch may include one or more standard one-dimensional convolutional layers. Stacked, each one-dimensional convolutional layer This is followed by an activation function (such as ReLU) and layer normalization (LayerNorm). The first branch employs a small first-time receptive field, which focuses on aggregating spatiotemporal features from neighboring time points to form a fine-grained contextual awareness for each time point t, accurately capturing short-term, local video quality details.

[0048] Based on this, for the first branch, after obtaining the spatiotemporal feature sequence X, X is input into a one-dimensional convolutional layer. In the middle, this one-dimensional convolutional layer It has a small kernel size k_s (e.g., 3) and a stride of 1. The convolution result is processed by the ReLU activation function, and the result after activation is normalized. Multiple such "convolution-activation-normalization" blocks can be stacked to deepen the model. For each one-dimensional convolutional layer... In other words, after obtaining this one-dimensional convolutional layer Local fusion information between the current frame and adjacent frames Then, the ReLU activation function can be used to optimize the feature aggregation results. The introduction of nonlinear expressive power allows the model to fit complex patterns of local distortion features, such as distinguishing feature differences between slight noise and severe block artifacts. Then, LayerNorm is applied to each time node. The corresponding features are normalized to stabilize the feature distribution, and finally, the features at each time point are obtained. Corresponding local mass change characteristics .

[0049] In one example, the characteristics of local mass variation are calculated. The method is as follows:

[0050]

[0051] in, Indicates the convolution kernel weights. Represents the first in the spatiotemporal feature sequence Feature vectors at each time point This represents the bias term of the convolution operation. Represents the ReLU activation function. Presentation layer normalization processing.

[0052] Then, based on local quality change characteristics Feature sequences for determining local quality change characteristics ,in, .

[0053] In this example, multiple stacked one-dimensional convolutional layers can be used to extract features from the spatiotemporal feature sequence based on the first time receptive field. This allows for the aggregation of adjacent features of the spatiotemporal features at each time point, thereby obtaining the local quality change features corresponding to each time point. This simulates the "close viewing" that humans do when watching videos, in order to accurately capture short-term, local quality details of the video.

[0054] Since local quality variation features are difficult to objectively reflect the overall and comprehensive quality of a video, this application embodiment also incorporates global quality variation features of the video. In some optional implementations, step S103 above, which involves performing feature extraction based on a second temporal receptive field on the spatiotemporal feature sequence to obtain the global quality variation features corresponding to each spatiotemporal feature, may include: performing feature extraction based on a second temporal receptive field on the spatiotemporal feature sequence, and aggregating non-adjacent features in each spatiotemporal feature sequence to obtain the global quality variation features corresponding to each spatiotemporal feature.

[0055] In this embodiment, non-adjacent feature aggregation can aggregate spatiotemporal features of non-adjacent time nodes to efficiently capture long-term, global quality change trends in videos, such as persistent blurring throughout the video, global brightness fluctuations, and quality fluctuations across multiple frames. For example, for time nodes... Time nodes Previous Frames, and time nodes After The spatiotemporal features of the frames are aggregated. The aggregation range of the second temporal receptive field can be set according to actual usage requirements, and will not be listed one by one in this application.

[0056] In one alternative approach, during the aggregation of non-adjacent spatiotemporal features, a dilated convolutional layer with a preset dilation rate can be used to extract features from the spatiotemporal feature sequence based on a second temporal receptive field. This allows for the aggregation of non-adjacent features in the spatiotemporal feature sequence to obtain the global quality change features corresponding to each spatiotemporal feature. Therefore, without increasing the number of parameters or computational cost, the second temporal receptive field can be increased exponentially, enabling more efficient aggregation of non-adjacent features.

[0057] Dilated convolutional layers significantly expand the receptive field without increasing the kernel size or the number of parameters by introducing "holes" (spacing) into the convolutional kernel. In temporal data, dilated convolution is applied along the time dimension. For example, for a given dilated convolutional layer... If the kernel size The preset void ratio is 3. This allows the second temporal receptive field to be expanded from 3 frames to 9 frames without increasing the kernel size. It should be understood that the preset porosity of a porous convolutional layer... The settings can be adjusted according to the actual usage scenario; no specific limitations are made in this application.

[0058] Taking the second branch of the aforementioned video quality assessment model as an example, the second branch may include a temporal residual dilated convolutional layer. By setting a large dilation rate r (e.g., 4 or 8), the temporal receptive field can be increased exponentially without increasing the number of parameters and computational cost, thereby aggregating features from more distant time points to efficiently capture long-term, global video quality change trends.

[0059] Taking the aforementioned spatiotemporal feature sequence X as an example, X is input into the one-dimensional dilated convolutional layer of the second branch. This approach leverages the inductive bias of dilated convolutions to efficiently aggregate contextual information within a given range, providing a more structured input for subsequent processing. This dilated convolutional layer... It has the same kernel size k_s as the first branch, but with a larger hole ratio r (e.g., 4 or 8).

[0060] In one example, the dilated convolution result is obtained. It can be represented as:

[0061] in, Indicates the convolution kernel weights. Represents the first in the spatiotemporal feature sequence The feature vector of the frame, This represents the bias term of dilated convolution.

[0062] Next, global long-range dependencies between video frames can be dynamically modeled using Transformer encoder layers. In one alternative approach, a cascaded structure of two Transformer encoder layers can be employed. For example, this Transformer encoder can capture the dependencies between the spatiotemporal features of any two frames (e.g., frame 1 and frame 64) in a T-frame video, and integrate the global contextual information of the 64-frame spatiotemporal feature sequence X. For instance, the first Transformer encoder layer can establish preliminary associations between video frames (e.g., motion in adjacent frames or global scene changes); the second Transformer encoder layer can further integrate information and extract global features relevant to video quality assessment based on the output of the first encoder layer (e.g., distinguishing between cases where the entire frame is blurred and cases where the first half is blurred but the second half is clear).

[0063] For example, each Transformer encoder layer contains a Multi-Head Self-Attention (MHSA) module and a Feedforward Neural Network (FFN), and is stabilized through residual connections and layer normalization. MHSA can calculate the correlation weights between each video frame in the sequence and all other video frames, thereby dynamically and globally aggregating information and effectively solving the problem that traditional methods cannot perform ultra-long-range dependency modeling of video quality-related features.

[0064] Next, a nonlinear transformation can be applied to the global quality variation characteristics aggregated by MHSA using FFN to fit complex global quality variation trends. Examples include combined distortion variations with blurriness in the first 30 frames and stuttering in the next 34 frames, and quality variations with an initial decrease followed by an increase in quality over 64 frames. In one example, the Transformer encoder outputs the values ​​at each time point. Corresponding global quality change characteristics The method is as follows:

[0065] in, This indicates the processing of the first layer Transformer encoder. This indicates that the second-layer Transformer encoder can perform processing on the output of the first-layer Transformer encoder.

[0066] Then, based on global quality change characteristics To determine the feature sequence of global quality change characteristics ,in, .

[0067] In this embodiment, a dilated convolutional layer with a preset dilation rate can be used to extract features from the spatiotemporal feature sequence based on the second temporal receptive field. This allows for the aggregation of non-adjacent features of the spatiotemporal features at each time point, thereby obtaining the global quality change features corresponding to each time point. This simulates the "scanning" of a human when watching a video, and accurately captures the long-term, global quality change trend of the video.

[0068] In some optional implementations, step S105 above, namely, the operation of adaptively fusing local quality change features and global quality change features to obtain a quality change feature sequence, can optionally be implemented as follows: based on the spatiotemporal feature sequence, obtain the fusion weight corresponding to each spatiotemporal feature in the spatiotemporal feature sequence, wherein the fusion weight is used to indicate the degree of influence of the video frame corresponding to each spatiotemporal feature on the global quality change indicated by the global quality change feature; based on the fusion weight, perform weighted fusion of the local quality change features and global quality change features corresponding to each spatiotemporal feature to obtain a quality change feature sequence.

[0069] If a video quality assessment model is used, then based on the adaptive fusion branch in the video quality assessment model, for each time node... The dynamic evaluation assesses which characteristic—global or local—has higher reliability, in order to obtain data at each time point. Corresponding fusion weights For example, this fusion weight The range can be (0, 1).

[0070] Further optionally, obtaining the fusion weights corresponding to each spatiotemporal feature in the spatiotemporal feature sequence based on the spatiotemporal feature sequence may include: performing gating processing on the spatiotemporal feature sequence to obtain the fusion weights corresponding to each spatiotemporal feature in the spatiotemporal feature sequence.

[0071] Gating processing can be based on the input spatiotemporal feature sequence. The system generates fusion weights between 0 and 1 for each frame's spatiotemporal features, thereby controlling the feature fusion ratio corresponding to different branches. In one optional approach, gating is implemented through a gating unit, which may include an MLP (Multilayer Perceptron) and a sigmoid activation function. The gating unit performs gating processing on the spatiotemporal feature sequence to obtain the local-global fusion weights corresponding to each frame's spatiotemporal features. For example, the gating unit can learn patterns related to quality distortion in the input features through an MLP and generate weight values ​​in the range of (0,1) using a sigmoid activation function to control the fusion ratio of local quality change features and global quality change features. For instance, when local noise is detected within a frame, the gating unit assigns a higher weight to the local quality change features for that frame; when a global blurring trend is detected across the entire video, a higher weight is assigned to the global quality change features.

[0072] For example, an MLP may include two linearization layers and one ReLU activation function layer. The first linearization layer performs a linear transformation of the features, such as linearly transforming the 768-dimensional features of each frame to reduce the dimensionality to 192 dimensions, to extract information more relevant to video quality, such as local noise and global blur features. Then, after non-linear mapping by the ReLU activation function layer, it is further transformed into 192-dimensional features by the second linearization layer.

[0073] After obtaining the features of the MLP output, the Sigmoid activation function can be used to convert the corresponding time points of each frame. The features are mapped to the (0,1) interval to obtain the spatiotemporal features of each frame at time points. Corresponding fusion weights .in, The higher the value, the greater the impact of global quality change characteristics; conversely, the lower the value, the smaller the complementary weight (1-) of local quality change characteristics. The higher the fusion weight, the greater the influence of local quality variation characteristics. In one example, the fusion weight... It can be represented as follows:

[0074] in, and These are learnable parameters. This represents the Sigmoid activation function. Then, it can be based on the fusion weights. Determine the fusion weight sequence ,in, .

[0075] After obtaining the fusion weights, the local quality change features and global quality change features corresponding to each spatiotemporal feature can be weighted and fused based on these fusion weights to obtain a quality change feature sequence. In this embodiment, the above-mentioned fusion weight sequence can be used... Characteristics of local quality changes Characteristics of global quality changes Perform frame-by-frame weighted summation to obtain the summation at each time point. Corresponding mass change characteristics In one example, the mass change characteristic It can be represented as follows:

[0076] Then, based on the characteristics of quality change Determine the characteristic sequence of quality change ,in, .

[0077] In this embodiment, the fusion weights corresponding to each spatiotemporal feature in the spatiotemporal feature sequence can be determined, and based on the fusion weights, the local quality change features and global quality change features corresponding to each spatiotemporal feature are weighted and fused. This enables the video quality assessment model to intelligently determine the proportion of global quality change features introduced at each time node, overcoming the shortcomings of long-range dependency loss, so as to better simulate the subjective feelings of humans when watching videos and improve the accuracy of video quality assessment results.

[0078] In some optional implementations, step S105 above, which determines the quality assessment result of the video frame sequence to be evaluated based on the quality change feature sequence, may include: performing a pooling operation on the quality change feature sequence to obtain candidate quality features corresponding to the video frame sequence; and performing quality prediction based on the candidate quality features to obtain the quality assessment result.

[0079] In this embodiment, the pooling operation is used to pool a multidimensional sequence of quality variation features into a single feature vector representing the global quality of the entire video, thereby obtaining candidate quality features. In practical applications, pooling operations can be performed using methods such as max pooling or global average pooling. Those skilled in the art can choose the appropriate pooling operation method according to actual needs.

[0080] Taking global average pooling as an example, this pooling operation can perform global average pooling on the sequence of quality change features, pooling the quality features of each frame into overall quality features at the video level. This avoids the excessive influence of a single frame on the overall score, thus reflecting the overall quality level of the video more accurately and objectively. In one example, global average pooling can be represented as follows:

[0081] Based on this, quality prediction is performed using candidate quality features to obtain the quality assessment result. In one example, the quality assessment result can be expressed as a video quality score. Characterization.

[0082] When using the aforementioned video quality assessment model, the quality change feature sequence can be pooled through the prediction output layer to obtain candidate quality features corresponding to the video frame sequence. Quality prediction is then performed based on these candidate quality features to obtain the quality assessment result. For example, the prediction output layer may include a pooling layer and an MLP (Multi-Level Pooling). The aforementioned global average pooling operation can then be implemented through this pooling layer; subsequently, the candidate quality features obtained based on the pooling layer can be... Input a simple multilayer perceptron (MLP) and predict the output video quality score using its regression head. For example, a video quality score. It can be represented as follows:

[0083] In this embodiment, a pooling operation can be performed on the quality change feature sequence to obtain candidate quality features corresponding to the video frame sequence. Based on these candidate quality features, quality prediction is then performed to obtain a video quality score for the video. This is to quantify the quality assessment results of the video.

[0084] Furthermore, for the aforementioned video quality assessment model, this application also provides a model training method. (Refer to...) Figure 3 The flowchart illustrates a video quality assessment model training method according to an embodiment of this application, such as... Figure 3 As shown, the method includes the following steps: Step S201: Obtain training samples, wherein the training samples include video frame sequence samples and video quality labels corresponding to the video frame sequence samples.

[0085] In the embodiments of this application, training samples can be obtained in any way, such as through a preset training dataset.

[0086] To achieve better training results, the training samples can optionally be diverse, covering different types of quality distortion (such as blur, noise, stuttering, blockiness, etc.), different degrees of distortion, and different video content (such as landscapes, faces, dynamic or static images, etc.), thereby avoiding overfitting of the model to a certain type of video or distortion, thus ensuring the generalization ability of the video quality assessment model.

[0087] Step S203: Input the video frame sequence samples into the video quality assessment model to be trained, so as to obtain the local quality change features and global quality change features corresponding to the video frame sequence samples through the video quality assessment model, and perform adaptive fusion based on the local quality change features and global quality change features, and predict the quality assessment result of the video frame sequence samples based on the adaptive fusion result.

[0088] In the embodiments of this application, the video quality assessment model includes a "detailed viewing" branch, i.e., the first branch, which simulates human short-term memory to evaluate video quality in segments, and a "rough viewing" branch, i.e., the second branch, which simulates human long-term memory to evaluate video quality globally.

[0089] An exemplary video quality assessment model has the following structure: Figure 4 As shown, it includes at least a spatiotemporal feature extraction layer; three parallel branches, namely the first branch, the second branch, and the adaptive fusion branch; and a prediction output layer.

[0090] The spatiotemporal feature extraction layer is used to extract spatiotemporal features from at least a portion of the video frames in the video frame sequence samples that have a temporal relationship, thereby obtaining corresponding spatiotemporal feature sequence samples. In this embodiment, the implementation method for extracting spatiotemporal features from the video frame sequence samples to obtain spatiotemporal feature sequence samples can be referred to the foregoing. Figure 2 The descriptions of the corresponding steps in the illustrated embodiments will not be repeated here.

[0091] The first branch is used to extract features from the spatiotemporal feature sequence samples based on the first time receptive field, obtaining local quality change feature samples corresponding to each spatiotemporal feature. In an optional approach, the first branch can be used to extract features from the spatiotemporal feature sequence samples based on the first time receptive field through multiple stacked convolutional layers, and to aggregate adjacent features among the spatiotemporal features in the spatiotemporal feature sequence samples to obtain local quality change feature samples corresponding to each spatiotemporal feature. The implementation of this first branch can be referred to the aforementioned method. Figure 2 The descriptions of the corresponding steps in the illustrated embodiments will not be repeated here.

[0092] The second branch is used to extract features from the spatiotemporal feature sequence samples based on the second temporal receptive field, obtaining global quality change feature samples corresponding to each spatiotemporal feature. The first temporal receptive field is smaller than the second temporal receptive field. In one optional approach, the second branch can be used to extract features from the spatiotemporal feature sequence samples based on the second temporal receptive field through a dilated convolutional layer with a preset dilation rate, aggregating non-adjacent features in the spatiotemporal feature sequence samples to obtain global quality change feature samples corresponding to each spatiotemporal feature. The implementation of this second branch can be referred to the aforementioned method. Figure 2 The descriptions of the corresponding steps in the illustrated embodiments will not be repeated here.

[0093] The adaptive fusion branch is used to adaptively fuse local quality change feature samples and global quality change feature samples to obtain a quality change feature sequence sample. In one optional approach, the adaptive fusion branch can be used to perform gating processing on the spatiotemporal feature sequence sample to obtain the fusion weights corresponding to each spatiotemporal feature in the spatiotemporal feature sequence sample; based on the fusion weights, the local quality change features and global quality change features corresponding to each spatiotemporal feature are weighted and fused to obtain the quality change feature sequence. The implementation of this adaptive fusion branch can be referred to the aforementioned method. Figure 2 The descriptions of the corresponding steps in the illustrated embodiments will not be repeated here.

[0094] The prediction output layer is used to predict the quality assessment result of video frame sequence samples based on quality change feature sequence samples. This quality assessment result can be used to train the video quality assessment model based on the difference between the predicted quality assessment result and the video quality label. The specific implementation of the prediction output layer predicting the quality assessment result can also refer to the above. Figure 2 The descriptions of the corresponding steps in the illustrated embodiments will not be repeated here.

[0095] After predicting the quality assessment results based on the samples, the video quality assessment model can be trained in a supervised manner based on the results and the video quality labels corresponding to the video frame sequence samples.

[0096] Step S205: Train the video quality assessment model based on the difference between the predicted quality assessment results and the video quality labels.

[0097] In this embodiment, the video quality assessment model can be trained in a supervised manner based on the difference between the predicted quality assessment result and the video quality label. In actual training, those skilled in the art can use any appropriate supervised training loss function according to actual needs. Based on this loss function, and the predicted quality assessment result and video quality label, a loss value is calculated, which characterizes the difference between the two. Then, based on this loss value, backpropagation training of the model is performed to update the model parameters until the model training termination condition is met.

[0098] For example, loss functions such as cross-entropy loss function and mean squared error (MSE) loss function can be used.

[0099] As described above, in this embodiment of the application, training samples including video frame sequence samples and video quality labels corresponding to the video frame sequence samples can be obtained. The video quality assessment model to be trained can be trained based on the training samples so that the trained video quality assessment model can execute the above-mentioned video quality assessment method and improve the confidence of the quality assessment results output by the video quality assessment model.

[0100] The following combination Figure 5 The present application provides an exemplary scenario description of a video quality assessment method provided in its embodiments.

[0101] exist Figure 5 Taking the quality assessment of course videos as an example, the process begins with receiving a video uploaded by a user (in this example, a learning course video) through an application or platform on the user's device. The uploaded video is then sent to the backend server. The server deploys a video quality assessment model. Upon receiving the video, the server samples it to obtain a temporally related sequence of video frames (exemplarily, the number of frames T=64). Then, the spatiotemporal feature extraction layer in the video quality assessment model extracts spatiotemporal features from the video frame sequence, yielding a corresponding spatiotemporal feature sequence. Next, the first and second branches extract features based on the first and second temporal receptive fields, respectively, to obtain local and global quality change features corresponding to each spatiotemporal feature. Finally, the adaptive fusion branch integrates the local and global quality change features to obtain a quality change feature sequence. The prediction output layer then determines the quality assessment result of the video frame sequence based on this quality change feature sequence. The quality assessment results can then be pushed to the user's device for further processing. For example, video optimization suggestions can be generated based on the assessment results to improve video quality.

[0102] As can be seen, the solution provided in this application, through the temporal dilated convolution of the second branch, can aggregate the contextual information of the video across a very long time distance (such as tens of seconds), explicitly modeling the global evolution trend of video quality and overcoming the problem of missing long-range dependencies in traditional video quality assessment. This enables the video quality assessment model to perceive at least two different temporal patterns: "bitrate continuously decreasing from high to low" and "bitrate suddenly decreasing and then recovering in the middle," which is more consistent with human subjective perception.

[0103] The first branch performs a detailed analysis of local video segments, while the second branch provides a global context from a more distant past. Furthermore, the adaptive fusion branch intelligently determines how much global information to incorporate based on the content of the current video segment, overcoming the limitation of contextual information caused by the isolated processing of each video segment in traditional video quality assessment. For example, suppose the video quality assessment model is evaluating a fast-moving video segment using the first branch. If the segment is blurry due to a low bitrate, the adaptive fusion branch might increase the weight of the second branch if it captures clear texture information from a similar static scene a few seconds earlier. This helps the video quality assessment model determine that the blur is due to coding distortion rather than motion blur, thus providing a more accurate quality assessment result.

[0104] For example, a stutter in a video (such as several consecutive frames with identical content) will create a unique "flat" pattern in the spatiotemporal feature sequence X. The first branch can immediately capture this local anomaly, and when the adaptive fusion branch detects this anomaly, it will increase the weight of the first branch to amplify the impact of the anomaly. This overcomes the problem in traditional video quality assessment where average pooling dilutes the signal of instantaneous distortions such as stutters, making the quality assessment insensitive to temporal distortions. Therefore, the video quality assessment model in this embodiment can become more sensitive to temporal distortions such as stutters, frame freezes, and screen tearing during scene transitions.

[0105] As can be seen, the video quality assessment model and its corresponding video quality assessment scheme in this application systematically solve the problem that traditional video quality assessment cannot perceive the temporal dynamics of video quality, and significantly improve the robustness and accuracy of assessment for various types of temporal distortions.

[0106] Reference Figure 6 The diagram shows a structural schematic of an electronic device according to this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0107] like Figure 6 As shown, the electronic device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0108] in: The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508.

[0109] Communication interface 504 is used to communicate with other electronic devices or servers.

[0110] The processor 502 is used to execute program 510, which can specifically execute the relevant steps in the above-described video quality assessment method or video quality assessment model training method embodiments.

[0111] Specifically, program 510 may include program code that includes computer operation instructions.

[0112] The processor 502 may be a CPU, a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The electronic device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0113] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0114] Program 510 may include multiple computer instructions. Specifically, program 510 may use multiple computer instructions to cause processor 502 to perform the operations corresponding to the video quality assessment method or video quality assessment model training method described in any of the foregoing multiple method embodiments.

[0115] The specific implementation of each step in program 510 can be found in the corresponding steps and units described in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0116] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0117] This application also provides a computer program product, including computer instructions that instruct a computing device to perform any of the video quality assessment methods or video quality assessment model training methods described in the above-described multiple method embodiments.

[0118] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0119] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0120] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, implementing the methods described herein when the software or computer code is accessed and executed by the computer, processor, or hardware. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0121] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0122] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A video quality assessment method, comprising: A video frame sequence to be evaluated is obtained, and spatiotemporal features are extracted from at least a portion of the video frames in the video frame sequence that have a temporal relationship to obtain the corresponding spatiotemporal feature sequence. The spatiotemporal feature sequence is subjected to feature extraction based on a first time receptive field to obtain local quality change features corresponding to each spatiotemporal feature; and the spatiotemporal feature sequence is subjected to feature extraction based on a second time receptive field to obtain global quality change features corresponding to each spatiotemporal feature, wherein the first time receptive field is smaller than the second time receptive field. The local quality change features and the global quality change features are adaptively fused to obtain a quality change feature sequence, and the quality assessment result of the video frame sequence to be evaluated is determined based on the quality change feature sequence.

2. The method according to claim 1, wherein, The adaptive fusion of the local quality change features and the global quality change features to obtain a quality change feature sequence includes: Based on the spatiotemporal feature sequence, the fusion weight corresponding to each spatiotemporal feature in the spatiotemporal feature sequence is obtained, wherein the fusion weight is used to indicate the degree of influence of the video frame corresponding to each spatiotemporal feature on the global quality change indicated by the global quality change feature; Based on the fusion weights, the local quality change features and global quality change features corresponding to each spatiotemporal feature are weighted and fused to obtain the quality change feature sequence. The step of obtaining the fusion weights corresponding to each spatiotemporal feature in the spatiotemporal feature sequence based on the spatiotemporal feature sequence includes: The spatiotemporal feature sequence is subjected to gating processing to obtain the fusion weights corresponding to each spatiotemporal feature in the spatiotemporal feature sequence.

3. The method according to claim 1, wherein, The step of performing feature extraction based on a first time receptive field on the spatiotemporal feature sequence to obtain local quality change features corresponding to each spatiotemporal feature includes: By performing feature extraction based on the first time receptive field on the spatiotemporal feature sequence, adjacent features of each spatiotemporal feature in the spatiotemporal feature sequence are aggregated to obtain the local quality change features corresponding to each spatiotemporal feature.

4. The method according to claim 3, wherein, The step involves extracting features from the spatiotemporal feature sequence based on a first time receptive field, aggregating adjacent features of each spatiotemporal feature in the spatiotemporal feature sequence, and obtaining the local quality change features corresponding to each spatiotemporal feature, including: By stacking multiple convolutional layers, feature extraction based on the first time receptive field is performed on the spatiotemporal feature sequence to aggregate adjacent features of each spatiotemporal feature in the spatiotemporal feature sequence, thereby obtaining the local quality change features corresponding to each spatiotemporal feature.

5. The method according to claim 1, wherein, The step of performing feature extraction based on the second temporal receptive field on the spatiotemporal feature sequence to obtain the global quality change features corresponding to each spatiotemporal feature includes: By performing feature extraction based on the second time receptive field on the spatiotemporal feature sequence, non-adjacent features in the spatiotemporal feature sequence are aggregated to obtain the global quality change features corresponding to each spatiotemporal feature.

6. The method according to claim 5, wherein, The step involves extracting features from the spatiotemporal feature sequence based on a second temporal receptive field, aggregating non-adjacent features in the spatiotemporal feature sequence, and obtaining global quality change features corresponding to each spatiotemporal feature, including: By using a dilated convolutional layer with a preset dilation rate, feature extraction based on a second temporal receptive field is performed on the spatiotemporal feature sequence to aggregate non-adjacent features of each spatiotemporal feature in the spatiotemporal feature sequence, thereby obtaining the global quality change features corresponding to each spatiotemporal feature.

7. The method according to claim 1, wherein, The process of determining the quality assessment result of the video frame sequence to be evaluated based on the quality change feature sequence includes: Pooling is performed on the quality change feature sequence to obtain candidate quality features corresponding to the video frame sequence; Based on the candidate quality features, quality prediction is performed to obtain the quality assessment result.

8. A method for training a video quality assessment model, comprising: Obtain training samples, wherein the training samples include video frame sequence samples and video quality labels corresponding to the video frame sequence samples; The video frame sequence samples are input into the video quality assessment model to be trained, so as to obtain the local quality change features and global quality change features corresponding to the video frame sequence samples through the video quality assessment model, and adaptively fuse the local quality change features and global quality change features, and predict the quality assessment result of the video frame sequence samples based on the adaptive fusion result. The video quality assessment model is trained based on the difference between the predicted quality assessment result and the video quality label.

9. The method according to claim 8, wherein, The video quality assessment model includes at least a spatiotemporal feature extraction layer, a prediction output layer, and parallel first branches, second branches, and adaptive fusion branches; The spatiotemporal feature extraction layer is used to extract spatiotemporal features from at least a portion of the video frames in the video frame sequence samples that have a temporal relationship, and obtain the corresponding spatiotemporal feature sequence samples. The first branch is used to perform feature extraction on the spatiotemporal feature sequence sample based on the first time receptive field to obtain local quality change feature samples corresponding to each spatiotemporal feature; The second branch is used to perform feature extraction based on the second time receptive field on the spatiotemporal feature sequence sample to obtain global quality change feature samples corresponding to each spatiotemporal feature, wherein the first time receptive field is smaller than the second time receptive field. The adaptive fusion branch is used to adaptively fuse the local quality change feature samples and the global quality change feature samples to obtain quality change feature sequence samples. The prediction output layer is used to predict the quality assessment result of the video frame sequence sample based on the quality change feature sequence sample, so that the video quality assessment model is trained based on the difference between the predicted quality assessment result and the video quality label.

10. The method according to claim 9, wherein, The first branch is used to extract features from the spatiotemporal feature sequence sample based on the first time receptive field through multiple stacked convolutional layers, and to aggregate adjacent features of each spatiotemporal feature in the spatiotemporal feature sequence sample to obtain local quality change feature samples corresponding to each spatiotemporal feature. The second branch is used to extract features from the spatiotemporal feature sequence samples based on the second time receptive field through a dilated convolutional layer with a preset dilation rate, and to perform non-adjacent feature aggregation on each spatiotemporal feature in the spatiotemporal feature sequence samples to obtain global quality change feature samples corresponding to each spatiotemporal feature, wherein the first time receptive field is smaller than the second time receptive field.

11. The method according to claim 9 or 10, wherein, The adaptive fusion branch is used to perform gating processing on the spatiotemporal feature sequence samples to obtain the fusion weights corresponding to each spatiotemporal feature in the spatiotemporal feature sequence samples; based on the fusion weights, the local quality change features and global quality change features corresponding to each spatiotemporal feature are weighted and fused to obtain the quality change feature sequence.

12. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-11.

13. A computer storage medium having a computer program stored thereon, which, when executed by a processor, performs the operation corresponding to the method described in any one of claims 1-11.

14. A computer program product comprising computer instructions that instruct a computing device to perform an operation corresponding to the method described in any one of claims 1-11.

Images