Video quality prediction method, device, electronic device, and storage medium

Abstract

The present disclosure relates to a video quality prediction method, device, electronic device, storage medium and computer program product, the method comprising: obtaining a frame set corresponding to the video to be predicted; the frame set is obtained by sampling the interval frames of the video to be predicted; multiple key frame sequences are determined based on the frame set, the feature correlation between the key frames contained in each key frame sequence is determined as the corresponding branch inter-frame feature, and the overall inter-frame feature corresponding to the frame set is obtained from the multiple branch inter-frame features; wherein each key frame sequence contains at least two key frames selected from the frame set, and the key frames corresponding to the multiple key frame sequences have different sparsity levels in time sequence; based on the overall inter-frame feature corresponding to the frame set, the visual quality prediction result of the video to be predicted is obtained. The method can accurately locate video frames containing different types of low-quality features for analysis, effectively improving the ability to process videos with mixed low-quality features.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to a video quality prediction method, device, electronic device, storage medium and computer program product. Background Art

[0002] With the development of the multimedia field, every user can become a producer of video content. The content produced by ordinary users is called user-generated content (UGC). Compared with the video content generated by professional producers, UGC videos are shot in poor condition, with ordinary shooting equipment and unstable transmission status. It is necessary to use video quality assessment (VQA, Visual Quality Transformer) to screen out videos with objectively low quality for processing.

[0003] Since UGC videos contain many types of low-quality features distributed in different frames and have different durations, the video quality assessment task is difficult. The traditional method is to model the low-quality features based on traditional manually designed features, and the video evaluation relies on the averaging of the prediction results of all frames. The computational complexity is high and it is also relatively limited in actual application scenarios, making the video quality assessment task difficult. Summary of the invention

[0004] The present disclosure provides a video quality prediction method, device, electronic device, storage medium and computer program product to at least solve the problem that video quality assessment tasks are difficult in related technologies. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a video quality prediction method is provided, comprising:

[0006] Acquire a frame set corresponding to the video to be predicted; the frame set is obtained by performing interval frame sampling on the video to be predicted;

[0007] Determine a plurality of key frame sequences based on the frame set, determine feature correlations between key frames included in each of the key frame sequences as corresponding branch inter-frame features, and obtain overall inter-frame features corresponding to the frame set from the plurality of branch inter-frame features; wherein each of the key frame sequences includes at least two key frames selected from the frame set, and the key frames corresponding to the plurality of key frame sequences have different sparsity levels in time sequence;

[0008] Based on the overall inter-frame features corresponding to the frame set, a visual quality prediction result of the video to be predicted is obtained.

[0009] In a possible implementation manner, determining a plurality of key frame sequences based on the frame set includes:

[0010] Acquire the number of frames included in the frame set and a preset sparse parameter; the preset sparse parameter is used to characterize the maximum sparseness of key frames selected from the frame set;

[0011] According to the number of frames included in the frame set and the preset sparsity parameter, the number of branches used to obtain the branch inter-frame features is determined; each of the multiple branches is used to obtain the inter-frame feature correlation between key frames having a sparsity degree in time sequence as the branch inter-frame feature corresponding to each branch, and the sparsity degree is less than the maximum sparsity degree;

[0012] A plurality of key frames are determined from the frame set, and a plurality of key frame sequences corresponding to the number of branches are determined based on the plurality of key frames.

[0013] In a possible implementation, determining a plurality of key frames from the frame set, and determining a plurality of key frame sequences corresponding to the number of branches based on the plurality of key frames, includes:

[0014] For each branch, obtaining the sparsity of the key frames corresponding to each branch;

[0015] Determining the number of key frames corresponding to each branch according to the sparseness of the key frames corresponding to each branch;

[0016] Based on the sparsity of the key frames corresponding to each branch and the number of key frames corresponding to each branch, key frames corresponding to the number of key frames are selected from the frame set to obtain a key frame sequence corresponding to each branch.

[0017] In a possible implementation, obtaining the sparsity of the key frame corresponding to each branch includes:

[0018] Obtaining a sequence number of each branch among the multiple branches;

[0019] Based on the sequence number of each branch in the multiple branches and the preset sparse parameter, the sparseness of the key frame corresponding to each branch is determined.

[0020] In a possible implementation, determining the feature correlation between the key frames included in each of the key frame sequences as the corresponding branch inter-frame feature includes:

[0021] For each key frame sequence, determining the feature correlation between the key frames included in the key frame sequence according to the importance of each key frame included in the key frame sequence as the branch inter-frame feature corresponding to the key frame sequence;

[0022] Among them, the importance corresponding to each key frame is obtained by calculating the importance of each frame in the frame set in the attention map using the attention mechanism.

[0023] In a possible implementation, obtaining the overall inter-frame features corresponding to the frame set from the plurality of branch inter-frame features includes:

[0024] Performing feature splicing on a plurality of branch inter-frame features corresponding to a plurality of the key frame sequences to obtain spliced ​​inter-frame features;

[0025] Performing feature filling on the spliced ​​inter-frame features, wherein the number of frames corresponding to the inter-frame features after the feature filling is the same as the number of frames included in the frame set;

[0026] The overall inter-frame features corresponding to the frame set are obtained based on the inter-frame features after the feature filling.

[0027] In a possible implementation, determining a plurality of key frame sequences based on the frame set, determining feature correlations between key frames included in each of the key frame sequences as corresponding branch inter-frame features, and obtaining overall inter-frame features corresponding to the frame set from the plurality of branch inter-frame features, includes:

[0028] The frame set is input into a visual quality prediction model, wherein the visual quality prediction model includes a plurality of branch structures, each of which determines a key frame sequence based on the frame set, and determines a feature correlation between key frames included in the key frame sequence as a branch inter-frame feature corresponding to each branch structure, and obtains an overall inter-frame feature corresponding to the frame set from a plurality of branch inter-frame features corresponding to the plurality of branch structures;

[0029] The obtaining the visual quality prediction result of the video to be predicted based on the overall inter-frame features corresponding to the frame set includes:

[0030] Based on the output information of the visual quality prediction model, a visual quality prediction result of the video to be predicted is determined; the output information of the visual quality prediction model is determined based on the overall inter-frame features.

[0031] In a possible implementation manner, before the step of determining the visual quality prediction result of the video to be predicted based on the output information of the visual quality prediction model, the method further includes:

[0032] Determine the intra-frame feature correlation of each frame in the frame set, and obtain the intra-frame features corresponding to the frame set based on the intra-frame feature correlation of multiple frames;

[0033] Determine the enhanced features of each frame in the frame set, and obtain the frame feature expression corresponding to the frame set based on the enhanced features of multiple frames; the enhanced features of each frame are features obtained by performing nonlinear transformation processing on the initial features of each frame;

[0034] According to the overall inter-frame features, the intra-frame features and the frame feature expressions corresponding to the frame set, the visual quality parameters of the video to be predicted are determined as output information of the visual quality prediction model.

[0035] In a possible implementation, the visual quality prediction model includes a plurality of stacked encoding modules; determining the visual quality parameter of the video to be predicted as output information of the visual quality prediction model according to the overall inter-frame features, the intra-frame features, and the frame feature expressions corresponding to the frame set, including:

[0036] Determine the overall inter-frame features, the intra-frame features, and the frame feature expressions corresponding to the frame set based on the current encoding module, and use the overall inter-frame features, the intra-frame features, and the frame feature expressions determined by the current encoding module as input information of a next encoding module of the current encoding module;

[0037] Determine, based on the overall inter-frame features, intra-frame features, and frame feature expressions determined by the last encoding module in the plurality of stacked encoding modules, a visual quality parameter of the video to be predicted as output information of the visual quality prediction model;

[0038] Each of the encoding modules includes a temporal attention mechanism unit, a spatial attention mechanism unit and a multi-layer perception unit; the temporal attention mechanism unit includes a plurality of the branch structures;

[0039] In each of the encoding modules, the overall inter-frame features are determined based on the temporal attention mechanism unit;

[0040] The intra-frame features are determined based on the spatial attention mechanism unit;

[0041] The frame feature expression is determined based on the multi-layer perception unit.

[0042] In a possible implementation manner, before determining the overall inter-frame features, the intra-frame features, and the frame feature expressions corresponding to the frame set based on the first encoding module in the multiple stacked encoding modules, the method further includes:

[0043] Extracting serialized features from each frame in the frame set to obtain a set of serialized features corresponding to the frame set;

[0044] A set of serialized features corresponding to the frame set is used as input information of the first encoding module.

[0045] According to a second aspect of an embodiment of the present disclosure, a video quality prediction device is provided, including:

[0046] A frame set acquisition unit is configured to acquire a frame set corresponding to the video to be predicted; the frame set is obtained by performing interval frame sampling on the video to be predicted;

[0047] The inter-frame feature obtaining unit is configured to determine a plurality of key frame sequences based on the frame set, determine the feature correlation between the key frames included in each of the key frame sequences as the corresponding branch inter-frame features, and obtain the overall inter-frame features corresponding to the frame set from the plurality of branch inter-frame features; wherein each of the key frame sequences includes at least two key frames selected from the frame set, and the key frames corresponding to the plurality of key frame sequences have different sparsity levels in time sequence;

[0048] The visual quality prediction result obtaining unit is configured to obtain the visual quality prediction result of the video to be predicted based on the overall inter-frame features corresponding to the frame set.

[0049] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0050] processor;

[0051] a memory for storing instructions executable by the processor;

[0052] The processor is configured to execute the instructions to implement the video quality prediction method as described in any one of the above items.

[0053] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any of the video quality prediction methods described above.

[0054] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product includes instructions, and when the instructions are executed by a processor of an electronic device, the electronic device is enabled to perform any of the video quality prediction methods described above.

[0055] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0056] The scheme disclosed in the present invention obtains a frame set corresponding to the video to be predicted, where the frame set is obtained by sampling the interval frames of the video to be predicted, and then determines multiple key frame sequences based on the frame set, determines the feature correlation between the key frames contained in each key frame sequence as the corresponding branch inter-frame feature, and obtains the overall inter-frame feature corresponding to the frame set from the multiple branch inter-frame features, wherein each key frame sequence contains at least two key frames selected from the frame set, and the key frames corresponding to the multiple key frame sequences have different sparsity levels in time sequence, and then obtains the visual quality prediction result of the video to be predicted based on the overall inter-frame feature corresponding to the frame set. In this way, key frames can be screened out from redundant video frames for analysis based on the sparse temporal attention mechanism, and video frames containing different types of low-quality features can be accurately located. Moreover, multiple low-quality features can be modeled simultaneously based on the temporal multi-branch network, effectively improving the ability to process videos with mixed low-quality features.

[0057] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0059] Figure 1 is a flowchart of a video quality prediction method according to an exemplary embodiment;

[0060] Figure 2 is a schematic diagram showing the distribution of low-quality features in a UGC video according to an exemplary embodiment;

[0061] Figure 3 is a schematic diagram showing a visual quality prediction model structure according to an exemplary embodiment;

[0062] Figure 4 is a flowchart of another video quality prediction method according to an exemplary embodiment;

[0063] Figure 5 is a block diagram of a video quality prediction device according to an exemplary embodiment;

[0064] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0065] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0066] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure.

[0067] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0068] Figure 1 is a flowchart of a video quality prediction method according to an exemplary embodiment. The method can be used in a computer device such as a terminal. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through interaction between the terminal and the server. Taking the method applied to the server as an example, Figure 1 As shown, the method includes the following steps.

[0069] In step S110, a frame set corresponding to the video to be predicted is obtained;

[0070] The frame set may be obtained by sampling the video to be predicted at intervals. For example, multiple frames may be extracted from the video to be predicted at preset sampling intervals (such as intervals of 8 frames or 16 frames) to form a frame set.

[0071] As an example, the video to be predicted may be a UGC video shot by a user, and the UGC video may contain various types of low-quality features, such as Figure 2 As shown in the figure, different types of low-quality features (such as large aperture, motion blur, block effects, etc.) are distributed in different frames in the UGC video, and the duration of different types of low-quality features varies, which makes the video quality assessment task have the following difficulties: (1) There is a large gap between the average quality assessment score based on a single frame and the average subjective quality score MOS (Mean Opinion Score), and the assessment accuracy is poor; (2) Different types of low-quality features need to be distinguished according to different video durations. For example, block effects, dirty shots, and noise can be identified based on a single frame, while blur and out-of-focus require multiple frames to identify; (3) There is a large amount of redundant information between video frames. Too high a sampling frequency will lead to high computational complexity, and a low sampling frequency may lose key video frames containing low-quality features.

[0072] In practical applications, a video to be predicted can be obtained, and then interval frame sampling can be performed on the video to be predicted to obtain a frame set. For example, by performing interval frame sampling on the video to be predicted, a frame set consisting of T frames can be obtained, which is input as input data into a visual quality prediction model for video quality identification.

[0073] Specifically, the server can respond to a video quality prediction request and obtain the video to be predicted, and then extract T frames from the video to be predicted according to a preset sampling interval to obtain a frame set, and then input the frame set into a visual quality prediction model. The visual quality prediction model is used to perform feature extraction on the frame set, and visual quality prediction is performed based on the extracted video frame features, so as to obtain a visual quality prediction result of the video to be predicted.

[0074] In step S120, multiple key frame sequences are determined based on the frame set, feature correlations between key frames included in each key frame sequence are determined as corresponding branch inter-frame features, and overall inter-frame features corresponding to the frame set are obtained from the multiple branch inter-frame features;

[0075] Each key frame sequence may include at least two key frames selected from the frame set, and the key frames corresponding to the multiple key frame sequences may have different sparsity levels in time sequence.

[0076] After obtaining a frame set, multiple key frame sequences can be determined based on the frame set, and then for each key frame sequence, the feature correlation between the key frames contained in each key frame sequence can be determined as the branch inter-frame feature corresponding to the frame set, and then the overall inter-frame feature corresponding to the frame set can be obtained based on the multiple branch inter-frame features. For example, the frame set can be input into a visual quality prediction model and processed by the temporal attention mechanism unit of each encoding module in the visual quality prediction model, so that the overall inter-frame feature corresponding to the frame set can be obtained based on the multiple branch inter-frame features.

[0077] In one example, based on the temporal attention mechanism unit of each encoding module in the visual quality prediction model, the correlation between different frames can be modeled to obtain the inter-frame feature correlation (i.e., the overall inter-frame feature), such as the distribution of different groups of related frames between the overall frames of the frame set can be characterized. The temporal attention mechanism unit can be constructed based on two parts of the network, one is a sparse temporal attention mechanism unit (STA), through which key frames with different information can be selected to represent low-quality features (i.e., branch inter-frame features), and the sparse temporal attention calculation mechanism can reduce the computational complexity, and the other is a temporal multi-branch network (MPTN, Multi-Pathway Temporal Network), through which key frame sequences of different temporal lengths can be obtained in parallel, and then different low-quality features can be analyzed based on key frame sequences of different temporal lengths (i.e., the overall inter-frame features corresponding to the frame set are obtained according to multiple branch inter-frame features), so that by introducing STA and MPTN into the network calculation, the modeling effect of low-quality features affecting the video quality is effectively improved and the computational complexity is reduced.

[0078] In step S130, a visual quality prediction result of the video to be predicted is obtained based on the overall inter-frame features corresponding to the frame set.

[0079] In a specific implementation, the output information of the visual quality prediction model can be determined based on the overall inter-frame features, and then the visual quality prediction result of the video to be predicted, such as the visual quality parameters for the video to be predicted, can be obtained according to the output information of the visual quality prediction model.

[0080] In one example, the output information of the visual quality prediction model may be a visual quality parameter for the video to be predicted, such as a video quality token of 1-5 points, to characterize the visual quality of the video to be predicted. For example, the higher the evaluation score, the better the visual quality of the video.

[0081] Compared with the traditional method, the network structure of the visual quality prediction model proposed in the technical solution of this embodiment can effectively model the temporal characteristics of the video, capture the various types of low-quality features that exist simultaneously in the UGC scene for video quality assessment, and has strong generalization ability for different data, which greatly improves the prediction accuracy and relevance. By using the video self-attention mechanism to model the low-quality features contained in the video, it can effectively handle complex video quality assessment tasks; based on the sparse temporal attention mechanism unit, it can screen out key frames from redundant video frames for analysis, effectively reducing the computational complexity, and at the same time, it can more accurately locate video frames containing different types of low-quality features. Based on the temporal multi-branch network, multiple low-quality features can be modeled simultaneously in one network structure, effectively improving the ability to process videos with mixed low-quality features.

[0082] In the above-mentioned video quality prediction method, a frame set corresponding to the video to be predicted is obtained by sampling the interval frames of the video to be predicted, and then multiple key frame sequences are determined based on the frame set, and the feature correlation between the key frames contained in each key frame sequence is determined as the corresponding branch inter-frame feature, and the overall inter-frame feature corresponding to the frame set is obtained from the multiple branch inter-frame features, wherein each key frame sequence contains at least two key frames selected from the frame set, and the key frames corresponding to the multiple key frame sequences have different sparsity levels in time sequence, and then based on the overall inter-frame feature corresponding to the frame set, the visual quality prediction result of the video to be predicted is obtained. In this way, key frames can be screened out from redundant video frames for analysis based on the sparse temporal attention mechanism, and video frames containing different types of low-quality features can be accurately located. Moreover, multiple low-quality features can be modeled simultaneously based on the temporal multi-branch network, which effectively improves the ability to process videos with mixed low-quality features.

[0083] In an exemplary embodiment, in the above step S110, multiple key frame sequences are determined based on a frame set, including: obtaining the number of frames contained in the frame set and a preset sparse parameter; determining the number of branches used to obtain branch inter-frame features based on the number of frames contained in the frame set and the preset sparse parameter; determining multiple key frames from the frame set, and determining multiple key frame sequences corresponding to the number of branches based on the multiple key frames.

[0084] As an example, the preset sparse parameter may be used to characterize the maximum sparseness of selecting key frames from a frame set.

[0085] Among them, each of the multiple branches can be used to obtain inter-frame feature correlation between key frames with a certain sparsity degree in time sequence as the branch inter-frame feature corresponding to each branch, and the sparsity degree is less than the maximum sparsity degree.

[0086] In the practical application of the disclosed method, in order to solve the problem that different types of low-quality video features have different durations in time series, multiple STAs with different sparsity levels are used based on MPTN to simultaneously model low-quality types of different time series features. The STA branches can be configured to be m (i.e., the number of branches), and the number of m can be calculated as follows:

[0087]

[0088] Among them, T can represent the number of frames contained in the frame set, and the maximum sparsity coefficient logT / T (i.e., the preset sparsity parameter) can be set according to the Johnson-Lindenstrauss lemma, which can be used to select the most critical multiple frames from the feature sequence of T frames (i.e., a set of serialized features corresponding to the frame set), thereby introducing sparsity and retaining sufficient information capacity.

[0089] The technical solution of this embodiment obtains the number of frames contained in a frame set and a preset sparse parameter, and then determines the number of branches used to obtain branch inter-frame features based on the number of frames contained in the frame set and the preset sparse parameter, and then determines multiple key frames from the frame set, and determines multiple key frame sequences corresponding to the number of branches based on the multiple key frames, thereby achieving simultaneous modeling of multiple low-quality features in a network structure, which can effectively improve the ability to process videos with mixed low-quality features.

[0090] In an exemplary embodiment, multiple key frames are determined from a frame set, and multiple key frame sequences corresponding to the number of branches are determined based on the multiple key frames, including: for each branch, obtaining the sparsity of the key frames corresponding to each branch; determining the number of key frames corresponding to each branch according to the sparsity of the key frames corresponding to each branch; based on the sparsity of the key frames corresponding to each branch and the number of key frames corresponding to each branch, selecting the key frames corresponding to the number of key frames from the frame set, to obtain a key frame sequence corresponding to each branch.

[0091] In one example, the number of key frames corresponding to each branch STA may be determined in the following manner:

[0092] 2 n logT

[0093] Among them, n is the sequence number of the current branch among multiple branches. For example, the value range of the sequence number corresponding to m branches is 1 to m.

[0094] Specifically, by obtaining the sparsity of the key frames corresponding to each branch STA, the number of key frames corresponding to the branch STA can be determined according to the sparsity, such as Figure 3 As shown in 2 m logT×d,2 mlogT can represent the number of key frames corresponding to the branch STA, d can represent the extracted feature dimension, and then for each branch STA, based on the sparsity of the key frames corresponding to each branch STA and the number of key frames corresponding to each branch, the key frames corresponding to the number of key frames can be selected from the frame set to form a key frame sequence corresponding to each branch STA.

[0095] In an optional embodiment, 2 0 logT~2 m Multiple key frame sequences corresponding to the branch STA where logT is located, such as Figure 3 logT×d……2 m logT×d, and then the m key frame sequences determined based on the frame set can be obtained (1+…+2 m )logT×d, which is the concatenated inter-frame feature.

[0096] The technical solution of this embodiment obtains the sparsity of the key frames corresponding to each branch, and then determines the number of key frames corresponding to the branch according to the sparsity. Then, based on the sparsity corresponding to each branch, the key frames corresponding to the number of key frames are selected from the frame set to form a key frame sequence corresponding to the branch. Key frame sequences of different time series lengths can be obtained, which provides a data basis for subsequent analysis of different types of low-quality features.

[0097] In an exemplary embodiment, obtaining the sparsity of the key frame corresponding to each branch includes: obtaining the sequence number of each branch in multiple branches; and determining the sparsity of the key frame corresponding to each branch based on the sequence number of each branch in multiple branches and a preset sparsity parameter.

[0098] In an example, the sparseness of the key frame corresponding to each branch STA may be determined in the following manner:

[0099] 2 n logT / T

[0100] Among them, n is the sequence number of the current branch among multiple branches, such as m branches corresponding to the sequence numbers 1 to m.

[0101] For example, by obtaining the sequence number of each branch in the multiple branches, the sparseness of the key frame corresponding to each branch STA can be determined based on the sequence number of each branch in the multiple branches and a preset sparse parameter.

[0102] The technical solution of this embodiment obtains the serial number of each branch in multiple branches, and then determines the sparsity of the key frames corresponding to each branch based on the serial number of each branch in multiple branches and preset sparsity parameters. A temporal multi-branch network can be constructed based on the sparsity corresponding to each branch, thereby achieving simultaneous modeling of different temporal lengths corresponding to different types of low-quality features, thereby improving the efficiency of video quality prediction.

[0103] In an exemplary embodiment, in the above step S120, the feature correlation between the key frames contained in each key frame sequence is determined as the corresponding branch inter-frame feature, including: for each key frame sequence, according to the corresponding importance of each key frame contained in the key frame sequence, the feature correlation between the key frames contained in the key frame sequence is determined as the branch inter-frame feature corresponding to the key frame sequence.

[0104] Among them, the importance corresponding to each key frame can be obtained by calculating the importance of each frame in the attention map in the frame set using the attention mechanism.

[0105] As an example, the branch inter-frame feature can be as follows Figure 3 V1', V m '.

[0106] In the specific implementation, for each branch STA, through each branch temporal attention mechanism unit, the feature correlation between the key frames contained in the key frame sequence of the branch can be determined according to the corresponding importance of each key frame contained in the key frame sequence of the branch, as the branch inter-frame feature corresponding to the key frame sequence. If the correlation between different frames is modeled based on the temporal attention mechanism unit, the inter-frame feature correlation corresponding to the key frame sequence can be extracted.

[0107] In one example, for each branch STA, N frames can be selected from the frame set as key frames based on the sparsity corresponding to the branch, where N is equal to the number of key frames corresponding to the branch, such as Figure 3 logT,2 m logT, etc., and then the key frame sequences corresponding to each branch can be constructed according to the selected key frames, such as Figure 3 In the equation, logT×d…2 m The sparse sampling matrix corresponding to logT×d can be used to obtain the importance of each key frame in the key frame sequence for each key frame sequence. For example, the attention mechanism can be used to calculate the importance of each frame in the attention map in the frame set.

[0108] In the practical application of the disclosed method, for each branch STA, such as Figure 3As shown in , the Q and K features of the attention mechanism can be used to calculate the coefficients of the sparse sampling matrix, that is, the importance of each frame in the attention map can be measured by the degree of deviation of q[i] and k[j] from the uniform distribution. If the deviation from the uniform distribution is greater, the attention weight between two frames can be represented to be greater. The measurement method can use KL divergence:

[0109]

[0110] Among them, T can represent the number of frames contained in the frame set, d can represent the feature dimension extracted from a single frame, i and j represent two frames in the frame set respectively, K can represent the K features in the attention calculation process, U can represent the uniform distribution, p(K|q i ) can characterize K and q i of relevance.

[0111] In another example, for each branch STA, a set of serialized features corresponding to a frame set can be used as an input feature vector, and then the attention mechanism can be used to generate three feature vectors, such as K-key feature vector, Q-query feature vector, and V-value vector can be created by multiplying feature embedding with three weight matrices, such as Figure 3 K1-T×d, Q1-T×d, V1-T×d in the image, and then according to the K-key feature vector and the Q-query feature vector, by measuring the importance of each frame in the frame set in the attention map, the most important multiple key frames can be selected from the frame set to obtain a sparsely sampled key frame sequence.

[0112] The technical solution of this embodiment determines the feature correlation between the key frames contained in the key frame sequence based on the self-attention mechanism for each key frame sequence, and uses it as the branch inter-frame feature corresponding to the key frame sequence. It can reduce the computational complexity based on the sparse temporal attention calculation mechanism and improve the efficiency of video quality prediction.

[0113] In an exemplary embodiment, in the above step S120, the overall inter-frame features corresponding to the frame set are obtained from the multiple branch inter-frame features, including: performing feature splicing on the multiple branch inter-frame features corresponding to the multiple key frame sequences to obtain the spliced ​​inter-frame features; performing feature filling on the spliced ​​inter-frame features, and the number of frames corresponding to the inter-frame features after feature filling is the same as the number of frames included in the frame set; and obtaining the overall inter-frame features corresponding to the frame set based on the inter-frame features after feature filling.

[0114] In one example, the features extracted from different STAs are concatenated to obtain the concatenated inter-frame features, such as Figure 3 As shown in (1+...+2 m)logT×d, and can be subjected to mean-padding operation to obtain the inter-frame features after feature filling, such as Figure 3 As shown in Z l -T×d, so that the number of frames corresponding to the inter-frame features after feature filling can be the same as the number of frames contained in the frame set.

[0115] For example, the concatenation operation can be expressed as follows:

[0116]

[0117] Among them, V1', ..., V m ' are multiple branch inter-frame features corresponding to multiple key frame sequences.

[0118] As another example, the average supplement operation can be expressed as follows:

[0119]

[0120] Among them, Z l It is the inter-frame feature after feature filling, that is, the overall inter-frame feature.

[0121] The technical solution of this embodiment is to obtain the spliced ​​inter-frame features by performing feature splicing on multiple branch inter-frame features corresponding to multiple key frame sequences, and then perform feature filling on the spliced ​​inter-frame features. The number of frames corresponding to the inter-frame features after feature filling is the same as the number of frames included in the frame set. Then, the overall inter-frame features corresponding to the frame set are obtained based on the inter-frame features after feature filling. The overall inter-frame features extracted for different types of low-quality features can be obtained, which provides a data basis for video quality prediction and analysis.

[0122] In an exemplary embodiment, in the above step S120, multiple key frame sequences are determined based on the frame set, and the feature correlation between the key frames contained in each key frame sequence is determined as the corresponding branch inter-frame feature, and the overall inter-frame feature corresponding to the frame set is obtained from the multiple branch inter-frame features, including: inputting the frame set into the visual quality prediction model, the visual quality prediction model includes multiple branch structures, each branch structure determines a key frame sequence based on the frame set, and determines the feature correlation between the key frames contained in a key frame sequence as the branch inter-frame feature corresponding to each branch structure, and the overall inter-frame feature corresponding to the frame set is obtained from the multiple branch inter-frame features corresponding to the multiple branch structures.

[0123] As an example, the visual quality prediction model may be VQT (Visual Quality Transformer, a visual quality network based on self-attention mechanism), and its network structure may be as follows: Figure 3As shown, VQT may include a serialized feature layer, multiple stacked encoding modules for feature fusion extraction, and a fully connected layer, wherein each encoding module may be composed of a temporal attention mechanism unit, a spatial attention mechanism unit, and a multi-layer perception unit.

[0124] After obtaining the frame set, the frame set can be input into the visual quality prediction model. Through the temporal attention mechanism unit of each encoding module in the visual quality prediction model, multiple key frame sequences can be determined based on the frame set. Then, for each key frame sequence, the feature correlation between the key frames contained in each key frame sequence can be determined as the branch inter-frame feature corresponding to the frame set, and then the overall inter-frame feature corresponding to the frame set can be obtained according to the multiple branch inter-frame features.

[0125] For example, based on the temporal attention mechanism unit of each coding module in the visual quality prediction model, the correlation between different frames can be modeled to obtain the inter-frame feature correlation. Based on the spatial attention mechanism unit of each coding module in the visual quality prediction model, the correlation between different regions within the frame can be modeled. Based on the multi-layer perception unit of each coding module in the visual quality prediction model, the representation capability of each frame feature can be further improved through nonlinear transformation.

[0126] In one example, a temporal attention mechanism unit can be constructed based on two parts of the network. One is a sparse temporal attention mechanism unit, which can select key frames with different information to represent low-quality features through STA, and the sparse temporal attention calculation mechanism can reduce the computational complexity. The other is a temporal multi-branch network, which can obtain key frame sequences of different temporal lengths in parallel through MPTN, and then analyze different low-quality features based on key frame sequences of different temporal lengths. Therefore, by introducing STA and MPTN into network calculation, the modeling effect of low-quality features that affect video quality is effectively improved, and the computational complexity is reduced.

[0127] In the above step S130, based on the overall inter-frame features corresponding to the frame set, the visual quality prediction result of the video to be predicted is obtained, including: based on the output information of the visual quality prediction model, the visual quality prediction result of the video to be predicted is determined; the output information of the visual quality prediction model is determined based on the overall inter-frame features.

[0128] As an example, the output information of the visual quality prediction model may be a visual quality parameter for the video to be predicted, such as a video quality label of 1-5 points, to characterize the visual quality of the video to be predicted. For example, the higher the evaluation score, the better the visual quality of the video.

[0129] In a specific implementation, the output information of the visual quality prediction model can be determined based on the overall inter-frame features, and then the visual quality prediction result of the video to be predicted, such as the visual quality parameters for the video to be predicted, can be obtained according to the output information of the visual quality prediction model.

[0130] The technical solution of this embodiment is to input a frame set into a visual quality prediction model, where the visual quality prediction model includes multiple branch structures, each branch structure determines a key frame sequence based on the frame set, and determines the feature correlation between the key frames contained in a key frame sequence as the branch inter-frame feature corresponding to each branch structure, and obtains the overall inter-frame feature corresponding to the frame set from the multiple branch inter-frame features corresponding to the multiple branch structures, and then determines the visual quality prediction result of the video to be predicted based on the output information of the visual quality prediction model. The network structure of the visual quality prediction model can effectively model the video temporal features, and can capture multiple different types of low-quality features that exist simultaneously in the UGC scenario to perform video quality assessment. It has strong generalization ability for different data, and greatly improves the prediction accuracy and relevance.

[0131] In an exemplary embodiment, before the step of determining the visual quality prediction result of the video to be predicted based on the output information of the visual quality prediction model, it also includes: determining the intra-frame feature correlation of each frame in the frame set, and obtaining the intra-frame features corresponding to the frame set based on the intra-frame feature correlation of multiple frames; determining the enhanced features of each frame in the frame set, and obtaining the frame feature expression corresponding to the frame set based on the enhanced features of multiple frames; determining the visual quality parameters of the video to be predicted according to the overall inter-frame features, intra-frame features and frame feature expressions corresponding to the frame set as the output information of the visual quality prediction model.

[0132] The enhanced features of each frame may be features obtained by performing nonlinear transformation processing on the initial features of each frame.

[0133] In the specific implementation, for each encoding module, the correlation between different frames can be modeled through the temporal attention mechanism unit to obtain the overall inter-frame features corresponding to the frame set, and the correlation between different regions within the frame can be modeled through the spatial attention mechanism unit. The intra-frame features corresponding to the frame set are obtained based on the intra-frame feature correlation of multiple frames. The multi-layer perception unit can be used to further improve the feature representation capability by using nonlinear transformation processing, and the frame feature expression corresponding to the frame set is obtained based on the enhanced features of multiple frames. Then, the visual quality parameters of the video to be predicted can be determined according to the overall inter-frame features, intra-frame features and frame feature expressions corresponding to the frame set, as the output information of the visual quality prediction model.

[0134] The technical solution of this embodiment determines the intra-frame feature correlation of each frame in a frame set, obtains the intra-frame features corresponding to the frame set based on the intra-frame feature correlation of multiple frames, determines the enhanced features of each frame in the frame set, obtains the frame feature expression corresponding to the frame set based on the enhanced features of multiple frames, and then determines the visual quality parameters of the video to be predicted according to the overall inter-frame features, intra-frame features and frame feature expressions corresponding to the frame set. As the output information of the visual quality prediction model, it can effectively improve the ability to process videos with mixed low-quality features.

[0135] In an exemplary embodiment, the visual quality prediction model may include multiple stacked encoding modules, and determine the visual quality parameters of the video to be predicted according to the overall inter-frame features, intra-frame features and frame feature expressions corresponding to the frame set as the output information of the visual quality prediction model, including: determining the overall inter-frame features, intra-frame features and frame feature expressions corresponding to the frame set based on the current encoding module, and using the overall inter-frame features, intra-frame features and frame feature expressions determined by the current encoding module as the input information of the next encoding module of the current encoding module; determining the visual quality parameters of the video to be predicted based on the overall inter-frame features, intra-frame features and frame feature expressions determined by the last encoding module among the multiple stacked encoding modules, as the output information of the visual quality prediction model.

[0136] In one example, VQT may include multiple stacked encoding modules for feature fusion extraction, each encoding module may be composed of a temporal attention mechanism unit, a spatial attention mechanism unit and a multi-layer perception unit. For each encoding module, the overall inter-frame features, intra-frame features and frame feature expressions corresponding to the frame set can be determined based on the current encoding module, and the overall inter-frame features, intra-frame features and frame feature expressions determined by the current encoding module can be used as input information of the next encoding module of the current encoding module, and then based on the overall inter-frame features, intra-frame features and frame feature expressions determined by the last encoding module in the multiple stacked encoding modules, the visual quality parameters of the video to be predicted can be obtained as the output information of the visual quality prediction model.

[0137] For example, the overall inter-frame features, intra-frame features, and frame feature expressions determined by the last encoding module can be input into the fully connected layer. By performing fully-connected processing on the overall inter-frame features, intra-frame features, and frame feature expressions through the fully-connected layer, the visual quality parameters of the video to be predicted can be obtained as the output information of the visual quality prediction model.

[0138] In another example, each encoding module may include a temporal attention mechanism unit, a spatial attention mechanism unit, and a multi-layer perception unit; the temporal attention mechanism unit may include multiple branch structures.

[0139] In practical applications, in each encoding module, the overall inter-frame features can be determined based on the temporal attention mechanism unit, the intra-frame features can be determined based on the spatial attention mechanism unit, and the frame feature expression can be determined based on the multi-layer perception unit.

[0140] Specifically, each encoding module can be composed of a temporal attention mechanism unit, a spatial attention mechanism unit and a multi-layer perception unit. The temporal attention mechanism unit can be used to model the correlation between different frames and obtain the overall inter-frame features corresponding to the frame set. The spatial attention mechanism unit can be used to model the correlation between different regions within the frame and obtain the intra-frame features corresponding to the frame set. The multi-layer perception unit can be used to further improve the feature representation capability based on nonlinear transformation processing and obtain the frame feature expression corresponding to the frame set, thereby effectively capturing various types of low-quality features that exist simultaneously in the UGC scenario for video quality assessment, thereby improving the efficiency of video quality prediction.

[0141] The technical solution of this embodiment determines the overall inter-frame features, intra-frame features and frame feature expressions corresponding to the frame set based on the current encoding module, and uses the overall inter-frame features, intra-frame features and frame feature expressions determined by the current encoding module as the input information of the next encoding module of the current encoding module, and then determines the visual quality parameters of the video to be predicted based on the overall inter-frame features, intra-frame features and frame feature expressions determined by the last encoding module in multiple stacked encoding modules, as the output information of the visual quality prediction model, which can further improve the feature expression based on the processing of multiple encoding modules, thereby improving the prediction accuracy and correlation.

[0142] In an exemplary embodiment, before the first encoding module in multiple stacked encoding modules determines the overall inter-frame features, intra-frame features and frame feature expressions corresponding to a frame set, it also includes: performing serialization feature extraction on each frame in the frame set to obtain a set of serialization features corresponding to the frame set; and using the set of serialization features corresponding to the frame set as input information of the first encoding module.

[0143] In a specific implementation, a frame set is obtained by sampling the video to be predicted at intervals, and after the frame set is input into the visual quality prediction model, serialized features of each frame in the frame set can be extracted to obtain a set of serialized features corresponding to the frame set, such as Figure 3 As shown in , a set of serialized features corresponding to the frame set can be used as input information of the first encoding module of the visual quality prediction model.

[0144] In an optional embodiment, before inputting the frame set into the visual quality prediction model, the visual quality prediction model can be pre-trained to optimize the visual quality prediction model. Training data can be obtained, and the training data can include at least two sample videos. Then, the at least two sample videos can be input into the visual quality prediction model to perform video visual quality prediction, and then visual quality parameters corresponding to each sample video and visual quality relative parameters can be obtained. The visual quality relative parameters can be determined based on the degree of difference in visual quality between at least two sample videos. Then, a target loss function can be determined based on the visual quality parameters corresponding to each sample video and the visual quality relative parameters. Based on the target loss function, the visual quality prediction model can be trained to obtain an optimized visual quality prediction model for use in video quality prediction in actual scenarios, thereby ensuring the relative consistency of evaluation judgments during video quality prediction.

[0145] For example, the optimization process of VQT can be expressed as follows:

[0146]

[0147] Among them, L1 can represent the regression loss function, which can be determined by the visual quality parameter, L rank The PLCC loss function can be characterized, which can be determined by the visual quality relative parameter, X i It can characterize the evaluated video 1, X j It can represent the evaluated video 2, that is, two sample videos, and F(X) can represent the processed by VQT network, y i is a visual quality parameter that can characterize the evaluated video 1, y j It can characterize the visual quality parameters corresponding to the evaluated video 2, and Lrank can constrain y i With y j The relative quality relationship is consistent with the real evaluation, such as the relative visual quality parameters can be obtained based on the evaluated video 1 and the evaluated video 2.

[0148] The technical solution of this embodiment extracts serialized features from each frame in a frame set to obtain a set of serialized features corresponding to the frame set, and then uses the set of serialized features corresponding to the frame set as input information of the first encoding module, which can provide data support for subsequent feature processing based on the serialized features.

[0149] Figure 4 is a flowchart of another video quality prediction method according to an exemplary embodiment. Figure 4 As shown, the method is used in a computer device such as a server, and includes the following steps.

[0150] In step S410, a frame set corresponding to the video to be predicted is obtained; the frame set is obtained by sampling the interval frames of the video to be predicted. In step S420, the frame set is input into the visual quality prediction model, and the visual quality prediction model determines multiple key frame sequences based on the frame set. In step S430, for each key frame sequence, according to the importance of each key frame contained in the key frame sequence, the feature correlation between the key frames contained in the key frame sequence is determined as the branch inter-frame feature corresponding to the key frame sequence. In step S440, multiple branch inter-frame features corresponding to multiple key frame sequences are feature spliced ​​to obtain spliced ​​inter-frame features, and feature filling is performed on the spliced ​​inter-frame features, and the overall inter-frame features corresponding to the frame set are obtained based on the inter-frame features after feature filling. In step S450, the intra-frame feature correlation of each frame in the frame set is determined, and the intra-frame features corresponding to the frame set are obtained based on the intra-frame feature correlation of multiple frames. In step S460, the enhanced features of each frame in the frame set are determined, and the frame feature expression corresponding to the frame set is obtained based on the enhanced features of multiple frames. In step S470, the visual quality parameters of the video to be predicted are determined according to the overall inter-frame features, intra-frame features, and frame feature expressions corresponding to the frame set, as the output information of the visual quality prediction model. In step S480, the visual quality prediction result of the video to be predicted is determined based on the output information of the visual quality prediction model. It should be noted that the specific definition of the above steps can refer to the specific definition of a video quality prediction method above, which will not be repeated here.

[0151] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0152] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0153] Based on the same inventive concept, an embodiment of the present disclosure further provides a video quality prediction device for implementing the above-mentioned video quality prediction method.

[0154] Figure 5 is a block diagram of a video quality prediction device according to an exemplary embodiment. Figure 5 , the device comprises:

[0155] The frame set acquisition unit 501 is configured to acquire a frame set corresponding to the video to be predicted; the frame set is obtained by performing interval frame sampling on the video to be predicted;

[0156] The inter-frame feature obtaining unit 502 is configured to determine a plurality of key frame sequences based on the frame set, determine the feature correlation between the key frames included in each of the key frame sequences as the corresponding branch inter-frame features, and obtain the overall inter-frame features corresponding to the frame set from the plurality of branch inter-frame features; wherein each of the key frame sequences includes at least two key frames selected from the frame set, and the key frames corresponding to the plurality of key frame sequences have different sparsity levels in time sequence;

[0157] The visual quality prediction result obtaining unit 503 is configured to determine the visual quality prediction result of the to-be-predicted video based on the overall inter-frame features corresponding to the frame set.

[0158] In one possible implementation, the inter-frame feature obtaining unit 502 is specifically configured to execute the acquisition of the number of frames contained in the frame set and a preset sparsity parameter; the preset sparsity parameter is used to characterize the maximum sparsity degree of key frames selected from the frame set; according to the number of frames contained in the frame set and the preset sparsity parameter, the number of branches used to obtain the branch inter-frame features is determined; each of the multiple branches is used to obtain the inter-frame feature correlation between key frames having a certain sparsity degree in time sequence, as the branch inter-frame feature corresponding to each branch, the sparsity degree being less than the maximum sparsity degree; multiple key frames are determined from the frame set, and multiple key frame sequences corresponding to the number of branches are determined based on the multiple key frames.

[0159] In one possible implementation, the inter-frame feature is obtained 502, which is specifically configured to execute, for each branch, obtaining the sparsity of the key frames corresponding to each branch; determining the number of key frames corresponding to each branch according to the sparsity of the key frames corresponding to each branch; and selecting the key frames corresponding to the number of key frames from the frame set based on the sparsity of the key frames corresponding to each branch and the number of key frames corresponding to each branch, to obtain a key frame sequence corresponding to each branch.

[0160] In one possible implementation, the inter-frame feature is obtained 502, which is specifically configured to obtain the serial number of each branch among the multiple branches; based on the serial number of each branch among the multiple branches and the preset sparsity parameter, determine the sparsity of the key frame corresponding to each branch.

[0161] In one possible implementation, the inter-frame feature is obtained 502, which is specifically configured to perform, for each key frame sequence, determining, according to the importance corresponding to each key frame contained in the key frame sequence, a feature correlation between the key frames contained in the key frame sequence as a branch inter-frame feature corresponding to the key frame sequence; wherein the importance corresponding to each key frame is obtained by calculating the importance of each frame in the frame set in the attention map using the attention mechanism.

[0162] In one possible implementation, the inter-frame features are obtained 502, which are specifically configured to perform feature splicing of multiple branch inter-frame features corresponding to multiple key frame sequences to obtain spliced ​​inter-frame features; perform feature filling on the spliced ​​inter-frame features, and the number of frames corresponding to the inter-frame features after the feature filling is the same as the number of frames included in the frame set; and obtain the overall inter-frame features corresponding to the frame set based on the inter-frame features after the feature filling.

[0163] In a possible implementation, the inter-frame feature is obtained 502, which is specifically configured to execute inputting the frame set into a visual quality prediction model, wherein the visual quality prediction model includes a plurality of branch structures, each branch structure determines a key frame sequence based on the frame set, and determines the feature correlation between the key frames included in the key frame sequence as the branch inter-frame feature corresponding to each branch structure, and obtains the overall inter-frame feature corresponding to the frame set from the plurality of branch inter-frame features corresponding to the plurality of branch structures;

[0164] The visual quality prediction result obtaining unit 503 is specifically configured to determine the visual quality prediction result of the video to be predicted based on the output information of the visual quality prediction model; the output information of the visual quality prediction model is determined based on the overall inter-frame features.

[0165] In a possible implementation, the video quality prediction device further includes:

[0166] An intra-frame feature obtaining unit is specifically configured to determine the intra-frame feature correlation of each frame in the frame set, and obtain the intra-frame features corresponding to the frame set based on the intra-frame feature correlation of multiple frames;

[0167] a frame feature expression obtaining unit, specifically configured to determine the enhanced features of each frame in the frame set, and obtain the frame feature expression corresponding to the frame set based on the enhanced features of multiple frames; the enhanced features of each frame are features obtained by performing nonlinear transformation processing on initial features of each frame;

[0168] The visual quality parameter determination unit is specifically configured to determine the visual quality parameters of the video to be predicted as output information of the visual quality prediction model according to the overall inter-frame features, the intra-frame features and the frame feature expressions corresponding to the frame set.

[0169] In a possible implementation, the visual quality prediction model includes a plurality of stacked encoding modules; the visual quality parameter determination unit is specifically configured to determine the overall inter-frame features, the intra-frame features, and the frame feature expressions corresponding to the frame set based on the current encoding module, and use the overall inter-frame features, the intra-frame features, and the frame feature expressions determined by the current encoding module as input information of the next encoding module of the current encoding module; based on the overall inter-frame features, the intra-frame features, and the frame feature expressions determined by the last encoding module in the plurality of stacked encoding modules, determine the visual quality parameters of the video to be predicted as output information of the visual quality prediction model;

[0170] Each of the encoding modules includes a temporal attention mechanism unit, a spatial attention mechanism unit and a multi-layer perception unit; the temporal attention mechanism unit includes a plurality of the branch structures;

[0171] In each of the encoding modules, the overall inter-frame features are determined based on the temporal attention mechanism unit;

[0172] The intra-frame features are determined based on the spatial attention mechanism unit;

[0173] The frame feature expression is determined based on the multi-layer perception unit.

[0174] In a possible implementation, the video quality prediction device further includes:

[0175] A serialization feature extraction unit is specifically configured to perform serialization feature extraction on each frame in the frame set to obtain a set of serialization features corresponding to the frame set;

[0176] The serialization feature input unit is specifically configured to execute a set of serialization features corresponding to the frame set as input information of the first encoding module.

[0177] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0178] Each module in the above video quality prediction device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0179] Figure 6 1 is a block diagram of an electronic device 600 for implementing a video quality prediction method according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0180] Reference Figure 6 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0181] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0182] The memory 604 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, optical disk, or graphene memory.

[0183] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0184] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0185] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting an audio signal.

[0186] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0187] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of the components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or the electronic device 600 component, the presence or absence of contact between the user and the electronic device 600, the device 600 orientation or acceleration / deceleration and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0188] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0189] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0190] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the above instructions can be executed by the processor 620 of the electronic device 600 to perform the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0191] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by the processor 620 of the electronic device 600 to complete the above method.

[0192] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. may also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.

[0193] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0194] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video quality prediction method, characterized in that: The method comprises: Acquire a frame set corresponding to the video to be predicted; the frame set is obtained by performing interval frame sampling on the video to be predicted; The number of frames included in the frame set and a preset sparsity parameter are obtained; the number of branches used to obtain branch inter-frame features is determined according to the number of frames included in the frame set and the preset sparsity parameter; each of the multiple branches is used to obtain inter-frame feature correlations between key frames having a sparsity degree in time sequence, as branch inter-frame features corresponding to each branch, the sparsity degree being less than a maximum sparsity degree; multiple key frames are determined from the frame set, and multiple key frame sequences corresponding to the number of branches are determined based on the multiple key frames; Determine the feature correlation between the key frames included in each of the key frame sequences as the corresponding branch inter-frame features, and obtain the overall inter-frame features corresponding to the frame set from the multiple branch inter-frame features; wherein each of the key frame sequences includes at least two key frames selected from the frame set, and the key frames corresponding to the multiple key frame sequences have different sparsity levels in time sequence, and the branch inter-frame features represent the low-quality features corresponding to the key frame sequences; the preset sparsity parameter is used to represent the maximum sparsity level of the key frames selected from the frame set; Based on the overall inter-frame features corresponding to the frame set, a visual quality prediction result of the video to be predicted is obtained.

2. The method according to claim 1, characterized in that The step of determining a plurality of key frames from the frame set, and determining a plurality of key frame sequences corresponding to the number of branches based on the plurality of key frames, comprises: For each branch, obtaining the sparsity of the key frames corresponding to each branch; Determining the number of key frames corresponding to each branch according to the sparseness of the key frames corresponding to each branch; Based on the sparsity of the key frames corresponding to each branch and the number of key frames corresponding to each branch, key frames corresponding to the number of key frames are selected from the frame set to obtain a key frame sequence corresponding to each branch.

3. The method according to claim 2, characterized in that The obtaining the sparsity of the key frame corresponding to each branch includes: Obtaining a sequence number of each branch among the multiple branches; Based on the sequence number of each branch in the multiple branches and the preset sparse parameter, the sparseness of the key frame corresponding to each branch is determined.

4. The method according to claim 1, characterized in that The determining of the feature correlation between the key frames included in each of the key frame sequences as the corresponding branch inter-frame features includes: For each key frame sequence, determining the feature correlation between the key frames included in the key frame sequence according to the importance of each key frame included in the key frame sequence as the branch inter-frame feature corresponding to the key frame sequence; Among them, the importance corresponding to each key frame is obtained by calculating the importance of each frame in the frame set in the attention map using the attention mechanism.

5. The method according to claim 1, characterized in that The obtaining the overall inter-frame features corresponding to the frame set from the plurality of branch inter-frame features comprises: Performing feature splicing on a plurality of branch inter-frame features corresponding to a plurality of the key frame sequences to obtain spliced ​​inter-frame features; Performing feature filling on the spliced ​​inter-frame features, wherein the number of frames corresponding to the inter-frame features after the feature filling is the same as the number of frames included in the frame set; The overall inter-frame features corresponding to the frame set are obtained based on the inter-frame features after the feature filling.

6. The method according to claim 1, characterized in that The determining of the feature correlation between the key frames included in each of the key frame sequences as the corresponding branch inter-frame features, and obtaining the overall inter-frame features corresponding to the frame set from the plurality of branch inter-frame features, includes: The frame set is input into a visual quality prediction model, wherein the visual quality prediction model includes a plurality of branch structures, each branch structure determines a key frame sequence based on the frame set, and determines a feature correlation between key frames included in the key frame sequence as a branch inter-frame feature corresponding to each branch structure, and obtains an overall inter-frame feature corresponding to the frame set from a plurality of branch inter-frame features corresponding to the plurality of branch structures; The obtaining the visual quality prediction result of the video to be predicted based on the overall inter-frame features corresponding to the frame set includes: Based on the output information of the visual quality prediction model, a visual quality prediction result of the video to be predicted is determined; the output information of the visual quality prediction model is determined based on the overall inter-frame features.

7. The method according to claim 6, characterized in that Before the step of determining the visual quality prediction result of the video to be predicted based on the output information of the visual quality prediction model, the method further includes: Determine the intra-frame feature correlation of each frame in the frame set, and obtain the intra-frame features corresponding to the frame set based on the intra-frame feature correlation of multiple frames; Determine the enhanced features of each frame in the frame set, and obtain the frame feature expression corresponding to the frame set based on the enhanced features of multiple frames; the enhanced features of each frame are features obtained by performing nonlinear transformation processing on the initial features of each frame; According to the overall inter-frame features, the intra-frame features and the frame feature expressions corresponding to the frame set, the visual quality parameters of the video to be predicted are determined as output information of the visual quality prediction model.

8. The method according to claim 7, characterized in that The visual quality prediction model comprises a plurality of stacked encoding modules; the determining, according to the overall inter-frame features, the intra-frame features and the frame feature expressions corresponding to the frame set, the visual quality parameters of the video to be predicted as output information of the visual quality prediction model comprises: Determine the overall inter-frame features, the intra-frame features, and the frame feature expressions corresponding to the frame set based on the current encoding module, and use the overall inter-frame features, the intra-frame features, and the frame feature expressions determined by the current encoding module as input information of a next encoding module of the current encoding module; Determine, based on the overall inter-frame features, intra-frame features, and frame feature expressions determined by the last encoding module in the plurality of stacked encoding modules, a visual quality parameter of the video to be predicted as output information of the visual quality prediction model; Each of the encoding modules includes a temporal attention mechanism unit, a spatial attention mechanism unit and a multi-layer perception unit; the temporal attention mechanism unit includes a plurality of the branch structures; In each of the encoding modules, the overall inter-frame features are determined based on the temporal attention mechanism unit; The intra-frame features are determined based on the spatial attention mechanism unit; The frame feature expression is determined based on the multi-layer perception unit.

9. The method according to claim 8, characterized in that Before determining the overall inter-frame features, the intra-frame features, and the frame feature expressions corresponding to the frame set based on the first encoding module in the plurality of stacked encoding modules, the method further includes: Extracting serialized features from each frame in the frame set to obtain a set of serialized features corresponding to the frame set; A set of serialized features corresponding to the frame set is used as input information of the first encoding module.

10. A video quality prediction device, characterized in that: The device comprises: A frame set acquisition unit is configured to acquire a frame set corresponding to the video to be predicted; the frame set is obtained by performing interval frame sampling on the video to be predicted; The inter-frame feature obtaining unit is configured to execute the acquisition of the number of frames included in the frame set and the preset sparsity parameter; determine the number of branches used to obtain branch inter-frame features according to the number of frames included in the frame set and the preset sparsity parameter; each of the multiple branches is used to obtain the inter-frame feature correlation between key frames with a sparsity degree in time sequence as the branch inter-frame feature corresponding to each branch, and the sparsity degree is less than the maximum sparsity degree; determine multiple key frames from the frame set, and determine multiple key frame sequences corresponding to the number of branches based on the multiple key frames; determine the feature correlation between the key frames included in each of the key frame sequences as the corresponding branch inter-frame feature, and obtain the overall inter-frame feature corresponding to the frame set from the multiple branch inter-frame features; wherein each of the key frame sequences includes at least two key frames selected from the frame set, and the key frames corresponding to the multiple key frame sequences have different sparsity degrees in time sequence, and the branch inter-frame feature characterizes the low-quality features corresponding to the key frame sequence; the preset sparsity parameter is used to characterize the maximum sparsity degree of the key frames selected from the frame set; The visual quality prediction result obtaining unit is configured to obtain the visual quality prediction result of the video to be predicted based on the overall inter-frame features corresponding to the frame set.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video quality prediction method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the video quality prediction method according to any one of claims 1 to 9.

13. A computer program product, comprising instructions, characterized in that: When the instruction is executed by a processor of an electronic device, the electronic device is enabled to execute the video quality prediction method according to any one of claims 1 to 9.

Images