Video quality evaluation method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a video quality evaluation method and device, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, the video quality features of the to-be-evaluated video frame are extracted, and the image region extraction is performed on the to-be-evaluated video frame, so that the image region masks corresponding to the one or more preset evaluation targets are obtained, and fusion processing can be performed on the video quality features and the image region masks; obtaining region quality features corresponding to the one or more preset evaluation targets, and determining image region quality evaluation results corresponding to the one or more preset evaluation targets of the to-be-evaluated video frame according to the region quality features, and according to the image area quality evaluation result, determining a video area quality evaluation result of the to-be-evaluated video corresponding to one or more preset evaluation targets. The video quality evaluation fineness and the video quality evaluation effect can be effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a video quality assessment method, apparatus, device, storage medium and product. Background Art

[0002] With the evolution of basic network technology, the improvement of hardware performance of mobile devices, and people's pursuit and yearning for quality of life, the behavior of watching short videos, live videos and long video content on mobile devices has become widely popular.

[0003] As a core indicator of video transmission, video quality assessment is widely used in media centers to describe various distortion behaviors and subjective quality scores in videos, and then assist in subsequent processing links such as video encoding, recommendation and distribution. At present, video quality assessment mainly focuses on the subjective quality evaluation of the whole image, such as inputting the video to be evaluated into the neural network and directly obtaining the quality assessment result of the video to be evaluated through the neural network. However, this video quality assessment scheme can only reflect the overall quality of the video, and the video quality assessment is relatively rough and the video quality assessment effect is poor. Summary of the invention

[0004] The embodiments of the present application provide a video quality assessment method, apparatus, device, storage medium and product to solve the technical problems in related technologies that video quality assessment mainly focuses on the subjective quality evaluation of the whole image, the video quality assessment is relatively rough, and the video quality assessment effect is poor. It can effectively improve the precision of video quality assessment and the video quality assessment effect, and effectively assist video processing and content distribution.

[0005] In a first aspect, an embodiment of the present application provides a video quality assessment method, comprising:

[0006] Obtaining a video frame to be evaluated in the video to be evaluated, and extracting video quality features of the video frame to be evaluated;

[0007] Extracting an image region from the video frame to be evaluated to obtain an image region mask corresponding to one or more preset evaluation targets;

[0008] The video quality feature and the image region mask are fused to obtain region quality features corresponding to one or more preset evaluation targets;

[0009] Determine, according to the regional quality features, an image region quality assessment result of the to-be-assessed video frame corresponding to one or more preset assessment targets;

[0010] According to the image area quality assessment result, a video area quality assessment result of the video to be assessed corresponding to one or more preset assessment targets is determined.

[0011] In a second aspect, an embodiment of the present application provides a video quality assessment device, including a feature extraction module, a region extraction module, a region fusion module, an image assessment module and a video assessment module, wherein:

[0012] The feature extraction module is configured to obtain a video frame to be evaluated and extract video quality features of the video frame to be evaluated;

[0013] The region extraction module is configured to extract the image region of the video frame to be evaluated to obtain an image region mask corresponding to one or more preset evaluation targets;

[0014] The regional fusion module is configured to fuse the video quality features and the image region mask to obtain regional quality features corresponding to one or more preset evaluation targets;

[0015] The image evaluation module is configured to determine the image region quality evaluation result of the video frame to be evaluated corresponding to one or more preset evaluation targets according to the region quality feature;

[0016] The video evaluation module is configured to determine the video area quality evaluation results of the video to be evaluated corresponding to one or more preset evaluation targets according to the image area quality evaluation results.

[0017] In a third aspect, an embodiment of the present application provides a video quality assessment device, including: a memory and one or more processors;

[0018] The memory is used to store one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the video quality assessment method as described in the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are used to perform the video quality assessment method as described in the first aspect when executed by a computer processor.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of a device reads and executes the computer program from the computer-readable storage medium, so that the device performs the video quality assessment method as described in the first aspect.

[0022] The embodiment of the present application extracts video quality features of a video frame to be evaluated, and performs image region extraction on the video frame to be evaluated to obtain image region masks corresponding to one or more preset evaluation targets. The video quality features and the image region masks can be fused to obtain region quality features corresponding to one or more preset evaluation targets. Image region quality evaluation results of the video frame to be evaluated corresponding to one or more preset evaluation targets are determined according to the region quality features, and video region quality evaluation results of the video to be evaluated corresponding to one or more preset evaluation targets are determined according to the image region quality evaluation results. High-quality regionalized quality perception of different preset evaluation targets can be achieved through the image region masks corresponding to one or more preset evaluation targets, which can effectively improve the precision of video quality evaluation and the effect of video quality evaluation, and effectively assist video processing and content distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of a video quality assessment method provided by an embodiment of the present application;

[0024] Figure 2 is a flowchart of another video quality assessment method provided by an embodiment of the present application;

[0025] Figure 3 It is a schematic diagram of the structure of a feature extraction module provided in an embodiment of the present application;

[0026] Figure 4 It is a structural diagram of a video quality assessment model in a training phase provided in an embodiment of the present application;

[0027] Figure 5 This is a schematic diagram of the structure of a video quality assessment model after training provided in an embodiment of the present application;

[0028] Figure 6 is a structural schematic diagram of a video quality assessment device provided in an embodiment of the present application;

[0029] Figure 7 It is a structural diagram of a video quality assessment device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The above process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The above process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0031] The video quality assessment method provided in this application can be applied to live video, short video platforms, etc., by providing video area quality assessment results of areas corresponding to different preset assessment targets of the video, providing picture quality indicators for the recommended business line, guiding the sorting of content such as low-score suppression and high-score weighting, and at the same time, the video area quality assessment results of different areas can be used to guide the development of downstream video enhancement and encoding services. The video quality assessment method can be deployed in the transcoding link link of the background, and interact with other business lines through regular reporting.

[0032] Existing video quality assessment schemes mainly focus on the subjective quality evaluation of the whole picture, such as inputting the video to be evaluated into the neural network, and directly obtaining the quality assessment result of the video to be evaluated through the neural network. With the increasing popularity of live broadcasts and short videos, people, as the main body of video viewing, show different behavioral habits, usually paying more attention to the main part related to "people" in the video, and paying less attention to or ignoring the background. In addition, the differential algorithm processing of the video from generation to the final viewing process, such as using a higher bit rate and a better algorithm for the face area to improve clarity, and using a lower bit rate for the background to save costs, all lead to uneven distribution of image quality in the whole picture. However, the current video quality assessment scheme can only reflect the overall quality of the video, and the video quality assessment is relatively rough, resulting in some videos with high quality background and blurred foreground still obtaining high quality scores, which violates people's subjective feelings, and the video quality assessment effect is poor. Based on this, a video quality assessment method of an embodiment of the present application is provided to solve the technical problem that the video quality assessment effect of the existing video quality assessment scheme is poor.

[0033] Figure 1A flowchart of a video quality assessment method provided in an embodiment of the present application is given. The video quality assessment method provided in an embodiment of the present application can be performed by a video quality assessment device, which can be implemented in hardware and / or software and integrated in a video quality assessment device.

[0034] The following description is made by taking a video quality assessment device executing a video quality assessment method as an example. Figure 1 , the video quality assessment method includes:

[0035] S110: Obtain a video frame to be evaluated in the video to be evaluated, and extract video quality features of the video frame to be evaluated.

[0036] Exemplarily, a video to be evaluated that needs to be evaluated for video quality is obtained, and a video frame to be evaluated in the video to be evaluated is obtained, so as to perform regional quality evaluation on the video frame to be evaluated for one or more preset evaluation targets (the evaluation target may be a portrait, a human body, a face, hair, a foreground, a background, a global image, etc. in the image). Optionally, regional quality evaluation may be performed on multiple video frames to be evaluated in the video to be evaluated at the same time, or may be performed on multiple video frames to be evaluated in sequence.

[0037] In one embodiment, feature extraction is performed on the acquired video frame to be evaluated to obtain video quality features corresponding to the video frame to be evaluated. Optionally, the video quality features of the video frame to be evaluated may be extracted by a preset feature extraction algorithm or by a trained neural network model.

[0038] In a possible embodiment, before extracting the video quality features of the video frame to be evaluated, the video quality assessment method provided by the present solution may also perform a whitening operation on the video frame to be evaluated based on the video frame to be evaluated and other video frames in the video to be evaluated. Correspondingly, the video quality assessment method provided by the present solution extracts the video quality features of the video frame to be evaluated by: extracting the video quality features of the video frame to be evaluated after the whitening operation.

[0039] Exemplarily, after obtaining the video frame to be evaluated in the video to be evaluated, the video frame to be evaluated may be whitened according to the pixel values ​​of each pixel point in the video frame to be evaluated and other video frames in the video to be evaluated. For example, the video frame to be evaluated may be whitened using the following formula:

[0040]

[0041] Where (x, y) is the pixel coordinate of the video to be evaluated, I′(x, y) is the video quality feature after the whitening operation, and I(x, y) is the video quality feature before the whitening operation. is the average value of all the video frames to be evaluated at the (x, y) position in the video to be evaluated, and σ(D(x, y)) is the standard deviation of all the video frames to be evaluated at the (x, y) position in the video to be evaluated. This scheme reduces the correlation between frames and improves the accuracy of video quality assessment by whitening the video frames to be evaluated.

[0042] In one embodiment, after obtaining the video frame to be evaluated in the video to be evaluated, the video frame to be evaluated may be normalized based on a preset frame size and frame format, and then the video quality features of the normalized video frame to be evaluated may be extracted. For example, the video frame to be evaluated is normalized to a format of H×W×3RGB24, and its value range is in the interval [0, 255], where H and W are preset image height and width.

[0043] Optionally, after obtaining the video to be evaluated, the video to be evaluated may be converted into a preset video format and / or color format, for example, the video format of the video to be evaluated may be converted into a video format such as MP4, AVI, RMVB, X264, etc., and the color format of the video to be evaluated may be converted into a color format such as RGB24, YUV420, YUV444, etc.

[0044] S120: Extracting image regions from the video frame to be evaluated, and obtaining image region masks corresponding to one or more preset evaluation targets.

[0045] Exemplarily, based on the preset evaluation target, the image region extraction is performed on the video frame to be evaluated to obtain one or more image region masks corresponding to one or more preset evaluation targets in the image region extraction, wherein the size of the image region mask is consistent with the size of the video quality feature.

[0046] Optionally, the image semantic segmenter can be used to segment the video frame to be evaluated based on the preset evaluation target to achieve image region extraction of the video frame to be evaluated. For example, the video frame to be evaluated is sent to the image semantic segmenter, and the image semantic segmenter performs one or more combinations of portrait, human body, face, hair, foreground, background, etc. on the video frame to be evaluated to obtain one or more image region masks. The value range of the pixel value corresponding to the image region mask is [0, 1]. The larger the value corresponding to the pixel point, the higher the possibility that the high pixel point is the pixel point corresponding to the preset evaluation target.

[0047] S130: Fusing the video quality features and the image region mask to obtain region quality features corresponding to one or more preset evaluation targets.

[0048] Exemplarily, after obtaining the video quality features and the image area masks corresponding to one or more preset evaluation targets, for each preset evaluation target, the video quality features and the image area masks are fused to obtain the area quality features corresponding to each preset evaluation target.

[0049] In one embodiment, the video quality feature and the image area mask are fused by performing pixel-level dot multiplication on the video quality feature and one or more image area masks, respectively, to accurately obtain the video quality feature that reflects the quality feature corresponding to the preset evaluation target.

[0050] S140: Determine, according to the regional quality features, an image region quality assessment result of the video frame to be assessed corresponding to one or more preset assessment targets.

[0051] Exemplarily, for each preset evaluation target, the image region quality evaluation result of the video frame to be evaluated under the corresponding preset evaluation target can be determined according to the corresponding regional quality feature. Optionally, the image region quality evaluation result corresponding to each preset evaluation target can be obtained by analyzing and processing the regional quality feature through a trained neural network model.

[0052] S150: Determine, according to the image region quality assessment result, a video region quality assessment result of the video to be assessed corresponding to one or more preset assessment targets.

[0053] Exemplarily, for each preset evaluation target, the video area quality evaluation result of the video to be evaluated under the corresponding preset evaluation target can be determined based on the above-determined image area quality evaluation result. For example, for a preset evaluation target, the average value of the image area quality evaluation results corresponding to the preset evaluation target is used as the video area quality evaluation result of the video to be evaluated under the preset evaluation target, or the weighted sum of the image area quality evaluation results corresponding to the preset evaluation target is used as the video area quality evaluation result of the video to be evaluated under the preset evaluation target, or the product of the image area quality evaluation results corresponding to the preset evaluation target is used as the video area quality evaluation result of the video to be evaluated under the preset evaluation target. Among them, the larger the numerical value corresponding to the video area quality evaluation result, the higher the video quality under the corresponding preset evaluation target.

[0054] In one embodiment, the video region quality assessment results corresponding to one or more preset assessment targets of the video to be assessed may be determined according to the image region quality assessment results corresponding to all or part of the video frames to be assessed in the video to be assessed.

[0055] In the above, by extracting the video quality features of the video frame to be evaluated, and performing image area extraction on the video frame to be evaluated, image area masks corresponding to one or more preset evaluation targets are obtained, and the video quality features and the image area masks can be fused to obtain regional quality features corresponding to one or more preset evaluation targets. The image area quality evaluation results of the video frame to be evaluated corresponding to one or more preset evaluation targets are determined according to the regional quality features, and the video area quality evaluation results of the video to be evaluated corresponding to one or more preset evaluation targets are determined according to the image area quality evaluation results. High-quality regionalized quality perception of different preset evaluation targets can be achieved through the image area masks corresponding to one or more preset evaluation targets, which can effectively improve the precision of video quality evaluation and the video quality evaluation effect, and effectively assist video processing and content distribution.

[0056] Based on the above embodiments, Figure 2 A flowchart of another video quality assessment method provided in an embodiment of the present application is given, and the video quality assessment method is a specific embodiment of the above-mentioned video quality assessment method. Figure 2 , the video quality assessment method includes:

[0057] S210: Obtain a video frame to be evaluated in the video to be evaluated, and send the video frame to be evaluated to a trained video quality assessment model. The video quality assessment model performs feature extraction and downsampling processing of a first preset level on the video frame to be evaluated based on a feature extraction module, and performs feature fine extraction and upsampling processing of a second preset level on the video frame to be evaluated, so as to obtain video quality features of the video frame to be evaluated.

[0058] In one embodiment, the present solution can obtain the video quality features of the video frame to be evaluated through a trained video quality assessment model. The video quality assessment model is configured with a feature extraction module (for example, constructed based on an encoder) and a video quality regressor. The feature extraction module can be used to extract the quality features of the input image, and the video quality regressor can be used to output the corresponding quality assessment result according to the video quality features of the input end. Among them, when the feature extraction module extracts the quality features of the input image, it will perform feature extraction and downsampling processing of the first preset level, and feature fine extraction and upsampling processing of the second preset level on the input image and output the corresponding quality features. The feature extraction and downsampling processing of the first preset level is feature extraction and downsampling processing alternately connected in series, and the feature fine extraction and upsampling processing of the second preset level is feature fine extraction and upsampling processing alternately connected in series.

[0059] Exemplarily, after obtaining the video frame to be evaluated, the video frame to be evaluated is sent to the trained video quality evaluation model. After receiving the video frame to be evaluated, the video quality evaluation model sends the video frame to be evaluated to the feature extraction module, and the feature extraction module performs feature extraction and downsampling processing of the first preset level on the video frame to be evaluated, and fine feature extraction and upsampling processing of the second preset level on the video frame to be evaluated, so as to obtain the video quality features of the video frame to be evaluated. In one embodiment, the first preset level is greater than the second preset level.

[0060] This solution uses a video quality assessment model to perform feature extraction and downsampling processing at a first preset level on the video frame to be evaluated based on a feature extraction module, as well as fine feature extraction and upsampling processing at a second preset level. The low-resolution but highly compressed semantic feature information obtained through the feature extraction and downsampling processing at the first preset level is transmitted back to the feature information at the upper level, so that the video quality features have video quality features with higher resolution and high-compression semantics, effectively improving the quality of video quality assessment.

[0061] In a possible embodiment, the feature extraction module in the video quality assessment model provided by the present solution performs feature extraction and downsampling processing of a first preset level on the video frame to be assessed, and fine feature extraction and upsampling processing of a second preset level on the video frame to be assessed, and obtains the video quality features of the video frame to be assessed, which may be:

[0062] S211: performing feature extraction and down-sampling processing of a first preset level and fine feature extraction and up-sampling processing of a second preset level on the video frame to be evaluated, to obtain fine features of the target video.

[0063] S212: extracting frequency band quality features of target video fine features in multiple frequency spectra through a feature extractor with a preset number of convolution kernels of different sizes, and performing mixed processing on the multiple frequency band quality features to obtain video quality features of the evaluation video frame.

[0064] Exemplarily, after receiving the video frame to be evaluated, the feature extraction module performs a first preset level of feature extraction and downsampling processing on the video frame to be evaluated, and performs a second preset level of feature fine extraction and upsampling processing on the output result of the first preset level of feature extraction and downsampling processing to obtain fine features of the target video.

[0065] Among them, the levels corresponding to the first preset level increase from the front to the back, and the levels corresponding to the second preset level decrease from the front to the back.

[0066] Optionally, when the first preset level of feature extraction and downsampling processing is performed on the video frame to be evaluated, after the feature extraction of the video frame to be evaluated is performed, the feature extraction is sequentially downsampled and feature extracted until the first preset level of feature extraction is completed. The input of the downsampling is the output result of the feature extraction of the previous layer, and the input of the feature extraction is the output of the downsampling of the previous layer. The feature extraction can be the product of the output of the downsampling of the previous layer and the feature extraction weight obtained by the training of the corresponding layer.

[0067] Optionally, when performing fine feature extraction and upsampling processing at the second preset level, the input of upsampling is the output of fine feature extraction at the next level, the input of fine feature extraction is the output of upsampling at the next level and the output of feature extraction at the same level, and the input of fine feature extraction at the last level is the output of feature extraction at the last level. Fine feature extraction can be performed by multiplying the fine feature extraction weights obtained by training at the corresponding level by the sum of the output of upsampling at the next level and the output of feature extraction at the same level.

[0068] In one embodiment, after obtaining the target video fine features, the target video fine features can be sent to a preset number of feature extractors (filters) with different sizes of convolution kernels, and the frequency band quality features of the target video fine features in multiple spectra are extracted by different feature extractors. After obtaining the frequency band quality features of the target video fine features in multiple spectra, the multiple frequency band quality features can be mixed to obtain the video quality features of the evaluation video frame. Optionally, the multiple frequency band quality features can be mixed to obtain the video quality features of the evaluation video frame, which can be obtained by splicing the multiple frequency band quality features together to obtain the target video fine features.

[0069] This solution obtains target video fine features by performing feature extraction and down-sampling processing at a first preset level and feature fine extraction and up-sampling processing at a second preset level on the video frame to be evaluated, and extracts frequency band quality features of the target video fine features in multiple spectrums through a feature extractor with a preset number of convolution kernels of different sizes, and obtains video quality features of the evaluation video frame by mixing the multiple frequency band quality features. The target video fine features have video quality features with higher resolution and high compression semantics, and the target video fine features are also subjected to feature extraction and mixing processing of multiple frequency bands to obtain video quality features of the evaluation video frame, which can effectively improve the quality of video quality assessment.

[0070] In one possible embodiment, Figure 3As shown in a schematic diagram of a feature extraction module structure provided by the present solution, the first preset level provided by the present solution is 4 layers, and the second preset level is 3 layers, wherein the first preset level of the 4th layer corresponds to the second preset level of the 3rd layer. The feature extraction module in the video quality assessment model provided by the present solution includes a first feature extraction unit (F1), a second downsampling unit (D2), a second feature extraction unit (F2), a third downsampling unit (D3), a third feature extraction unit (F3), a fourth downsampling unit (D4) and a fourth feature extraction unit (F4) connected in sequence, and a fourth feature fine extraction unit (P4), a fourth upsampling unit (U4), a third feature fine extraction unit (P3), a third upsampling unit (U3) and a second feature fine extraction unit (P2) connected in sequence, and the fourth feature extraction unit is connected to the fourth feature fine extraction unit, the third feature extraction unit is connected to the third feature fine extraction unit, and the second feature extraction unit is connected to the second feature fine extraction unit. The first feature extraction unit, the second feature extraction unit, the third feature extraction unit and the fourth feature extraction unit respectively correspond to the first feature extraction weight (W) confirmed in the training stage. F1 ), the second feature extraction weight (W F2 ), the third feature extraction weight (W F3 ) and the fourth feature extraction weight (W F4 ), the fourth feature fine extraction unit, the third feature fine extraction unit and the second feature fine extraction unit respectively correspond to the fourth feature fine extraction weight (W) confirmed in the training stage P4 ), the third feature fine extraction weight (W P3 ) and the second feature refinement weight (W P2 ). Similarly, the network parameters of the video quality regressor are also determined during the training phase.

[0071] Correspondingly, the feature extraction module in the video quality assessment model provided by the present solution performs feature extraction and downsampling processing of the first preset level on the video frame to be assessed, and fine feature extraction and upsampling processing of the second preset level to obtain fine features of the target video, which can be:

[0072] S2111: extracting a first video feature of a video frame to be evaluated by a first feature extraction unit, downsampling the first video feature by a second downsampling unit and a second feature extraction unit to obtain a second video feature, downsampling the second video feature by a third downsampling unit and a third feature extraction unit to obtain a third video feature, and downsampling the third video feature by a fourth downsampling unit and a fourth feature extraction unit to obtain a fourth video feature.

[0073] S2112: Performing fine feature extraction on the fourth video feature through the fourth feature fine extraction unit to obtain the fourth video fine feature, performing fine feature extraction on the fourth video fine feature up-sampled by the fourth upsampling unit and the third feature fine extraction unit to obtain the third video fine feature, and performing fine feature extraction on the third video fine feature up-sampled by the third upsampling unit and the second feature fine extraction unit to obtain the target video fine feature.

[0074] Exemplarily, the feature extraction module sends the video frame to be evaluated to the first feature extraction unit, and the first video feature of the video frame to be evaluated is extracted by the first feature extraction unit (for example, the first video feature is obtained by multiplying the video frame to be evaluated by the first feature extraction weight). Then, the first video feature is downsampled and extracted by the second downsampling unit and the second feature extraction unit to obtain the second video feature (for example, after downsampling the first video feature by the second downsampling unit, the downsampling result is multiplied by the second feature extraction weight to obtain the second video feature), and the second video feature is sent to the third downsampling unit and the second feature fine extraction unit. The second video feature is downsampled and extracted by the third downsampling unit and the third feature extraction unit to obtain the third video feature (for example, after downsampling the second video feature by the third downsampling unit, the downsampling result is multiplied by the third feature extraction weight to obtain the third video feature), and the third video feature is sent to the fourth downsampling unit and the third feature fine extraction unit. The third video feature is downsampled and extracted by the fourth downsampling unit and the fourth feature extraction unit to obtain the fourth video feature (for example, after the third video feature is downsampled by the fourth downsampling unit, the downsampling result is multiplied by the fourth feature extraction weight to obtain the fourth video feature), and the fourth video feature will be sent to the fourth feature fine extraction unit.

[0075] In one embodiment, the feature extraction module performs feature fine extraction on the fourth video feature through the fourth feature fine extraction unit to obtain the fourth video fine feature (for example, the fourth video feature is multiplied by the fourth feature fine extraction weight to obtain the fourth video fine feature). Then, the fourth video fine feature is upsampled and finely extracted through the fourth upsampling unit and the third feature fine extraction unit to obtain the third video fine feature (for example, after upsampling the fourth video fine feature through the fourth upsampling unit, the sum of the upsampling result and the third video feature is multiplied by the third feature fine extraction weight to obtain the third video fine feature), and the third video fine feature is upsampled and finely extracted through the third upsampling unit and the second feature fine extraction unit to obtain the target video fine feature (for example, after upsampling the third video fine feature through the third upsampling unit, the sum of the upsampling result and the second video feature is multiplied by the second feature fine extraction weight to obtain the target video fine feature (i.e., the second video fine feature)).

[0076] This solution obtains low-resolution features by progressively extracting and downsampling the video frames to be evaluated, and then performs progressive fine feature extraction and upsampling on the low-resolution features to obtain fine features of the target video with higher resolution and high compression semantics, effectively improving the quality of video quality assessment.

[0077] In a possible embodiment, the feature extraction module in the video quality assessment model provided by the present solution extracts the frequency band quality features of the fine features of the target video in multiple spectrums through a feature extractor with a preset number of convolution kernels of different sizes, which may include: extracting the high-frequency quality features of the fine features of the target video through a feature extractor with a 3*3 convolution kernel; extracting the mid-frequency quality features of the fine features of the target video through a feature extractor with a 5*5 convolution kernel; extracting the low-frequency quality features of the fine features of the target video through a feature extractor with a 7*7 convolution kernel.

[0078] Exemplarily, the fine features of the target video are respectively input into the feature extractor of 3*3 convolution kernel, the feature extractor of 5*5 convolution kernel and the feature extractor of 7*7 convolution kernel. While the resolution of the fine features of the target video remains consistent, the high-frequency quality features, mid-frequency quality features and low-frequency quality features of the fine features of the target video are extracted by the feature extractors of these convolution kernels of different sizes. The quality features of different frequency bands are mixed to obtain the video quality features of the evaluated video frame. The video quality evaluation of the video frame to be evaluated is more comprehensive, which can effectively improve the quality of video quality evaluation.

[0079] S220: Extracting image regions from the video frame to be evaluated to obtain image region masks corresponding to one or more preset evaluation targets.

[0080] S230: Fusing the video quality features and the image region mask to obtain region quality features corresponding to one or more preset evaluation targets.

[0081] S240: Send the regional quality features to the video quality regressor of the video quality assessment model, and determine the image region quality assessment result of the video frame to be assessed corresponding to the preset assessment target through the video quality regressor according to the regional quality features.

[0082] Exemplarily, after obtaining regional quality features corresponding to one or more preset evaluation targets, the regional quality features can be sent simultaneously or sequentially to the video quality regressor of the video quality assessment model, and the regional quality features are analyzed and processed by the video quality regressor to obtain the image area quality assessment results of the video frame to be evaluated corresponding to the preset evaluation target.

[0083] like Figure 4A structural schematic diagram of a video quality assessment model in a training phase is provided, wherein the output end of the feature extraction module in the video quality assessment model is connected to the output end of the video quality regressor. After constructing the video quality assessment data set, the video quality assessment data set can be divided to obtain a training set and a validation set, and the video quality assessment model can be trained using the training set and the validation set (for example, regression training is performed using the L1 norm, and network parameter optimization is performed using the ADAM optimizer), and the network parameters of the feature extraction module and the video quality regressor are determined. The video quality assessment model may also include a quality feature mixing unit that extracts the frequency band quality features of the target video fine features in multiple frequency spectra through a feature extractor with a preset number of convolution kernels of different sizes, and mixes the multiple frequency band quality features to obtain the video quality features of the assessment video frame, which is not shown in the figure. Optionally, before the sample video frames in the training set are input into the video quality assessment model, the sample video frames may be whitened and / or normalized.

[0084] like Figure 5 A structural schematic diagram of a video quality assessment model after training is provided, wherein in the training stage, the feature extraction unit Fx, the feature fine extraction unit Px and the video quality regressor C2 have the ability to perceive the quality of the entire image with high-resolution features. This scheme introduces an image semantic segmentor into the video quality assessment model, and the output end of the feature extraction module is connected to the image semantic segmentor. At the same time, the video frame to be evaluated is also input into the image semantic segmentor. The image semantic segmentor extracts the image region of the video frame to be evaluated to obtain image region masks corresponding to multiple preset evaluation targets (the background mask, foreground mask, face mask, hair mask and full image mask are described as examples in the figure). After the video quality features and the image region masks are fused to obtain the region quality features corresponding to different preset evaluation targets, the video quality regressor is used to determine the image region quality assessment results of the video frame to be evaluated corresponding to different preset evaluation targets for the region quality features. Among them, the network parameters of the video quality regressors corresponding to different preset evaluation targets are the same, and the processing of regional quality features corresponding to different preset evaluation targets can reuse the same video quality regressor. For example, multiple identical video quality regressors are configured in the video quality assessment model, which are used to process regional quality features corresponding to different preset evaluation targets respectively.

[0085] S250: Determine, according to the image region quality assessment result, a video region quality assessment result of the video to be assessed corresponding to one or more preset assessment targets.

[0086] In the above, by extracting the video quality features of the video frame to be evaluated, and performing image region extraction on the video frame to be evaluated, an image region mask corresponding to one or more preset evaluation targets is obtained, and the video quality features and the image region mask can be fused to obtain the regional quality features corresponding to one or more preset evaluation targets, and the image region quality evaluation results of the video frame to be evaluated corresponding to one or more preset evaluation targets are determined according to the regional quality features, and the video region quality evaluation results of the video to be evaluated corresponding to one or more preset evaluation targets are determined according to the image region quality evaluation results. The image region masks corresponding to one or more preset evaluation targets can realize high-quality regionalized quality perception of different preset evaluation targets, which can effectively improve the precision of video quality evaluation and the effect of video quality evaluation, and effectively assist video processing and content distribution. At the same time, the video quality evaluation model performs feature extraction and downsampling processing of the first preset level of the video frame to be evaluated based on the feature extraction module, and fine feature extraction and upsampling processing of the second preset level, so that the low-resolution but highly compressed semantic feature information obtained by the feature extraction and downsampling processing of the first preset level is returned to the feature information of the upper layer, so that the video quality features have video quality features with higher resolution and high compression semantics, which effectively improves the quality of video quality evaluation.

[0087] Figure 6 is a schematic diagram of the structure of a video quality assessment device provided in an embodiment of the present application. Figure 6 The video quality assessment device includes a feature extraction module 61, a region extraction module 62, a region fusion module 63, an image assessment module 64 and a video assessment module 65.

[0088] Among them, the feature extraction module 61 is configured to obtain the video frame to be evaluated and extract the video quality features of the video frame to be evaluated; the region extraction module 62 is configured to perform image region extraction on the video frame to be evaluated and obtain image region masks corresponding to one or more preset evaluation targets; the region fusion module 63 is configured to fuse the video quality features and the image region masks and obtain region quality features corresponding to one or more preset evaluation targets; the image evaluation module 64 is configured to determine the image region quality evaluation results of the video frame to be evaluated corresponding to one or more preset evaluation targets based on the region quality features; the video evaluation module is configured to determine the video region quality evaluation results of the video to be evaluated corresponding to one or more preset evaluation targets based on the image region quality evaluation results.

[0089] In the above, by extracting the video quality features of the video frame to be evaluated, and performing image area extraction on the video frame to be evaluated, image area masks corresponding to one or more preset evaluation targets are obtained, and the video quality features and the image area masks can be fused to obtain regional quality features corresponding to one or more preset evaluation targets. The image area quality evaluation results of the video frame to be evaluated corresponding to one or more preset evaluation targets are determined according to the regional quality features, and the video area quality evaluation results of the video to be evaluated corresponding to one or more preset evaluation targets are determined according to the image area quality evaluation results. High-quality regionalized quality perception of different preset evaluation targets can be achieved through the image area masks corresponding to one or more preset evaluation targets, which can effectively improve the precision of video quality evaluation and the video quality evaluation effect, and effectively assist video processing and content distribution.

[0090] In a possible embodiment, the feature extraction module 61 extracts the video quality features of the video frame to be evaluated, and is configured as follows:

[0091] The video frame to be evaluated is sent to the trained video quality assessment model. The video quality assessment model performs feature extraction and downsampling processing of the first preset level on the video frame to be evaluated based on the feature extraction module, and fine feature extraction and upsampling processing of the second preset level on the video frame to be evaluated, so as to obtain the video quality features of the video frame to be evaluated.

[0092] In a possible embodiment, the feature extraction module performs feature extraction and downsampling processing of a first preset level on the video frame to be evaluated, and fine feature extraction and upsampling processing of a second preset level on the video frame to be evaluated, to obtain the video quality features of the video frame to be evaluated, and is configured as follows:

[0093] Performing feature extraction and down-sampling processing of a first preset level on the video frame to be evaluated, and fine feature extraction and up-sampling processing of a second preset level, to obtain fine features of the target video;

[0094] The frequency band quality features of the target video fine features in multiple spectrums are extracted by a feature extractor with a preset number of convolution kernels of different sizes, and the multiple frequency band quality features are mixed to obtain the video quality features of the evaluated video frame.

[0095] In a possible embodiment, the feature extraction module performs feature extraction and downsampling processing of a first preset level on the video frame to be evaluated, and fine feature extraction and upsampling processing of a second preset level to obtain fine features of the target video, and is configured as follows:

[0096] Extract a first video feature of the video frame to be evaluated by a first feature extraction unit, downsample the first video feature by a second downsampling unit and a second feature extraction unit to obtain a second video feature, downsample the second video feature by a third downsampling unit and a third feature extraction unit to obtain a third video feature, and downsample the third video feature by a fourth downsampling unit and a fourth feature extraction unit to obtain a fourth video feature;

[0097] The fourth video feature is finely extracted by the fourth feature fine extraction unit to obtain the fourth video fine feature, the fourth video fine feature is upsampled and finely extracted by the fourth upsampling unit and the third feature fine extraction unit to obtain the third video fine feature, and the third video fine feature is upsampled and finely extracted by the third upsampling unit and the second feature fine extraction unit to obtain the target video fine feature.

[0098] In a possible embodiment, the feature extraction module extracts the frequency band quality features of the target video fine features in multiple frequency spectra through a feature extractor with a preset number of convolution kernels of different sizes, and is configured as follows:

[0099] The high-frequency quality features of the target video’s fine features are extracted through a feature extractor with a 3*3 convolution kernel;

[0100] The mid-frequency quality features of the target video’s fine features are extracted through a feature extractor with a 5*5 convolution kernel;

[0101] The low-frequency quality features of the target video fine features are extracted through the feature extractor of 7*7 convolution kernel.

[0102] In a possible embodiment, the image evaluation module 64 determines the image region quality evaluation result corresponding to one or more preset evaluation targets of the video frame to be evaluated according to the region quality feature, and is configured as follows:

[0103] The regional quality features are sent to the video quality regressor of the video quality assessment model, and the video quality regressor determines the image region quality assessment result of the video frame to be evaluated corresponding to the preset assessment target according to the regional quality features.

[0104] In a possible embodiment, the region fusion module 63 performs fusion processing on the video quality feature and the image region mask, and is configured to perform dot multiplication processing on the video quality feature and one or more image region masks respectively.

[0105] In a possible embodiment, the video quality assessment device further includes a data pre-processing module, and the data pre-processing module is configured to: perform a whitening operation on the video frame to be assessed according to the video frame to be assessed and other video frames in the video to be assessed;

[0106] Correspondingly, the feature extraction module 61 extracts the video quality features of the video frame to be evaluated, and is configured to extract the video quality features of the video frame to be evaluated after the whitening operation.

[0107] It is worth noting that in the embodiment of the above-mentioned video quality assessment device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0108] The embodiment of the present application further provides a video quality assessment device, which can be integrated with the video quality assessment apparatus provided in the embodiment of the present application. Figure 7 Schematic diagram of a video quality assessment device provided by an embodiment of the present application. Figure 7 The video quality assessment device is configured as follows: an input device 73, an output device 74, a memory 72, and one or more processors 71; the memory 72 is used to store one or more programs; when one or more programs are executed by one or more processors 71, the one or more processors 71 implement the video quality assessment method provided in the above embodiment. The video quality assessment device, equipment, and computer provided above can be used to execute the video quality assessment method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0109] The embodiments of the present application also provide a non-volatile storage medium storing computer executable instructions, which are used to execute the video quality assessment method provided in the above embodiments when executed by a computer processor. Of course, the non-volatile storage medium storing computer executable instructions provided in the embodiments of the present application, whose computer executable instructions are not limited to the video quality assessment method provided above, can also execute the related operations in the video quality assessment method provided in any embodiment of the present application. The video quality assessment apparatus, device and storage medium provided in the above embodiments can execute the video quality assessment method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the video quality assessment method provided in any embodiment of the present application.

[0110] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the video quality assessment method provided in each embodiment of the present application.

Claims

1. A video quality assessment method, characterized in that: include: Obtaining a video frame to be evaluated in the video to be evaluated, and extracting video quality features of the video frame to be evaluated; Extracting an image region from the video frame to be evaluated to obtain an image region mask corresponding to one or more preset evaluation targets; The video quality feature and the image region mask are fused to obtain region quality features corresponding to one or more preset evaluation targets; Determine, according to the regional quality features, an image region quality assessment result of the to-be-assessed video frame corresponding to one or more preset assessment targets; According to the image area quality assessment result, a video area quality assessment result of the video to be assessed corresponding to one or more preset assessment targets is determined.

2. The video quality assessment method according to claim 1, characterized in that: The extracting the video quality feature of the video frame to be evaluated includes: The video frame to be evaluated is sent to a trained video quality assessment model. The video quality assessment model performs feature extraction and downsampling processing of a first preset level, and feature fine extraction and upsampling processing of a second preset level on the video frame to be evaluated based on a feature extraction module, so as to obtain video quality features of the video frame to be evaluated.

3. The video quality assessment method according to claim 2, characterized in that: The step of performing feature extraction and downsampling processing at a first preset level, and fine feature extraction and upsampling processing at a second preset level on the video frame to be evaluated to obtain the video quality features of the video frame to be evaluated includes: Performing feature extraction and downsampling processing of a first preset level, and fine feature extraction and upsampling processing of a second preset level on the video frame to be evaluated, to obtain fine features of the target video; The frequency band quality features of the target video fine features in multiple frequency spectra are extracted by a feature extractor with a preset number of convolution kernels of different sizes, and the multiple frequency band quality features are mixed to obtain the video quality features of the evaluation video frame.

4. The video quality assessment method according to claim 3, characterized in that: The step of performing feature extraction and downsampling processing at a first preset level and fine feature extraction and upsampling processing at a second preset level on the video frame to be evaluated to obtain fine features of the target video includes: Extracting a first video feature of the video frame to be evaluated by a first feature extraction unit, performing downsampling feature extraction on the first video feature by a second downsampling unit and a second feature extraction unit to obtain a second video feature, performing downsampling feature extraction on the second video feature by a third downsampling unit and a third feature extraction unit to obtain a third video feature, and performing downsampling feature extraction on the third video feature by a fourth downsampling unit and a fourth feature extraction unit to obtain a fourth video feature; The fourth video feature is finely extracted by the fourth feature fine extraction unit to obtain the fourth video fine feature, the fourth upsampling unit and the third feature fine extraction unit are used to upsample the fourth video fine feature to obtain the third video fine feature, and the third upsampling unit and the second feature fine extraction unit are used to upsample the third video fine feature to obtain the target video fine feature.

5. The video quality assessment method according to claim 3, characterized in that: The feature extractor using a preset number of convolution kernels of different sizes to extract the frequency band quality features of the target video fine features in multiple frequency spectra includes: Extracting high-frequency quality features of the target video fine features through a feature extractor with a 3*3 convolution kernel; Extracting the intermediate frequency quality features of the target video fine features through a feature extractor with a 5*5 convolution kernel; The low-frequency quality features of the target video fine features are extracted by a feature extractor with a 7*7 convolution kernel.

6. The video quality assessment method according to claim 2, characterized in that: The determining, according to the regional quality feature, an image region quality assessment result of the video frame to be assessed corresponding to one or more preset assessment targets includes: The regional quality features are sent to the video quality regressor of the video quality assessment model, and the video quality regressor determines the image region quality assessment result of the video frame to be assessed corresponding to a preset assessment target according to the regional quality features.

7. The video quality assessment method according to claim 1, characterized in that: The fusing process of the video quality feature and the image area mask comprises: The video quality feature is subjected to dot multiplication processing with one or more of the image region masks respectively.

8. The video quality assessment method according to claim 1, characterized in that: Before extracting the video quality feature of the video frame to be evaluated, the method further includes: According to the video frame to be evaluated and other video frames in the video to be evaluated, performing a whitening operation on the video frame to be evaluated; Correspondingly, extracting the video quality feature of the video frame to be evaluated includes: The video quality features of the video frame to be evaluated after the whitening operation are extracted.

9. A video quality assessment device, characterized in that: It includes feature extraction module, region extraction module, region fusion module, image evaluation module and video evaluation module, among which: The feature extraction module is configured to obtain a video frame to be evaluated and extract video quality features of the video frame to be evaluated; The region extraction module is configured to perform image region extraction on the video frame to be evaluated to obtain image region masks corresponding to one or more preset evaluation targets; The regional fusion module is configured to fuse the video quality features and the image region mask to obtain regional quality features corresponding to one or more preset evaluation targets; The image evaluation module is configured to determine the image region quality evaluation result of the video frame to be evaluated corresponding to one or more preset evaluation targets according to the region quality feature; The video evaluation module is configured to determine the video area quality evaluation results of the video to be evaluated corresponding to one or more preset evaluation targets according to the image area quality evaluation results.

10. A video quality assessment device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video quality assessment method according to any one of claims 1 to 8.

11. A non-volatile storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the video quality assessment method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video quality assessment method according to any one of claims 1 to 8 is implemented.

Images