Video quality assessment methods, devices, electronic equipment and storage media

By filtering the multi-dimensional videos of the video library to generate data sets, the problem of poor generalization performance of the existing video quality evaluation model is solved, more accurate video quality evaluation is achieved, and user experience is improved.

CN117729323BActive Publication Date: 2024-11-22SHUXING TECH (BEIJING) CO LTD
View PDF 15 Cites 0 Cited by

Patent Information

Application Number
CN202310416288.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-18
Publication Date
2024-11-22
Estimated Expiration
2043-04-18

AI Technical Summary

Technical Problem

The existing video quality evaluation model has insufficient training video volume and diversity, resulting in poor generalization performance, and it is impossible to accurately evaluate the quality of user-generated content videos, which affects the user experience.

Method used

By filtering videos in the video library, multi-dimensional screening is carried out based on video quality indicators, attributes and characteristics, and a data set covering various quality types, attribute characteristics and detailed characteristics are generated, which is used to train video quality evaluation models and increase the model's learning ability and generalization performance.

Benefits of technology

The generalization performance of the video quality evaluation model is improved, so that it can provide more accurate evaluation results in multiple video scenarios and improve users' video consumption experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117729323B_ABST
    Figure CN117729323B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a data set generation method and device, a video quality evaluation model training method and device, a video quality evaluation method and device, and an electronic device and a storage medium. The data set generation method includes: according to the video quality index, a first video set is selected from a video library, and the index value of the video in the first video set under the video quality index covers the value range of the video quality index; according to the video attribute, a second video set is selected from the first video set, and the attribute value of the video in the second video set under the video attribute covers the value range of the video attribute; according to the video feature, a third video set is selected from the second video set, and the feature value of the video in the third video set under the video feature covers the value range of the video feature; a data set is generated according to the third video set, and the data set is used to train the video quality evaluation model. By adopting the embodiment of the present application, the generalization performance of the model can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of video quality assessment, and in particular to a method and device for generating a data set, a method and device for training a video quality assessment model, a method and device for video quality assessment, and an electronic device and storage medium. Background Art

[0002] In video search or recommendation scenarios, in order to provide users with better quality videos, it is necessary to evaluate the quality of online videos. With the development of deep learning in the field of video quality evaluation, video quality evaluation models (such as no-reference scoring models) have become an important way to evaluate video quality.

[0003] However, the current video quality assessment model has great limitations due to the amount of videos used for training and the diversity of videos, resulting in poor model generalization performance and cannot be effectively applied to some video scenarios. For example, the quality assessment results in user-generated content (UGC) video scenarios are not accurate enough, affecting users' video consumption experience. Summary of the invention

[0004] The present application provides a data set generation method and device, a video quality assessment model training method and device, a video quality assessment method and device, and an electronic device and a storage medium, which can improve the model generalization performance.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a data set, the method comprising:

[0006] According to the video quality index, a first video set is selected from a video library, wherein index values ​​of the videos in the first video set under the video quality index cover a value range of the video quality index;

[0007] According to the video attribute, a second video set is selected from the first video set, wherein attribute values ​​of the videos in the second video set under the video attribute cover a value range of the video attribute;

[0008] According to the video feature, a third video set is selected from the second video set, wherein the feature values ​​of the videos in the third video set under the video feature cover the value range of the video feature;

[0009] A data set is generated according to the third video set, and the data set is used to train a video quality assessment model.

[0010] Through the above-mentioned embodiments, videos are screened from coarse to fine by comprehensively considering multiple dimensions such as video quality indicators, video attributes, and video features. The final screened data set covers videos of various quality types, various attribute characteristics, and various detail characteristics, enriching the diversity of videos. Using this data set to train a video quality evaluation model can enhance the model's learning ability for various videos, thereby improving the model's generalization performance, enabling the model to be effectively applied to a variety of video scenarios, obtaining more accurate video quality evaluation results, and helping to improve users' video consumption experience.

[0011] In some possible implementations, the video quality index includes: a video quality multi-method evaluation fusion index, a peak signal-to-noise ratio, and a bit rate;

[0012] The step of selecting a first video set from a video library according to the video quality indicator includes:

[0013] Dividing the value range of the video quality indicator into multiple value intervals;

[0014] For each value range of the video quality indicator, a corresponding number of videos satisfying the value range are selected from the video library to obtain the first video set.

[0015] Through the above implementation, the value range of the video quality index is divided into multiple value intervals. For each value interval, a corresponding number of videos that meet the value interval are selected from the video library. The video set thus screened covers videos of various quality types, which helps to ensure the diversity of the video set in the quality dimension.

[0016] In some possible implementations, the video attributes include: category, bit rate, resolution, and horizontal or vertical screen;

[0017] The step of selecting the second video set from the first video set according to the video attribute includes:

[0018] Dividing the value range of the video attribute into multiple value intervals;

[0019] For each value range of the video attribute, a corresponding number of videos satisfying the value range are selected from the first video set to obtain the second video set.

[0020] Through the above implementation, the value range of the video attribute is divided into multiple value intervals. For each value interval, a corresponding number of videos that meet the value interval are selected from the first video set. The video set thus screened covers videos with various attribute characteristics, which helps to ensure the diversity of the video set in the attribute dimension.

[0021] In some possible implementations, the video features include: scene and target features, wherein the target features include: blockiness, energy, blur, exposure, contrast, edge and color;

[0022] The step of selecting a third video set from the second video set according to the video feature includes:

[0023] Dividing the value range of each target feature into a plurality of value intervals, and determining a corresponding video subset from the second video set based on the value interval of each target feature;

[0024] Selecting the current video subset from the unselected video subsets;

[0025] Selecting a current video from videos not selected in the current video subset;

[0026] When the number of videos added to the video set to be filled does not reach a preset number, the following processing is performed until the number of videos added to the video set to be filled reaches the preset number, and the video set to be filled is determined as the third video set; the processing includes:

[0027] If the current video meets the condition, the current video is added to the to-be-filled video set, and when there are unselected videos in the current video subset, the step of selecting the current video from the unselected videos in the current video subset is returned, or when there are no unselected videos in the current video subset, the step of selecting the current video subset from the unselected video subset is returned;

[0028] If the current video does not satisfy the condition, then if there are unselected videos in the current video subset, return to the step of selecting the current video from the unselected videos in the current video subset, or if there are no unselected videos in the current video subset, return to the step of selecting the current video subset from the unselected video subset;

[0029] Among them, the conditions include: the current video is not in the video set to be filled, the ratio of the number of videos corresponding to the scenes of the current video in the video set to be filled to the preset number does not exceed a preset ratio, and the Euclidean distance between the target feature of the current video and the target feature of each video added to the video set to be filled is greater than a preset threshold.

[0030] Through the above implementation, videos that meet the conditions are screened out from the second video set. The conditions take into account the differences in scene features and target features of the videos. The video set screened out in this way covers videos with various detailed features, which helps to ensure the diversity of the video set in the feature dimension.

[0031] In some possible implementations, generating a data set according to the third video set includes:

[0032] Screening out videos to be degraded from the third video set, and performing quality degradation on the videos to be degraded to obtain a fourth video set;

[0033] A data set is generated according to the third video set and the fourth video set.

[0034] Through the above implementation, adding degraded videos to the data set used to train the video quality assessment model helps to enhance the model's learning ability for low-quality videos caused by video technology defects, thereby further improving the model's generalization performance.

[0035] In some possible implementations, the step of downgrading the to-be-downgraded video to obtain a fourth video set includes:

[0036] Extracting at least one frame of image from the video to be degraded, and performing image dimension degradation processing on each frame of image to obtain a first degraded video, wherein the image dimension degradation processing includes: image encoding, downsampling, Gaussian blurring, and Gaussian noise addition;

[0037] Performing a video dimension degradation process on the video to be degraded to obtain a second degraded video, wherein the video dimension degradation process includes: video encoding, downsampling, Gaussian blurring and Gaussian noise addition;

[0038] A fourth video set is obtained according to the first degraded video and the second degraded video.

[0039] Through the above implementation, combined with image degradation and video degradation, a rich variety of low-quality videos caused by video technology defects can be generated, which is conducive to covering more low-quality scenes and enhancing the model's generalization ability for low-quality videos.

[0040] In a second aspect, an embodiment of the present application provides a method for training a video quality assessment model, the method comprising:

[0041] Obtain a sample data set, wherein each sample video in the sample data set is annotated with a first scoring result, and the sample data set is a data set generated according to the method in the first aspect and any possible implementation manner thereof;

[0042] The sample video is scored using the video quality evaluation model to be trained to obtain a second scoring result of the sample video, the loss is calculated based on the second scoring result of the sample video and its corresponding first scoring result, and the parameters of the video evaluation model to be trained are adjusted based on the loss until the training end condition is met to obtain the video quality evaluation model.

[0043] Through the above embodiments, the data set used to train the video quality evaluation model is a data set obtained by screening the video from coarse to fine based on multiple dimensions such as comprehensive video quality indicators, video attributes, and video features. The data set covers videos of various quality types, various attribute characteristics, and various detail characteristics, enriching the diversity of videos. Using this data set to train the video quality evaluation model can enhance the model's learning ability for various videos, thereby improving the model's generalization performance, so that the model can be effectively applied to a variety of video scenarios, and obtain more accurate video quality evaluation results, which helps to improve the user's video consumption experience.

[0044] In a third aspect, an embodiment of the present application provides a video quality method, the method comprising:

[0045] Get the video to be evaluated;

[0046] The video to be evaluated is scored using the video quality evaluation model trained according to the method in the second aspect to obtain a quality evaluation result of the video to be evaluated.

[0047] Through the above-mentioned embodiment, the video quality evaluation model is used to evaluate the quality of the video to be evaluated. The data set used for training the video quality evaluation model is a data set obtained by screening the video from coarse to fine based on multiple dimensions such as comprehensive video quality indicators, video attributes, and video features. The data set covers videos of various quality types, various attribute characteristics, and various detail characteristics, enriching the diversity of videos. Using the data set to train the video quality evaluation model can enhance the model's learning ability for various videos, thereby improving the model's generalization performance, so that the model can be effectively applied to a variety of video scenarios, and obtain more accurate video quality evaluation results, which helps to improve the user's video consumption experience.

[0048] In a fourth aspect, an embodiment of the present application provides a data set generation device, the device comprising:

[0049] A first screening unit, configured to screen out a first video set from a video library according to a video quality indicator, wherein index values ​​of videos in the first video set under the video quality indicator cover a value range of the video quality indicator;

[0050] A second screening unit, configured to screen out a second video set from the first video set according to a video attribute, wherein attribute values ​​of videos in the second video set under the video attribute cover a value range of the video attribute;

[0051] A third screening unit, configured to screen a third video set from the second video set according to a video feature, wherein a feature value of the video in the third video set under the video feature covers a value range of the video feature;

[0052] A generating unit is used to generate a data set according to the third video set, where the data set is used to train a video quality assessment model.

[0053] In some possible implementations, the video quality index includes: a video quality multi-method evaluation fusion index, a peak signal-to-noise ratio, and a bit rate; the first screening unit is specifically used to: divide the value range of the video quality index into multiple value intervals; for each value interval of the video quality index, select a corresponding number of videos that meet the value interval from the video library to obtain the first video set.

[0054] In some possible implementations, the video attributes include: category, bit rate, resolution, and horizontal or vertical screen; the second screening unit is specifically used to: divide the value range of the video attribute into multiple value intervals; for each value interval of the video attribute, select a corresponding number of videos that meet the value interval from the first video set to obtain the second video set.

[0055] In some possible implementations, the video features include: scene and target features, wherein the target features include: blockiness, energy, blur, exposure, contrast, edge and color; the third screening unit is specifically used for:

[0056] Dividing the value range of each target feature into a plurality of value intervals, and determining a corresponding video subset from the second video set based on the value interval of each target feature;

[0057] Selecting the current video subset from the unselected video subsets;

[0058] Selecting a current video from videos not selected in the current video subset;

[0059] When the number of videos added to the video set to be filled does not reach a preset number, the following processing is performed until the number of videos added to the video set to be filled reaches the preset number, and the video set to be filled is determined as the third video set; the processing includes:

[0060] If the current video meets the condition, the current video is added to the to-be-filled video set, and when there are unselected videos in the current video subset, the step of selecting the current video from the unselected videos in the current video subset is returned, or when there are no unselected videos in the current video subset, the step of selecting the current video subset from the unselected video subset is returned;

[0061] If the current video does not satisfy the condition, then if there are unselected videos in the current video subset, return to the step of selecting the current video from the unselected videos in the current video subset, or if there are no unselected videos in the current video subset, return to the step of selecting the current video subset from the unselected video subset;

[0062] Among them, the conditions include: the current video is not in the video set to be filled, the ratio of the number of videos corresponding to the scenes of the current video in the video set to be filled to the preset number does not exceed a preset ratio, and the Euclidean distance between the target feature of the current video and the target feature of each video added to the video set to be filled is greater than a preset threshold.

[0063] In some possible implementations, the generating unit is specifically configured to: filter out videos to be degraded from the third video set, and degrade the videos to be degraded to obtain a fourth video set; and generate a data set based on the third video set and the fourth video set.

[0064] In some possible implementations, when the generation unit downgrades the video to be downgraded to obtain a fourth video set, it is specifically used to: extract at least one frame of image from the video to be downgraded, downgrade each frame of image in image dimension to obtain a first downgraded video, wherein the image dimension downgrading includes: image encoding, downsampling, Gaussian blur and Gaussian noise; downgrade the video to be downgraded in video dimension to obtain a second downgraded video, wherein the video dimension downgrading includes: video encoding, downsampling, Gaussian blur and Gaussian noise; obtain a fourth video set based on the first downgraded video and the second downgraded video.

[0065] In a fifth aspect, an embodiment of the present application provides a training device for a video quality assessment model, the device comprising:

[0066] An acquisition unit, configured to acquire a sample data set, wherein each sample video in the sample data set is annotated with a first scoring result, and the sample data set is a data set generated according to the method in the first aspect and any possible implementation manner thereof;

[0067] A training unit is used to score the sample video using the video quality evaluation model to be trained to obtain a second scoring result of the sample video, calculate the loss according to the second scoring result of the sample video and its corresponding first scoring result, and adjust the parameters of the video quality evaluation model to be trained according to the loss until the training end condition is met to obtain the video quality evaluation model.

[0068] In a sixth aspect, an embodiment of the present application provides a video quality evaluation device, the device comprising:

[0069] An acquisition unit, used for acquiring a video to be evaluated;

[0070] An evaluation unit is used to score the video to be evaluated by using the video quality evaluation model trained according to the method in the second aspect above, so as to obtain a quality evaluation result of the video to be evaluated.

[0071] In the seventh aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method in the above-mentioned first aspect and any possible implementation manner, or the method in the above-mentioned second aspect, or the method in the above-mentioned third aspect.

[0072] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method in the above-mentioned first aspect and any possible implementation manner thereof, or the method in the above-mentioned second aspect, or the method in the above-mentioned third aspect is implemented.

[0073] In the ninth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the method in the above-mentioned first aspect and any possible implementation manner, or the method in the above-mentioned second aspect, or the method in the above-mentioned third aspect.

[0074] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0076] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0077] Figure 1 It is a flowchart of a method for generating a data set provided in an embodiment of the present application;

[0078] Figure 2 is a schematic diagram of video scene feature distribution in a third data set provided in an embodiment of the present application;

[0079] Figure 3 is a distribution diagram of video target features in a third data set provided in an embodiment of the present application;

[0080] Figure 4 It is a flowchart of a method for training a video quality assessment model provided in an embodiment of the present application;

[0081] Figure 5 It is a flowchart of a video quality evaluation method provided in an embodiment of the present application;

[0082] Figure 6 is a schematic diagram of a video quality evaluation system provided in an embodiment of the present application;

[0083] Figure 7 is a structural schematic diagram of a data set generating device provided in an embodiment of the present application;

[0084] Figure 8 It is a structural schematic diagram of a training device for a video quality assessment model provided in an embodiment of the present application;

[0085] Fig. 9 is a structural schematic diagram of a video quality evaluation device provided in an embodiment of the present application;

[0086] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0088] The terms "first", "second", etc. in the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0089] It should be understood that in the present application, "at least one (item)" means one or more, "more than one" means two or more, and "at least two (items)" means two or three and more than three. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " can indicate that the previous and next associated objects are in an "or" relationship, which refers to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one item (item) of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0090] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0091] In video search or recommendation scenarios, in order to provide users with better quality videos, it is necessary to evaluate the quality of online videos. With the development of deep learning in the field of video quality evaluation, video quality evaluation models (such as no-reference scoring models) have become an important way to evaluate video quality.

[0092] However, the current video quality assessment model has great limitations due to the amount of videos used for training and the diversity of videos, resulting in poor model generalization performance and cannot be effectively applied to some video scenarios. For example, the quality assessment results in user-generated content (UGC) video scenarios are not accurate enough, affecting users' video consumption experience.

[0093] Based on this, the embodiments of the present application provide a data set generation method, a video quality assessment model training method, and a video quality assessment method, so as to improve the generalization performance of the model and help improve the accuracy of the video quality assessment results.

[0094] The execution subject of the data set generation method may be a data set generation device, for example, the data set generation method may be executed by a terminal device or a server or other processing device. The execution subject of the video quality assessment model training method may be a video quality assessment model training device, for example, the video quality assessment model training method may be executed by a terminal device or a server or other processing device. The execution subject of the video quality assessment method may be a video quality assessment device, for example, the video quality assessment method may be executed by a terminal device or a server or other processing device. Among them, the terminal device may be, but is not limited to, various personal computers, laptops, tablet computers, smart phones and portable wearable devices, and the server may be implemented with an independent server or a server cluster composed of multiple servers. In some possible implementations, the data set generation method, the video quality assessment model training method, and the video quality assessment method may all be implemented by a processor calling computer-readable instructions stored in a memory.

[0095] See also Figure 1 , Figure 1 It is a flowchart of a data set generation method provided in an embodiment of the present application. The data set generation method includes the following steps S101 to S104.

[0096] S101, selecting a first video set from a video library according to a video quality index, wherein index values ​​of the videos in the first video set under the video quality index cover a value range of the video quality index.

[0097] The video library refers to a video collection containing a large number of videos. In one example, the video library can be a video collection consisting of all videos on a certain platform. Optionally, the platform is a platform that can support UGC videos, and the video library contains all UGC videos on the platform.

[0098] The video quality index refers to an index used to measure video quality. In one example, the video quality index may include, but is not limited to, one or more of a Visual Multimethod Assessment Fusion (VMAF) index, a Peak Signal to Noise Ratio (PSNR), and a bitrate.

[0099] The quality of the video can be represented by the index value under the video quality index. Under a video quality index, different index values ​​represent different video qualities. For example, for the video quality index of peak signal-to-noise ratio, the larger the value of the peak signal-to-noise ratio, the higher the video quality, and the smaller the value of the peak signal-to-noise ratio, the lower the video quality.

[0100] Videos are selected from the video library according to the video quality index to form the first video set, so that the index values ​​of the videos in the first video set under the video quality index cover the value range of the video quality index. The index values ​​of the videos in the first video set under the video quality index cover the value range of the video quality index, which can be understood as the first video set containing videos of various qualities. For example, the video quality can be divided into three types: low quality, medium quality, and high quality according to different value intervals under the video quality index, and the first video set contains videos of these three quality types.

[0101] In one example, the number of videos in the first video set is in the order of 100,000, for example, 400,000. Optionally, the number of videos of each quality type in the first video set is not less than the corresponding preset number to ensure that there are sufficient samples of videos of each quality type. The preset number corresponding to each quality type can be the same or different, and the preset number can be set in combination with actual conditions, which is not limited here.

[0102] S102: Filter out a second video set from the first video set according to video attributes, wherein attribute values ​​of the videos in the second video set under the video attributes cover a value range of the video attributes.

[0103] Video attributes may include attributes of the video itself and attributes of the video content. In one example, video attributes may include, but are not limited to, one or more of category, bit rate, resolution, and horizontal or vertical screen. Under a video attribute, different attribute values ​​represent different attribute characteristics. For example, for the video attribute of category, multiple attribute characteristics such as technology, sports, and life may be included; for another example, for the video attribute of horizontal or vertical screen, two attribute characteristics of horizontal screen and vertical screen may be included.

[0104] According to the video attributes, videos are selected from the first video set to form the second video set, so that the attribute values ​​of the videos in the second video set under the video attributes cover the value range of the video attributes. The attribute values ​​of the videos in the second video set under the video attributes cover the value range of the video attributes, which can be understood as the second video set containing videos with various attribute characteristics. The number of videos in the second video set is less than the number of videos in the first video set, so that the amount of subsequent calculations can be reduced while ensuring the video quality and the diversity of attribute characteristics.

[0105] In one example, the number of videos in the second video set is in the hundreds of thousands, for example, 100,000. Optionally, the number of videos of each attribute characteristic in the second video set is not less than the corresponding preset number to ensure that there are sufficient samples of videos of each attribute characteristic. The preset number corresponding to each attribute characteristic can be the same or different, and the preset number can be set in combination with actual conditions, which is not limited here.

[0106] S103: Filter out a third video set from the second video set according to the video feature, wherein feature values ​​of the videos in the third video set under the video feature cover a value range of the video feature.

[0107] Video features are used to describe video details, such as scenes, colors, textures, artifacts, vertical categories, and other video details. In one example, video features may include, but are not limited to, one or more of scenes, block effects, energy, blur, exposure, contrast, edges, and colors. Under a video feature, different feature values ​​represent different detail characteristics. For example, for the video feature of energy, the larger the energy value, the richer the texture of the video, and the smaller the energy value, the less rich the texture of the video.

[0108] Videos are selected from the second video set according to the video features to form the third video set, so that the feature values ​​of the videos in the third video set under the video features cover the value range of the video features. The index values ​​of the videos in the third video set under the video features cover the value range of the video features, which can be understood as the third video set containing videos with various detail features. The number of videos in the third video set is less than the number of videos in the second video set, so that the amount of subsequent calculations can be reduced while ensuring the video quality, video attribute characteristics, and the diversity of video detail characteristics.

[0109] In one example, the number of videos in the third video set is in the thousands, for example, 5000. Optionally, the number of videos of each detail feature in the third video set is not less than the corresponding preset number to ensure that there are enough samples of videos of each attribute feature. The preset number corresponding to each detail feature can be the same or different, and the preset number can be set according to the actual situation, which is not limited here.

[0110] S104, generating a data set according to the third video set, where the data set is used to train a video quality assessment model.

[0111] In one example, the third video set can be used as a data set for training a video quality assessment model. In other examples, some videos in the third data set can also be post-processed, such as degrading, and the processed videos and the third video set together form a data set for training a video quality assessment model.

[0112] The video quality evaluation model refers to a model used to evaluate the quality of a video. Optionally, the video quality evaluation model is a no-reference scoring model, which is used to score the quality of the video. After obtaining the data set, volunteers with certain subjective image quality evaluation experience can score each video in the data set, and annotate each video in the data set according to the scoring results to obtain an annotated data set that is ultimately used to train the no-reference evaluation model. After the no-reference scoring model is trained, the quality of the video is scored by the no-reference scoring model, and the quality score of the video is output, thereby obtaining the quality evaluation result of the video.

[0113] Through the above-mentioned embodiments, videos are screened from coarse to fine by comprehensively considering multiple dimensions such as video quality indicators, video attributes, and video features. The final screened data set covers videos of various quality types, various attribute characteristics, and various detail characteristics, enriching the diversity of videos. Using this data set to train a video quality evaluation model can enhance the model's learning ability for various videos, thereby improving the model's generalization performance, enabling the model to be effectively applied to a variety of video scenarios, obtaining more accurate video quality evaluation results, and helping to improve users' video consumption experience.

[0114] In a possible implementation, the video quality index includes: a video quality multi-method evaluation fusion index, a peak signal-to-noise ratio, and a bit rate; based on the video quality index, a first video set is screened out from a video library, which may specifically include: dividing the value range of the video quality index into multiple value intervals; for each value interval of the video quality index, a corresponding number of videos that meet the value interval are selected from the video library to obtain a first video set.

[0115] For each video quality index, a value range of the video quality index may be determined first, and then the value range may be divided into a plurality of value intervals using at least one segmentation node. For each value interval of the video quality index, a corresponding number of videos satisfying the value interval are selected from the video library, wherein a video satisfying the value interval means that the index value of the video under the video quality index is within the value interval.

[0116] Taking the peak signal-to-noise ratio as an example of a video quality indicator, assuming that the peak signal-to-noise ratio ranges from 0 to 100, it is divided into three value intervals of 0-30, 30-70 and 70-100, representing low quality, medium quality and high quality respectively. The corresponding numbers of videos to be selected are N1, N2 and N3 respectively. Then, N1 videos with a peak signal-to-noise ratio in the range of 0-30, N2 videos with a peak signal-to-noise ratio in the range of 30-70, and N3 videos with a peak signal-to-noise ratio in the range of 70-100 can be selected from the video library to form a first video set.

[0117] In one example, the corresponding videos can be first screened out from the video library according to a certain video quality index, and these videos are collected to obtain data set A. It should be understood that the number of videos in each value interval that meets the video quality index in data set A reaches the corresponding number. Then, it is verified whether the number of videos in each value interval that meets other video quality indexes in data set A reaches the corresponding number. If so, data set A is used as the first video set; if not, the corresponding videos are screened out from the video library according to other video quality indexes, and added to data set A, until the number of videos in each value interval that meets each video quality index in data set A reaches the corresponding number, and data set A is used as the first video set.

[0118] In other examples, corresponding videos may be screened out from the video library according to each video quality index, and all screened videos may be collected and deduplicated to obtain a first video set.

[0119] Through the above implementation, the value range of the video quality index is divided into multiple value intervals. For each value interval, a corresponding number of videos that meet the value interval are selected from the video library. The video set thus screened covers videos of various quality types, which helps to ensure the diversity of the video set in the quality dimension.

[0120] In a possible implementation, the video attributes include: category, bit rate, resolution, and horizontal or vertical screen; based on the video attributes, the second video set is screened out from the first video set, which may specifically include: dividing the value range of the video attribute into multiple value intervals; for each value interval of the video attribute, a corresponding number of videos that meet the value interval are selected from the first video set to obtain the second video set.

[0121] For each video attribute, the value range of the video attribute can be determined first. If the value range is continuous, the value range can be divided into multiple value intervals using at least one segmentation node. If the value range is discrete, each discrete value can be used as a value interval. For each value interval of the video attribute, a corresponding number of videos that meet the value interval are selected from the video library, wherein the video that meets the value interval means that the attribute value of the video under the video attribute is within the value interval.

[0122] Taking the video attribute of horizontal and vertical screen as an example, the value range of horizontal and vertical screen is discrete, including two discrete values ​​of horizontal screen and vertical screen, that is, two value intervals, and the corresponding number of videos to be selected are N4 and N5 respectively. Then, N4 horizontal screen videos and N5 vertical screen videos are selected from the first video set to form the second video set.

[0123] In one example, the corresponding videos can be first screened out from the first data set according to a certain video attribute, and these videos are collected to obtain data set B. It should be understood that the number of videos in each value interval that satisfies the video attribute in data set B reaches the corresponding number. Then, it is verified whether the number of videos in each value interval that satisfies other video attributes in data set B reaches the corresponding number. If so, data set B is used as the second video set; if not, the corresponding videos are screened out from the first data set according to other video attributes and added to data set B, until the number of videos in each value interval that satisfies each video quality index in data set B reaches the corresponding number, and data set B is used as the second video set.

[0124] In other examples, corresponding videos may be screened out from the first data set according to the attributes of each video, and all the screened videos may be collected and deduplicated to obtain a second video set.

[0125] Through the above implementation, the value range of the video attribute is divided into multiple value intervals. For each value interval, a corresponding number of videos that meet the value interval are selected from the first video set. The video set thus screened covers videos with various attribute characteristics, which helps to ensure the diversity of the video set in the attribute dimension.

[0126] In a possible implementation, video features include: scene and target features, wherein the target features include: block effect, energy, blur, exposure, contrast, edge and color; based on the video features, a third video set is selected from the second video set, including: dividing the value range of each target feature into multiple value intervals, and based on the value interval of each target feature, determining a corresponding video subset from the second video set; selecting a current video subset from unselected video subsets; selecting a current video from unselected videos in the current video subset; when the number of videos added to the video set to be filled does not reach a preset number, executing the following processing process until the number of videos added to the video set to be filled reaches a preset number, and determining the video set to be filled as the third video set; the processing process includes: if the current video meets the conditions, adding the current video to the video set to be filled, and if there is an unselected video in the current video subset, If a video is selected, the step of selecting the current video from the unselected videos in the current video subset is returned, or if there is no unselected video in the current video subset, the step of selecting the current video from the unselected video subset is returned; if the current video does not meet the conditions, if there is an unselected video in the current video subset, the step of selecting the current video from the unselected videos in the current video subset is returned, or if there is no unselected video in the current video subset, the step of selecting the current video from the unselected video subset is returned; wherein the conditions include: the current video is not in the video set to be filled, the ratio of the number of videos corresponding to the scene of the current video in the video set to be filled to the preset number does not exceed the preset ratio, and the Euclidean distance between the target feature of the current video and the target feature of each video added to the video set to be filled is greater than a preset threshold.

[0127] In one example, the scene of the video can be identified by any existing or future scene recognition model to obtain the scene to which the video belongs, such as people, animals, food, travel, fitness, etc. For example, the open source Places CNN classification model can be used to identify the scene of each video in the second video set. It can be understood that the value range of the scene is discrete, and each discrete value represents a scene category.

[0128] The target features include seven features: blocking effect, energy, blur, exposure, contrast, edge and color. Among them, blocking effect, blur and exposure are used to describe video defects, such as the deterioration of video quality due to transcoding, transmission, etc.; energy and edge are used to describe video texture details; contrast and color are used to describe video color details.

[0129] For each video in the second video set, calculate the feature value of the video under each target feature. Optionally, when calculating the feature value of the video under a certain target feature, first extract frames of the video, then calculate the feature value of each extracted frame under the target feature, and then average these feature values ​​to obtain the feature value of the video under the target feature.

[0130] In one example, the eigenvalue of an image under the energy feature can be calculated by the following formula:

[0131]

[0132] Among them, Energy represents energy characteristics; P i,j Represents the value at (i, j) in the gray-level co-occurrence matrix (GLCM) of the image, i represents the horizontal pixel value in the gray-level co-occurrence matrix, j represents the vertical pixel value in the gray-level co-occurrence matrix; M represents the maximum pixel value in the image. By calculating the sum of the squares of all values ​​in the gray-level co-occurrence matrix of the image, the eigenvalue of the image under the energy feature is obtained.

[0133] In one example, the feature value of an image under edge features can be calculated by the following formula:

[0134] Edge=rms[Laplacian(x,y)]

[0135] Laplacian(x,y)=f(x+1,y)+f(x-1,y)+f(x,y+1)+f(x,y-1)-4f(x,y)

[0136] Where Edge represents edge features; f(x,y) represents the pixel value of the pixel at (x,y) in the image; Laplacian(x,y) represents the Laplacian operator detection value of the pixel (x,y); rms represents the root mean square function. By calculating the root mean square of the Laplacian operator detection values ​​of all pixels in the image, the feature value of the image under edge features is obtained.

[0137] In one example, the feature value of an image under the color feature can be calculated by the following formula:

[0138]

[0139]

[0140]

[0141]

[0142] Colorful=σ rgyb +α*μ rgyb

[0143] Wherein, Colorful represents the color feature; R, G and B represent the color values ​​of the red, green and blue channels respectively; rg represents the color difference value between red and green, recorded as the first color difference value; yb represents the color difference value between red, green and blue, recorded as the second color difference value; μ rg and σ rg Respectively represent the mean and standard deviation of the first color difference values ​​of all pixels in the image; μ yb and σ yb They respectively represent the mean value and standard deviation of the second color difference values ​​of all pixels in the image; α represents a hyperparameter, optionally, α=0.36.

[0144] After obtaining the characteristic value of each video in the second data set under each target feature, for each target feature, the value range of the target feature is divided into multiple value intervals, and based on each value interval of the target feature, a video subset corresponding to the value interval is screened out from the second video set, wherein the video subset corresponding to the value interval means that the characteristic values ​​of the videos in the video subset under the target feature are within the value interval.

[0145] Optionally, before dividing the value interval of each target feature, the eigenvalues ​​under each target feature are normalized first, so that the eigenvalues ​​under each target feature are unified to the range of [0,1]. Correspondingly, the value range of each target feature is also normalized to [0,1], and then the normalized value range of each target feature is divided, for example, evenly divided into 20 value intervals.

[0146] In one example, the feature value under each target feature can be normalized by the following formula:

[0147] F ′ =(F-min(F)) / (percentile(F,99)-min(F))

[0148] Where F and F' respectively represent the feature value and normalized feature value of a certain video in the second data set under a certain target feature; min(F) represents the minimum value of the feature values ​​of all videos in the second data set under the target feature; percentile(F,99) represents the 99th percentile value of the feature values ​​of all videos in the second data set under the target feature. If the normalized feature value obtained by the above formula is greater than 1, it is assigned a value of 1.

[0149] After selecting the video subsets corresponding to each value interval under each target feature from the second video set, N video subsets can be determined, N = Z*K, Z represents the number of target features, and K represents the number of value intervals. In an example, the number of target features is 7, and each target feature is divided into 20 value intervals, so 140 video subsets (recorded as bins) can be determined. Videos that meet the corresponding conditions are selected from all determined video subsets according to the scene and target features to form a third video set. The screening process may include the following steps one to three.

[0150] Step 1: Select the current video subset from the unselected video subsets.

[0151] After all video subsets are determined, video subsets are selected one by one from all video subsets for subsequent processing. It can be understood that the video subsets not selected initially refer to all video subsets, and when a certain video subset is selected, the video subsets not selected refer to the remaining video subsets after removing the selected video subset from all video subsets.

[0152] Optionally, all video subsets are randomly arranged, and video subsets are selected in sequence according to the arranged order as the current video subset. Initially, the first video subset after the arrangement is selected as the current video subset.

[0153] Step 2: Select the current video from the videos that have not been selected in the current video subset.

[0154] After the current video subset is determined, videos are selected one by one from the current video subset for subsequent processing. It can be understood that the videos not selected initially refer to all videos in the current video subset, and when a video is selected, the videos not selected are the remaining videos in the current video subset after removing the selected video.

[0155] Optionally, all videos in the current video subset are randomly arranged, and videos are selected in sequence according to the arranged order as the current video, and initially the first video after the arrangement is selected as the current video.

[0156] Step three: when the number of videos added to the video set to be filled does not reach the preset number, perform the following processing until the number of videos added to the video set to be filled reaches the preset number, and determine the video set to be filled as the third video set.

[0157] The preset number refers to the number of videos that the third video set needs to include, that is, the number of videos that need to be screened out from the second video set. The video set to be filled refers to a collection for storing videos screened out from the second video set. It can be understood that the number of videos in the video set to be filled is zero initially. When a video is added to the video set to be filled, the number of videos in the video set to be filled increases. When the number of videos added to the video set to be filled reaches the preset number, the video set to be filled at this time is used as the third video set.

[0158] The processing process includes: if the current video meets the conditions, the current video is added to the video set to be filled. The conditions include the following conditions 1, 2 and 3. If the current video meets the three conditions at the same time, it is determined that the current video meets the conditions.

[0159] Condition 1: The current video is not in the set of videos to be filled.

[0160] Condition 2: The ratio of the number of videos corresponding to the scene of the current video in the video set to be filled to the preset number does not exceed the preset ratio.

[0161] Specifically, condition 2 can be expressed as follows:

[0162]

[0163] Wherein, sum(current scene) represents the number of videos corresponding to the scene of the current video in the video set to be filled; Q represents a preset number, optionally, Q=5000; φ represents a preset ratio, optionally, φ=3.5%.

[0164] The scene of the current video is recorded as current scene, and the number of videos with the scene of current scene added to the video set to be filled is counted, and sum(current scene) can be determined according to the number of videos with the scene of current scene added to the video set to be filled. Optionally, sum(current scene) is equal to the number of videos with the scene of current scene added to the video set to be filled plus one.

[0165] The preset ratio can be understood as the maximum ratio of videos of the same scene in the third video set. If the proportion of videos of a certain scene exceeds the maximum ratio, there is no need to add the videos of the scene to the video set to be filled. Based on this, the number of videos of the same scene in the third video set finally obtained can be controlled to ensure the richness of the video scenes and the uniformity of the number of videos of each scene.

[0166] Condition three: The Euclidean distance between the target feature of the current video and the target feature of each video added to the video set to be filled is greater than a preset threshold.

[0167] Specifically, condition three can be expressed as follows:

[0168]

[0169] Where Distance represents the Euclidean distance between the target feature of the current video and the target feature of each video that has been added to the video set to be filled; F i represents the i-th target feature; N represents the number of target features; C(F i ) represents the feature value of the current video under the i-th target feature; L(F i ) represents the feature value of each video added to the video set to be filled under the i-th target feature; T represents a preset threshold, optionally, T = 0.3.

[0170] Euclidean distance can characterize feature similarity. If the Euclidean distance is smaller, the feature similarity is higher, and thus the features can be considered similar; if the Euclidean distance is larger, the feature similarity is lower, and thus the features can be considered dissimilar. The preset threshold can be understood as the maximum Euclidean distance that characterizes feature similarity, or as the minimum Euclidean distance that characterizes feature dissimilarity. If the Euclidean distance between the target feature of the current video and the target feature of each video added to the video set to be filled is greater than the preset threshold, it can be considered that the video set to be filled does not contain videos with features similar to the current video; if the Euclidean distance between the target feature of the current video and the target feature of any one or more videos added to the video set to be filled is not greater than the preset threshold, it can be considered that the video set to be filled already contains videos with features similar to the current video, and at this time, there is no need to add the current video to the video set to be filled. Based on this, the number of videos with similar features in the third video set finally obtained can be controlled to ensure the richness of video features.

[0171] After adding the current video to the set of videos to be filled, determine whether there are unselected videos in the current video subset. If there are unselected videos in the current video subset, return to the above step 2, that is, select a new current video from the unselected videos in the current video subset, and then repeat the above step 3 until the number of videos added to the set of videos to be filled reaches a preset number. If there are no unselected videos in the current video subset, return to the above step 1, that is, select a new current video subset from the unselected video subset, and then repeat the above steps 2 to 3 until the number of videos added to the set of videos to be filled reaches a preset number.

[0172] The processing process also includes: if the current video does not meet the conditions, the current video is not added to the video set to be filled, and then it is determined whether there are unselected videos in the current video subset. If there are unselected videos in the current video subset, return to the above step 2, that is, select a new current video from the unselected videos in the current video subset, and then repeat the above step 3 until the number of videos added to the video set to be filled reaches a preset number. If there are no unselected videos in the current video subset, return to the above step 1, that is, select a new current video subset from the unselected video subset, and then repeat the above steps 2 to 3 until the number of videos added to the video set to be filled reaches a preset number.

[0173] Through the above implementation, videos that meet the conditions are screened out from the second video set. The conditions take into account the differences in scene features and target features of the videos. The video set screened out in this way covers videos with various detailed features, which helps to ensure the diversity of the video set in the feature dimension.

[0174] See also Figure 2 , Figure 2 : This is a schematic diagram of the distribution of video scene features in a third data set provided by an embodiment of the present application. The horizontal axis represents the feature value under the scene feature (Scene), and the vertical axis represents the number of videos corresponding to each scene in the third data set. It can be seen from the figure that the third data set has a rich number of video scenes, and the number of videos corresponding to each scene accounts for no more than 3.5%.

[0175] See also Figure 3 , Figure 3 : This is a distribution diagram of the video target features in a third data set provided by an embodiment of the present application. Among them, Blockness, Energy, Blur, Exposure, Contrast, Edge and Colorful are all target features, representing block effect, energy, blur, exposure, contrast, edge and color features respectively. The horizontal axis represents the feature value under the target feature, and the vertical axis represents the number of videos corresponding to each feature in the third data set. It can be seen from the figure that the third data set has rich video features.

[0176] In a possible implementation, generating a data set based on the third video set may specifically include: screening out videos to be degraded from the third video set, and downgrading the videos to be degraded to obtain a fourth video set; and generating a data set based on the third video set and the fourth video set.

[0177] After obtaining the third video set, some videos can be selected from the third video set for degradation processing. The videos to be degraded are the selected videos to be degraded. In one example, the videos to be degraded include high-quality videos in various scenarios. The videos to be degraded are degraded to obtain degraded videos. The degraded videos can be understood as low-quality videos caused by defects in video technology (such as image synthesis, image coding, video synthesis, video coding, etc.). The degraded videos are collected to obtain a fourth video set, and then the third video set and the fourth video set are combined to obtain a data set.

[0178] Through the above implementation, adding degraded videos to the data set used to train the video quality assessment model helps to enhance the model's learning ability for low-quality videos caused by video technology defects, thereby further improving the model's generalization performance.

[0179] In a possible implementation, downgrading the video to be degraded to obtain a fourth video set may specifically include: extracting at least one frame of image from the video to be degraded, downgrading each frame of image in terms of image dimension to obtain a first degraded video, wherein the image dimension downgrading includes: image encoding, downsampling, Gaussian blur and Gaussian noise; downgrading the video to be degraded in terms of video dimension to obtain a second degraded video, wherein the video dimension downgrading includes: video encoding, downsampling, Gaussian blur and Gaussian noise; and obtaining a fourth video set based on the first degraded video and the second degraded video.

[0180] There may be multiple videos to be degraded, and each video to be degraded may be understood to be composed of multiple frames of images. Degradation processing includes two methods: image degradation and video degradation. Regarding image degradation, at least one frame of image may be extracted from each video to be degraded to obtain multiple frames of image, and image encoding, downsampling, Gaussian blurring and Gaussian noise are performed on each frame of image to obtain a degraded video, and the first degraded video includes all degraded videos obtained by image degradation.

[0181] Regarding video degradation, each video to be degraded may be subjected to video encoding, downsampling, Gaussian blurring and Gaussian noise addition to obtain a degraded video, and the second degraded video includes all degraded videos obtained by video degradation.

[0182] Through the above implementation, combined with image degradation and video degradation, a rich variety of low-quality videos caused by video technology defects can be generated, which is conducive to covering more low-quality scenes and enhancing the model's generalization ability for low-quality videos.

[0183] See also Figure 4 , Figure 4It is a flowchart of a training method for a video quality assessment model provided in an embodiment of the present application. The training method for the video quality assessment model includes the following steps S401 to S402.

[0184] S401, obtaining a sample data set, each sample video in the sample data set is annotated with a first scoring result.

[0185] The sample data set refers to a data set used to train a video quality assessment model, and can be specifically generated by the data set generation method provided in the previous embodiment, which will not be described in detail here.

[0186] The sample video refers to a video in the sample data set, and the first scoring result is used to represent the real subjective quality of the sample video. In an example, the range of the first scoring result is [1,5], where [1,2) represents extremely poor quality, [2,3) represents poor quality, [3,4) represents good quality, and [4,5] represents excellent quality.

[0187] Optionally, 15 volunteers with certain subjective image quality evaluation experience are selected to score each sample video in the sample data set through the data annotation platform. Each sample video can obtain 15 scores, and the average of the 15 scores is used as the first scoring result, thereby completing the data annotation. Optionally, after completing the data annotation, data cleaning can also be performed, such as removing the labeled data with low credibility.

[0188] S402, using the video quality evaluation model to be trained to score the sample video, obtain a second scoring result of the sample video, calculate the loss according to the second scoring result of the sample video and its corresponding first scoring result, adjust the parameters of the video evaluation model to be trained according to the loss, until the training end conditions are met, and obtain the video quality evaluation model.

[0189] The sample video may be frame sampled to obtain a sample frame block for use as a model input. In one example, the sample video is sampled in time domain and space domain, and the sample frame blocks obtained thereby are more evenly distributed in time domain and space domain, thereby better representing the sample video quality.

[0190] The sampled frame block is input into the video quality evaluation model to be trained, and the video quality evaluation model to be trained is used to score, and a second scoring result of the sample video is output, and the second scoring result is used to represent the predicted subjective quality of the sample video. The loss is calculated according to the second scoring result of the sample video and its corresponding first scoring result, and the parameters of the video evaluation model to be trained are adjusted according to the loss so that the second scoring result output by the video evaluation model to be trained is close to the corresponding first scoring result.

[0191] When the training end condition is met, the training is ended to obtain a trained video quality assessment model. In one example, the training framework of the video quality assessment model adopts a Swin Transformer structure, and the training end condition can be that the loss is less than a preset loss, or that the number of training times reaches a preset number, which is not limited in the present embodiment of the application.

[0192] Through the above embodiments, the data set used to train the video quality evaluation model is a data set obtained by screening the video from coarse to fine based on multiple dimensions such as comprehensive video quality indicators, video attributes, and video features. The data set covers videos of various quality types, various attribute characteristics, and various detail characteristics, enriching the diversity of videos. Using this data set to train the video quality evaluation model can enhance the model's learning ability for various videos, thereby improving the model's generalization performance, so that the model can be effectively applied to a variety of video scenarios, and obtain more accurate video quality evaluation results, which helps to improve the user's video consumption experience.

[0193] See also Figure 5 , Figure 5 It is a flow chart of a video quality evaluation method provided in an embodiment of the present application. The video quality evaluation method includes the following steps S501 to S502.

[0194] S501, obtaining a video to be evaluated.

[0195] The videos to be evaluated refer to videos that need to be evaluated for quality. In one example, the videos to be evaluated may be multiple videos that meet the user's search request in a search scenario. Through the embodiments of the present application, these videos may be evaluated for quality, and then the videos with higher quality may be recommended to the user first.

[0196] S502, using a video quality evaluation model to score the video to be evaluated, and obtaining a quality evaluation result of the video to be evaluated.

[0197] The video quality evaluation model can be obtained by training the video quality evaluation model training method provided in the previous embodiment, which will not be repeated here. After the video quality evaluation model is trained, it is deployed for inference, for example, deployed to an open source application container engine (Docker), and the video quality evaluation can be implemented by calling the Docker service later.

[0198] The video to be evaluated can be frame sampled to obtain sampled frame blocks for use as model input. In one example, the video to be evaluated is sampled in time domain and space domain, and the sampled frame blocks obtained thereby are more evenly distributed in time domain and space domain, thereby better representing the quality of the video to be evaluated.

[0199] The sampled frame block is input into the video quality evaluation model, and the video quality evaluation model is used to score the video, and the quality score of the video to be evaluated is output, thereby obtaining the quality evaluation result of the video to be evaluated. For example, if the quality score of the video to be evaluated is in the range of [1,2), it means that the quality of the video to be evaluated is extremely poor; if the quality score of the video to be evaluated is in the range of [2,3), it means that the quality of the video to be evaluated is poor; if the quality score of the video to be evaluated is in the range of [3,4), it means that the quality of the video to be evaluated is good; if the quality score of the video to be evaluated is in the range of [4,5], it means that the quality of the video to be evaluated is excellent.

[0200] Through the above-mentioned embodiment, the video quality evaluation model is used to evaluate the quality of the video to be evaluated. The data set used for training the video quality evaluation model is a data set obtained by screening the video from coarse to fine based on multiple dimensions such as comprehensive video quality indicators, video attributes, and video features. The data set covers videos of various quality types, various attribute characteristics, and various detail characteristics, enriching the diversity of videos. Using the data set to train the video quality evaluation model can enhance the model's learning ability for various videos, thereby improving the model's generalization performance, so that the model can be effectively applied to a variety of video scenarios, and obtain more accurate video quality evaluation results, which helps to improve the user's video consumption experience.

[0201] See also Figure 6 , Figure 6 : is a schematic diagram of a video quality evaluation system provided in an embodiment of the present application, and the video quality evaluation system includes four modules: data collection, data annotation, model training, and reasoning deployment, forming a comprehensive solution (pipeline). Among them, the data collection module is used to screen out videos that meet the requirements to generate a data set; the data annotation module is used to annotate the data set to obtain an annotated data set; the model training module is used to perform model training using the annotated data set to obtain a video quality evaluation model; the reasoning deployment module is used to deploy the video quality evaluation model to achieve quality evaluation of the video to be evaluated. For the specific description of the above modules, please refer to the corresponding description in the previous embodiment, which will not be repeated here.

[0202] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0203] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0204] See also Figure 7, Figure 7 is a structural diagram of a data set generating device provided in an embodiment of the present application. The data set generating device 700 includes: a first screening unit 701, a second screening unit 702, a third screening unit 703 and a generating unit 704, wherein:

[0205] A first screening unit 701 is used to screen out a first video set from a video library according to a video quality index, wherein index values ​​of videos in the first video set under the video quality index cover a value range of the video quality index;

[0206] The second screening unit 702 is used to screen out a second video set from the first video set according to the video attribute, wherein the attribute values ​​of the videos in the second video set under the video attribute cover the value range of the video attribute;

[0207] A third screening unit 703 is used to screen a third video set from the second video set according to the video feature, wherein the feature values ​​of the videos in the third video set under the video feature cover the value range of the video feature;

[0208] The generating unit 704 is configured to generate a data set according to the third video set, where the data set is used to train a video quality assessment model.

[0209] In some possible implementations, the video quality index includes: a multi-method video quality evaluation fusion index, a peak signal-to-noise ratio, and a bit rate; the first screening unit 701 is specifically used to: divide the value range of the video quality index into multiple value intervals; for each value interval of the video quality index, select a corresponding number of videos that meet the value interval from the video library to obtain a first video set.

[0210] In some possible implementations, the video attributes include: category, bit rate, resolution, and horizontal or vertical screen; the second screening unit 702 is specifically used to: divide the value range of the video attribute into multiple value intervals; for each value interval of the video attribute, select a corresponding number of videos that meet the value interval from the first video set to obtain a second video set.

[0211] In some possible implementations, the video features include: scene and target features, wherein the target features include: blocking effect, energy, blur, exposure, contrast, edge and color; the third screening unit 703 is specifically used for:

[0212] Dividing the value range of each target feature into a plurality of value intervals, and determining a corresponding video subset from the second video set based on the value interval of each target feature;

[0213] Selecting the current video subset from the unselected video subsets;

[0214] Select the current video from the videos that are not selected in the current video subset;

[0215] When the number of videos added to the video set to be filled does not reach the preset number, the following processing is performed until the number of videos added to the video set to be filled reaches the preset number, and the video set to be filled is determined as the third video set; the processing includes:

[0216] If the current video meets the condition, the current video is added to the video set to be filled, and if there are unselected videos in the current video subset, the step of selecting the current video from the unselected videos in the current video subset is returned, or if there are no unselected videos in the current video subset, the step of selecting the current video subset from the unselected video subset is returned;

[0217] If the current video does not meet the condition, then if there are unselected videos in the current video subset, return to the step of selecting the current video from the unselected videos in the current video subset, or if there are no unselected videos in the current video subset, return to the step of selecting the current video subset from the unselected video subset;

[0218] The conditions include: the current video is not in the video set to be filled, the ratio of the number of videos corresponding to the scene of the current video in the video set to be filled to a preset number does not exceed a preset ratio, and the Euclidean distance between the target feature of the current video and the target feature of each video added to the video set to be filled is greater than a preset threshold.

[0219] In some possible implementations, the generating unit 704 is specifically configured to: filter out videos to be degraded from the third video set, degrade the videos to be degraded to obtain a fourth video set; and generate a data set according to the third video set and the fourth video set.

[0220] In some possible implementations, when the generation unit 704 downgrades the video to be degraded to obtain the fourth video set, it is specifically used to: extract at least one frame of image from the video to be degraded, perform image dimension downgrading on each frame of image, to obtain a first degraded video, wherein the image dimension downgrading includes: image encoding, downsampling, Gaussian blur and Gaussian noise; perform video dimension downgrading on the video to be degraded to obtain a second degraded video, wherein the video dimension downgrading includes: video encoding, downsampling, Gaussian blur and Gaussian noise; and obtain the fourth video set based on the first degraded video and the second degraded video.

[0221] For the specific definition of the data set generation device, please refer to the definition of the data set generation method above, which will not be repeated here. Each unit in the above data set generation device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above units.

[0222] See also Figure 8 , Figure 8 800 is a structural diagram of a video quality assessment model training device provided in an embodiment of the present application. The video quality assessment model training device 800 includes: an acquisition unit 801 and a training unit 802, wherein:

[0223] An acquisition unit 801 is used to acquire a sample data set, each sample video in the sample data set is annotated with a first scoring result, and the sample data set is a data set generated according to the data set generation method provided in the above embodiment;

[0224] The training unit 802 is used to score the sample video using the video quality evaluation model to be trained, obtain a second scoring result of the sample video, calculate the loss according to the second scoring result of the sample video and its corresponding first scoring result, and adjust the parameters of the video quality evaluation model to be trained according to the loss until the training end condition is met to obtain the video quality evaluation model.

[0225] For the specific definition of the training device of the video quality assessment model, please refer to the definition of the training method of the video quality assessment model above, which will not be repeated here. Each unit in the above-mentioned training device of the video quality assessment model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned units.

[0226] See also Fig. 9 , Fig. 9 is a structural diagram of a video quality evaluation device provided in an embodiment of the present application. The video quality evaluation device 900 includes: an acquisition unit 901 and an evaluation unit 902, wherein:

[0227] An acquisition unit 901 is used to acquire a video to be evaluated;

[0228] The evaluation unit 902 is used to score the video to be evaluated by using the video quality evaluation model trained by the training method of the video quality evaluation model provided in the previous embodiment to obtain the quality evaluation result of the video to be evaluated.

[0229] For the specific definition of the video quality evaluation device, please refer to the definition of the video quality evaluation method above, which will not be repeated here. Each unit in the above-mentioned video quality evaluation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned units.

[0230] See also Fig.10 , Fig.10 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 1000 includes a memory 1001 and a processor 1002. Optionally, the electronic device 1000 also includes a communication interface 1003 and a bus 1004. The memory 1001, the processor 1002 and the communication interface 1003 are connected to each other through the bus 1004. The memory 1001 stores a computer program, and the processor 1002 is used to execute the computer program stored in the memory 1001 to implement the methods in the above-mentioned method embodiments.

[0231] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the methods in the above-mentioned method embodiments are implemented.

[0232] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method in the above-mentioned method embodiments is implemented.

[0233] It should be understood that the memory / readable storage medium in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a hard disk drive (HDD), a solid-state drive (SSD), a read-only memory (ROM) or a flash memory, etc. The volatile memory may be a random access memory (RAM) or an external cache memory. As an illustration and not limitation, RAM may be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc.

[0234] The processor in the embodiments of the present application may be a combination of one or more processing modules such as a central processing unit (CPU), a graphics processing unit (GPU) or a microprocessor (MPU), and may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0235] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0236] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A method for generating a data set, characterized in that: The method comprises: According to the video quality index, a first video set is selected from a video library, wherein index values ​​of the videos in the first video set under the video quality index cover a value range of the video quality index; According to the video attribute, a second video set is selected from the first video set, wherein attribute values ​​of the videos in the second video set under the video attribute cover a value range of the video attribute; According to the video feature, a third video set is selected from the second video set, wherein the feature values ​​of the videos in the third video set under the video feature cover the value range of the video feature; Generate a data set according to the third video set, where the data set is used to train a video quality assessment model; The video features include scene and target features, the current video in the second video set is added to the video set to be filled when the condition is met, and the third video set is determined according to the video set to be filled; The conditions include: the current video is not in the video set to be filled, the ratio of the number of videos corresponding to the scenes of the current video in the video set to be filled to a preset number does not exceed a preset ratio, and the Euclidean distance between the target feature of the current video and the target feature of each video added to the video set to be filled is greater than a preset threshold.

2. The method according to claim 1, characterized in that The video quality indicators include: video quality multi-method evaluation fusion index, peak signal-to-noise ratio and bit rate; The step of selecting a first video set from a video library according to the video quality indicator includes: Dividing the value range of the video quality indicator into multiple value intervals; For each value range of the video quality indicator, a corresponding number of videos satisfying the value range are selected from the video library to obtain the first video set.

3. The method according to claim 1, characterized in that The video attributes include: category, bit rate, resolution, and horizontal or vertical screen; The step of selecting the second video set from the first video set according to the video attribute includes: Dividing the value range of the video attribute into multiple value intervals; For each value range of the video attribute, a corresponding number of videos satisfying the value range are selected from the first video set to obtain the second video set.

4. The method according to claim 1, characterized in that: The target features include: blockiness, energy, blur, exposure, contrast, edge and color; The step of selecting a third video set from the second video set according to the video feature includes: Dividing the value range of each target feature into a plurality of value intervals, and determining a corresponding video subset from the second video set based on the value interval of each target feature; Selecting the current video subset from the unselected video subsets; Selecting a current video from videos not selected in the current video subset; When the number of videos added to the video set to be filled does not reach a preset number, the following processing is performed until the number of videos added to the video set to be filled reaches the preset number, and the video set to be filled is determined as the third video set; the processing includes: If the current video satisfies the condition, the current video is added to the to-be-filled video set, and if there are unselected videos in the current video subset, the step of selecting the current video from the unselected videos in the current video subset is returned, or if there are no unselected videos in the current video subset, the step of selecting the current video subset from the unselected video subset is returned; If the current video does not satisfy the condition, then when there are unselected videos in the current video subset, return to the step of selecting the current video from the unselected videos in the current video subset; or when there are no unselected videos in the current video subset, return to the step of selecting the current video subset from the unselected video subset.

5. The method according to any one of claims 1 to 4, characterized in that: Generating a data set according to the third video set includes: Screening out videos to be degraded from the third video set, and performing quality degradation on the videos to be degraded to obtain a fourth video set; A data set is generated according to the third video set and the fourth video set.

6. The method according to claim 5, characterized in that The step of downgrading the to-be-downgraded video to obtain a fourth video set includes: Extracting at least one frame of image from the video to be degraded, and performing image dimension degradation processing on each frame of image to obtain a first degraded video, wherein the image dimension degradation processing includes: image encoding, downsampling, Gaussian blurring, and Gaussian noise addition; Performing a video dimension degradation process on the video to be degraded to obtain a second degraded video, wherein the video dimension degradation process includes: video encoding, downsampling, Gaussian blurring and Gaussian noise addition; A fourth video set is obtained according to the first degraded video and the second degraded video.

7. A method for training a video quality assessment model, characterized in that: The method comprises: Obtain a sample data set, wherein each sample video in the sample data set is annotated with a first scoring result, and the sample data set is a data set generated by the method according to any one of claims 1 to 6; The sample video is scored using the video quality evaluation model to be trained to obtain a second scoring result of the sample video, the loss is calculated based on the second scoring result of the sample video and its corresponding first scoring result, and the parameters of the video evaluation model to be trained are adjusted based on the loss until the training end condition is met to obtain the video quality evaluation model.

8. A video quality evaluation method, characterized in that: The method comprises: Get the video to be evaluated; The video quality assessment model trained by the method according to claim 7 is used to score the video to be evaluated, thereby obtaining a quality assessment result of the video to be evaluated.

9. A data set generating device, characterized in that: The device comprises: A first screening unit, configured to screen out a first video set from a video library according to a video quality indicator, wherein index values ​​of videos in the first video set under the video quality indicator cover a value range of the video quality indicator; A second screening unit, configured to screen out a second video set from the first video set according to a video attribute, wherein attribute values ​​of videos in the second video set under the video attribute cover a value range of the video attribute; A third screening unit, configured to screen a third video set from the second video set according to a video feature, wherein a feature value of the video in the third video set under the video feature covers a value range of the video feature; A generating unit, configured to generate a data set according to the third video set, wherein the data set is used to train a video quality assessment model; The video features include scene and target features, the current video in the second video set is added to the video set to be filled when the condition is met, and the third video set is determined according to the video set to be filled; The conditions include: the current video is not in the video set to be filled, the ratio of the number of videos corresponding to the scenes of the current video in the video set to be filled to a preset number does not exceed a preset ratio, and the Euclidean distance between the target feature of the current video and the target feature of each video added to the video set to be filled is greater than a preset threshold.

10. A training device for a video quality assessment model, characterized in that: The device comprises: an acquiring unit, configured to acquire a sample data set, wherein each sample video in the sample data set is annotated with a first scoring result, and the sample data set is a data set generated by the method according to any one of claims 1 to 6; A training unit is used to score the sample video using the video quality evaluation model to be trained to obtain a second scoring result of the sample video, calculate the loss according to the second scoring result of the sample video and its corresponding first scoring result, and adjust the parameters of the video quality evaluation model to be trained according to the loss until the training end condition is met to obtain the video quality evaluation model.

11. A video quality evaluation device, characterized in that: The device comprises: An acquisition unit, used for acquiring a video to be evaluated; An evaluation unit is used to score the video to be evaluated by using the video quality evaluation model trained according to the method according to claim 7 to obtain a quality evaluation result of the video to be evaluated.

12. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Video quality diagnosis method based on deep learning

    CN111369555A

  • Model training method and device, expression recognition method and device, equipment and storage medium

    CN112241715A

  • Data set construction method and device, electronic equipment and storage medium

    CN112488033A

  • Quality evaluation method and device and electronic equipment

    CN112950581A

  • Object attribute identification method and device, storage medium and computer equipment

    CN113569111A