Video quality evaluation method, device, medium and equipment

Abstract

The present invention provides a video quality evaluation method, device, medium and equipment, the method comprising calculating the visual information fidelity value between a reference video frame sequence and a target video frame sequence; calculating the frame temporal complexity of the reference video frame sequence and the target video frame sequence respectively; obtaining a frame weight value corresponding to the visual information fidelity value according to the frame temporal complexity of the reference video frame sequence and the target video frame sequence; obtaining a quality evaluation result corresponding to the target video frame sequence according to the frame weight value and the visual information fidelity value. The embodiment provided by the present invention can obtain a video quality evaluation result that is closer to the attention mechanism of the human eye by calculating the frame temporal complexity to imitate the attention mechanism of the human eye without relying on a learning model.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing, and in particular relates to a video quality evaluation method, device, medium and equipment. Background Art

[0002] Video quality evaluation is used to measure visual indicators such as clarity, color reproduction, contrast, and detail performance of video content. These indicators are usually quantified as subjective visual quality scores or objective quality scores. Subjective evaluation is performed by human observers, while objective evaluation is performed by computer algorithms or specific hardware devices. For example, typical subjective evaluation methods such as the MOS (mean opinion score) method are close to the visual perception of the human eye and can accurately evaluate video quality, but their high cost, multiple steps, and poor real-time performance make them less practical.

[0003] Objective evaluation is mainly divided into three methods: full reference, partial reference, and no reference. They are mainly evaluated based on relevant indicators such as images and networks. For the full reference evaluation of videos, the existing full reference video evaluation method is mainly based on the image evaluation method, and the final video quality evaluation score is obtained by taking the average of each frame. The simple average method does not well reflect the viewer's attention mechanism during the dynamic playback of the video. If machine learning or deep learning is used to evaluate video quality, it relies on a large amount of training data or human annotation, and the complexity of the algorithm is too high.

[0004] In order to solve the above problems, a new video quality evaluation method is urgently needed. Summary of the invention

[0005] In order to overcome the above-mentioned defects in the prior art, the present invention provides a video quality evaluation method, which includes: calculating the visual information fidelity value between a reference video frame sequence and a target video frame sequence; respectively calculating the frame time domain complexity of the reference video frame sequence and the target video frame sequence; obtaining a frame weight value corresponding to the visual information fidelity value according to the frame time domain complexity of the reference video frame sequence and the target video frame sequence; and obtaining a quality evaluation result corresponding to the target video frame sequence according to the frame weight value and the visual information fidelity value.

[0006] The present invention also provides a video quality evaluation device, which includes: an information fidelity calculation unit, used to calculate the visual information fidelity value between a reference video frame sequence and a target video frame sequence; a time domain complexity calculation unit, used to respectively calculate the frame time domain complexity of the reference video frame sequence and the target video frame sequence; a frame weight value calculation unit, used to obtain a frame weight value corresponding to the visual information fidelity value based on the frame time domain complexity of the reference video frame sequence and the target video frame sequence; an evaluation result calculation unit, used to obtain a quality evaluation result corresponding to the target video frame sequence based on the frame weight value and the visual information fidelity value.

[0007] The present invention also provides one or more computer-readable media storing computer-executable instructions, which, when used by one or more computer devices, enable the one or more computer devices to execute the video quality assessment method as described above.

[0008] The present invention further provides an electronic device, which includes a memory and a processor, wherein: the memory stores a computer program, and the processor implements the video quality evaluation method as described above when executing the computer program.

[0009] The video quality evaluation method, device, medium and equipment provided by the present invention first calculate the visual information fidelity value between the reference video frame sequence and the target video frame sequence, then respectively calculate the frame temporal complexity of the reference video frame sequence and the target video frame sequence; then, according to the frame temporal complexity of the reference video frame sequence and the target video frame sequence, obtain the frame weight value corresponding to the visual information fidelity value; finally, according to the frame weight value and the visual information fidelity value, obtain the quality evaluation result corresponding to the target video frame sequence. The embodiment provided by the present invention can obtain an objective video quality evaluation result that is closer to the attention mechanism of the human eye by calculating the frame temporal complexity to imitate the attention mechanism of the human eye without relying on a learning model.

[0010] Furthermore, by calculating the temporal complexity and saliency in blocks and then calculating the temporal complexity of the frame, the attention mechanism of the human eye can be more perfectly aligned to the attention mechanism of the human eye from the local to the overall, and objective video quality evaluation results that are closer to the attention mechanism of the human eye can be obtained, with lower computational complexity and better evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0012] Figure 1 A schematic diagram showing an application scenario of the video quality evaluation method provided by an embodiment of the present invention is shown;

[0013] Figure 2 A schematic diagram of a process of a video quality evaluation method 200 provided in an embodiment of the present invention is shown;

[0014] Figure 3 A schematic diagram of a process of a video quality assessment method 300 provided in an embodiment of the present invention is shown;

[0015] Figure 4 A schematic diagram of a process of a video quality assessment method 400 provided in an embodiment of the present invention is shown;

[0016] Figure 5 A schematic diagram of a video quality assessment method framework provided by an embodiment of the present invention is shown;

[0017] Figure 6 A schematic diagram of a reference video frame sequence and a target video frame sequence provided by an embodiment of the present invention is shown;

[0018] Figure 7 A schematic diagram showing the structure of a video quality evaluation device 700 provided in an embodiment of the present invention

[0019] Figure 8 A schematic structural diagram of an electronic device 800 provided in an embodiment of the present invention is shown.

[0020] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0021] In order to better understand and explain the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings. The present invention is not limited to these specific embodiments. On the contrary, modifications or equivalent substitutions made to the present invention should all be included in the scope of the claims of the present invention.

[0022] It should be noted that numerous specific details are given in the following specific embodiments. Those skilled in the art should understand that the present invention can also be implemented without these specific details. In the multiple specific embodiments given below, the principles, structures and components well known in the art are not described in detail in order to highlight the main purpose of the present invention.

[0023] like Figure 1 As shown, server 101 is used to provide video. Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0024] The terminal device 103 is used to receive the video. The terminal device 103 may include but is not limited to mobile devices such as smart phones, tablet computers, smart glasses, and electronic devices such as desktop computers.

[0025] The network 102 is used to support the terminal device 103 to access the server 101 under any network conditions. It may include but is not limited to access agents, intelligent gateways and other devices. Optionally, the above-mentioned network 102 may include a wireless network or a wired network, and standard communication technologies and / or protocols are used through the wireless network or the wired network. The network is usually the Internet, and it can also be any network, including but not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or a virtual private network.

[0026] During video transmission, various factors affect video quality, such as network factors, image encoding, terminal performance, etc. The need to accurately evaluate video quality becomes increasingly important.

[0027] At present, some full-reference video evaluation methods are based on image evaluation methods, and the final video quality evaluation score is obtained by taking the average of each frame. This method does not take into account that the degree of stimulation to the human eye is different for each frame during the video playback process. Another part is the method of full-reference video evaluation through machine learning or deep learning, which can obtain quality evaluation results that are more in line with the attention of the human eye, but it relies on a large amount of training data or human annotation, and the evaluation methods based on machine learning or deep learning have problems such as high complexity, making it difficult to apply to scenarios such as video encoding and decoding.

[0028] Therefore, the present invention proposes a video quality evaluation method that can obtain video evaluation results that are closer to the attention mechanism of the human eye without the need for high computational complexity.

[0029] Please refer to Figure 2 , Figure 2 FIG. 2 is a flow chart of a video quality evaluation method 200 provided in an embodiment of the present invention, which can be implemented by a video quality evaluation device. Figure 2 As shown, the method includes:

[0030] Step S201, calculating the visual information fidelity value between the reference video frame sequence and the target video frame sequence.

[0031] In the above steps, Visual Information Fidelity (VIF) measures the visual quality of an image by calculating the mutual information between the image to be evaluated and the reference image.

[0032] VIF is based on the combination of natural scene statistical models and distortion models to quantify the shared information between the distorted image and the reference image. This shared information is closely related to visual quality, and VIF measurement can quantify the information extracted by the brain from the reference image. However, the VIF method does not respond to the structural information of the image, and the VIF method does not consider the local visual factors of the human eye, and cannot be close to the visual perception of the human eye.

[0033] To calculate the visual information fidelity value between the reference video frame sequence and the target video frame sequence, the reference video frame sequence and the target video frame sequence can be input into a pre-built visual information fidelity calculation module to obtain a VIF value corresponding to the target video frame sequence. The visual information fidelity calculation module can be obtained by, for example, processing a script of a MATLAB toolbox, or by a third-party image processing software that supports VIF and has a built-in VIF video quality evaluation index calculation function. It can also be obtained through an interface of a cloud service provider, using a video analysis service provided by a cloud service provider.

[0034] Step S202, respectively calculating the frame temporal complexity of the reference video frame sequence and the target video frame sequence.

[0035] In the above steps, the frame temporal complexity refers to the temporal complexity calculated in frames. The frame temporal complexity is used to evaluate the degree of temporal change of video frames.

[0036] For example, an algorithm for detecting moving targets in a video or image sequence is used to extract foreground targets to detect the motion changes of the target image. For example, the frame difference method can be used to calculate the temporal complexity of two adjacent frames. The frame difference method for calculating the temporal complexity includes but is not limited to converting continuous frames in a video or image sequence into grayscale images, performing differential operations, and binarization processing.

[0037] For example, dynamic targets can also be identified and tracked by background subtraction and optical flow methods. The optical flow method detects motion by calculating the displacement of each pixel between two consecutive frames (i.e., optical flow). This method can handle situations with background motion, such as scenes with camera movement.

[0038] The frame difference method can also be combined with the background subtraction method to improve the accuracy and robustness of motion detection. It can be applied to complex scenes that are difficult to handle with a single method, and the overall processing capability and adaptability can be improved through the complementarity between the methods.

[0039] Consider the characteristics of the actual application scenario, such as the dynamics of the environment, the limitations of computing resources, and the requirements for real-time performance. For example, for real-time monitoring, you may prefer to use the frame difference method or background subtraction method with lower computing cost and faster response speed; while in film and television production or applications with high precision requirements, you may choose the optical flow method to obtain more detailed motion analysis results.

[0040] Step S203: obtaining a frame weight value corresponding to the visual information fidelity value according to the frame temporal complexity of the reference video frame sequence and the target video frame sequence.

[0041] In the above steps, the intensity of the motion of adjacent video frames in the video can be measured according to the frame temporal complexity. Then, the frame weight value corresponding to the visual information fidelity value is assigned according to the frame temporal complexity, and the influence of the temporal complexity of the video frame is approximated as the attention of the human eye to the video frame with complex motion in the video. For example, when the temporal complexity of the frame is recognized, it can represent the obvious change of motion trend in the video playback process, such as Figure 6 The change from the second frame to the third frame in the distorted video shown has a relatively large frame temporal complexity. This motion trend makes people pay more attention to the third frame while watching the video. Therefore, a larger weight can be given to this frame in the video quality evaluation. For the first frame to the second frame, they have similar motion complexity. These frames have less stimulation to the human eye, so they are given a smaller weight in the process of video quality evaluation.

[0042] The weight assignment can be quantified based on the relationship between the frame temporal complexity and the threshold. For example, if the frame temporal complexity is higher than the threshold, the frame weight value corresponding to the visual information fidelity value is larger, and if the frame temporal complexity is less than or equal to the threshold, the frame weight value corresponding to the visual information fidelity value is smaller. Alternatively, the frame temporal complexity is divided into multiple level threshold ranges. If the frame temporal complexity reaches a certain level threshold range, the value corresponding to the level threshold range is assigned as the frame weight value corresponding to the visual information fidelity value.

[0043] Step S204: obtaining a quality evaluation result corresponding to the target video frame sequence according to the frame weight value and the visual information fidelity value.

[0044] In the above steps, the visual information fidelity values ​​can be weighted and summed according to the frame weight values ​​to obtain a quality evaluation result corresponding to the target video frame sequence.

[0045] Different video quality evaluation indicators can reflect different aspects of video quality, such as evaluating the similarity of images, evaluating the structural information of images, or comprehensively evaluating the quality indicators of images. When selecting evaluation indicators, it is necessary to select the most appropriate indicators based on actual needs and evaluation objectives. For example, for real-time transmission and processing scenarios, it is necessary to select an evaluation method with low computational complexity and fast processing speed. The embodiment provided by the present invention performs weighted evaluation of the fidelity of visual information through the temporal complexity of the frame, and obtains an objective video quality evaluation result that is closer to the subjective evaluation result of the human visual system. It has low computational complexity and fast processing speed, and is more suitable for real-time transmission encoding and decoding scenarios.

[0046] The embodiment provided by the present invention calculates the frame time domain complexity of the target video frame sequence to obtain a frame weight value corresponding to the visual information fidelity value, and introduces the human eye's attention mechanism to the video into the model for objectively evaluating the video quality. The embodiment provided by the present invention has low computational complexity and fast processing speed, and can obtain evaluation video quality that is closer to the human visual system without relying on a learning model.

[0047] Please refer to Figure 3 , Figure 3 The flowchart of a video quality evaluation method 300 provided by an embodiment of the present invention is shown. The method can be implemented by a video quality evaluation device. The method includes the following steps:

[0048] Step S301, calculating the visual information fidelity value between the reference video frame sequence and the target video frame sequence.

[0049] Step S302, block processing is performed on each video frame in the reference video frame sequence and the target video frame sequence to obtain N image blocks corresponding to each video frame in the reference video frame sequence and N image blocks corresponding to the target video frame sequence, where N is a natural number.

[0050] In the above steps, each video frame is processed by block processing, for example, a fixed size block method, an adaptive size block method, or a feature-based block method can be used, and this application does not limit the block. Assume that a fixed size block method is used, and the size of each block is defined as 16x16.

[0051] Step S303 , calculating the temporal complexity and the saliency value for each image block of each video frame in the reference video frame sequence and the target video frame sequence respectively.

[0052] In the above steps, calculating the temporal complexity may include, for each image block, obtaining the temporal complexity corresponding to the image block according to the square mean of the difference between pixels of two adjacent frames. For example, calculating the temporal complexity by block may be performed according to the following formula:

[0053]

[0054] in, represents the temporal complexity of the jth block of the i-th frame, Represents the pixel value of the kth pixel in the jth block of the i-th frame.

[0055] The temporal complexity of the jth block of the i-th frame is calculated by calculating the square of the difference between the pixel value of the kth pixel in the jth block of the i-th frame and the pixel value of the kth pixel in the jth block of the i-1-th frame. The calculated temporal complexity indicates the intensity of motion of the previous and next frames, and the human eye will receive more stimulation from areas with intense motion.

[0056] In the above steps, calculating the saliency value may include obtaining a saliency value corresponding to each image block according to an algorithm that simulates the human visual attention mechanism. The algorithm that simulates the human visual attention mechanism may be, for example, the GBVS (Graph-Based Visual Saliency) algorithm, or other saliency calculation algorithms. The main idea of ​​the GBVS algorithm is to divide the image into multiple regions and construct these regions into a graph. Then, by analyzing the structure and properties of the graph, determine which regions attract people's attention the most. Assuming the GBVS algorithm is used to calculate block-level saliency as an example, it can be calculated using the following formula:

[0057]

[0058] in, represents the saliency value of the jth block in the i-th frame, Represents the saliency value of the k-th pixel in the j-th block of the i-th frame (limited to 0 to 255).

[0059] By calculating the saliency value of the kth pixel in the jth block of the i-th frame; then calculating the saliency value of the jth block of the i-th frame based on the saliency value of the kth pixel in the jth block of the i-th frame, and then using such saliency, important image blocks can be found in the video frame. Since the human eye has different sensitivities to different image blocks, the human eye is usually more sensitive to the temporal complexity of these important image blocks. The present invention proposes to calculate temporal complexity and saliency based on block division to better simulate the human eye's attention mechanism to local information.

[0060] Step S304, calculates the frame temporal complexity corresponding to each video frame of the reference video frame sequence and the frame temporal complexity corresponding to each video frame of the target video frame sequence according to the temporal complexity and significance value of each image block of each video frame in the reference video frame sequence and the target video frame sequence, respectively.

[0061] In the above steps, it is comprehensively considered that people will pay more attention to images at important positions in the video frame, and during the playback of the video, areas with complex movements also attract more attention from people. Therefore, the temporal complexity of the image block is weighted by the saliency of the detail attention mechanism to approximate the feeling of the human eye when watching the video.

[0062] The temporal complexity of the j-th image block of the i-th frame of the reference video frame sequence and the target video frame sequence is calculated respectively, and the first temporal complexity of the reference video frame sequence (i.e., the first block-level temporal complexity) and the second temporal complexity of the target video frame sequence (i.e., the second block-level temporal complexity) are obtained; the saliency value of the j-th image block of the i-th frame of the reference video frame sequence and the target video frame sequence is calculated respectively, and the first saliency value of the reference video frame sequence (i.e., the first block-level saliency) and the second saliency value of the target video frame sequence (i.e., the second block-level saliency) are obtained; according to the first block-level temporal complexity and the first block-level saliency, the first frame temporal complexity of the i-th frame of the reference video frame sequence is obtained; according to the second block-level temporal complexity and the second block-level saliency, the second frame temporal complexity of the i-th frame of the target video frame sequence is obtained, wherein the value of i is any frame of the reference video frame sequence and the target video frame sequence, and the value of j is any one of 1 to N.

[0063] After obtaining the temporal complexity and saliency for each image block, assume that all saliencies of the i-th frame are summed, and then use the ratio of the saliency value of the j-th block of the i-th frame to the sum of the saliencies of the i-th frame as the weight of the temporal complexity of the j-th block of the i-th frame, and then use the weight of the temporal complexity of the j-th block of the i-th frame and the temporal complexity of the j-th block of the i-th frame for weighted summation to obtain the frame temporal complexity of the i-th frame, as shown in the following formula:

[0064]

[0065] in, represents the temporal complexity of the jth block of the i-th frame, represents the saliency value of the jth block in the i-th frame. The block-level temporal complexity is weighted by the spatial saliency to obtain the frame-level temporal complexity.

[0066] In the above steps, the ratio of the saliency value of the jth image block of the i-th frame to the sum of the saliency values ​​of the N image blocks of the i-th frame is used as the block-level weight value of the block-level temporal complexity of the jth image block of the i-th frame; the block-level weight value is multiplied and summed with the block-level temporal complexity of the jth image block of the i-th frame to obtain the frame temporal complexity of the i-th frame. The block-level weight value is the first weight value corresponding to each video frame of the reference video frame sequence, or the second weight value corresponding to each video frame of the target video frame sequence.

[0067] In some embodiments, Figure 5 As shown, each image block of each video frame in the reference video frame sequence and each image block of each video frame in the target video frame sequence can be respectively input into the time domain complexity calculation model to obtain the first time domain complexity of each image block of each video frame in the reference video frame sequence and the second time domain complexity of each image block of each video frame in the target video frame sequence. The time domain complexity calculation model, for example, uses a frame difference method calculation model.

[0068] Each image block of each video frame in the reference video frame sequence and each image block of each video frame in the target video frame sequence can also be input into the saliency calculation model to obtain a first saliency value of each image block of each video frame in the reference video frame sequence and a second saliency value of each image block of each video frame in the target video frame sequence. The saliency calculation model, for example, uses the GBVS algorithm calculation model. The GBVS algorithm realizes saliency calculation by comparing the similarity or difference between regional features and global features, and also considers the connection relationship between regions to determine which regions attract people's attention the most.

[0069] Then, according to the saliency value of each image block of each video frame of the reference video frame sequence, a first weight value corresponding to each video frame of the reference video frame sequence is obtained, for example

[0070] Then, according to the saliency value of each image block of each video frame of the target video frame sequence, a second weight value corresponding to each video frame of the target video frame sequence is obtained, for example,

[0071] According to the first weight value and the time domain complexity of each image block of each video frame of the reference video frame sequence, the time domain complexity of the first frame is obtained, that is, fCom i ;

[0072] According to the second weight value and the time domain complexity of each image block of each video frame of the target video, the time domain complexity of the second frame is obtained, that is, fCom i The first and second here are for distinguishing the representations, and the same algorithm can be used for their calculation.

[0073] The present invention proposes to calculate the temporal complexity and saliency by block-by-block calculation, and better simulate the attention mechanism of the human eye to local information according to the ratio of the sum of the block saliency and the saliency of the entire video frame. At the same time, the local attention mechanism is introduced into the video frame to simulate the attention mechanism of the human eye to the global changes of the video frame. Through lower computational complexity, video quality evaluation results that are closer to the local attention mechanism and global attention mechanism of the human eye to the played video are obtained.

[0074] Step S305 , obtaining a frame weight value corresponding to the visual information fidelity value according to the frame temporal complexity of the reference video frame sequence and the target video frame sequence.

[0075] Step S306: Obtain a quality evaluation result corresponding to the target video frame sequence according to the frame weight value and the visual information fidelity value.

[0076] The embodiment provided by the present invention performs block processing by calculating the target video frame sequence, first calculates the temporal complexity and significance of the block, and then calculates the frame temporal complexity according to the temporal complexity and significance of the block. The block temporal complexity is weighted by the block significance, which is similar to the attention of the human eye to important positions in the video frame. Finally, according to the influence of the frame temporal complexity on the fidelity value of the visual information, from local to global, the attention mechanism of the human eye to the video is introduced into the model of objectively evaluating the video quality, so as to obtain the evaluated video quality that is closer to the human visual system.

[0077] Please refer to Figure 4 , Figure 4 The flowchart of a video quality evaluation method 400 provided by an embodiment of the present invention is shown. The method can be implemented by a video quality evaluation device. The method includes the following steps:

[0078] Step S401, calculating the visual information fidelity value between the reference video frame sequence and the target video frame sequence.

[0079] Step S402, block processing is performed on each video frame in the reference video frame sequence and the target video frame sequence to obtain N image blocks corresponding to each video frame in the reference video frame sequence and N image blocks corresponding to the target video frame sequence, where N is a natural number.

[0080] Step S403 : calculating the temporal complexity and the saliency value for each image block of each video frame in the reference video frame sequence and the target video frame sequence respectively.

[0081] Step S404, calculates the frame temporal complexity corresponding to each video frame of the reference video frame sequence and the frame temporal complexity corresponding to each video frame of the target video frame sequence based on the temporal complexity and significance value of each image block of each video frame in the reference video frame sequence and the target video frame sequence.

[0082] Step S405, obtaining a complexity factor according to the frame temporal complexity of the reference video frame sequence and the target video frame sequence, wherein the complexity factor is used to describe the relationship between the difference between the frame temporal complexity of the reference video frame sequence and the target video frame sequence and the frame temporal complexity of the reference video frame sequence.

[0083] In the above steps, after calculating the frame temporal complexity corresponding to each video frame of the reference video frame sequence and the frame temporal complexity corresponding to each video frame of the target video frame sequence, the relationship between the target video frame complexity and the reference video complexity, i.e., the complexity factor, is calculated. It can be expressed as:

[0084]

[0085] Among them, Ori represents the reference video, Dis represents the target video; represents the temporal complexity of the i-th frame of the reference video; represents the temporal complexity of the i-th frame of the target video; comFac i It represents the ratio between the difference between the temporal complexity of the i-th frame of the reference video and the temporal complexity of the i-th frame of the target video and the temporal complexity of the i-th frame of the reference video. i is a complexity factor, which is used to describe the relationship between the difference between the frame temporal complexity of the reference video frame sequence and the target video frame sequence and the frame temporal complexity of the reference video frame sequence.

[0086] The focus of the full-reference quality assessment algorithm is to consider the difference between the two. When comFac is large, it means that the motion of the reference video and the target video is quite different. Therefore, when evaluating the video quality, the video frames with a large gap will receive more attention.

[0087] Step S406: determining a frame weight value corresponding to the visual information fidelity value according to the complexity factor.

[0088] In the above steps, if Figure 6 As shown, it is assumed that the upper side is a reference video frame sequence, which includes {frame 1, frame 2, frame 3} arranged in time sequence, and the lower side is a distorted video frame sequence, that is, a target video frame sequence, which also includes {frame 1, frame 2, frame 3} arranged in time sequence.

[0089] In the reference video frame sequence, the temporal complexity from the second frame to the third frame is small. In the target video frame sequence, the temporal complexity from the second frame to the third frame is small. When watching the video, people will pay more attention to the third frame in the target video frame sequence. Therefore, the evaluation of video quality should have a greater weight on this frame. In the two video frame sequences, there is similar motion complexity from the first frame to the second frame. Therefore, for the second frame, it has less stimulation to the human eye and can be given a smaller weight in the process of video quality evaluation. For example, different weights can be provided for the visual information fidelity VIF of each frame according to the complexity factor, including:

[0090]

[0091] Among them, ω i That is, the frame weight value, which indicates the weight assigned to each frame, and M indicates the total number of frames in the video; comFac i represents the complexity factor of the i-th frame; Represents the sum of the complexity factors of the reference video frame sequence. ω i It is expressed as the ratio of the complexity factor of the i-th frame to the sum of the complexity factors of the reference video frame sequence, which is used to indicate the proportion of the temporal complexity of the i-th frame in the entire video.

[0092] Step S407 , weighted summing of the visual information fidelity values ​​is performed according to the frame weight values ​​to obtain a quality evaluation result corresponding to the target video frame sequence.

[0093] In the above steps, the frame weight value ω of the i-th frame i The visual information fidelity value VIF for the i-th frame i The weighted sum can be calculated as follows:

[0094]

[0095] Among them, VVIF represents the video-level visual information fidelity value for the target video, that is, the quality evaluation result corresponding to the target video frame sequence, and VIF i Represents the fidelity value of the visual information of the i-th frame; ω i Represents the weight value assigned to the i-th video frame.

[0096] The embodiment provided by the present invention performs block processing by calculating the target video frame sequence, first calculates the temporal complexity and significance of the block, and then calculates the frame temporal complexity according to the temporal complexity and significance of the block. The block temporal complexity is weighted by the block significance, which is similar to the attention of the human eye to important positions in the video frame. Finally, according to the influence of the frame temporal complexity on the fidelity value of the visual information, from local to global, the attention mechanism of the human eye to the video is introduced into the model of objectively evaluating the video quality, so as to obtain the evaluated video quality that is closer to the human visual system.

[0097] Please refer to Figure 7 , Figure 7 FIG. 7 is a schematic diagram showing the structure of a video quality assessment device 700 provided in an embodiment of the present invention. The device comprises:

[0098] An information fidelity calculation unit 701, used to calculate a visual information fidelity value between a reference video frame sequence and a target video frame sequence;

[0099] A time domain complexity calculation unit 702, used to calculate the frame time domain complexity of the reference video frame sequence and the target video frame sequence respectively;

[0100] A frame weight value calculation unit 703, used to obtain a frame weight value corresponding to the visual information fidelity value according to the frame temporal complexity of the reference video frame sequence and the target video frame sequence;

[0101] The evaluation result calculation unit 704 is used to obtain a quality evaluation result corresponding to the target video frame sequence according to the frame weight value and the visual information fidelity value.

[0102] Optionally, the time domain complexity calculation unit 702 is used to perform block processing on each video frame in the reference video frame sequence and the target video frame sequence, respectively, to obtain N image blocks corresponding to each video frame in the reference video frame sequence and N image blocks corresponding to the target video frame sequence, where N is a natural number; calculate the time domain complexity and significance value of each image block of each video frame in the reference video frame sequence and the target video frame sequence, respectively; calculate the frame time domain complexity corresponding to each video frame in the reference video frame sequence and the frame time domain complexity corresponding to each video frame in the target video frame sequence based on the time domain complexity and significance value of each image block of each video frame in the reference video frame sequence and the target video frame sequence.

[0103] Optionally, the time domain complexity calculation unit 702 further includes a subunit for calculating the time domain complexity, which is used to obtain, for each image block, a time domain complexity corresponding to the image block according to the square mean of the difference between pixels of two adjacent frames.

[0104] Optionally, the subunit for calculating the saliency value is used to obtain, for each image block, a saliency value corresponding to the image block in a one-to-one manner according to an algorithm that simulates a human visual attention mechanism.

[0105] The time domain complexity calculation unit 702 also includes a unit for: obtaining a first weight value corresponding to each video frame of the reference video frame sequence according to the saliency value of each image block of each video frame of the reference video frame sequence; obtaining a second weight value corresponding to each video frame of the target video frame sequence according to the saliency value of each image block of each video frame of the target video frame sequence; obtaining the time domain complexity of the first frame according to the weighted summation of the first weight value and the time domain complexity of each image block of each video frame of the reference video frame sequence; obtaining the time domain complexity of the second frame according to the weighted summation of the second weight value and the time domain complexity of each image block of each video frame of the target video.

[0106] Optionally, the frame weight value calculation unit 703 is used to obtain a complexity factor based on the frame time domain complexity of the reference video frame sequence and the target video frame sequence, and the complexity factor is used to describe the relationship between the difference between the frame time domain complexity of the reference video frame sequence and the target video frame sequence and the frame time domain complexity of the reference video frame sequence; determine the frame weight value corresponding to the visual information fidelity value based on the complexity factor.

[0107] The evaluation result calculation unit 704 is used to perform weighted summation on the visual information fidelity values ​​according to the frame weight values ​​to obtain a quality evaluation result corresponding to the target video frame sequence.

[0108] The video quality evaluation device provided by the present invention performs block processing by calculating the target video frame sequence, first calculates the temporal complexity and significance of the block, and then calculates the frame temporal complexity according to the temporal complexity and significance of the block. The block temporal complexity is weighted by the block significance, which is similar to the attention of the human eye to important positions in the video frame. Finally, according to the influence of the frame temporal complexity on the fidelity value of visual information, from local to global, the attention mechanism of the human eye to the video is introduced into the model of objectively evaluating the video quality, so as to obtain the evaluated video quality that is closer to the human visual system.

[0109] Please refer to Figure 8 , Figure 8 The electronic device 800 provided by the embodiment of the present invention is shown in the structural diagram. The electronic device 800 may be a terminal device for receiving a video, including but not limited to mobile devices such as smart phones, tablet computers, smart glasses, and electronic devices such as desktop computers. Figure 8 It does not limit the structure of the electronic device. Figure 8 As shown, the electronic device at least includes a memory 801 and a processor 802. For example, the electronic device may also include Figure 8 More or fewer components (such as network interfaces, display devices, etc.) are shown. An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned Figure 2-Figure 4 Describe the method.

[0110] In particular, according to the embodiment provided by the present invention, the above reference flow chart Figure 2-Figure 4 The described process can be implemented as a computer software program. For example, the embodiments provided by the present invention include a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program contains program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the system of the present invention are executed.

[0111] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and computer program products described in various embodiments provided by the present invention. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the aforementioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The units or modules involved in the embodiments described in the present invention may be implemented by software or by hardware. The described units or modules may also be arranged in a processor. For example, they may be described as: a processor includes an information fidelity calculation unit, a temporal complexity calculation unit, a frame weight value calculation unit, and an evaluation result calculation unit. Among them, the names of these units or modules do not constitute a limitation on the units or modules themselves in certain cases. For example, the rule file acquisition unit may also be described as "a unit for calculating the visual information fidelity value between a reference video frame sequence and a target video frame sequence."

[0114] As another aspect, an embodiment of the present invention further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently and not be assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the video quality assessment method described in the present invention.

[0115] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

Claims

1. A video quality evaluation method, characterized in that: The method includes: Calculating the visual information fidelity value between the reference video frame sequence and the target video frame sequence; Calculating the frame temporal complexity of the reference video frame sequence and the target video frame sequence respectively; Obtaining a frame weight value corresponding to the visual information fidelity value according to the frame temporal complexity of the reference video frame sequence and the target video frame sequence; Obtaining a quality evaluation result corresponding to the target video frame sequence according to the frame weight value and the visual information fidelity value; The frame temporal complexity of the reference video frame sequence and the target video frame sequence is calculated respectively, including: Calculating the temporal complexity and the saliency value for each image block of each video frame in the reference video frame sequence and the target video frame sequence respectively, and calculating the frame temporal complexity corresponding to each video frame of the reference video frame sequence and the frame temporal complexity corresponding to each video frame of the target video frame sequence according to the temporal complexity and the saliency value; The temporal complexity is calculated, including obtaining the temporal complexity corresponding to each image block according to the square mean of the difference between the pixels of two adjacent frames, and calculating the temporal complexity by block according to the following formula: in, represents the temporal complexity of the jth block of the i-th frame, represents the pixel value of the kth pixel in the jth block of the i-th frame; The saliency value is calculated, including obtaining the saliency value corresponding to each image block according to the GBVS algorithm that simulates the human visual attention mechanism, and calculating it by the following formula: in, represents the saliency value of the jth block in the i-th frame, represents the saliency value of the k-th pixel in the j-th block of the i-th frame; After obtaining the temporal complexity and saliency for each image block, assume that all saliencies of the i-th frame are summed, and then use the ratio of the saliency value of the j-th block of the i-th frame to the sum of the saliencies of the i-th frame as the weight of the temporal complexity of the j-th block of the i-th frame, and then use the weight of the temporal complexity of the j-th block of the i-th frame and the temporal complexity of the j-th block of the i-th frame for weighted summation to obtain the frame temporal complexity of the i-th frame, as shown in the following formula: in, represents the temporal complexity of the jth block of the i-th frame, represents the saliency value of the jth block in the i-th frame. The block-level temporal complexity is weighted by the spatial saliency to obtain the frame-level temporal complexity.

2. The method according to claim 1, characterized in that: The calculating the frame temporal complexity of the reference video frame sequence and the target video frame sequence respectively comprises: Performing block processing on each video frame in the reference video frame sequence and the target video frame sequence respectively, to obtain N image blocks corresponding to each video frame in the reference video frame sequence and N image blocks corresponding to the target video frame sequence, where N is a natural number; Calculating the temporal complexity and the saliency value for each image block of each video frame in the reference video frame sequence and the target video frame sequence respectively; The frame temporal complexity corresponding to each video frame of the reference video frame sequence and the frame temporal complexity corresponding to each video frame of the target video frame sequence are calculated based on the temporal complexity and significance value of each image block of each video frame in the reference video frame sequence and the target video frame sequence.

3. The method according to claim 2, characterized in that in, Calculating the frame temporal complexity corresponding to each video frame of the reference video frame sequence and the frame temporal complexity corresponding to each video frame in the target video frame sequence according to the temporal complexity and the saliency value of each image block of each video frame in the reference video frame sequence and the target video frame sequence, comprising: Obtaining a first weight value corresponding to each video frame of the reference video frame sequence according to a saliency value of each image block of each video frame of the reference video frame sequence; Obtaining a second weight value corresponding to each video frame of the target video frame sequence according to the saliency value of each image block of each video frame of the target video frame sequence; Obtaining a first frame temporal complexity by weighted summing the first weight value and the temporal complexity of each image block of each video frame of the reference video frame sequence; The second frame temporal complexity is obtained by weighted summing the second weight value and the temporal complexity of each image block of each video frame of the target video.

4. The method according to claim 1, characterized in that: The obtaining, according to the frame temporal complexity of the reference video frame sequence and the target video frame sequence, a frame weight value corresponding to the visual information fidelity value comprises: Obtaining a complexity factor according to the frame temporal complexity of the reference video frame sequence and the target video frame sequence, wherein the complexity factor is used to describe the relationship between the difference between the frame temporal complexity of the reference video frame sequence and the target video frame sequence and the frame temporal complexity of the reference video frame sequence; A frame weight value corresponding to the visual information fidelity value is determined according to the complexity factor.

5. The method according to claim 1, characterized in that Obtaining a quality evaluation result corresponding to the target video frame sequence according to the frame weight value and the visual information fidelity value, including: The visual information fidelity values ​​are weighted and summed according to the frame weight values ​​to obtain a quality evaluation result corresponding to the target video frame sequence.

6. A video quality evaluation device, characterized in that: The device includes: An information fidelity calculation unit, used to calculate a visual information fidelity value between a reference video frame sequence and a target video frame sequence; A time domain complexity calculation unit, used to calculate the frame time domain complexity of the reference video frame sequence and the target video frame sequence respectively; A frame weight value calculation unit, configured to obtain a frame weight value corresponding to the visual information fidelity value according to the frame temporal complexity of the reference video frame sequence and the target video frame sequence; An evaluation result calculation unit, used to obtain a quality evaluation result corresponding to the target video frame sequence according to the frame weight value and the visual information fidelity value; The time domain complexity calculation unit calculates the frame time domain complexity of the reference video frame sequence and the target video frame sequence respectively, including: Calculating the temporal complexity and the saliency value for each image block of each video frame in the reference video frame sequence and the target video frame sequence respectively, and calculating the frame temporal complexity corresponding to each video frame of the reference video frame sequence and the frame temporal complexity corresponding to each video frame of the target video frame sequence according to the temporal complexity and the saliency value; The temporal complexity is calculated, including obtaining the temporal complexity corresponding to each image block according to the square mean of the difference between the pixels of two adjacent frames, and calculating the temporal complexity by block according to the following formula: in, represents the temporal complexity of the jth block of the i-th frame, represents the pixel value of the kth pixel in the jth block of the i-th frame; The saliency value is calculated, including obtaining the saliency value corresponding to each image block according to the GBVS algorithm that simulates the human visual attention mechanism, and calculating it by the following formula: in, represents the saliency value of the jth block in the i-th frame, represents the saliency value of the k-th pixel in the j-th block of the i-th frame; After obtaining the temporal complexity and saliency for each image block, assume that all saliencies of the i-th frame are summed, and then use the ratio of the saliency value of the j-th block of the i-th frame to the sum of the saliencies of the i-th frame as the weight of the temporal complexity of the j-th block of the i-th frame, and then use the weight of the temporal complexity of the j-th block of the i-th frame and the temporal complexity of the j-th block of the i-th frame for weighted summation to obtain the frame temporal complexity of the i-th frame, as shown in the following formula: in, represents the temporal complexity of the jth block of the i-th frame, represents the saliency value of the jth block in the i-th frame. The block-level temporal complexity is weighted by the spatial saliency to obtain the frame-level temporal complexity.

7. One or more computer-readable media storing computer-executable instructions, characterized in that: When the instructions are used by one or more computer devices, the one or more computer devices execute the video quality assessment method according to any one of claims 1 to 5.

8. An electronic device, comprising a memory and a processor, characterized in that: in: The memory stores a computer program, and the processor implements the video quality assessment method according to any one of claims 1 to 5 when executing the computer program.

Images