Method, device, apparatus and storage medium for performing video quality assessment

Abstract

The present application provides a method, device, apparatus and storage medium for performing video quality assessment (VQA). The method includes obtaining multiple images of a video; determining a first image to which parameter-based VQA is to be applied, the first image including all images in the multiple images; determining a first score for each of the first images based on applying the parameter-based VQA to each of the first images; determining a second image in the multiple images, the second image to which sample-based VQA is to be applied; determining a second score for each of the second images based on applying the sample-based VQA to each of the second images; and outputting a final score of at least one of the multiple images based on the first score and the second score.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims priority to U.S. Provisional Application No. 63 / 063,712 filed in the U.S. Patent and Trademark Office on August 10, 2020 and U.S. Application No. 17 / 362,145 filed in the U.S. Patent and Trademark Office on June 29, 2021, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to the field of advanced multimedia technology, and in particular to a method, device, apparatus and storage medium for quality assessment of compressed or uncompressed videos or images. Background Art

[0004] With the popularity of affordable and reliable user acquisition devices and social media platforms, videos and images have become an everyday part of life for many people. As the size of visual data has increased with the improvement of cameras, video compression is often applied to raw visual data (e.g., images or videos acquired by a camera) in order to transmit them from one device to another at a faster rate. However, compression systems often introduce artifacts or cause degradation in the visual quality of the raw visual data. Therefore, it may be desirable to identify the presence of certain compression artifacts or degradations in videos and images in order to repair and improve the quality of the video.

[0005] One method for understanding and identifying compression artifacts or compression degradation is by performing video quality assessment ("VQA"). To evaluate the quality of a video or image, a person can visually inspect the video under certain environments and test conditions (e.g., lighting conditions, distance, etc.) and use, for example, a mean opinion score ("MOS") to rate the video. However, the disadvantage of this manual operation is that it is time-consuming and incurs a lot of cost when the test scale is large. In order to provide a fast and economical evaluation of the quality of visual data without the use of human eyes, objective methods are used to replace this manual operation. In general, objective methods in VQA are divided into three categories: (1) no-reference VQA where no reference data is provided, (2) full-reference VQA where ground-truth data is provided, and (3) partial (reduced) reference VQA where some but not all reference data is provided.

[0006] In no-reference VQA, the evaluation method is purely based on the decoded image without any knowledge of the original image. Some of the features considered in evaluating the decoded image may include sharpness, gradient, etc. A score may be generated by mapping the behavior of one or more features to a numerical value between, for example, 0 and 100.

[0007] In full-reference VQA, the evaluation method is based on the difference between the test data and the reference data. The larger the difference between the test data and the reference data, the greater the degradation of the compression. An exemplary metric can be Video Multimethod Assessment Fusion ("VMAF"), where the subject video quality is predicted based on the reference and distorted video sequences. This metric can be used to evaluate the quality of different video encoders.

[0008] In partial reference VQA, only part of the information in the reference data is used to evaluate the video quality. One example may be to use the codec parameters of the bitstream (e.g., quantization parameter (QP), group of pictures (GOP) size, frame rate, resolution, bit rate, etc.) to estimate the quality of the video. Another example may be to use a subset of the reference video, such as a downsampled version of the original video and / or images.

[0009] For subjective evaluation of videos and images, video quality assessment methods with low computational complexity and low system latency are usually preferred. However, the accuracy of subjective evaluation may decrease when more intensive calculations need to be performed. Moreover, even if no-reference VQA is applied, some applications may not be able to support intensive calculations on a pixel-by-pixel basis to evaluate the quality of the video. Therefore, technical solutions are needed to achieve lightweight quality evaluation while maintaining the reliability of the evaluation. Summary of the invention

[0010] According to one embodiment, a method for performing video quality assessment (VQA) is provided. The method includes: obtaining multiple images of a video; determining a first image to which parameter-based VQA is to be applied, the first image including all images in the multiple images; determining a first score for each of the first images based on applying the parameter-based VQA to each of the first images; determining a second image in the multiple images, the second image to which sample-based VQA is to be applied; determining a second score for each of the second images based on applying the sample-based VQA to each of the second images; and outputting a final score of at least one image in the multiple images based on the first score and the second score.

[0011] According to one embodiment, a device for performing video quality assessment VQA is provided. The device includes: at least one memory storing computer program code; and at least one processor configured to access the at least one memory and operate according to the instructions of the computer program code to perform the method for performing video quality assessment VQA according to the embodiment of the present application.

[0012] According to one embodiment, a device for performing video quality assessment (VQA) is provided. The device includes: an acquisition module, which obtains multiple images of a video; a first determination module, which determines a first image to which parameter-based VQA is to be applied, the first image includes all images in the multiple images, and determines a first score for each of the first images based on applying the parameter-based VQA to each of the first images; a second determination module, which determines a second image in the multiple images, the second image to which sample-based VQA is to be applied, and determines a second score for each of the second images based on applying the sample-based VQA to each of the second images; and an output module, which outputs a final score of at least one image in the multiple images based on the first score and the second score.

[0013] According to one embodiment, a non-transitory computer-readable storage medium is provided, storing computer program code, and when the computer program code is executed by at least one processor, the at least one processor is configured to: obtain multiple images of a video; determine a first image to which parameter-based VQA is to be applied, the first image including all images of the multiple images; determine a first score for each of the first images based on applying the parameter-based VQA to each of the first images; determine a second image of the multiple images, the second image to which sample-based VQA is to be applied; determine a second score for each of the second images based on applying the sample-based VQA to each of the second images; and output a final score of at least one image of the multiple images based on the first score and the second score. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The following description briefly introduces the accompanying drawings, which illustrate one or more embodiments of the disclosure.

[0015] Figure 1 is a schematic diagram of a video quality assessment device according to an embodiment.

[0016] Figure 2 is a schematic diagram of a method for performing video quality assessment according to an embodiment.

[0017] Figure 3 A flow chart showing the operation of video quality assessment according to an embodiment is shown.

[0018] Figure 4 is a schematic diagram of a method for performing video quality assessment including calibrating a score according to an embodiment.

[0019] Figure 5A flow diagram of operations for video quality assessment including calibrating a score is shown in accordance with an embodiment.

[0020] Figure 6 is a flowchart of a method for performing video quality assessment based on video complexity according to an embodiment.

[0021] Figure 7 A flow chart showing the operation of video quality assessment based on video complexity according to an embodiment.

[0022] Figure 8 A method for executing the Figure 3 A block diagram of a computer program code for the method.

[0023] Fig. 9 A method for executing the Figure 5 A block diagram of a computer program code for the method.

[0024] Fig.10 A method for executing the Figure 7 A block diagram of a computer program code for the method. DETAILED DESCRIPTION

[0025] This document describes example embodiments in detail with reference to the accompanying drawings. It should be understood that one or more embodiments described herein are only examples and should not be interpreted as limiting the scope of the present disclosure.

[0026] Figure 1 is a schematic diagram of a video quality assessment device according to an embodiment.

[0027] refer to Figure 1 , a video quality assessment (VQA) device 100 may include a processor 110, a volatile memory 120, a non-volatile memory 130, an input interface 140, an output interface 150, a communication interface 160, and a bus 170. The VQA device 100 may be implemented as part of an encoder and / or a decoder to assess the quality of a video or an image. However, one or more embodiments are not limited thereto, and the VQA device 100 may be connected to the encoder and / or the decoder, but may be physically separated from the encoder and / or the decoder. In addition, the VQA device 100 may be an algorithm stored in the volatile memory 120 in the form of software and executed by the processor 110 connected to the volatile memory 120.

[0028] The processor 110 is implemented in hardware, firmware, or a combination of hardware and software. The processor 110 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In some embodiments, the processor 110 includes one or more processors that can be programmed to perform functions.

[0029] Volatile memory 120 includes random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory), which stores information and / or instructions for use by processor 110.

[0030] The non-volatile memory 130 stores information and / or software related to the operation and use of the VQA device 100. For example, the non-volatile memory 130 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-volatile computer-readable medium, and a corresponding drive.

[0031] The input interface 140 may include components that allow the VQA device 100 to receive information, such as through user input, such as a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone. Additionally or alternatively, the input interface 140 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator).

[0032] Output interface 150 may include components that provide output information from VQA device 100, such as a display, a speaker, and / or one or more light emitting diodes (LEDs).

[0033] The communication interface 160 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the VQA device 100 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 160 can allow the VQA device 100 to receive information from another device and / or provide information to another device. For example, the communication interface 160 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0034] The bus 170 includes components that allow communication between components of the VQA device 100 .

[0035] The VQA device 100 may perform one or more operations described herein. The VQA device 100 may perform the above operations in response to the processor 110 executing software instructions stored by a non-volatile computer-readable medium (e.g., volatile memory 120 and / or non-volatile memory 130). Computer-readable media is defined herein as a non-volatile memory device. A memory device includes storage space within a single physical storage device or storage space distributed across multiple physical storage devices.

[0036] The software instructions may be read into the volatile memory 120 and / or the non-volatile memory 130 from another computer-readable medium or from another device through the communication interface 160. When executed, the software instructions stored in the volatile memory 120 and / or the non-volatile memory 130 may cause the processor 110 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0037] Figure 1 The number and arrangement of components shown are provided as examples. Figure 1 The VQA device 100 may include more components, fewer components, different components, or components arranged differently than the components shown. Additionally or alternatively, one set of components (e.g., one or more components) of the VQA device 100 may perform one or more functions described as being performed by another set of components of the VQA device 100.

[0038] Figure 2 is a schematic diagram of a method for performing video quality assessment according to an embodiment.

[0039] refer to Figure 2 , the VQA algorithm 200 may include a parameter-based VQA model 210 and a sample-based VQA model 220. Parameter-based VQA may refer to video quality assessment that does not require pixel-level information but only requires bitstream-level information. For example, pixel-level information may include information about the color and position of each pixel in the image, and bitstream-level information may include information about bitrate, quantization parameters, frame resolution, frame rate, etc. Sample-based VQA may refer to video quality assessment that requires pixel-level analysis. Generally, sample-based VQA is more accurate than parameter-based VQA, but when there are many samples to be analyzed, sample-based VQA requires more intensive computation.

[0040] According to an embodiment, both the parameter-based VQA model 210 and the sample-based VQA model 220 can be used for quality evaluation of one or more images. Specifically, the VQA algorithm 200 can be designed to apply the parameter-based VQA model more frequently than the sample-based VQA model. For example, the video to be evaluated may include a first group 230a and a second group 230b of images. The first group 230a may include images A, B, and C, and the second group 230b may include images A′, B′, and C′. Although in Figure 2 Only two sets of images are shown in , but there may be more than two sets of images.

[0041] In a specific example, the parameter-based VQA model 210 may be applied to each image frame in each group, and the sample-based VQA model 220 may be applied to the first image frame in each group. Figure 2 As shown, the parameter-based VQA model 210 is applied to each of the images A, B, and C in the first group 230a and each of the images A′, B′, and C′ in the second group 230b. In contrast, the sample-based VQA model 220 is applied to the first image A in the first group 230a and the first image A′ in the second group 230b. Since some applications may not be able to support intensive calculations on a pixel-by-pixel basis for quality assessment, the sample-based VQA model 220 can be selectively applied to some of the multiple images in the video. Therefore, when both the parameter-based VQA model 210 and the sample-based VQA model 220 are used to evaluate the quality of the video, the burden of the processing load is reduced. However, one or more embodiments are not limited thereto. For example, the parameter-based VQA model 210 can be applied to each of the multiple images in the video, while the sample-based VQA model 220 is only applied to a subset of the images, such as one per second, one every 30 frames, etc. In other words, the sample-based VQA 220 can be applied to images selected at predetermined time intervals or predetermined frame intervals.

[0042] According to an embodiment, the parameter-based VQA model 210 may be applied to each image frame of a video, and the sample-based VQA model 220 may be applied only to images with reduced resolution. For example, the sample-based VQA model 220 may be applied to 1:4 sub-sampled images.

[0043] According to an embodiment, the parameter-based VQA model 210 may be applied to each image frame of a video, and the sample-based VQA model 220 may be applied only to a set of regions within one or more images. For example, a target region of interest (ROI) of an image may be identified based on user input, and the sample-based VQA model 220 may be applied only to the target ROI in the image. Although it has been described that the target ROI may be identified based on user input, one or more embodiments are not limited thereto. For example, the target ROI may be predetermined to be in the middle of the screen because the middle of the screen is where the human eye is usually guided or focused when viewing an image or video. In addition, the target ROI may be determined based on a motion vector. That is, when the value of the motion vector of a region changes between consecutive image frames at a rate greater than a predetermined value, the region may be determined as a target ROI. Here, a motion vector that changes between consecutive frames at a rate greater than a predetermined value means that there is a large amount of motion in the image frame.

[0044] Figure 3 A flow chart showing the operation of video quality assessment according to an embodiment is shown.

[0045] refer to Figure 3 , illustrates a method 300 for performing video quality assessment according to an embodiment.

[0046] In S310, the method 300 may include obtaining images of a video to be analyzed. Here, the images of the video may be grouped into one or more groups, as described above with reference to Figure 2 as described.

[0047] In S320, the method 300 may include determining a first image to which parameter-based VQA should be applied. According to an embodiment, the first image may include all images of a plurality of images in the video. In other words, parameter-based VQA may be applied to each of the plurality of images in the video. For example, referring back to Figure 2 , the first image may be images A, A′, B, B′, C, and C′. However, one or more embodiments are not limited thereto, and parameter-based VQA may be selectively applied to some of the plurality of images of a video.

[0048] At S330, the method 300 may include determining a first score for each of the first images based on applying the parameter-based VQA to each of the first images. Figure 4 This operation is described in more detail.

[0049] At S340, the method 300 may include determining a second image to which the sample-based VQA should be applied. As described above, determining the second image may cause the sample-based VQA to be selectively applied to the images in the group. Here, the second image may overlap with some of the first images in the plurality of images in the video. For example, referring back to Figure 2 , the first image may be images A, A′, B, B′, C, and C′, and the second image may be images A and A′. Both parameter-based VQA and sample-based VQA are applied to images A and A′ included in the first image and the second image. In addition, the second image may be determined based on time, frame sequence, image resolution, and region of interest, as described above with reference to Figure 2 as described.

[0050] In S350, the method 300 may include determining a second score for each second image based on applying the sample-based VQA to each of the second images. Figure 4 This operation is described in more detail.

[0051] In S360, the method 300 may include outputting a final score of at least one image based on the determined first score and second score. Figure 4 This operation is described in more detail.

[0052] Figure 4 is a schematic diagram of a method for performing video quality assessment including calibrating a score according to an embodiment. Figure 2 and Figure 3 Some features or operations of the method for performing video quality assessment have been described. Therefore, some repeated descriptions of these features or operations may be omitted.

[0053] refer to Figure 4 , the VQA algorithm 400 may include a parameter-based VQA model 410 and a sample-based VQA model 420. Here, both the parameter-based VQA model 410 and the sample-based VQA model 420 can be used to evaluate the quality of one or more images. Specifically, the parameter-based VQA model 410 is applied to each of the images A, B, and C in the first group 430a and each of the images A′, B′, and C′ in the second group 430b. The sample-based VQA model 420 is applied to the first image A in the first group 430a and the first image A′ in the second group 430b.

[0054] Based on performing parameter-based VQA on images A, A′, B, B′, C, and C′, the parameter-based VQA model 410 may output a first score 440. Based on performing sample-based VQA on images A and A′, the sample-based VQA model 420 may output a second score 450. It should be understood that for one set of images, the first score 440 may include multiple scores, and for another set of images, the second score 450 may include multiple scores.

[0055] According to an embodiment, when the second score 450 is output from the sample-based VQA model 420, the second score 450 may be used to calibrate the first score derived from the parameter-based VQA model 410. More specifically, the function of the sample-based VQA may add the second score 450 to the first score 440 derived from the parameter-based VQA model 410 to generate a final score 460. However, when the second score 450 is not available for at least one of the plurality of images in the video, a sample-based VQA-generated score of a previous image in the video may be used. For example, when the second score 450 is not available for image C in the first group 430a, the sample-based VQA score of image B in the first group 430a may be used as the second score 450. In addition, when the second score 450 is not available for at least one of the plurality of images in the video sequence, the function may be applied to N sample-based VQA scores of some previous images in the video sequence. The function may include an average of N sample-based VQA scores, a median of N sample-based VQA scores, a minimum and a maximum of N sample-based VQA scores, etc., where N is a positive integer. For example, when the second score 450 is not available for image C in the first group 430, the average of the two sample-based VQA scores of image A and image B in the first group 430 may be used.

[0056] The final score 460 can be determined based on two components, a base score and an incremental score. When a sample-based VQA score is available, the base score can be based in whole or in part on the sample-based VQA. In other words, the base score depends on the second score 450 output by the sample-based VQA model 420. The incremental score is based on the parameter-based VQA. That is, the incremental score can be the first score 440 output by the parameter-based VQA model 410.

[0057] As an example, assume that a first score 440 of image A derived from the parameter-based VQA model 410 is represented as Sp(A), and a second score 450 of image A derived from the sample-based VQA model 420 is represented as Ss(A). In the case where the parameter-based VQA model 410 and the sample-based VQA model 420 generate scores using the same scoring system, the final score of image A may be Ss(A). That is, when the parameter-based VQA model 410 and the sample-based VQA model 420 use the same range of rating scales (e.g., scores ranging from 1 to 10), the final score 460 may be the second score 450 output from the sample-based VQA model. In other words, when the second score 450 based on the sample-based model 420 is determined for the image, assuming that the score ranges of the two models are the same, the second score 450 may be prioritized over the first score 440, and the second score 450 is used as the final score 460. However, one or more embodiments are not limited thereto, and the final score 460 may be derived according to various calculation methods.

[0058] In addition, image B following image A in the video sequence may be evaluated without using the sample-based VQA model 420. For example, a first score 440 of image B derived from the parameter-based VQA model 410 may be expressed as Sp(B). When the scoring systems of the parameter-based VQA model 410 and the sample-based VQA model 420 are different from each other, a final score 460 of image B may be calculated according to the following equation:

[0059] Final score (image B) = Sp (B) + w* (Ss (A) - Sp (A)) + offset Equation (1)

[0060] Here, "w" is a weight that can be fixed or obtained from decoded information such as image type and quantization information. The weight w can be a value less than or equal to 1, such as 0.8. The offset is a constant value used to balance the scoring system of the two models. The value of the offset can be 0 or a non-zero value.

[0061] According to equation (1), in the absence of a score for image B from the sample-based VQA model 420, the VQA algorithm 400 can calculate a final score 460. Specifically, the difference between the scoring systems of the two models, i.e., Ss(A)-Sp(A), is calculated based on image A, and then the difference is used to estimate the difference between the scoring systems at image B, where only the first score 440 of image B from the parameter VQA model 410 is available. Therefore, when the final score is calculated by the VQA algorithm 400 in the absence of a score from the sample-based VQA model, the difference in the scoring system between the parameter-based VQA model and the sample-based VQA model can be calibrated to provide a more accurate assessment of video quality while reducing the computational load.

[0062] As another example, the final score 460 for image B may be derived according to the following equation:

[0063] Final score (image B) = w0*Ss(A)+w1*Sp(B)-w2*Sp(A)+offset Equation (2).

[0064] Here, w0, w1, and w2 are weights that can be fixed or obtained from decoded information such as image type and quantization information. The offset can be a constant value that balances the difference between the scoring systems of the two models. The value of the offset can be 0 or a non-zero value.

[0065] Figure 5 A flow chart showing the operation of video quality assessment including calibrating the score according to an embodiment is shown. Figure 2 and Figure 3 Some features or operations of methods for performing video quality assessment are described. Figure 5 Operations S510 to S550 and S570 may correspond to Figure 3 Therefore, some repeated descriptions of operations S310 to S350 and S360 may be omitted.

[0066] refer to Figure 5 , in S560, method 500 may include calibrating the first score from the parameter-based VQA model based on the second score from the sample-based VQA model. Because sample-based VQA involves pixel-by-pixel analysis of the image, it requires more intensive computation. Therefore, as described above with reference to Figure 4As described, the sample-based VQA model is applied only to some of the multiple images of the video, while the parameter-based VQA model is applied to each of the multiple images of the video. However, when there is a difference between the scoring systems of the sample-based VQA model and the parameter-based VQA model, there may be an inconsistency between the first score obtained from the parameter-based VQA model and the second score obtained from the sample-based VQA model, thereby reducing the accuracy of video quality assessment.

[0067] In view of this, when the second score for an image is not available from the sample-based VQA model, the first score from the parameter-based VQA model can be calibrated based on the difference in the scoring system calculated from a previous image frame for which both the first score from the parameter-based VQA model and the second score from the sample-based VQA model are available. In other words, returning to the reference Figure 4 , when there is no score from the sample-based VQA model 420 for image B, a difference in the scoring system can be determined based on image A for which a first score 440 from the parameter-based VQA model 410 and a second score 450 from the sample-based VQA model 420 are available. Based on determining the difference between the scoring systems of the two models, the first score 440 for image B from the parameter-based VQA model 410 can be calibrated to obtain a final score 460.

[0068] Figure 6 is a schematic diagram of a method for performing video quality assessment based on video complexity according to an embodiment. Figure 4 and Figure 5 Some features or operations of the method for performing video quality assessment have been described. Therefore, some repeated descriptions of these features or operations may be omitted.

[0069] refer to Figure 6 The VQA algorithm 600 may include a parameter-based VQA model 610 and a sample-based VQA model 620 , the parameter-based VQA model 610 outputting a first score 640 , and the sample-based VQA model 620 outputting a second score 650 based on evaluating a plurality of images 630 .

[0070] According to an embodiment, the VQA algorithm 600 may further include complexity metadata 670 received by the parameter-based VQA model 610. The complexity metadata 670 may include information about the complexity of the video, such as spatial textual complexity, temporal motion complexity, etc. The complexity of the video may be evaluated by the encoder and included in the video bitstream sent from the encoder to the decoder. The decoder may not require the complexity metadata 670 to perform decoding. However, one or more embodiments are not limited to this, and the complexity metadata may be sent to the decoder separately from the video bitstream via a network. Similarly, instead of complexity metadata, information about the complexity of the video may be transmitted via a shutter interval information (SEI) message, and the decoder does not need to perform decoding on the SEI message.

[0071] Based on receiving the complexity metadata including information about the complexity of the video, the parameter-based VQA model 610 may adjust the first score 640 based on the information about the complexity of the video. For example, if the complexity of the video is high, the first score 640 output from the parameter-based VQA model 610 may be adjusted to be lower. Alternatively, if the complexity of the video is low, the first score 640 output from the parameter-based VQA model 610 may be adjusted to be higher.

[0072] Figure 7 FIG. 4 is a flowchart showing the operation of video quality assessment based on video complexity according to an embodiment. Figure 4 and Figure 5 Some features or operations of methods for performing video quality assessment are described. Figure 7 The operations of S710 to S750 and S770 may correspond to Figure 5 Therefore, repeated description of S710 to S750 and S770 may be omitted.

[0073] refer to Figure 7 In S760, the method 700 may further include adjusting the first score according to information about the complexity of the video. The information about the complexity of the video may include information such as spatial text complexity, temporal motion complexity, etc. The complexity of the video may be evaluated by the encoder and included in the video bitstream sent from the encoder to the decoder. Moreover, instead of metadata, the information about the complexity of the video may be transmitted via an SEI message, and the decoder does not need to perform decoding on the SEI message.

[0074] Based on the complexity information of the received video, the first score of the parameter-based VQA model can be adjusted based on the information about the complexity of the video. Figure 6If the complexity of the video is high, the first score 640 output from the parameter-based VQA model 610 may be adjusted to be lower. Alternatively, if the complexity of the video is low, the first score 640 output from the parameter-based VQA model 610 may be adjusted to be higher. Here, the first score may be adjusted based on a predetermined threshold. For example, if the temporal motion complexity of the video is higher than a predetermined threshold, the first score may be adjusted to be lower.

[0075] Figure 8 A method for executing the Figure 3 A block diagram of a computer program code for the method. Figure 8 The computer program code of the present invention can be stored in the memory and can be accessed by at least one processor to execute the stored computer program code. Figure 2 and 3 Some features or operations of the method for performing video quality assessment have been described. Therefore, repeated description of computer program codes for these features or operations may be omitted.

[0076] The computer program code 800 may include an acquisition code 810 , a first determination code 820 , a second determination code 830 , and an output code 840 .

[0077] The acquisition code 810 may be configured to acquire images of a video to be analyzed.

[0078] The first determination code 820 may be configured to determine a first image to which parameter-based VQA should be applied. The first image may include all images of a plurality of images in the video. For example, the first image may be Figure 2 , A′, B, B′, C, and C′ in . However, one or more embodiments are not limited thereto, and parameter-based VQA may be selectively applied to some of the plurality of images. The first determination code 820 may be configured to determine a first score based on applying parameter-based VQA to the first image.

[0079] The second determination code 830 may be configured to determine a second image to which parameter-based VQA should be applied. Determining the second image may cause sample-based VQA to be selectively applied to one or more of the plurality of images. For example, the second image may be Figure 2 The second determining code 830 may be configured to determine a second score based on applying a sample-based VQA to the second image.

[0080] The output code 840 may be configured to output a final score based on the determined first score and the determined second score.

[0081] Fig. 9A method for executing the Figure 5 A block diagram of a computer program code for the method. Fig. 9 The computer program code of the present invention can be stored in the memory and can be accessed by at least one processor to execute the stored computer program code. Figure 4 and 5 Some features or operations of the method for performing video quality assessment have been described. Therefore, repeated description of these features or operations may be omitted.

[0082] The computer program code 900 may include an acquisition code 910 , a first determination code 920 , a second determination code 930 , a calibration code 940 , and an output code 950 .

[0083] The acquisition code 910 may be configured to acquire images of a video to be analyzed.

[0084] The first determination code 920 may be configured to determine a first image to which parameter-based VQA should be applied. The first image may include all images of a plurality of images in the video. For example, the first image may be Figure 2 , A′, B, B′, C, and C′ in the image. However, one or more embodiments are not limited thereto, and parameter-based VQA may be selectively applied to some of the plurality of images. The first determination code 920 may be configured to determine a first score based on applying parameter-based VQA to the first image.

[0085] The second determination code 930 may be configured to determine a second image to which parameter-based VQA should be applied. Determining the second image may cause sample-based VQA to be selectively applied to one or more of the plurality of images. For example, the second image may be Figure 2 The second determination code 930 may be configured to determine a second score based on applying a sample-based VQA to the second image.

[0086] The calibration code 940 may be configured to calibrate a first score of the parameter-based VQA model based on a second score of the sample-based VQA model.

[0087] The output code 950 may be configured to output a final score based on the determined first score and the determined second score.

[0088] Fig.10 A method for executing the Figure 7 A block diagram of a computer program code for the method. Fig.10 The computer program code of the present invention can be stored in the memory and can be accessed by at least one processor to execute the stored computer program code. Figures 6 to 7Some features or operations of the method for performing video quality assessment have been described. Therefore, repeated description of these features or operations may be omitted.

[0089] The computer program code 1000 may include a first acquisition code 1010 , a second acquisition code 1020 , a first determination code 1030 , a second determination code 1040 , an adjustment code 1050 , and an output code 1060 .

[0090] The first acquisition code 1010 may be configured to acquire an image of a video to be analyzed.

[0091] The second acquisition code 1020 can be configured to obtain information about the complexity of the video to be analyzed. Information about the complexity of the video may include information such as spatial text complexity, temporal motion complexity, etc. The complexity of the video can be evaluated by the encoder and included in the video bitstream sent from the encoder to the decoder. It is also possible not to perform decoding of metadata about the video to be decoded via the decoder. In addition, information about the complexity of the video can be transmitted via metadata or SEI messages, and the decoder does not need to perform decoding on the SEI message.

[0092] The first determination code 1030 may be configured to determine a first image to which parameter-based VQA should be applied. The first image may include all images of a plurality of images in the video. For example, the first image may be Figure 2 , A′, B, B′, C, and C′ in the image. However, one or more embodiments are not limited thereto, and parameter-based VQA may be selectively applied to some of the plurality of images. The first determination code 1030 may be configured to determine a first score based on applying parameter-based VQA to the first image.

[0093] The second determination code 1040 may be configured to determine a second image to which parameter-based VQA should be applied. Determining the second image may cause sample-based VQA to be selectively applied to one or more images of the plurality of images. For example, the second image may be Figure 2 The second determination code 1040 may be configured to determine a second score based on applying a sample-based VQA to the second image.

[0094] The adjustment code 1050 may be configured to adjust the first score according to information about the complexity of the video.

[0095] The output code 1060 may be configured to output a final score based on the determined first score and the determined second score.

[0096] Some embodiments of the present disclosure have been shown and described above. However, one or more embodiments of the present disclosure are not limited to the above-mentioned specific embodiments. It is understood that various modifications, substitutions, improvements and equivalents may be made thereto without departing from the spirit and scope of the present disclosure. It should be understood that such modifications, substitutions, improvements and their equivalents shall belong to the scope of protection of the present disclosure and shall not be interpreted independently of the inventive concept or prospect of this separation.

Claims

1. A method for performing video quality assessment (VQA), characterized in that: The method comprises: obtaining multiple images of a video; determining a first image to which parameter-based VQA is to be applied, the first image comprising all images of the plurality of images; determining a first score for each of the first images based on applying the parameter-based VQA to each of the first images; Determine a second image among the plurality of images, to which sample-based VQA is to be applied, the second image being a partial image among the plurality of images; determining a second score for each of the second images based on applying the sample-based VQA to each of the second images; and outputting a final score of at least one image among the plurality of images based on the first score and the second score; Based on determining that the second score is not applicable to at least one of the first images, calibrating the first score of the at least one of the first images, wherein The calibrating the first score of the at least one of the first images comprises: obtaining one or more first pictures in a sequence of the video, the one or more first pictures being earlier in decoding order than the at least one of the first pictures; obtaining one or more second images corresponding to the one or more first images; calculating a difference between a first scoring system for the parameter-based VQA and a second scoring system for the sample-based VQA based on the first scores for the one or more first images and the second scores for the one or more second images; and The first score for the at least one of the first images is calibrated based on the difference between the first scoring system and the second scoring system.

2. The method according to claim 1, wherein: The multiple images are divided into one or more groups; and determining the second image among the multiple images includes: selecting at least one image in each of the one or more groups, and using the selected at least one image as the second image.

3. The method according to claim 1, wherein: The determining the second image from the plurality of images includes: selecting an image from the plurality of images at a predetermined time interval or a predetermined frame interval, and using the selected image as the second image.

4. The method according to claim 1, wherein: The determining a second score for each of the second images based on applying the sample-based VQA to each of the second images comprises: Based on bitstream level information, the sample-based VQA is applied to each of the second images to determine a second score for each of the second images, wherein the bitstream level information includes at least one of the following: bit rate, quantization parameter, frame resolution and frame rate.

5. The method according to claim 1, wherein: Determining a first score for each of the first images based on applying the parameter-based VQA to each of the first images comprises: Based on pixel-level information of each image in the first images, the parameter-based VQA is applied to each image to determine a first score for each image, wherein the pixel-level information of each image includes: pixel color and pixel position.

6. The method according to any one of claims 1 to 5, further comprising: receiving information about the complexity of the video; as well as The first score is adjusted based on the information about the complexity of the video.

7. The method according to claim 6, wherein: The information about the complexity of the video includes at least one of: spatial text information of the video, temporal motion information of the video, and resolution of the video.

8. The method according to claim 6, wherein: The information about the complexity of the video is included in at least one of: metadata and a shutter interval information SEI message.

9. The method according to any one of claims 1 to 5, characterized in that At least one first image of the first images and at least one second image of the second images are identical.

10. A device for performing video quality assessment (VQA), comprising: at least one memory storing computer program code; and At least one processor is configured to access the at least one memory and operate according to the instructions of the computer program code to perform the method according to any one of claims 1 to 9.

11. An apparatus for performing video quality assessment (VQA), comprising: An acquisition module obtains multiple images of a video; a first determination module, determining a first image to which parameter-based VQA is to be applied, the first image comprising all images of the plurality of images, and determining a first score for each of the first images based on applying the parameter-based VQA to each of the first images; a second determination module, determining a second image among the plurality of images, to which the sample-based VQA is to be applied, and determining a second score for each of the second images based on applying the sample-based VQA to each of the second images; as well as an output module, outputting a final score of at least one image among the plurality of images based on the first score and the second score; Based on determining that the second score is not applicable to at least one of the first images, calibrating the first score of the at least one of the first images, wherein The calibrating the first score of the at least one of the first images comprises: obtaining one or more first pictures in a sequence of the video, the one or more first pictures being earlier in decoding order than the at least one of the first pictures; obtaining one or more second images corresponding to the one or more first images; calculating a difference between a first scoring system for the parameter-based VQA and a second scoring system for the sample-based VQA based on the first scores for the one or more first images and the second scores for the one or more second images; and The first score for the at least one of the first images is calibrated based on the difference between the first scoring system and the second scoring system.

12. A non-transitory computer-readable storage medium storing a computer program code, wherein when the computer program code is executed by at least one processor, the at least one processor is configured to perform the method according to any one of claims 1 to 9.

Images