Method and system for sensing and evaluating compression effect in high efficiency video coding

Abstract

The invention provides a compression effect perception and evaluation method and system in high-efficiency video coding, and the method comprises the steps: constructing a deep compression effect perception model combining attention learning and data reweighting, inputting positive and negative samples of each type of compression effect into a network, and carrying out the training of the positive and negative samples to obtain a corresponding perception model of the compression effect; calculating a video compression effect intensity value based on a compression effect perception model obtained by training and considering the influence of visual saliency and a visual fovea effect; and calculating an average subjective score of the video through a subjective test so as to establish a mapping relation between the video compression effect intensity value and the average subjective quality score of the video, thereby realizing compression effect perception-oriented video quality evaluation.

Description

Technical Field

[0001] The present invention belongs to the technical field of video quality evaluation, and in particular relates to a method and system for perceiving and evaluating compression effects in efficient video coding. Background Art

[0002] Videos are usually compressed to save storage and transmission bandwidth, and popular lossy video compression inevitably leads to video compression effects, also known as perceptual coding artifacts. The compression effect affects the user's subjective perception of the video and significantly reduces the user's video experience quality. However, there are many types of video compression effects and their characteristics are complex, making it difficult to construct a perceptual model by manually extracting features. Existing methods usually lack comprehensive consideration of spatiotemporal compression effects, have great defects in perceiving and integrating compression effect features, and cannot provide reliable data references for compressed video quality assessment, thus having limitations in compressed video quality evaluation. Therefore, perceiving and analyzing compression effects in efficient video coding, exploring more accurate perception methods, and analyzing compressed video quality will help promote the optimization and improvement of video coding compression technology. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a method and system for perceiving and analyzing compression effects in efficient video coding, wherein the method comprises the following steps: Step S1: constructing a deep compression effect perception model, inputting positive and negative samples of each type of compression effect into the network, and respectively training to obtain corresponding perception and recognition models of the compression effect; Step S2: calculating the video compression effect strength value based on the compression effect perception model and taking into account the influence of visual saliency and visual foveal effect; Step S3: calculating the average subjective score of the video through subjective testing, and establishing a mapping relationship between the video compression effect strength value and the video average subjective quality score; finally, according to the mapping relationship between the measurement value of the compression effect strength and the video average subjective quality score, realizing video quality evaluation oriented to compression effect perception; it can realize accurate compression effect perception to guide video quality assessment.

[0004] The technical solution specifically adopted by the present invention to solve the technical problem is:

[0005] A method for perceiving and evaluating compression effects in efficient video coding:

[0006] Construct a deep compression effect perception model that combines attention learning and data reweighting, input positive and negative samples of each type of compression effect into the network, and train them to obtain the corresponding compression effect perception model;

[0007] Based on the compression effect perception model obtained through training, and taking into account the influence of visual saliency and visual foveal effect, the video compression effect intensity value is calculated;

[0008] Through subjective testing, the average subjective score of the video is calculated to establish a mapping relationship between the video compression effect intensity value and the average subjective quality score of the video, thereby realizing video quality evaluation oriented to compression effect perception.

[0009] Furthermore, when training the deep compression effect perception model, the video frame samples including multiple compression effects are preprocessed as input, and the sample data is input into the YOLO detection backbone network to extract the compression effect features. The compression effect features are used to create a feature pyramid through the Transformer encoding block, and then the feature fusion is realized through the multi-scale structure, and the data is reweighted to solve the long-tail distribution problem of the samples.

[0010] The trained network model is verified to confirm whether it meets the preset requirements. If so, the model is saved as a compression effect detection model. If not, it is retrained until it meets the preset requirements.

[0011] Furthermore, the types of the compression effect samples include blur effect samples, block effect samples, ringing effect samples, color spill effect samples, flickering effect samples and floating effect samples.

[0012] Furthermore, the compression effect perception model obtained through training and considering the influence of visual saliency and visual foveal effect, calculating the video compression effect strength value includes:

[0013] Total pixel value of compression effect of video based on perceptual model statistics;

[0014] Calculate the saliency factor and fovea factor by considering visual saliency and visual concave effect;

[0015] Then calculate the weighted compression effect intensity value.

[0016] Furthermore, the total pixel value of the compression effect of the video based on the perception model statistics is specifically:

[0017] The compressed video to be tested is used as the input of the trained compression effect perception model, and the area where the compression effect is located in each video frame is detected, and then the total pixel value I of the compression effect of each video frame is counted. p :

[0018] Where T represents the type of compression effect, M represents the total number of compression effect areas in the video frame, w_box and h_box represent the width and height of the detection box respectively;

[0019] The calculation considers visual saliency and visual concave effect to calculate the saliency factor and the foveal factor specifically as follows:

[0020] By introducing the video saliency detection algorithm ACLNet, the region of interest in each video frame is extracted, and the area of ​​the salient region in each video frame is counted, corresponding to the saliency detection factor I S ;

[0021] Combined with the detection box height of the perception model, the visual fovea factor I is calculated E :

[0022]

[0023] Where H represents the height of the video frame, and y' represents the height of the prediction box of the perception model;

[0024] The weighted compression effect intensity value is specifically calculated as:

[0025] Based on the detection results of the compression effect perception model, considering visual saliency and visual foveal effect, the weighted compression effect intensity value I of the entire compressed video is calculated:

[0026]

[0027] Where x and y represent the central horizontal and vertical coordinates of the detection frame of the perception model, respectively. P The total pixel value representing the compression effect, I S Represents the significance detection factor, I E represents the visual fovea factor, W and H represent the width and height of the video frame respectively, and N represents the total number of compressed video frames.

[0028] Furthermore, the subjective test adopts a double stimulation method. After the subjective test is completed, the MOS values ​​of the experimenters are screened; the collected subjective scores are processed, and the MOS value of each test video is calculated by the following formula: Where S i Indicates the score given by each tester to the same video; N represents the number of testers; then the correlation coefficient between each tester's score and MOS is calculated to ensure the validity of the obtained data; finally, the test data is screened according to ITU-RBT.1788.

[0029] Further, the correlation coefficient includes PLCC and SROCC;

[0030] First, input the PLCC and SRCC values ​​of all testers; and calculate the mean values ​​r1 and r2 of PLCC and SRCC;

[0031] Then determine whether r1 is greater than r2. If yes, take r2 and calculate the standard deviation s2 of SRCC. If no, take r1 and calculate the standard deviation s1 of PLCC.

[0032] Then calculate the difference q between r and s, and determine whether q is greater than the maximum threshold:

[0033] If yes, set the rejection threshold = maximum threshold, and then determine whether PLCC(i) / SRCC(i) is greater than the maximum threshold. If yes, output the elimination result. If no, eliminate the i-th tester.

[0034] If not, set the rejection threshold = q, and then determine whether PLCC(i) / SRCC(i) is greater than q. If so, output the elimination result. If not, eliminate the i-th tester.

[0035] The mapping relationship between the measurement value of the compression effect strength and the average subjective quality score of the video is specifically to establish a mapping relationship between the compression effect strength value I and the average subjective opinion score after screening.

[0036] A compression effect perception and evaluation system in efficient video coding, comprising:

[0037] The training module is used to train the deep compression effect perception model that combines attention learning and data reweighting. The positive and negative samples of each type of compression effect are input into the network, and the corresponding compression effect perception models are trained respectively.

[0038] A video compression effect strength value calculation module, used to calculate the video compression effect strength value based on the compression effect perception model obtained through training and taking into account the influence of visual saliency and visual foveal effect;

[0039] The subjective test module is used to calculate the average subjective score of the video through subjective testing.

[0040] The evaluation module is used to establish a mapping relationship between the video compression effect intensity value and the video average subjective quality score, so as to realize video quality evaluation oriented to compression effect perception.

[0041] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the compression effect perception and evaluation method in high-efficiency video coding are implemented as described above.

[0042] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for perceiving and evaluating compression effects in high-efficiency video coding as described above.

[0043] Compared with the prior art, the present invention and its preferred solution establish a mapping relationship between the video compression effect intensity value and the video average subjective quality score; based on the mapping relationship between the measurement value of the compression effect intensity and the video average subjective quality score, a video quality evaluation oriented to compression effect perception is realized; it can realize accurate compression effect perception to guide video quality assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0045] Figure 1 is an overall implementation flow chart of an embodiment of the present invention;

[0046] Figure 2 is a diagram of a model for video quality assessment guided by compression effect perception and analysis in an embodiment of the present invention;

[0047] Figure 3 4 is a flowchart of subjective data screening in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the features and advantages of the present invention more clearly understood, the following embodiments are specifically described in detail as follows:

[0049] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0050] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0051] like Figure 1 As shown, an embodiment of the present invention provides a design of a compression effect perception and analysis method in efficient video coding, including the following steps:

[0052] Step S1: construct a deep compression effect perception model, input positive and negative samples of each type of compression effect into the network, and train them to obtain corresponding compression effect perception and recognition models respectively;

[0053] Step S2: Based on the compression effect perception model and taking into account the influence of visual saliency and visual foveal effect, the video compression effect intensity value is calculated;

[0054] Step S3: Calculate the average subjective score of the video through subjective testing, and establish a mapping relationship between the video compression effect intensity value and the average subjective quality score of the video;

[0055] Finally, based on the mapping relationship between the measurement value of the compression effect intensity and the average subjective quality score of the video, video quality evaluation oriented to compression effect perception is achieved.

[0056] In this embodiment, if Figure 2 As shown in the figure, when training the deep compression effect perception model, six compression effect samples are used as input. The sample data is input into the backbone network to extract the basic features of the compression effect. The compression effect features are used to create a feature pyramid through the Transformer encoding block, and then feature fusion is achieved through a multi-scale structure. The problem of long-tail distribution of samples is solved through data reweighting. Finally, the compression effect is accurately detected to guide the evaluation of compressed video quality. Specifically:

[0057] Step S11: using samples in a large-scale compression effect database as data input for training the network;

[0058] Step S12: Using the YOLO detection network as the basic framework, attention learning and data reweighting are used to deal with the complex diversity of compression effects and the long-tail distribution problem of samples;

[0059] Step S13: perform model verification on the trained network model to confirm whether it meets the preset requirements. If so, save the model as a compression effect detection model. If not, retrain until it meets the preset requirements.

[0060] In this embodiment, the six types of compression effect samples include blur effect samples, block effect samples, ringing effect samples, color spill effect samples, flickering effect samples and floating effect samples.

[0061] In a specific example, the model training is as follows: in the process of detecting model training, for each category of perceptible coding effect, 6 categories of compression effect samples are selected from a large-scale database, totaling 37,130 samples, of which 80% are used as training sets and 20% are used as test sets, with a ratio of positive and negative samples of 1:1. In the process of training the model, the SGD optimizer is used, the epoch is set to 300 batches during training, the input image size is set to 640×640, the patch is set to 8, the learning rate is set to 0.01, and the IoU threshold is set to 0.5. In the process of model training, the adjustment of network hyperparameters, fine-tuning of network layer structure, and selection of network optimizer can improve the accuracy and performance of the network.

[0062] In this embodiment, step S2 is specifically as follows:

[0063] Step S21: Take the compressed video to be tested as the input of the perception model, detect the area where the compression effect is located in each video frame, and then count the total pixel value I of the compression effect of each video frame of the video p :

[0064] Where T represents the type of compression effect, M represents the total number of compression effect areas in the video frame, w_box and h_box represent the width and height of the detection box respectively;

[0065] Step S22: Considering that people usually pay more attention to the region of interest, this is an important factor affecting the perceived quality of the video. By introducing the video saliency detection algorithm ACLNet to extract the region of interest in each video frame, the area of ​​the salient region in each video frame is counted, that is, the saliency detection factor I S ;

[0066] Step S23: Considering the foveal effect formed by the difference in visual acuity between the central vision and peripheral vision of the human eye, the visual foveal factor I is calculated in combination with the detection frame height of the perception model. E :

[0067]

[0068] Where H represents the height of the video frame, and y' represents the height of the prediction box of the perception model;

[0069] Step S24: Based on the detection result of the compression effect perception model, the weighted compression effect intensity value I of the entire compressed video is calculated by considering visual saliency and visual foveal effect:

[0070]

[0071] Where x and y represent the central horizontal and vertical coordinates of the detection frame of the perception model, respectively. P The total pixel value representing the compression effect, I S Represents the significance detection factor, I E represents the visual fovea factor, W and H represent the width and height of the video frame respectively, and N represents the total number of compressed video frames.

[0072] In this embodiment, step S3 is specifically as follows:

[0073] Step S31: Use the double stimulation method in the subjective test method, that is, provide a reference video and a test video to the tester each time. The reference video and the test video are 10 seconds long, and a 1-second black screen is set between the videos. After the test video is played, a 3-second scoring time is set to allow the tester to subjectively score the video. The experimental environment is set and calibrated according to the recommendations of ITU-RBT.500;

[0074] Step S32: After the subjective experiment is completed, the MOS values ​​of the experimenters are screened. The collected subjective scores are processed and the MOS value of each test video is calculated as follows: Where S i Indicates the score given by each tester to the same video; N represents the number of testers;

[0075] Step S33: Calculate the correlation coefficient between each tester's score and MOS to ensure the validity of the obtained data;

[0076] Step S34: Screening the test data according to the calculation result of step S33 and ITU-R BT.1788;

[0077] Step S35: Establish a mapping relationship between the compression effect intensity value I obtained in step S3 and the average subjective opinion score after the screening process in step S34.

[0078] As a preferred solution, in this embodiment, the correlation coefficient includes PLCC and SROCC.

[0079] Among them, PLCC (Pearson linear correlation coefficient) and SROCC (Spearman rank correlation coefficient) are both indicators for measuring the correlation between subjective scores and objective scores.

[0080] PLCC is used to describe the linear correlation between two sets of subjective and objective quality evaluation data. Its value range is (-1,1). A value of 0 means that the two sets of data are completely unrelated, and a value of 1 or -1 means that the two sets of data are completely correlated.

[0081] SROCC is used to describe the consistency of subjective and objective evaluation results. The value range is (0,1). A value of 0 indicates that the subjective and objective evaluation results are completely inconsistent, and a value of 1 indicates that they are completely consistent. SROCC does not measure the linear correlation between two sets of data, but measures the correlation between the data orders.

[0082] like Figure 3 As shown, step S33 specifically includes the following implementation process:

[0083] First, input the PLCC and SRCC values ​​of all testers; and calculate the mean values ​​r1 and r2 of PLCC and SRCC;

[0084] Then determine whether r1 is greater than r2. If yes, take r2 and calculate the standard deviation s2 of SRCC. If no, take r1 and calculate the standard deviation s1 of PLCC.

[0085] Then calculate the difference q between r and s, and determine whether q is greater than the maximum threshold:

[0086] If yes, set the rejection threshold = maximum threshold, and then determine whether PLCC(i) / SRCC(i) is greater than the maximum threshold. If yes, output the elimination result. If no, eliminate the i-th tester.

[0087] If not, set the rejection threshold = q, and then determine whether PLCC(i) / SRCC(i) is greater than q. If so, output the elimination result. If not, eliminate the i-th tester.

[0088] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0089] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0093] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

[0094] The present invention is not limited to the above-mentioned optimal implementation mode. Anyone can derive other forms of compression effect perception and evaluation methods and systems in high-efficiency video coding under the inspiration of the present invention. All equal changes and modifications made within the scope of the invention application of the present invention should fall within the scope of the present invention.

Claims

1. A method for perceiving and evaluating compression effects in efficient video coding, characterized in that: Construct a deep compression effect perception model that combines attention learning and data reweighting, input positive and negative samples of each type of compression effect into the network, and train them to obtain the corresponding compression effect perception model; Based on the compression effect perception model obtained through training, and taking into account the influence of visual saliency and visual foveal effect, the video compression effect intensity value is calculated; Through subjective testing, the average subjective score of the video is calculated to establish a mapping relationship between the video compression effect intensity value and the average subjective quality score of the video, thereby realizing video quality evaluation oriented to compression effect perception.

2. The method for perceiving and evaluating compression effects in high-efficiency video coding according to claim 1, characterized in that: When training the deep compression effect perception model, video frame samples including multiple compression effects are preprocessed as input, and the sample data is input into the YOLO detection backbone network to extract compression effect features. The compression effect features are used to create a feature pyramid through the Transformer encoding block, and then feature fusion is achieved through a multi-scale structure. The data is reweighted to solve the long-tail distribution problem of samples.

3. The method for perceiving and evaluating compression effects in high-efficiency video coding according to claim 2, wherein: The types of the compression effect samples include blurring effect samples, blocking effect samples, ringing effect samples, color spill effect samples, flickering effect samples and floating effect samples.

4. The method for perceiving and evaluating compression effects in high-efficiency video coding according to claim 1, wherein: The compression effect perception model obtained through training and considering the influence of visual saliency and visual foveal effect, calculating the video compression effect intensity value includes: Total pixel value of compression effect of video based on perceptual model statistics; Calculate the saliency factor and fovea factor by considering visual saliency and visual concave effect; Then calculate the weighted compression effect intensity value.

5. The method for perceiving and evaluating compression effects in high-efficiency video coding according to claim 4, characterized in that: The total pixel value of the compression effect of the video based on the perception model statistics is specifically: The compressed video to be tested is used as the input of the trained compression effect perception model, and the area where the compression effect is located in each video frame is detected, and then the total pixel value I of the compression effect of each video frame is counted. p : Where T represents the type of compression effect, M represents the total number of compression effect areas in the video frame, w_box and h_box represent the width and height of the detection box respectively; The calculation considers visual saliency and visual concave effect to calculate the saliency factor and the foveal factor specifically as follows: By introducing the video saliency detection algorithm ACLNet, the region of interest in each video frame is extracted, and the area of ​​the salient region in each video frame is counted, corresponding to the saliency detection factor I S ; Combined with the detection box height of the perception model, the visual fovea factor I is calculated E : Where H represents the height of the video frame, and y' represents the height of the prediction box of the perception model; The weighted compression effect intensity value is specifically calculated as: Based on the detection results of the compression effect perception model, considering visual saliency and visual foveal effect, the weighted compression effect intensity value I of the entire compressed video is calculated: Where x and y represent the central horizontal and vertical coordinates of the detection frame of the perception model, respectively. P The total pixel value representing the compression effect, I S Represents the significance detection factor, I E represents the visual fovea factor, W and H represent the width and height of the video frame respectively, and N represents the total number of compressed video frames.

6. The method for perceiving and evaluating compression effects in high-efficiency video coding according to claim 1, wherein: The subjective test adopts the double stimulation method. After the subjective test is completed, the MOS values ​​of the experimenters are screened; the collected subjective scores are processed, and the MOS value of each test video is calculated by the following formula: Where S i Indicates the score given by each tester to the same video; N represents the number of testers; then the correlation coefficient between each tester's score and MOS is calculated to ensure the validity of the obtained data; finally, the test data is screened according to ITU-RBT.1788.

7. The method for perceiving and evaluating compression effects in high-efficiency video coding according to claim 6, characterized in that: The correlation coefficients include PLCC and SROCC; First, input the PLCC and SRCC values ​​of all testers; and calculate the mean values ​​r1 and r2 of PLCC and SRCC; Then determine whether r1 is greater than r2. If yes, take r2 and calculate the standard deviation s2 of SRCC. If no, take r1 and calculate the standard deviation s1 of PLCC. Then calculate the difference q between r and s, and determine whether q is greater than the maximum threshold: If yes, set the rejection threshold = maximum threshold, and then determine whether PLCC(i) / SRCC(i) is greater than the maximum threshold. If yes, output the elimination result. If no, eliminate the i-th tester. If not, set the rejection threshold = q, and then determine whether PLCC(i) / SRCC(i) is greater than q. If so, output the elimination result. If not, eliminate the i-th tester.

8. A system for perceiving and evaluating compression effects in efficient video coding, characterized in that: include: The training module is used to train the deep compression effect perception model that combines attention learning and data reweighting. The positive and negative samples of each type of compression effect are input into the network, and the corresponding compression effect perception models are trained respectively. A video compression effect strength value calculation module, used to calculate the video compression effect strength value based on the compression effect perception model obtained through training and taking into account the influence of visual saliency and visual foveal effect; The subjective test module is used to calculate the average subjective score of the video through subjective testing. The evaluation module is used to establish a mapping relationship between the video compression effect intensity value and the video average subjective quality score, so as to realize video quality evaluation oriented to compression effect perception.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of a method for perceiving and evaluating compression effects in high-efficiency video coding as described in any one of claims 1-7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for perceiving and evaluating compression effects in high-efficiency video coding as described in any one of claims 1 to 7 are implemented.

Images