Method for evaluating quality of light field image based on primary and secondary hierarchical evaluation

Abstract

The application relates to a light field image quality evaluation method based on primary and secondary hierarchical evaluation, which comprises the following steps: step S1: initializing a model, and splitting the model and a perception score; step S2: for each sub-aperture image of a light field image, inputting the image into the model to obtain an overall quality evaluation and a detail quality, and further obtaining a quality score of the sub-aperture image; step S3: repeating step S2 until all the sub-aperture images in the light field image are traversed, and taking the mean value of the sub-aperture images as the quality score of the light field image; step S4: deducing a loss function in the secondary training, and optimizing the loss function proportion of the primary evaluation process and the loss function proportion of the secondary process by using an optimizer; and step S5: repeating steps S2 to S4 until the network weight of a neural network is stable, and the goal of the light field image quality evaluation is achieved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for evaluating the quality of light field images based on primary and secondary hierarchical evaluation. Background Technology

[0002] Light field imaging is gaining popularity due to its potential applications in various fields such as virtual reality, gaming, and digital cinema. However, the success of these applications largely depends on the quality of the light field images. Ensuring the quality of experience (QoE) of the light field is crucial in the processing and application of light fields to provide users with an immersive viewing experience. While traditional two-dimensional or three-dimensional image quality assessment (IQA) methods are commonly used, they cannot fully capture the unique properties of light field images. Furthermore, subjective assessment methods are costly and impractical for most applications.

[0003] Recent research has attempted to address the problem of light field image (LFI) using modern methods. These studies have extensively analyzed angular consistency, global naturalness, and other aspects of light field image composition, proposing a series of machine learning-based or statistical analysis-based methods to improve the accuracy of light field image prediction. However, these methods primarily focus on pixel-independent features, neglecting the dependencies between image pixels and information about pixel group composition, which contradicts the subjective evaluation process of humans. Therefore, the results still require improvement. Furthermore, these methods generally suffer from long prediction times and poor portability, thus necessitating the development of novel approaches. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a light field image quality assessment method based on primary and secondary hierarchical evaluation, which can effectively and accurately predict light field images.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a light field image quality assessment method based on primary and secondary hierarchical evaluation, comprising the following steps:

[0006] Step S1: Initialize the model and separate the model and perception score;

[0007] Step S2: For each sub-aperture image of the light field image, input the image into the model to obtain the overall quality evaluation and detail quality, and then obtain the quality score of the sub-aperture image;

[0008] Step S3: Repeat step S2 until all sub-aperture images in the light field image have been traversed, and use the average value of these sub-aperture images as the quality score of the light field image.

[0009] Step S4: Estimate the loss function in this training, and use the optimizer to optimize the proportion of the loss function for the main evaluation process and the proportion of the loss function for the secondary process;

[0010] Step S5: Repeat steps S2 to S4 until the network weights of the neural network are stable, achieving the goal of evaluating the quality of the light field image.

[0011] In a preferred embodiment, step S1 specifically comprises:

[0012] Step S11: For each model, divide the first half of the model into an overall evaluation part and the second half into a detailed evaluation part. The output of the overall evaluation part is denoted as M. main The output of the detailed evaluation part is denoted as M. bias ;

[0013] Step S12: Divide the known perception score into integer parts Score Int and decimal part Score Dec .

[0014] In a preferred embodiment, step S2 specifically comprises:

[0015] Step S21: For the light field data set One of the light field images For U, select all sub-aperture images SAI∈U;

[0016] Step S22: For the count_1th SAI in set U, input it into the model processed in step S1. According to step one, the overall evaluation part of this input result is called... and

[0017] Step S23: Record the output result M for this step. i ,in,

[0018]

[0019] In a preferred embodiment, step S3 specifically comprises:

[0020] Step S31: Repeat step S2 until the traversal of set U is complete;

[0021] Step S32: Estimate the overall mass fraction M of set U main and detail quality score M bias M main and M bias The calculation method follows the formula below:

[0022]

[0023]

[0024] Where N represents the number of elements in set U.

[0025] In a preferred embodiment, step S4 specifically involves:

[0026] Step S41: Set the image size for a single training session to [value]. Collect the perceptual scores of the generated images in step S3, and divide the scores into sub-scores according to the method in step S1. Int and Score Dec ;

[0027] Step S42: Equip the network from S1 with the loss function LOSS and the optimizer OPT.

[0028] The loss function is constructed in accordance with the following function:

[0029] LOSS=αMSE(M main Score Int )+βMSE(M bias Score Dec )

[0030]

[0031] Among them, y i and The two parameters accepted by the MSE function are y i ∈M and α and β are two trainable parameters with an initial value of 1;

[0032] The optimizer is constructed according to the following function:

[0033]

[0034]

[0035] Among them, g t The gradient of the parameters is β1, and β2 are the decay coefficients of the two exponentially weighted averages. and It is the bias-corrected moving average of the gradient, θ t+1 These are the updated parameters, η is the learning rate, and ∈ is a very small constant used to avoid division by zero;

[0036] Step S43: Backpropagate the calculated loss function result through the optimizer to obtain the result of this optimization.

[0037] In a preferred embodiment, step S5 specifically involves:

[0038] Step S51: Repeat the operations on the light field image set in S2 to S4 until the weights and loss function coefficients α and β of the neural network reach stability;

[0039] Step S52: Use the validation set to estimate the generalization error during or after training, and update the hyperparameters.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The proposed method utilizes an object detection network for LFIQA, achieving both accurate LFI quality score prediction and simplified preprocessing. Experimental results show that this method also significantly improves inference time compared to existing models.

[0042] 2. By combining a hierarchical evaluation process with an automatically adjusted loss function ratio, the proposed method effectively adjusts the ability of the experimental backbone network to capture image subject and detail features, while suppressing jitter in the prediction process.

[0043] 3. The general strategy proposed in this invention is applicable to a variety of backbone networks. It not only ensures the improvement of the strategy's performance indicators, but also greatly shortens the inference time and improves the generalization performance of the model by relying on the performance advantages of the backbone network. Attached Figure Description

[0044] Figure 1 This is a flowchart of a preferred embodiment of the present invention. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0047] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0048] Please refer to Figure 1 A method for evaluating the quality of light field images based on primary and secondary hierarchical evaluation includes the following steps:

[0049] Step S1: Initialize the model and separate the model and perception scores.

[0050] Step S2: For each sub-aperture image of the light field image, input the image into the model to obtain the overall quality evaluation and detail quality, and then obtain the quality score of the sub-aperture image.

[0051] Step S3: Repeat step S2 until all sub-aperture images in the light field image have been traversed, and use the average value of these sub-aperture images as the quality score of the light field image.

[0052] Step S4: Estimate the loss function in this training, and use the optimizer to optimize the proportion of the loss function for the main evaluation process and the proportion of the loss function for the secondary process;

[0053] Step S5: Repeat steps S2 to S4 until the network weights of the neural network are stable, achieving the goal of evaluating the quality of the light field image.

[0054] Step S1 specifically involves:

[0055] Step S11: For each model, divide the first half of the model into an overall evaluation part and the second half into a detailed evaluation part. The output of the overall evaluation part is denoted as M. main The output of the detailed evaluation part is denoted as M. bias .

[0056] Step S12: Divide the known perception score into integer parts Score Int and decimal part Score Dec .

[0057] Step S2 specifically involves:

[0058] Step S21: For the light field data set One of the light field images For U, select all sub-aperture images SAI∈U.

[0059] Step S22: For the count_1th SAI in set U, input it into the model processed in step S1. According to step one, the overall evaluation part of this input result is called... and

[0060] Step S23: Record the output result M for this step. i ,in,

[0061]

[0062] Step S3 specifically involves:

[0063] Step S31: Repeat step S2 until the traversal of set U is complete.

[0064] Step S32: Estimate the overall mass fraction M of set U main and detail quality score M bias M main and M bias The calculation method follows the formula below:

[0065]

[0066]

[0067] Where N represents the number of elements in set U.

[0068] Step S4 specifically involves:

[0069] Step S41: Set the image size for a single training session to [value]. Collect the perceptual scores of the generated images in step S3, and divide the scores into sub-scores according to the method in step S1. Int and Score Dec .

[0070] Step S42: Equip the network from S1 with the loss function LOSS and the optimizer OPT.

[0071] The loss function is constructed in accordance with the following function:

[0072] LOSS=αMSE(M main Score Int )+βMSE(M bias Score Dec )

[0073]

[0074] Among them, y i and The two parameters accepted by the MSE function are y i ∈M and α and β are two trainable parameters with an initial value of 1.

[0075] The optimizer is constructed according to the following function:

[0076]

[0077]

[0078] Among them, g tThe gradient of the parameters is β1, and β2 are the decay coefficients of the two exponentially weighted averages. and It is the bias-corrected moving average of the gradient, θ t+1 These are the updated parameters, η is the learning rate, and ∈ is a very small constant used to avoid division by zero.

[0079] Step S43: Backpropagate the calculated loss function result through the optimizer to obtain the result of this optimization.

[0080] Step S5 specifically involves:

[0081] Step S51: Repeat the operations on the light field image set in S2 to S4 until the weights and loss function coefficients α and β of the neural network reach stability.

[0082] Step S52: Use the validation set to estimate the generalization error during or after training, and update the hyperparameters.

Claims

1. A light field image quality assessment method based on primary and secondary hierarchical evaluation, characterized in that, Includes the following steps: Step S1: Initialize the model and separate the model and perception score; Step S2: For each sub-aperture image of the light field image, input the image into the model to obtain the overall quality evaluation and detail quality, and then obtain the quality score of the sub-aperture image; Step S3: Repeat step S2 until all sub-aperture images in the light field image have been traversed, and use the average value of these sub-aperture images as the quality score of the light field image. Step S4: Estimate the loss function in this training, and use the optimizer to optimize the proportion of the loss function for the main evaluation process and the proportion of the loss function for the secondary process; Step S5: Repeat steps S2 to S4 until the network weights of the neural network are stable, achieving the goal of light field image quality evaluation; Step S1 specifically involves: Step S11: For each model, divide the first half of the model into an overall evaluation part and the second half into a detailed evaluation part. Record the output of the overall evaluation part as... The output of the detailed evaluation section is recorded as follows: ; Step S12: Transfer the known perception scores Divide into integer parts and decimal part ; Step S4 specifically involves: Step S41: Set the image size for a single training session to [value]. Collect the perceptual scores of the images generated in step S3. ,Will According to the method in step S1, it is divided into and ; Step S42: Equip the network from S1 with the loss function LOSS and the optimizer OPT. The loss function is constructed in accordance with the following function: MSE , ) in, and These are the two parameters accepted by the MSE function. and ; and There are two trainable parameters, and their initial values ​​are 1; The optimizer is constructed according to the following function: OPT = in, It is the gradient of the parameters. and It is the decay coefficient of the two exponentially weighted averages. and It is the moving average after gradient bias correction. These are the updated parameters. It's the learning rate. It is a very small constant used to avoid division by zero; Step S43: Backpropagate the calculated loss function result through the optimizer to obtain the result of this optimization.

2. The light field image quality assessment method based on primary and secondary hierarchical evaluation according to claim 1, characterized in that, Step S2 specifically involves: Step S21: For the light field data set One of the light field images ,for Select all sub-aperture images included. ; Step S22: For the set The input is then fed into the model processed in step S1. According to step one, the overall evaluation of this input result is referred to as... and ; Step S23: Record the output result. ,in, 。 3. The light field image quality assessment method based on primary and secondary hierarchical evaluation according to claim 1, characterized in that, Step S3 specifically involves: Step S31: Repeat step S2 until... Collection traversal complete; Step S32: Estimated set Overall quality score and detail quality score , and The calculation method follows the formula below: Where N represents the set The number of elements in the array.

4. The light field image quality assessment method based on primary and secondary hierarchical evaluation according to claim 1, characterized in that, Step S5 specifically involves: Step S51: Repeat the operations on the light field image set in S2 to S4 until the weights and loss function coefficients of the neural network are finalized. and To achieve stability; Step S52: Use the validation set to estimate the generalization error during or after training, and update the hyperparameters.

Images