Reference-free light field image quality assessment method and system based on deep meta-learning

Abstract

The present invention provides a method and system for evaluating the quality of a light field image without reference based on deep meta-learning, respectively constructing a training set including a quality evaluation task without reference and a test set including a target evaluation task; constructing a deep meta-learning model including a Swin Transformer network structure; based on the training set, learning prior knowledge of distortion to obtain a quality prior model; fine-tuning the quality prior model on the test set to obtain a final light field image quality evaluation model; and using the light field image quality evaluation model to evaluate the quality of the light field image to be evaluated. The present invention overcomes the problem of insufficient combination of spatial and angular information of light field images, which leads to a large difference between experimental results and real results, as well as the fitting problem and inadaptability to various distortions that may be caused by using deep learning, and adapts to each type of distortion, with greatly enhanced performance.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and system for evaluating quality of a reference-free light field image based on deep meta-learning. A corresponding computer terminal and a computer-readable storage medium are also provided. Background Art

[0002] With the development of immersive multimedia technology and virtual reality technology, light field technology has become a research hotspot in the field of computer vision and has attracted the attention of many relevant researchers in recent years. Since light field is a collection of light in space, the three-dimensional (3D) world can be intuitively reproduced by acquiring and displaying light field images. Due to its rich spatial and angular information, light field is widely used in image applications, including depth estimation, refocusing, 3D reconstruction, etc. However, distortion effects are inevitable in each link of light field processing, which leads to a decrease in the perceived quality of light field content. In order to guide and supervise the acquisition, processing and application of light field images (LFI), it is crucial to design a light field image quality assessment (LFIQA) model that conforms to the human visual system.

[0003] However, most of the existing light field image quality assessment models are based on traditional machine learning to manually extract features, and then use quality regression models (such as support vector machines (SVR)) to map the input image features to their quality scores. Although the effect is very good, it does not achieve fully automatic objective evaluation and does not bring much convenience. At the same time, research has found that image quality assessment using deep convolutional neural networks has achieved good results on low-dimensional image assessment models. Unfortunately, LFIQA is a small sample problem, and most light field datasets contain relatively little image content. Therefore, most existing deep learning-based image quality assessment (IQA) models cannot adapt well to high-dimensional LFIQA models, resulting in underfitting or overfitting problems when evaluating different types of distortions. Moreover, many IQA models do not take into account the extremely rich information of light field images, and do not combine their spatial information and angle information for evaluation, resulting in poor results when evaluating certain distortions that destroy angle information such as reconstruction. At the same time, since image quality assessment relies on human eye recognition, it is particularly important whether the model conforms to the visual effects of the human eye. Some models do not combine the human eye recognition characteristics well for image quality assessment, so their effects cannot reach a certain level.

[0004] After searching, we found:

[0005] The Chinese invention patent application with publication number CN115937064A, "A method for evaluating the quality of light field images based on spatial and angular measurements", includes: extracting the spatial features of the light field image from the sub-aperture image array of the 4-dimensional light field image using a multi-band local binary pattern algorithm; extracting the features of the microlens image array of the 4-dimensional light field image as the angular features of the light field image using a feature extraction algorithm based on entropy weighted local phase quantization; performing feature fusion on the spatial features of the light field image and the angular features of the light field image to obtain a one-dimensional feature vector; performing support vector regression pooling on the one-dimensional feature vector to obtain the quality score of the light field image. The present invention extracts features from the sub-aperture image and the macro-pixel respectively to effectively quantify the spatial quality and angular consistency of the light field image, thereby avoiding the problem that the quality of the light field image is restricted by the spatial quality and angular consistency. This method still has the following technical problems:

[0006] This method still uses the traditional method of manual feature extraction. After the manual feature extraction, it is necessary to use regression methods such as support vector regression machine to map it to the quality score of the image.

[0007] This method does not adapt well to images with different distortion types and may result in lower scores on some distortions. Summary of the invention

[0008] In view of the above-mentioned deficiencies in the prior art, the present invention provides a reference-free light field image quality assessment method and system based on deep meta-learning.

[0009] According to one aspect of the present invention, a method for evaluating quality of a reference-free light field image based on deep meta-learning is provided, comprising:

[0010] Construct a training set containing a no-reference light field image quality assessment task and a test set containing a target assessment task respectively;

[0011] Build a deep meta-learning model containing the Swin Transformer network structure;

[0012] Based on the training set, learning prior knowledge of distortion to obtain a quality prior model;

[0013] Fine-tuning the quality prior model on the test set to obtain a final light field image quality assessment model;

[0014] The light field image quality assessment model is used to perform light field image quality assessment on the light field image to be assessed.

[0015] Preferably, the separately constructing of a training set including a no-reference light field image quality assessment task and a test set including a target assessment task comprises:

[0016] The light field images are divided into multiple non-reference light field image quality assessment tasks according to their distortion types, which are used to construct a training set; wherein, in the same data set, light field images with the same distortion type are divided into a group, and each group is used as a light field image quality assessment task;

[0017] Light field images with unknown distortion types are used as target evaluation tasks to construct the test set;

[0018] The light field images used for the no-reference light field image quality assessment task and the light field images used for the target assessment task are converted into MacPI modes respectively, and the training set and test set required for the no-reference light field image quality assessment are obtained.

[0019] Preferably, the construction of a deep meta-learning model including a Swin Transformer network structure comprises:

[0020] Construct a Swin Transformer network and a fully connected layer network; among them:

[0021] The Swin Transformer network includes: a PatchEmbed module, four basicLayer modules and a Linear layer connected in sequence; wherein the first three basicLayer modules each include a Swin Transformer block submodule and a Patch Merging submodule, and the fourth basicLayer module includes a Transformer block submodule; and finally connected to a fully connected layer;

[0022] The PatchEmbed module is used to divide the input light field image of dimension H x W x C into N patches of size P*P*C and then flatten them, where Then input to the SwinTransformer block submodule of the first basicLayer module;

[0023] The Patch Merging submodule is used to reduce the image resolution and adjust the number of channels after each of the swin Transformer block submodules to form a hierarchical image;

[0024] The fully connected layer network is used to map the features of the image output by the Swin Transformer network to the output result, and the final output value is the predicted quality score value of the light field image.

[0025] Preferably, the numbers of Swin Transformer block sub-modules in the first three basicLayer modules are 2, 2 and 18 respectively.

[0026] Preferably, the number of Transformer block sub-modules in the fourth basicLayer module is 2.

[0027] Preferably, the fully connected layer network includes 3 fully connected layers, and the image output by the Swin Transformer network maps the features of the light field image layer by layer to the final output result through the 3 fully connected layers, and the final output value is the quality score value of the light field image prediction.

[0028] Preferably, the learning of prior knowledge of distortion based on the training set includes:

[0029] The light field image x in the training set is used as input, and passes through the Swin Transformer network and the fully connected layer network in sequence to generate the output value Among them, the output value is the predicted quality score of the light field image x, defined as:

[0030]

[0031] In the formula, θ is the initialized model parameter, f θ It is a deep meta-learning model that includes the Swin Transformer network structure;

[0032] Using the mean square error (MSE) loss function Through continuous training by gradient descent method, the quality score is predicted Minimize the difference between the actual quality score y of the input image x; where the mean square error loss function is

[0033]

[0034] Where s is the number of input images in each task, and i is the current input image;

[0035] The no-reference light field image quality assessment task with different distortion types in the light field image dataset is divided into training sets in, are the support set and query set for each task respectively. There are a total of Z no-reference light field image quality assessment tasks;

[0036] Using Adam optimizer on the support set Iteratively update the neural network weights, where the gradient g of Adam at the t+1th time stept for:

[0037]

[0038] Current model parameters θ t The update formula is:

[0039]

[0040] In the formula, θ ′ is the value of the updated model parameters, α is the learning rate, ∈ is set to 10 -8 To avoid division by zero, m t is momentum, v t is the variance of the gradient change, m t and v t are the first moment and second moment estimates of the gradient, respectively, expressed as:

[0041]

[0042]

[0043] Where β1 and β2 coefficients are exponential decay rates, controlling the weight distribution and the influence of the previous squared gradient respectively; and are the values ​​of β1 and β2 at the tth time step, and their values ​​tend to 0 as t increases;

[0044] Using Adam optimizer in query set The model parameters are updated for the second time, then:

[0045]

[0046] The updated gradients of the Z no-reference light field image quality assessment tasks in the light field image dataset are iterated, and the final model parameter θ is obtained as:

[0047]

[0048] θ i is the updated model parameter value after the i-th task training. Finally, the final model parameter θ is obtained based on all updates, and the final deep meta-learning model f containing the Swin Transformer network structure is obtained. θ , which is the quality prior model.

[0049] Preferably, the fine-tuning of the quality prior model on the test set includes:

[0050] Divide M light field images with quality score values ​​from the test set as training samples;

[0051] The predicted quality score of the light field image is minimized using the mean squared error loss function The error between y and the true mass fraction y;

[0052] The Adam optimizer is used to update the network parameters of the quality prior model and fine-tune the quality prior model to obtain the model with the best predicted quality score. A light field image quality assessment model.

[0053] According to another aspect of the present invention, a reference-free light field image quality assessment system based on deep meta-learning is provided, comprising:

[0054] A dataset construction module, which is used to construct a training set containing a no-reference light field image quality assessment task and a test set containing a target assessment task;

[0055] Model building module, which is used to build a deep meta-learning model containing the Swin Transformer network structure;

[0056] A model meta-training module, which learns prior knowledge of distortion based on the training set to obtain a prior quality model;

[0057] A model fine-tuning module, which is used to fine-tune the quality prior model on a test set to obtain a final light field image quality assessment model;

[0058] A quality assessment module is used to perform light field image quality assessment on the light field image to be assessed using the light field image quality assessment model.

[0059] According to a third aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any of the above-mentioned methods or to run any of the above-mentioned systems when executing the program.

[0060] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described above, or to run any of the systems described above.

[0061] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0062] The present invention bids farewell to the traditional machine learning method of manually extracting image features, and adopts deep learning technology in the field of machine learning. By learning the inherent laws and representation levels of sample image data, the interpretation information of image data is obtained in the learning process, and the required performance can be achieved autonomously like a robot. Due to the use of a deep learning solution, it is possible to fully automatically learn the knowledge of light field image distortion, evaluate the distorted image, and solve the inconvenience of manually extracting features in traditional machine learning.

[0063] The present invention adopts a deep meta-learning model to solve the overfitting or underfitting problems caused by the small sample characteristics of light field images in deep learning, and has good generalization ability for images with different distortion types. Due to the use of meta-learning, it is possible to achieve quality evaluation of various distortions in light field images, solve the fitting problems that may arise in deep learning due to the small sample characteristics of light field datasets, and solve the inadaptability to various distortions.

[0064] The present invention converts the input image into a MacPI (macro pixel image) mode. MacPI is composed of fixed spatial information and angle information, which can well combine the two information to make the information extraction more comprehensive. Due to the use of the MacPI mode, the spatial and angle information of the light field image can be well combined, solving the problem that the model ignores the image angle information, resulting in poor results in the evaluation process of certain angle distortions.

[0065] The present invention adds the Swin Transformer network structure, and its attention mechanism characteristics make the model consistent with the visual characteristics of the human eye, greatly improving the experimental effect. Due to the use of the Swin Transformer network structure, the evaluation effect can be made consistent with the visual characteristics of the human eye, solving the problem that the model does not conform to the visual characteristics of the human eye, resulting in unrealistic results or low performance.

[0066] The present invention adopts a deep learning solution, which can realize fully automatic learning of light field image distortion knowledge, evaluate the distorted image, and solve the inconvenience of manual feature extraction in traditional machine learning.

[0067] The present invention adopts meta-learning to achieve quality assessment of various distortions in light field images without causing inadaptability due to the distortion type. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0069] Figure 1 This is a workflow diagram of a reference-free light field image quality assessment method based on deep meta-learning in a preferred embodiment of the present invention.

[0070] Figure 2 This is a working principle diagram of a reference-free light field image quality assessment method based on deep meta-learning in a preferred embodiment of the present invention.

[0071] Figure 3 Schematic diagram of the structure of the Swin Transformer network in a preferred embodiment of the present invention.

[0072] Figure 4 This is a scatter plot of the experimental results of the present invention; the horizontal axis is the quality score predicted by the model, and the vertical axis is the subjective evaluation value. Among them, (a) is the test result of the Win5-LID dataset, (b) is the test result of the SHU dataset, (c) is the test result of different distortion types in the Win5-LID dataset, and (d) is the test result of different distortion types in the SHU dataset.

[0073] Figure 5 Schematic diagram of component modules of a reference-free light field image quality assessment system based on deep meta-learning in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0074] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0075] An embodiment of the present invention provides a reference-free light field image quality assessment (LFIQA) method based on deep meta-learning. When evaluating the quality of various distorted images, the method can learn distortion knowledge through previous evaluation tasks and then easily adapt to subsequent unknown distortions. The method has been verified through extensive experiments to be largely superior to other current advanced LFIQA models and has good generalization ability for various distortions.

[0076] like Figure 1 As shown, the reference-free light field image quality assessment method based on deep meta-learning provided by this embodiment may include:

[0077] S1, construct a training set containing the no-reference light field image quality assessment task and a test set containing the target assessment task respectively;

[0078] S2, build a deep meta-learning model containing the Swin Transformer network structure;

[0079] S3, based on the training set, learns the prior knowledge of various distortions and obtains the quality prior model;

[0080] S4, fine-tune the quality prior model on the test set to obtain the final light field image quality assessment model;

[0081] S5, performing light field image quality assessment on the light field image to be assessed using a light field image quality assessment model.

[0082] In a preferred embodiment of S1, constructing a training set including a no-reference light field image quality assessment task and a test set including a target assessment task respectively may include:

[0083] The light field images are divided into multiple no-reference light field image quality assessment tasks according to their distortion types, which are used to construct a training set;

[0084] Light field images with unknown distortion types are used as target evaluation tasks to construct the test set;

[0085] The light field images used for the quality assessment task of the non-reference light field images and the light field images used for the target assessment task are respectively converted into MacPI mode to obtain the training set and test set required for the quality assessment of the non-reference light field images;

[0086] in:

[0087] In the training set, light field images with the same distortion type in the same dataset are divided into a group, and each group is used as a light field image quality assessment task;

[0088] In the test set, the ground-truth types of light field images in the dataset are unknown.

[0089] In this preferred embodiment, the light field image dataset used to construct the training set and the test set may include:

[0090] In the Win5-LID dataset, the distortion types include JPEG2000, HEVC, LINEAR, NN, and two CNN models, a total of six distortions.

[0091] In the SHU dataset, there are five types of distortions: JPEG2000, JPEG, Gaussian blur, White noise, and Motion blur.

[0092] In the NBU-LF1.0 dataset, there are five types of distortions: NNI, BI, EPICNN, Zhang and VDSR.

[0093] In the same dataset, light field images of the same distortion type are divided into a group, and each group is used as an evaluation task. The model is trained on two of the datasets and fine-tuned and tested on the other dataset, for example, trained on Win5-LID and NBU-LF1.0 datasets, tested on SHU dataset; trained on SHU and NBU-LF1.0 datasets, tested on Win5-LID dataset. As a training set, its task division is as shown above; as a test set, its distortion type is unknown.

[0094] In a preferred embodiment of S2, a deep meta-learning model including a Swin Transformer network structure is constructed, including:

[0095] Construct a Swin Transformer network and a fully connected layer network; among them:

[0096] The Swin Transformer network includes: a PatchEmbed module, four basicLayer modules and a Linear layer connected in sequence; the first three basicLayer modules include a Swin Transformer block submodule and a Patch Merging submodule, and the fourth basicLayer module includes a Transformer block submodule; and finally connected to the Linear layer;

[0097] The PatchEmbed module is used to split the input light field image of dimension H x W x C into N patches of size P*P*C and then flatten them, where Then input it to the Swin Transformerblock submodule of the first basicLayer module;

[0098] The Patch Merging submodule is used to reduce the image resolution and adjust the number of channels after each Swin Transformer block submodule to form a hierarchical image;

[0099] The fully connected layer network is used to map the image features output by the Swin Transformer network to the output result, and the final output value is the predicted score value of the light field image.

[0100] Furthermore, in a preferred embodiment, the numbers of Swin Transformerblock submodules in the first three basicLayer modules are 2, 2, and 18, respectively.

[0101] Furthermore, in a preferred embodiment, the number of Transformer block sub-modules in the fourth basicLayer module is 2.

[0102] Furthermore, in a preferred embodiment, the fully connected layer network includes three fully connected layers, and the image output by the SwinTransformer network maps the features of the light field image layer by layer to the final output result through the three fully connected layers, for example, from 1000→1024→512→1, and the final output value is the quality score value of the light field image prediction.

[0103] In a preferred embodiment of S3, prior knowledge of various distortions is learned based on the training set, including:

[0104] The model is trained in the divided no-reference light field image quality assessment task. After the neural network model is initialized, the model parameters are adjusted in the support set and query set of the training set by the double-layer gradient descent method. Specifically:

[0105] The light field image x in the training set is taken as input and passed through the Swin Transformer network and the fully connected layer network to generate the output value Among them, the output to is the predicted quality score of the light field image x, defined as:

[0106]

[0107] In the formula, θ is the initialized model parameter, f θ It is a deep meta-learning model that includes the Swin Transformer network structure;

[0108] Using the mean square error (MSE) loss function Through continuous training by gradient descent method, the quality score is predicted Minimize the difference between the actual quality score y of the input image x; where the mean square error loss function is

[0109]

[0110] Where s is the number of input images in each task, and i is the current input image;

[0111] The no-reference light field image quality assessment task with different distortion types in the light field image dataset is divided into training sets in, are the support set and query set for each task respectively. There are a total of Z no-reference light field image quality assessment tasks;

[0112] Using Adam optimizer on the support set Iteratively update the neural network weights, where the gradient g of Adam at the t+1th time step t for:

[0113]

[0114] Current model parameters θ t The update formula is:

[0115]

[0116] In the formula, θ ′ is the value of the updated model parameters, α is the learning rate, ∈ is set to 10 -8 To avoid division by zero, m t is momentum, v t is the variance of the gradient change, m t and v t are the first moment and second moment estimates of the gradient, respectively, expressed as:

[0117]

[0118]

[0119] Where β1 and β2 coefficients are exponential decay rates, controlling the weight distribution and the influence of the previous squared gradient respectively; and are the values ​​of β1 and β2 at the tth time step, and their values ​​tend to 0 as t increases;

[0120] Using Adam optimizer in query set The model parameters are updated for the second time, then:

[0121]

[0122] The updated gradients of the Z no-reference light field image quality assessment tasks in the light field image dataset are combined, and the final model parameters are updated as follows:

[0123]

[0124] θ i is the updated model parameter value after the i-th task training. Finally, the final model parameter θ is obtained based on all updates. The final deep meta-learning model f containing the Swin Transformer network structure θ , which is the quality prior model.

[0125] In this preferred embodiment, the process of aggregation refers to an iterative process, where the updated θ at each step is used for the next task, and then θ is updated again until all tasks are trained. The last updated θ is the final parameter of the model.

[0126] In a preferred embodiment of S4, fine-tuning the quality prior model on the target reference-free light field image quality assessment task includes:

[0127] Based on the obtained quality prior model, the parameters are optimized by fine-tuning the model in the target no-reference light field image quality assessment task; specifically:

[0128] M light field images with quality scores are divided from the target no-reference light field image quality assessment task as training samples; the quality score is the quality score value given by the official dataset;

[0129] The predicted quality score of the light field image is minimized using the mean squared error loss function The error between y and the true mass fraction y;

[0130] The Adam optimizer is used to update the network parameters of the quality prior model and fine-tune the quality prior model to obtain the best predicted quality score. A light field image quality assessment model.

[0131] The technical solutions provided by the above embodiments of the present invention are further described in detail below.

[0132] like Figure 2 As shown, the above embodiment of the present invention first establishes a model that can learn the prior knowledge of light field image distortion types, and then fine-tunes the model in a completely new unknown distortion type to obtain the final light field image quality assessment model.

[0133] The model's network consists of two networks: the Swin Transformer network and the fully connected layer.

[0134] 1) Swin Transformer network, consisting of four basicLayer modules and one linear layer. The first three basicLayer modules consist of Swin Transformer block submodules and Patch Merging submodules. The number of Swin Transformer block submodules is 2, 2, and 18 respectively. The fourth basicLayer module consists of two Transformerblock submodules, without Patch Merging submodule, and finally connected to a Linear layer. Figure 3 shown.

[0135] When a distorted light field image is input, it will pass through a PatchEmbed module to cut the image into tiles and embed vectors. It is sent to the basicLayer module, where the Patch Merging submodule mainly reduces the image resolution and adjusts the number of channels after each SwinTransformer block submodule, thereby forming a hierarchical design and saving a certain amount of computation. After passing through the Swin Transformer network, the image dimension changes from HxW to 1x1000.

[0136] 2) Fully connected layer, there are three layers in total, which convert the output of the previous layer from 1000->1024->512->1, and the final output result is the predicted score value of the light field image.

[0137] The training of the model consists of two phases: the meta-training phase and the fine-tuning phase.

[0138] 1) Meta-training stage: In this stage, the model will be trained in the divided NR-IQA training task. After initializing the network model, the model parameters are adjusted in the support set and query set of the training set through the double-layer gradient descent method.

[0139] 2) Fine-tuning stage: In this stage, the parameters are optimized by fine-tuning the model in the target task based on the previously learned prior model.

[0140] First, in the data preprocessing stage, the light field image is converted into a sub-aperture image (SAI) array composed of multiple sub-aperture images, and the SAI array is converted into a macro pixel image (MacPI) form. The macro pixel image is generated by organizing each SAI at the same spatial position according to its angular position. The macro pixel form can well combine the spatial and angular information of the light field image to make full use of the information contained in the image. Then deep training is performed, which includes two steps. First, according to the distortion type of the image in the light field dataset, it is divided into multiple tasks as a meta-training set. This training set will be divided into a support set and a query set, and the prior model will be trained on the training set by a two-level gradient descent method. Then, we fine-tune the previously trained model on the target NR-IQA task to obtain the final quality model.

[0141] 1) Meta-learning training phase

[0142] This stage is similar to the pre-training stage of image processing using deep learning. First, an initialized model is built, and then pre-training is performed on various LFIQA distortion tasks. The deep regression network of the training model consists of a SwinTransformer network layer and a fully connected layer. Swin Transformer is a new Transformer architecture that uses a sliding window and a hierarchical structure mechanism, making it a new backbone in machine vision, reaching the SOTA level in a variety of machine vision tasks such as image classification and target detection. Moreover, its attention mechanism is consistent with the human eye perception characteristics in the image quality assessment process. It is added to the model of the present invention to enhance the evaluation performance of the model. Then, a fully connected layer is added after the Transformer layer to generate the output of the regression network as the value of the light field image quality score.

[0143] Specifically, a light field image x is used as input, which is fed into two networks to generate output values this is the predicted quality score of the image, defined as:

[0144]

[0145] Among them, θ is the initialized network parameter, f θ It is a network evaluation model, which is continuously trained through the gradient descent method to make its evaluation value Close to the true value y of the image score. Using the mean square error (MSE) loss function, continuous training makes the predicted quality score The difference between the actual quality score y of the input image x is getting smaller and smaller. This loss function is the most commonly used error function in the regression loss function in deep learning. It is the predicted value f θ The average of the sum of the squares of the differences between (x) and the target value y. The formula is as follows:

[0146]

[0147] First, several NR-LFIQA tasks with different distortion types in the light field dataset are divided into training sets in are the support set and query set for each task respectively, and there are Z NR-LFIQA tasks in total.

[0148] Since the quality assessment model is relatively complex compared to the recognition model, the stochastic gradient descent method is used to optimize the function. Adam is a first-order optimization algorithm that can replace the traditional stochastic gradient descent process. It uses the gradient of the first-order moment estimate and the second-order moment estimate to dynamically adjust the learning rate of each parameter during the training process. Therefore, the present invention uses the Adam optimizer on the support set Iterate to update the neural network weights. The gradient of Adam at the t+1th time step is:

[0149]

[0150] The update formula is:

[0151]

[0152] Where α is the learning rate and ∈ is set to 10 -8 to avoid division by zero. t is momentum, v t is the variance of the gradient change, they are the first and second moment estimates of the gradient, respectively, expressed as:

[0153]

[0154]

[0155] The β1 and β2 coefficients are exponential decay rates that control the influence of the weight distribution (momentum and current gradient) and the square of the previous gradient, respectively.

[0156] After optimizing the model parameters on the support set, in order to make the optimized model have higher performance on the query set, the present invention performs a second parameter optimization on the query set. ′ In the query set The Adam optimizer is also used for updating.

[0157]

[0158] Then, the updated gradients of the Z tasks in each batch are combined to update the final prior model, which is defined as:

[0159]

[0160] Through the above steps, the present invention iteratively samples different distortion tasks on the meta-training set to train the deep meta-regression network f θ Finally, a prior model that has learned different distorted LFIs is obtained.

[0161] 2) Fine-tuning phase of deep meta-learning

[0162] This stage will fine-tune the model on some unknown NR-LFIQA tasks. The model is trained on a specific distortion task to learn various prior knowledge about the distortion, but it also lacks light training on the target task. Therefore, we divide M light field images with subjective scores from the target NR-IQA task for training to fine-tune the model. Similarly, the MSE loss function is used to minimize the output value The error between the true value y and the Adam optimizer is used to update the model parameters. The distortion LFIx in the quality assessment model After fine-tuning, the best prediction quality score can be obtained The two stages of the model use the same learning method and do not require learning additional parameters, making the model efficient and having excellent fitting characteristics.

[0163] In order to study the performance effectiveness of the method proposed in the present invention, an evaluation experiment was conducted on the method in two public datasets, including Win5-LID and SHU. In this evaluation experiment, the Spearman rank correlation coefficient (SROCC), linear correlation coefficient (PLCC) and root mean square error (RMSE) were used as evaluation criteria to intuitively display the experimental performance. Table 1 shows the experimental performance of the quality assessment method proposed in the present invention on the Win5-LID and SHU datasets, as well as the comparison results with other state-of-the-art IQA methods, where the best performance is indicated in bold. The present invention selects nine representative state-of-the-art comparison indicators. It can be seen from the experimental results that the quality assessment method proposed in the present invention outperforms several comparable advanced IQA methods in overall performance. In addition, in order to verify that the proposed meta-deep learning model can adapt to various distortion types, the SROCC values ​​of different distortion types tested on the Win5-LID dataset are listed in Table 2. It can be seen that the quality assessment method proposed in the present invention can also maintain high performance on all types of distortion. In order to more directly show the experimental performance of the quality assessment method proposed in the present invention, the present invention draws a scatter plot of the overall performance of the method and different distortions, as shown in Table 2. Figure 4 Based on a large number of experimental results, it can be concluded that the reference-free light field image quality assessment method based on deep meta-learning proposed in the present invention has a strong practical value for the quality assessment of LFI.

[0164] Table 1 Performance comparison on Win5-LID and SHU datasets

[0165]

[0166] Table 2 SROCC performance of different distortion types on Win5-LID dataset

[0167]

[0168] In order to further study whether the MacPI mode and the Swin Transformer network structure have an enhancing effect on the model performance in the method proposed by the present invention, the present invention conducted two ablation experiments. Specifically, first, the network structure was changed to the most common Resnet network in deep learning in the MacPI mode to verify the effectiveness of the Swin Transformer network. Secondly, the Swin Transformer network was directly used in the original SAI array diagram mode to verify whether the conversion to the MacPI mode has an enhancing effect on the experimental results. The experimental results are shown in Table 3. It can be seen that after adding the Swin Transformer network structure, the experimental effect is better to a certain extent. In the Win5-LID dataset, the effect is significantly improved, while in the SHU dataset, only ordinary deep networks can achieve very high effects. One possible reason is that the spatial distortion of the SHU dataset is higher, so it is easy to achieve higher performance in an ordinary network structure. At the same time, it can be observed that the conversion to the MacPI mode has a significant gain in the experimental results of the two datasets. Because the generation of MacPI needs to include the spatial and angular information of the light field image, the deep learning in the MacPI mode can adapt well to the characteristics of the light field image, making the experimental effect better. In summary, deep learning of the network structure composed of Swin Transformer in MacPI mode can greatly improve the overall performance of the model.

[0169] Table 3 Verification experiments on the impact of MacPI and Swin Transformer on experimental performance

[0170]

[0171] An embodiment of the present invention provides a reference-free light field image quality assessment system based on deep meta-learning, such as Figure 5 As shown, the system may include:

[0172] A dataset construction module, which is used to construct a training set containing a no-reference light field image quality assessment task and a test set containing a target assessment task;

[0173] Model building module, which is used to build a deep meta-learning model containing the Swin Transformer network structure;

[0174] Model meta-training module, which learns prior knowledge of distortion based on the training set and obtains a priori quality model;

[0175] A model fine-tuning module, which is used to fine-tune the quality prior model on the test set to obtain the final light field image quality assessment model;

[0176] The quality assessment module uses a light field image quality assessment model to perform light field image quality assessment on the light field image to be assessed.

[0177] It should be noted that the steps in the method provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system, that is, the embodiments in the method can be understood as preferred examples for constructing the system, which will not be elaborated here.

[0178] An embodiment of the present invention provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the processor can be used to execute any method of the above-mentioned embodiments of the present invention, or to run any system of the above-mentioned embodiments of the present invention.

[0179] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0180] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0181] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0182] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0183] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0184] The method and system for non-reference light field image quality assessment based on deep meta-learning provided by the above-mentioned embodiments of the present invention adopt technologies such as deep learning, deep meta-learning, MacPI mode, Swin Transformer network, etc., so that the light field image quality assessment technology can achieve higher performance in quality assessment on data sets such as Win5-LID and SHU, and adapt to each type of distortion. Compared with the latest advanced LQA model, the performance is greatly enhanced. The method and system for non-reference light field image quality assessment based on deep meta-learning provided by the above-mentioned embodiments of the present invention overcome the problems in the prior art that the spatial and angular information of the light field image is not sufficiently combined, resulting in a large difference between the experimental results and the actual results, as well as the fitting problems and inadaptability to various distortions that may be caused by the use of deep learning.

[0185] Those skilled in the art know that, in addition to implementing the system and its various devices provided by the present invention in a purely computer-readable program code, the system and its various devices provided by the present invention can be made to implement the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices provided by the present invention can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component; the devices for implementing various functions can also be considered as both software modules for implementing the method and structures within the hardware component.

[0186] All matters not covered in the above embodiments of the present invention are well known in the art.

[0187] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A reference-free light field image quality assessment method based on deep meta-learning, characterized in that: include: Construct a training set containing a no-reference light field image quality assessment task and a test set containing a target assessment task respectively; Build a deep meta-learning model containing the Swin Transformer network structure; Based on the training set, learning prior knowledge of distortion to obtain a quality prior model; Fine-tuning the quality prior model on the test set to obtain a final light field image quality assessment model; Performing light field image quality assessment on the light field image to be assessed using the light field image quality assessment model; in: The method of respectively constructing a training set including a no-reference light field image quality assessment task and a test set including a target assessment task comprises: The light field images are divided into multiple non-reference light field image quality assessment tasks according to their distortion types, which are used to construct a training set; wherein, in the same data set, light field images with the same distortion type are divided into a group, and each group is used as a light field image quality assessment task; Light field images with unknown distortion types are used as target evaluation tasks to construct the test set; The light field images used for the quality assessment task of the non-reference light field images and the light field images used for the target assessment task are respectively converted into MacPI mode to obtain the training set and test set required for the quality assessment of the non-reference light field images; The construction of a deep meta-learning model containing a Swin Transformer network structure includes: Construct a Swin Transformer network and a fully connected layer network; among them: The Swin Transformer network includes: a PatchEmbed module, four basicLayer modules and a Linear layer connected in sequence; wherein the first three basicLayer modules each include a Swin Transformer block submodule and a Patch Merging submodule, and the fourth basicLayer module includes a Swin Transformer block submodule; and finally connected to a fully connected layer; The PatchEmbed module is used to divide the input light field image of dimension H x W x C into N patches of size P*P*C and then flatten them, where Then input to the SwinTransformer block submodule of the first basicLayer module; The Patch Merging submodule is used to reduce the image resolution and adjust the number of channels after each of the swin Transformer block submodules to form a hierarchical image; The fully connected layer network is used to map the features of the image output by the Swin Transformer network to the output result, and the final output value is the predicted quality score value of the light field image; The number of Swin Transformer block submodules in the first three basicLayer modules are 2, 2, and 18 respectively; The fourth basicLayer module has two Swin Transformer block sub-modules.

2. The method for quality assessment of reference-free light field images based on deep meta-learning according to claim 1, characterized in that: The fully connected layer network includes three fully connected layers. The image output by the Swin Transformer network maps the features of the light field image layer by layer to the final output result through the three fully connected layers. The final output value is the quality score value of the light field image prediction.

3. The method for quality assessment of reference-free light field images based on deep meta-learning according to claim 1, characterized in that: The learning of prior knowledge of distortion based on the training set includes: The light field image x in the training set is used as input, and passes through the Swin Transformer network and the fully connected layer network in sequence to generate the output value Among them, the output value is the predicted quality score of the light field image x, defined as: In the formula, θ is the initialized model parameter, f θ It is a deep meta-learning model that includes the Swin Transformer network structure; Using the mean square error loss function Through continuous training by gradient descent method, the quality score is predicted Minimize the difference between the actual quality score y of the input image x; where the mean square error loss function is Where s is the number of input images in each task, and i is the current input image; The no-reference light field image quality assessment task with different distortion types in the light field image dataset is divided into training sets in, are the support set and query set for each task respectively. There are a total of Z no-reference light field image quality assessment tasks; Using Adam optimizer on the support set Iteratively update the neural network weights, where the gradient g of Adam at the t+1th time step t for: Current model parameters θ t The update formula is: In the formula, θ ′ is the value of the updated model parameters, α is the learning rate, ∈ is set to 10 -8 To avoid division by zero, m t is momentum, v t is the variance of the gradient change, m t and v t They are the first moment and second moment estimates of the gradient, expressed as: Where β1 and β2 coefficients are exponential decay rates, controlling the weight distribution and the influence of the previous squared gradient respectively; and are the values ​​of β1 and β2 at the tth time step, and their values ​​tend to 0 as t increases; Using Adam optimizer in query set The model parameters are updated for the second time, then: The updated gradients of the Z no-reference light field image quality assessment tasks in the light field image dataset are iterated, and the final model parameters are obtained as follows: θ i is the updated model parameter value after the i-th task training. Finally, the final model parameter θ is obtained based on all updates, and the final deep meta-learning model f containing the Swin Transformer network structure is obtained. θ , which is the quality prior model.

4. The method for quality assessment of reference-free light field images based on deep meta-learning according to claim 1, characterized in that: The quality prior model is fine-tuned on the test set, including: Divide M light field images with quality score values ​​from the test set as training samples; The predicted quality score of the light field image is minimized using the mean squared error loss function The error between y and the true mass fraction y; The Adam optimizer is used to update the network parameters of the quality prior model and fine-tune the quality prior model to obtain the model with the best predicted quality score. A light field image quality assessment model.

5. A reference-free light field image quality assessment system based on deep meta-learning, characterized in that: include: A dataset construction module, which is used to construct a training set containing a no-reference light field image quality assessment task and a test set containing a target assessment task; A model building module, which is used to build a deep meta-learning model including a Swin Transformer network structure; a model meta-training module, which learns prior knowledge of distortion based on the training set to obtain a quality prior model; A model fine-tuning module, which is used to fine-tune the quality prior model on a test set to obtain a final light field image quality assessment model; A quality assessment module, which uses the light field image quality assessment model to perform light field image quality assessment on the light field image to be assessed; in: The method of respectively constructing a training set including a no-reference light field image quality assessment task and a test set including a target assessment task comprises: The light field images are divided into multiple non-reference light field image quality assessment tasks according to their distortion types, which are used to construct a training set; wherein, in the same data set, light field images with the same distortion type are divided into a group, and each group is used as a light field image quality assessment task; Light field images with unknown distortion types are used as target evaluation tasks to construct the test set; The light field images used for the quality assessment task of the non-reference light field images and the light field images used for the target assessment task are respectively converted into MacPI mode to obtain the training set and test set required for the quality assessment of the non-reference light field images; The construction of a deep meta-learning model containing a Swin Transformer network structure includes: Construct a Swin Transformer network and a fully connected layer network; among them: The Swin Transformer network includes: a PatchEmbed module, four basicLayer modules and a Linear layer connected in sequence; wherein the first three basicLayer modules each include a Swin Transformer block submodule and a Patch Merging submodule, and the fourth basicLayer module includes a Swin Transformer block submodule; and finally connected to a fully connected layer; The PatchEmbed module is used to divide the input light field image of dimension H x W x C into N patches of size P*P*C and then flatten them, where Then input to the SwinTransformer block submodule of the first basicLayer module; The Patch Merging submodule is used to reduce the image resolution and adjust the number of channels after each of the Swin Transformer block submodules to form a hierarchical image; The fully connected layer network is used to map the features of the image output by the Swin Transformer network to the output result, and the final output value is the predicted quality score value of the light field image; The number of Swin Transformer block submodules in the first three basicLayer modules are 2, 2, and 18 respectively; The fourth basicLayer module has two Swin Transformer block sub-modules.

6. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the method described in any one of claims 1 to 4, or to run the system described in claim 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 4, or to run the system described in claim 5.

Images