Multi-mode and multi-task combined light field image blind quality evaluation method

Abstract

The invention discloses a multi-mode and multi-task combined light field image blind quality evaluation method. The method comprises the following steps: acquiring a pseudo video sequence of a light field image; the pseudo video sequence is processed, a visual feature vector is generated, and the visual feature vector is used for identifying a distortion type; using a contrast language-image pre-training CLIP model to obtain a text feature vector of the light field image; calculating the cosine similarity of the visual feature vector and the text feature vector; and converting the cosine similarity into a quality grade probability, and obtaining an image quality score based on the quality grade probability and the distortion type. According to the method, multi-modal fusion of visual features and language features of the light field image is fully utilized, and the accuracy of light field image quality evaluation is remarkably improved through model parameter sharing and loss weight optimization.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and quality assessment, and in particular relates to a multi-modal and multi-task combined light field image blind quality assessment method. Background Art

[0002] Light field imaging can capture four-dimensional information of a scene, including the spatial and angular information of light, which is crucial for reproducing three-dimensional visual effects. However, in practical applications, such as compression and transmission, this technology often encounters various distortion problems, such as parallax distortion and depth distortion, which reduce the visual quality of the image. Traditional image quality evaluation methods are mainly aimed at ordinary two-dimensional images and are not suitable for light field images with special distortion characteristics.

[0003] With the development of deep learning technology, convolutional neural networks have shown great potential in image quality assessment, but their application is limited by the need for a large amount of annotated data. For light field images, it is particularly difficult to obtain these data because the existing light field image datasets are small. Therefore, a multi-modal and multi-task combined light field image blind quality assessment method is urgently needed. Summary of the invention

[0004] To solve the above technical problems, the present invention proposes a multi-modal and multi-task combined light field image blind quality evaluation method, which significantly improves the accuracy of light field image quality evaluation by fusing visual and text information and utilizing model parameter sharing and loss weight optimization.

[0005] The present invention provides a multi-modal and multi-task combined light field image blind quality assessment method, comprising:

[0006] Obtaining a pseudo video sequence of light field images;

[0007] Processing the pseudo video sequence to generate a visual feature vector, wherein the visual feature vector is used to identify the distortion type;

[0008] Obtaining a text feature vector of the light field image using a contrastive language-image pre-trained CLIP model;

[0009] Calculating the cosine similarity between the visual feature vector and the text feature vector;

[0010] The cosine similarity is converted into a quality level probability, and an image quality score is obtained based on the quality level probability and the distortion type.

[0011] Optionally, obtaining a pseudo video sequence of a light field image includes:

[0012] Based on the light field image, acquiring a sub-aperture image array;

[0013] The sub-aperture image array is extracted according to the target direction to form the pseudo video sequence.

[0014] Optionally, processing the pseudo video sequence to generate a visual feature vector includes:

[0015] Inputting the pseudo video sequence into the ViT model, the ViT model segmenting the pseudo video sequence to obtain a plurality of sequence blocks;

[0016] Encoding the sequence blocks and extracting visual features;

[0017] Based on the visual feature, the visual feature vector is obtained.

[0018] Optionally, based on the visual feature, obtaining the visual feature vector includes:

[0019] splicing the visual features to obtain overall visual features;

[0020] Perform mean pooling processing on the overall visual feature to obtain the visual feature vector.

[0021] Optionally, using a contrastive language-image pre-trained CLIP model, obtaining a text feature vector of the light field image includes:

[0022] Use the contrastive language-image pre-trained CLIP model to obtain text descriptions of quality levels;

[0023] The text description is converted into a Token sequence, and the Token sequence is mapped and encoded to obtain a text feature vector.

[0024] Optionally, calculating the cosine similarity between the visual feature vector and the text feature vector includes:

[0025] The visual feature vector and the text feature vector are normalized, and the cosine similarity of the processed visual feature vector and the text feature vector is calculated, where the cosine similarity is used to calculate the degree of matching between the visual feature vector and the text feature vector.

[0026] Optionally, converting the cosine similarity to quality level probability includes:

[0027] Based on the cosine similarity, obtaining a similarity score matrix;

[0028] The similarity score matrix is ​​converted into a probability distribution using a softmax function to obtain the quality level probability.

[0029] Optionally, based on the quality level probability and the distortion type, obtaining the image quality score includes:

[0030] The quality level probability and the corresponding numerical level are weightedly summed, and combined with the distortion type to obtain the image quality score, wherein the corresponding numerical level is a preset numerical level.

[0031] Compared with the prior art, the present invention has the following advantages and technical effects:

[0032] The present invention achieves a more accurate evaluation effect by combining visual and text features. The method does not require reference images, has a wide range of applicability, and improves the generalization ability and robustness of the model through multi-task learning. After the introduction of the CLIP model, it makes full use of visual and language prior knowledge, can effectively handle a variety of distortion types, and significantly reduces the demand for training data and resource consumption. Overall, the invention significantly improves the efficiency and accuracy of light field image quality evaluation, and has important application value and broad promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0034] Figure 1 It is a flow chart of a multi-modal and multi-task combined light field image blind quality assessment method according to an embodiment of the present invention;

[0035] Figure 2 is a flow chart of implementing the method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0037] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] This paper proposes a multi-modal and multi-task combined light field image blind quality assessment method. Figure 1 As shown, the specific steps include:

[0039] Obtaining a pseudo video sequence of light field images;

[0040] Input the pseudo video sequence into the ViT model to obtain the visual feature vector;

[0041] Use the contrastive language-image pre-trained CLIP model to obtain text feature vectors of light field images;

[0042] Calculate the cosine similarity between the visual feature vector and the text feature vector;

[0043] The Softmax function is used to convert the cosine similarity into quality level probability, and the image quality score is obtained based on the quality level probability.

[0044] Specifically, S1, arraying the sub-aperture images of the light field image to form a pseudo video sequence with the sub-aperture images in horizontal, vertical, left diagonal, and right diagonal directions;

[0045] S2, use ViT to process the pseudo video sequence, extract visual features, perform mean pooling on the output of ViT, and generate a visual feature vector of fixed length;

[0046] S3. After ViT processes the pseudo video sequence and generates visual feature vectors, these feature vectors are used for auxiliary tasks such as distortion type identification to improve the accuracy of light field image quality evaluation.

[0047] S4. Use the Textual Encoder model to process text descriptions related to image quality and generate text feature vectors. The text descriptions contain different quality level descriptions, such as "bad", "poor", "fair", "good", and "perfect".

[0048] S5, calculating the cosine similarity between the visual feature vector and the text feature vector to quantify the matching degree between the image and each quality description;

[0049] S6. Convert the cosine similarity score into a probability form through the Softmax function to represent the probability of the image belonging to different quality levels, and then perform weighted summation to obtain the final image quality score;

[0050] This method can not only improve the performance of light field image quality assessment through model parameter sharing and loss function weighting, but also effectively solve the problem of insufficient labeled data by utilizing rich prior knowledge in visual and text embedding. By introducing auxiliary tasks, such as distortion type recognition, the method of the present invention can further improve the performance of blind quality assessment of light field images.

[0051] Furthermore, obtaining a pseudo video sequence of a light field image includes:

[0052] Based on the light field image, a sub-aperture image array is obtained;

[0053] The sub-aperture image array is extracted according to direction to obtain a pseudo video sequence.

[0054] Specifically, the sub-aperture image array of the light field image is extracted along the horizontal direction, vertical direction, left diagonal direction and right diagonal direction to form four pseudo video sequences in different directions. Each pseudo video sequence is composed of sub-aperture images arranged in a certain order to simulate the form of stacked video frames. Then, a series of preprocessing operations are performed on the pseudo video sequence, including random cropping, rotation, scaling, etc., to enhance the robustness of the model. Finally, the dimension of each frame of the preprocessed pseudo video sequence is (batch_size, 3, 224, 224).

[0055] Furthermore, the pseudo video sequence is input into the ViT model to obtain the visual feature vector including:

[0056] The ViT model segments the pseudo video sequence and obtains several sequence blocks;

[0057] Encode the sequence blocks and extract visual features;

[0058] Based on the visual features, a visual feature vector is obtained.

[0059] Furthermore, based on the visual features, obtaining the visual feature vector includes:

[0060] Splice the visual features to obtain the overall visual features;

[0061] Perform mean pooling on the overall visual features to obtain the visual feature vector.

[0062] Specifically, the pseudo video sequence is fed into the Vision Transformer (ViT) model for feature extraction. ViT divides each frame into multiple small blocks (patches), encodes these patches, and extracts rich visual feature information. Subsequently, the features of each frame are spliced ​​to integrate visual information from different directions, and these spliced ​​features are mean-pooled to generate a visual feature vector of fixed length. This vector contains the global visual features of the pseudo video sequence, providing key input for the subsequent similarity calculation with the text description.

[0063] After using Vision Transformer (ViT) to extract rich visual features from pseudo video sequences, these features are not only used for subsequent image quality evaluation, but also for an auxiliary task - distortion type identification. To this end, the output of ViT is fed into a specially designed classification network, which maps visual features to different distortion types through a series of fully connected layers. The model learns to predict the type of distortion contained in the image by calculating the cross-entropy loss function, which measures the difference between the predicted probability distribution of the distortion type and the actual label. Distortion type identification and blind image quality evaluation complement each other. Distortion identification provides background information for quality assessment. For example, identifying the presence of motion blur or noise in an image can help the evaluation model to more accurately judge the overall quality of the image. Different types of distortion may have different effects on human perception, and identifying the distortion type can make the quality evaluation more targeted. Distortion identification provides detailed background information for quality evaluation, which can reflect the specific impact of distortion on the image.

[0064] Furthermore, using the contrastive language-image pre-trained CLIP model, the text feature vector of the light field image is obtained, including:

[0065] Use the contrastive language-image pre-trained CLIP model to obtain text descriptions of quality levels;

[0066] Convert the text description into a Token sequence, map and encode the Token sequence, and obtain the text feature vector.

[0067] Specifically, the textual encoder in the CLIP model is used to encode the text description related to image quality. The text description contains different quality levels, such as "bad", "poor", "fair", "good" and "perfect". Each text description is first converted into a series of tokens by the CLIP tokenizer, and then mapped to a high-dimensional space to generate a 512-dimensional embedding vector. Finally, the text descriptions of five different quality levels form a text feature matrix with a matrix dimension of (batch_size, 5, 512), which is used to describe the feature information of different quality levels.

[0068] Furthermore, calculating the cosine similarity between the visual feature vector and the text feature vector includes:

[0069] Normalizing the visual feature vector and the text feature vector, and calculating the matching degree between the processed visual feature vector and the text feature vector;

[0070] Based on the matching degree, the cosine similarity is obtained.

[0071] Specifically, the cosine similarity between the visual feature vector and the text feature matrix is ​​calculated to quantify the degree of match between the visual features and the text description. Specifically, the visual features and each text feature are normalized, and then the cosine similarity between them is calculated to obtain a similarity score matrix. The dimension of this matrix is ​​(batch_size, 5), and each value represents the degree of similarity between the image and a specific quality description.

[0072] Further, converting the cosine similarity into quality level probability includes:

[0073] Based on cosine similarity, obtain the similarity score matrix;

[0074] The similarity score matrix is ​​converted into a probability distribution using the softmax function to obtain the quality level probability.

[0075] Specifically, we calculate the cosine similarity between the visual features of the light field image and the text descriptions of different quality levels to form a similarity score matrix. Then, we use the Softmax function to convert these scores into probability distributions, indicating the likelihood that the image belongs to each quality level.

[0076] Further, based on the quality level probability, obtaining the image quality score includes:

[0077] The quality level probability and the corresponding numerical level are weighted and summed to obtain the image quality score.

[0078] Specifically, the Softmax function is used to convert the cosine similarity scores into a probability form to represent the likelihood that the image belongs to each quality level. The Softmax function is applied to the cosine similarity score vector for each image, converting each score to a value between 0 and 1 that sums to 1, representing the probability of belonging to each quality level. After the conversion, a probability vector with the dimension (batch_size, 5) is obtained, where each element corresponds to the probability that the image belongs to the "bad", "poor", "fair", "good", or "perfect" level. Next, the final quality score for each image is calculated by multiplying each probability with the corresponding quality level value and summing the results. This weighted summation process integrates the likelihood of multiple quality levels based on the predicted probability of each quality level to generate a single value representing the overall quality of the image, with an output dimension of (batch_size, 1).

[0079] The multi-task learning method of the present invention can effectively mine the spatial and angular features of light field images, and combine these visual information with language descriptions to achieve a comprehensive evaluation of the quality of light field images. Compared with traditional light field image quality evaluation methods, the present invention has the following significant advantages: First, by integrating visual and language features, it can more accurately evaluate image quality from a multimodal perspective, overcoming the limitations of single visual evaluation; second, the present invention adopts parameter sharing and multi-task learning frameworks to reduce computational complexity, improve evaluation efficiency, and can adapt to real-time quality evaluation requirements in different scenarios; finally, the method of the present invention verifies its superiority on multiple light field image data sets, demonstrating its wide applicability in applications such as digital refocusing, depth extraction, three-dimensional reconstruction, and virtual reality. The present invention not only improves the accuracy and reliability of light field image quality evaluation, but also provides innovative solutions for the fields of image processing and computer vision.

[0080] like Figure 2 As shown, this embodiment specifically includes:

[0081] (1) The original light field image can be expressed as L(u,v,h,w,c), where (u,v) represents the angular dimension (sub-aperture view) of the light field image, (h,w) is the spatial dimension, and c is the number of color channels (such as RGB channels). The light field image is extracted by direction, horizontally, vertically, left diagonally, and right diagonally, and four pseudo video sequences are generated, denoted as L0, L90, L45, and L135 respectively. Each pseudo video sequence contains sub-aperture images collected from different angles, forming a four-dimensional tensor with a dimension of L(n,h,w,c), where n=9 represents the number of frames, i.e., the number of sub-aperture images contained in each direction; the spatial resolution of each frame is 512×512 pixels, and it has 3 color channels, so the size of each frame is 512×512×3;

[0082] (2) First, the Vision Transformer (ViT) model is applied to each frame of the pseudo video sequence obtained in step (1) to extract visual features. The ViT model requires the input image size to be 224×224×3, so each frame is center-cropped and the original size is converted from 512×512×3 to 224×224×3. Each 224×224 image is divided into 14×14 patches, and the dimension of each patch is 16×16 pixels. After each image block passes through the linear embedding layer and position encoding of ViT, it is encoded into a 768-dimensional feature vector. Finally, the output of each frame is represented as a 196×768 feature matrix; in order to generate a fixed-length visual feature vector, the features of these 196 image blocks are further averaged. Average pooling is to average the feature vectors of all patches in the feature dimension, calculate a global visual feature representation, and integrate these local features into an overall feature vector. After mean pooling, the 196 feature vectors are compressed into a global feature vector with a dimension of 1 × 768. This fixed-length visual feature vector represents the overall visual information of each frame of the image, which is then used for matching calculations with text features to complete the evaluation of light field image quality;

[0083] (3) The visual feature vector obtained in step (2) is fed into a classification network consisting of two fully connected layers. The first fully connected layer reduces the feature dimension to 256, and the second fully connected layer further maps the feature to the number of distortion types C, outputting a vector of dimension C, which represents the probability distribution of the image belonging to each distortion type. The parameters of the classification network are optimized using the cross entropy loss function, which calculates the difference between the probability distribution of the distortion type predicted by the model and the true distortion type label. In the joint optimization process of multi-task learning, the loss function Ldistortion of the distortion recognition task is combined with the loss function Lquality of the main task image quality evaluation to form a total loss function Ltotal=Lquality+Ldistortion for end-to-end model training, where λ is a weight parameter used to balance the importance of the two tasks.

[0084] (4) In this step, the present invention uses the Textual Encoder in the CLIP model to perform feature extraction on a series of text descriptions related to image quality. Specifically, these text descriptions include five different quality levels, namely "bad", "poor", "fair", "good" and "perfect". First, these text descriptions are input into the tokenizer of the CLIP model, and the tokenizer converts each text description into a fixed-length token sequence. The tokenizer of the CLIP model converts each text description into a token sequence of length 77 (i.e., each sentence generates 77 tokens), which represent the positions of each word or tag in the text. Next, these token sequences are mapped to a high-dimensional space, and each token is encoded into a 768-dimensional embedding vector. This mapping process uses the multi-layer Transformer architecture in the Textual Encoder to gradually extract the semantic information and structural features of the text, and finally combines the embedding vectors of the 77 tokens into a fixed-length text feature representation. For each text description, the output after Transformer encoding is a 768-dimensional global feature vector with a dimension of 1×768. The feature vector of each text description is processed and encoded independently to form a text feature representing the description. In order to generate the final text feature matrix, the text feature vectors of the five different quality level descriptions are concatenated along the quality level dimension to form a tensor containing all the description features. Specifically, the five text feature vectors correspond to five quality level descriptions, and are combined into a three-dimensional tensor along a new dimension with a dimension of 5×768, where "5" represents five quality level descriptions, and the feature vector length of each description is 768. For each batch of input (ie, batch size), the dimension of the final generated text feature matrix is ​​batch_size×5×768. This text feature matrix is ​​used for subsequent similarity calculations with visual features, and then participates in the quality evaluation of light field images;

[0085] (5) In this step, the visual feature vector and the text feature vector are similarly calculated to quantify the degree of match between the image and the descriptions of different quality levels. The initial dimension of the visual feature vector is batch_size×1×768. In order to make it match the dimension of the text feature matrix, the visual feature vector needs to be expanded along the dimension of the quality level, and the expanded visual feature matrix becomes batch_size×5×768. Here, 5 represents the descriptions of five different quality levels ("bad", "poor", "fair", "good", "perfect"). Subsequently, the similarity between the expanded visual feature matrix and the text feature matrix is ​​calculated. The dimension of the text feature matrix is ​​batch_size×5×768, and the matching degree of the feature vector of each description is calculated one by one with the expanded visual feature, and the output is a similarity matrix with the dimension of batch_size×5. Each row of the similarity matrix contains the similarity score between the image and the five quality level descriptions, reflecting the matching degree between the image and each quality description.

[0086] (6) Convert the similarity score matrix batch_size×5 obtained in step (5) into probability form. Through the Softmax operation, the similarity score is normalized into probability, and the sum of the five probability values ​​in each row is equal to 1. The dimension of the converted probability matrix is ​​still batch_size×5, where each element represents the probability that the image belongs to the corresponding quality level. Next, to obtain the final quality score of the image, the probability of each quality level and its corresponding numerical level (from 1 to 5) are weighted summed. This operation combines the probability matrix with the level value and outputs a comprehensive score. The output dimension of the final image quality score is batch_size×1, and the score result in each batch is a scalar, indicating the overall quality level of the image.

[0087] The multi-task learning method based on the contrastive language-image pre-training (CLIP) model proposed in this embodiment makes full use of the multimodal fusion of the visual features and language features of the light field image, and significantly improves the accuracy of the light field image quality evaluation through model parameter sharing and loss weight optimization. First, the present invention extracts the sub-aperture images of the light field image in four directions: horizontal, vertical, left diagonal, and right diagonal to form a pseudo video sequence, and uses the VisionTransformer (ViT) model to extract visual features from each frame of the sub-aperture image to obtain a visual feature vector of fixed length. Subsequently, the textual encoder of the CLIP model processes the text description related to the image quality, including five quality levels of "bad", "poor", "fair", "good" and "perfect", and generates the corresponding text feature vector. This embodiment calculates the cosine similarity between the visual features and the text features, converts the similarity score into a probability form through the Softmax function, quantifies the possibility that the image belongs to different quality levels, and finally obtains the comprehensive quality score of the image through weighted summation.

[0088] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-modal and multi-task light field image blind quality assessment method, characterized in that: include: Obtaining a pseudo video sequence of light field images; Processing the pseudo video sequence to generate a visual feature vector, wherein the visual feature vector is used to identify the distortion type; Obtaining a text feature vector of the light field image using a contrastive language-image pre-trained CLIP model; Calculating the cosine similarity between the visual feature vector and the text feature vector; The cosine similarity is converted into a quality level probability, and an image quality score is obtained based on the quality level probability and the distortion type.

2. According to claim 1, a multi-modal and multi-task combined light field image blind quality assessment method is characterized in that: The pseudo video sequence for obtaining light field images includes: Based on the light field image, acquiring a sub-aperture image array; The sub-aperture image array is extracted according to the target direction to form the pseudo video sequence.

3. According to the multi-modal and multi-task combined light field image blind quality assessment method of claim 1, it is characterized in that: Processing the pseudo video sequence to generate a visual feature vector includes: Inputting the pseudo video sequence into the ViT model, the ViT model segmenting the pseudo video sequence to obtain a plurality of sequence blocks; Encoding the sequence blocks and extracting visual features; Based on the visual feature, the visual feature vector is obtained.

4. The multi-modal and multi-task combined light field image blind quality assessment method according to claim 3 is characterized in that: Based on the visual feature, obtaining the visual feature vector includes: splicing the visual features to obtain overall visual features; Perform mean pooling processing on the overall visual feature to obtain the visual feature vector.

5. The multi-modal and multi-task combined light field image blind quality assessment method according to claim 1, characterized in that: Using the contrastive language-image pre-trained CLIP model, obtaining the text feature vector of the light field image includes: Use the contrastive language-image pre-trained CLIP model to obtain text descriptions of quality levels; The text description is converted into a Token sequence, and the Token sequence is mapped and encoded to obtain a text feature vector.

6. The multi-modal and multi-task combined light field image blind quality assessment method according to claim 1, characterized in that: Calculating the cosine similarity between the visual feature vector and the text feature vector includes: The visual feature vector and the text feature vector are normalized, and the cosine similarity of the processed visual feature vector and the text feature vector is calculated, where the cosine similarity is used to calculate the degree of matching between the visual feature vector and the text feature vector.

7. The multi-modal and multi-task combined light field image blind quality assessment method according to claim 1, characterized in that: Converting cosine similarity to quality grade probability involves: Based on the cosine similarity, obtaining a similarity score matrix; The similarity score matrix is ​​converted into a probability distribution using a softmax function to obtain the quality level probability.

8. The multi-modal and multi-task combined light field image blind quality assessment method according to claim 1, characterized in that: Based on the quality level probability and the distortion type, obtaining an image quality score includes: The quality level probability and the corresponding numerical level are weightedly summed, and combined with the distortion type to obtain the image quality score, wherein the corresponding numerical level is a preset numerical level.

Images