Light field image quality evaluation method based on visual language model

Abstract

The invention provides a light field image quality evaluation method based on a visual language model. According to the method, multi-dimensional comprehensive evaluation of a light field image is realized through the following steps of: (a) evaluating the definition, brightness, angle consistency and the like of the light field image through antonym prompt pairing by utilizing a text prompt function of a visual language model; (b) removing fixed position embedding of a visual language model image encoder so as to improve the adaptability of the model to light field images with different space and angle resolutions; (c) finely adjusting the antonym prompt pair by using a context optimization technology to adapt to the multi-view characteristic of the light field image, and keeping the network weight of the visual language model unchanged; and (d) comprehensively evaluating the light field image based on multi-dimensional evaluation indexes such as brightness, color and angle consistency, and generating an overall quality score. The method can be used for efficient coding, transmission, rendering and other light field image processing scenes, and a new solution is provided for light field image quality evaluation.

Description

Technical Field

[0001] The present invention relates to a method for light field image quality evaluation, in particular to a method for light field image quality evaluation based on a visual language model, which is suitable for evaluating multi-dimensional characteristics (such as clarity, brightness, angle consistency, etc.) of light field images. Background Art

[0002] Current image quality evaluation methods are mainly applied to two-dimensional images, focusing on image clarity, color and other indicators. However, light field images have unique angular domain characteristics and are composed of a collection of images from multiple perspectives, so quality evaluation is more complicated than traditional two-dimensional images. Light field images have a wide range of application scenarios in virtual reality, augmented reality and other fields. Their multi-perspective characteristics place higher requirements on image quality in dimensions such as color consistency and angle consistency. However, existing image quality evaluation methods have limitations when processing light field images. It is difficult to fully consider the multi-dimensional characteristics of light field images such as clarity, brightness, and angle consistency, and often cannot fully reflect their actual quality. To this end, this patent proposes a light field image quality evaluation method based on a visual language model. This method utilizes the multi-dimensional feature prompting capabilities of the visual language model, and can directly perform a comprehensive evaluation of the quality of the light field image without designing a complex model specifically for the light field image, thereby more comprehensively reflecting the quality of the light field image while maintaining the adaptability of the model. Summary of the invention

[0003] The present invention provides a light field image quality assessment method based on a visual language model (VLM), which makes full use of the rich prior knowledge in the visual language model to evaluate the quality of light field images in multiple dimensions. The method is applicable to light field images with angular domain characteristics by introducing antonym prompt pairing strategy and removing position embedding optimization means, ensuring that the quality assessment results are more accurate and comprehensive.

[0004] Specifically, the present invention utilizes the text prompt function of the visual language model to design a series of antonym pairings (such as "clear / blurry", "bright / dim", "smooth / abrupt", "fluent / hard", etc.). These prompt words serve as evaluation dimensions, which intuitively and accurately reflect the key quality characteristics of light field images, thereby guiding the model to achieve multi-dimensional and refined quality evaluation of light field images.

[0005] In view of the multi-view characteristics of light field images, the present invention optimizes the structure of the visual language model and removes the original fixed position embedding. This improvement enables the model to flexibly process light field image inputs of different sizes and resolutions, significantly enhancing the model's generalization ability and robustness in processing complex light field images.

[0006] In addition, the present invention also proposes a prompt word fine-tuning method based on context optimization technology (such as CoOp, etc.). By fine-tuning the prompt words to adapt to the light field image quality evaluation task, the present invention further improves the evaluation accuracy of the model and ensures the reliability of the evaluation results.

[0007] During the evaluation process, the present invention evaluates the quality of light field images from multiple dimensions and finally obtains a comprehensive quality score. This process ensures the comprehensiveness and objectivity of the evaluation and provides strong support for the quality evaluation of light field images.

[0008] The present invention provides a light field image quality assessment method based on a visual language model, which has the advantages of multi-dimensionality and high accuracy, and can comprehensively and accurately assess the quality of light field images. This method has brought important technological innovations to the field of light field image quality assessment, and has broad application prospects and significant practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a flow chart of a light field image quality evaluation method based on a visual language model of the present invention; DETAILED DESCRIPTION

[0010] Reference Figure 1 The light field image quality evaluation method based on the visual language model of the present invention comprises:

[0011] Step S1: Antonym prompt pairing: In this embodiment, a series of antonym prompt pairs are designed for light field image quality evaluation using the text prompt function of a visual language model (such as CLIP, etc.). Specifically, these antonym pairs include but are not limited to "clear / blurry", "bright / dim", "smooth / abrupt", "fluent / rigid", etc., which are used to evaluate the clarity, brightness, and angle consistency of light field images. By inputting these antonym pairs as prompt words into the model, the model can generate a multi-dimensional evaluation of the quality of light field images, covering multiple key quality features of light field images.

[0012] Step S2: Remove position embedding: To adapt to the multi-view characteristics of light field images, the present invention optimizes the structure of the visual language model. Specifically, we remove the original fixed position embedding in the model and adjust the structure of the visual encoder to enable it to process light field image inputs of different sizes and resolutions. This adjustment makes the model more flexible in evaluating light field images, can more keenly capture subtle differences in images, and improve the perception of multi-dimensional quality characteristics of light field images.

[0013] Step S3: Fine-tune the prompt pairing words. In order to further improve the accuracy of the model in light field image quality assessment, we used context optimization techniques (such as CoOp, etc.) to fine-tune the prompt words. With the support of a small amount of training data, the prompt words are optimized through the standard back-propagation algorithm, while the network weights of the visual language model remain unchanged. Through this process, the prompt words are better adapted to the multi-view characteristics of light field images, thereby optimizing the prompt information and making it highly matched with the characteristics of light field images, improving the effect and stability of the model in quality assessment tasks.

[0014] Step S4: The present invention uses multi-dimensional evaluation indicators to comprehensively analyze the clarity, brightness, and angle consistency of the light field image. Through these evaluation dimensions, the model generates an overall quality score for the light field image, realizes quantitative analysis of the quality of the light field image, and provides a comprehensive and objective evaluation result for the quality of the light field image.

[0015] In summary, the light field image quality evaluation method based on the visual language model of the present invention can comprehensively and accurately reflect the quality of light field images in multiple dimensions, and provides reliable technical support for quality monitoring and optimization of light field images in various application fields.

Claims

1. A light field image quality evaluation method based on a visual language model, characterized in that: The method comprises the following steps: a) Antonym prompt pairing: Utilizing the text prompt capability of the visual language model, the multi-dimensional evaluation of light field images (such as clarity, brightness, etc.) is performed by inputting antonym prompt pairings. The consistency of the unique angular domain (i.e., multi-viewing angle) of light field images is also considered, thereby achieving a comprehensive evaluation of the quality of light field images. b) Removing position embedding: In view of the multi-view features of light field images, the fixed position embedding of the image encoder of the visual language model is removed, so that the model can adapt to light field images of different spatial resolutions and the unique angular domain of light field images, thus avoiding the limitations brought by fixed position embedding. c) Fine-tuning the cue pairs: The antonym cue pairs are fine-tuned using contextual optimization techniques to optimize the cue information under the multi-view characteristics of light field images. The cue words are updated using standard back-propagation, but the network weights of the visual language model remain unchanged to adapt the antonym cue pairs to the characteristics of light field images. d) Light field image quality evaluation: Based on multi-dimensional evaluation indicators, the light field image is comprehensively evaluated in terms of brightness, color, etc. The overall quality score of the light field image is obtained through the above evaluation, and then the image quality is quantitatively analyzed.

2. The method according to claim 1, wherein the antonym prompt pairing includes but is not limited to using antonym pairs such as "clear / blurry", "bright / dim", and antonym pairs such as "smooth / abrupt", "fluent / hard", etc. specifically used to evaluate the consistency of light field angles, so as to more comprehensively reflect the quality of the light field image.

3. According to the method of claim 1, the step of removing the position embedding further includes removing or replacing the absolute position encoding in the visual language model to improve the adaptability and robustness of the model in processing light field images with different spatial and angular resolutions.

4. According to the method of claim 1, the step of fine-tuning the prompt pairing words further includes training a visual language model based on different methods (such as CoOp, etc.) and light field image datasets (such as DUT-LF, etc.) to optimize the prompt effect during the prompt word adjustment process, thereby improving the accuracy of the model in light field image quality evaluation.

5. The method according to claim 1, wherein the light field image quality evaluation step further comprises using a multi-dimensional evaluation method, including but not limited to indicators such as clarity, brightness, and angle consistency to generate a comprehensive evaluation.

6. The light field image quality assessment method according to any one of claims 1 to 5, wherein the method can be applied to light field image processing scenarios such as efficient encoding, transmission, and rendering, to achieve real-time quality monitoring and optimization adjustment of light field images.

Images