AI-based multi-modal panoramic image blind quality evaluation method and system based on description generation

Abstract

The application provides a multi-modal panoramic image blind quality evaluation method and system based on AI-generated description, which comprises the following steps: generating a quality description text based on a distorted panoramic image; obtaining an image block containing rich visual semantic information from the distorted panoramic image; pre-training a visual language model by using the distorted panoramic image and the quality description text to obtain a pre-trained visual language model; extracting the quality description text and the image block containing rich visual semantic information by using the pre-trained visual language model to obtain text features supervised by image information; obtaining image block features through an equal distance deformable convolution block; and obtaining a quality prediction score based on the image block features and the text features supervised by image information. The application explores the powerful visual representation capability of a multi-modal large language, and extensive experiments are conducted in a panoramic image quality evaluation database, and the results show that the method has a significant advantage in evaluating panoramic image data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and multimedia digital image processing, and particularly relates to a multi-modal panoramic image blind quality evaluation method and system based on AI-generated description. BACKGROUND

[0002] With the rapid development and popularization of virtual reality (VR) technology, panoramic images (OIs) have shown their unique value in many application scenarios as an image form that can provide a 360-degree all-around visual experience. Whether it is virtual tourism leading users to experience the beauty of the world without leaving home, online education providing an immersive learning environment for students through panoramic classrooms, or real estate display allowing potential buyers to view properties online, OIs play a crucial role.

[0003] However, during the entire life cycle of acquisition, transmission, and display, OIs are inevitably affected by various factors, including but not limited to performance limitations of devices, bandwidth fluctuations of network transmission, and resolution differences of display devices, which collectively cause image quality degradation and affect user quality of experience (QoE). SUMMARY

[0004] In view of the above situation, the main purpose of the present application is to propose a multi-modal panoramic image blind quality evaluation method and system based on AI-generated description to solve the above technical problems.

[0005] The present application proposes a multi-modal panoramic image blind quality evaluation method based on AI-generated description, which comprises the following steps:

[0006] Step 1, generating a quality description text based on a distorted panoramic image;

[0007] By using a positioning and cropping method, an image block containing rich visual semantic information is obtained from the distorted panoramic image;

[0008] Step 2, pre-training a visual language model using the distorted panoramic image and the quality description text to obtain a pre-trained visual language model;

[0009] Step 3, extracting the quality description text and the image block containing rich visual semantic information using the pre-trained visual language model to obtain text features supervised by image information;

[0010] Step 4, extracting the image block using an isometric deformable convolution block to obtain image block features;

[0011] The image block features and the text features supervised by image information are spliced to obtain perceptual features after splicing and fusion.

[0012] The spliced and fused perceptual features are subjected to quality regression to obtain a quality prediction score.

[0013] The application further provides a multi-modal panoramic image blind quality evaluation system based on AI-generated description, which comprises:

[0014] A data generation module is configured to:

[0015] generate a quality description text based on the distorted panoramic image;

[0016] obtain an image block containing rich visual semantic information from the distorted panoramic image by adopting a positioning and cropping manner;

[0017] A pre-training module is configured to:

[0018] pre-train a visual language model by using the distorted panoramic image and the quality description text, to obtain a pre-trained visual language model;

[0019] A feature extraction module is configured to:

[0020] extract the quality description text and the image block containing rich visual semantic information by using the pre-trained visual language model, to obtain text features supervised by image information;

[0021] A quality regression module is configured to:

[0022] extract the image block by using an isometric deformable convolution block, to obtain image block features;

[0023] splice the image block features and the text features supervised by image information, to obtain spliced and fused perceptual features;

[0024] The spliced and fused perceptual features are subjected to quality regression to obtain a quality prediction score.

[0025] Compared with the prior art, the application has the following beneficial effects:

[0026] 1. The application develops a set of quality descriptions that can be generated for a single-modal panoramic image dataset by means of the powerful capability of a multi-modal large language model, constructs a multi-modal database, and enriches the information representation of panoramic image quality evaluation.

[0027] 2. The application directly performs a block operation on the original ERP image without performing viewport extraction, thereby avoiding additional computational burden and improving the universality of the model.

[0028] 3. The application pre-trains a visual language model by using the constructed multi-modal panoramic image database, thereby realizing efficient cross-modal feature extraction and fusion in the panoramic image quality evaluation task.

[0029] 4、The application can more accurately capture the local features of the image block by introducing the equidistant variable convolution module, and solve the problem of polar distortion of panoramic images.

[0030] 5、The application constructs a multi-modal feature fusion module, which not only enhances the richness of feature representation, but also effectively aligns the feature spaces of images and texts through the attention mechanism, ensuring the effective integration of multi-modal information.

[0031] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following description and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 Flow chart of the multi-modal panoramic image blind quality evaluation method based on AI generated description proposed by the application;

[0033] Figure 2 Quality description generation and pre-training Long-CLIP framework of the multi-modal panoramic image blind quality evaluation method based on AI generated description proposed by the application;

[0034] Figure 3 Long-CLIP and equidistant deformable convolution block double-branch prediction framework of the multi-modal panoramic image blind quality evaluation method based on AI generated description proposed by the application;

[0035] Figure 4 Overall framework schematic diagram of the multi-modal panoramic image blind quality evaluation system based on AI generated description proposed by the application. DETAILED DESCRIPTION

[0036] Embodiments of the application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the application, and cannot be understood as a limitation of the application.

[0037] These and other aspects of embodiments of the application will become apparent from the following description and drawings. In these descriptions and drawings, some specific implementations of embodiments of the application are specifically disclosed to represent some ways in which the principles of embodiments of the application can be implemented, but it should be understood that the scope of embodiments of the application is not limited thereto.

[0038] Please refer to Figure 1 and Figure 2 The multi-modal panoramic image blind quality evaluation method based on AI generated description proposed by the embodiments of the application comprises the following steps:

[0039] Step 1, generating quality description text based on distorted panoramic image;

[0040] By adopting the positioning clipping mode, the image block containing rich visual semantic information is obtained from the distorted panoramic image;

[0041] In step 1, the quality description text is generated based on the distorted panoramic image, and the corresponding process exists the following relationship:

[0042] ;

[0043] Among them, represents the image-text multimodal dataset, represents the processing of the multimodal large language model, represents the first image in the input dataset, represents the prompt text input to the multimodal large language model, represents the number of images in the dataset, represents the image data in the dataset, represents the text data in the dataset.

[0044] Further, in this step, the distorted panoramic image is obtained by ERP projection, and the number and size of the sampled image blocks are set to extract 10 image blocks with a size of 0.1Hx0.05W. In order to prevent overfitting, 80% of the images in the multimodal panoramic image database are used for training, and 20% are used for testing.

[0045] Step 2, pre-training the visual language model using the distorted panoramic image and the quality description text to obtain the pre-trained visual language model;

[0046] In step 2, the visual language model is pre-trained using the distorted panoramic image and the quality description text to obtain the pre-trained visual language model, which includes the following sub-steps:

[0047] Each distorted panoramic image and quality description text is input into the visual language model, and the feature representation is learned by comparing the differences and similarities between different image-text pairs. The corresponding process exists the following relationship:

[0048] ;

[0049] Among them, represents the feature representation of the quality description text of the image, represents the feature representation of the image, denotes a learnable temperature parameter, denotes the image-text contrastive learning loss function, denotes the text-image contrastive learning loss function, denotes a transpose symbol, denotes a feature representation of a quality description text of the th other image, denotes a feature representation of the th other image, denotes a sample number of the other image;

[0050] The final contrastive learning loss expression is as follows:

[0051] ;

[0052] wherein, denotes the final contrastive learning loss function.

[0053] By comparing the differences and similarities between different image-text pairs, the feature representations are learned so that the corresponding image-text pairs are close in the feature space, while the non-corresponding image-text pairs are far apart in the feature space. Finally, the visual language model parameters are optimized by minimizing the final contrastive learning loss function, and the pre-trained visual language model

[0054] Step 3, using the pre-trained visual language model to extract the quality description text and the image block containing rich visual semantic information, to obtain the text features supervised by image information;

[0055] In step 3, the pre-trained visual language model is used to extract the quality description text and the image block containing rich visual semantic information, to obtain the text features supervised by image information, and the specific steps are as follows:

[0056] The quality description text is labeled by the pre-trained visual language model, and then extracted by the text encoder to obtain the text features extracted by the pre-trained visual language model. The relationship between the corresponding processes is as follows:

[0057] ;

[0058] wherein, denotes the text features extracted by the pre-trained visual language model, processed by the text encoder of the pre-trained visual language model, processed by the text labeler of the pre-trained visual language model, denotes the pre-trained visual language model;

[0059] The perception features of the image blocks are extracted by using the image encoder of the pre-trained visual language model to obtain intermediate image block features, and a relationship formula corresponding to the process is:

[0060] ;

[0061] wherein, represents the intermediate image block feature, represents processing by the image encoder of the pre-trained visual language model, represents the i-th image block, represents the number of image blocks;

[0062] All obtained intermediate image block features are spliced and fused to obtain a fused image feature, and a relationship formula corresponding to the process is:

[0063] ;

[0064] wherein, represents the fused image feature, all represent intermediate image block features;

[0065] The key sequence and the value sequence are obtained based on the text features extracted by the pre-trained visual language model, and the query sequence is obtained by fusing the image feature, and a relationship formula corresponding to the process is:

[0066] ;

[0067] wherein, represents processing by a convolutional layer with a size of 1x1, represents the query sequence, represents the key sequence, represents the value sequence;

[0068] The query sequence, the key sequence and the value sequence are fused by attention to obtain a text feature supervised by image information, and a relationship formula corresponding to the process is:

[0069] ;

[0070] wherein, represents the text feature supervised by image information, represents a scaling factor in the attention mechanism, represents processing by an attention function.

[0071] In this step, the visual language model uses Long-CLIP;

[0072] The size of the text feature extracted by the pre-trained visual language model is 1x1x512;​

[0073] In the step of obtaining the intermediate image block feature, first, the image block is down-sampled to 224x224 size, and then input to the Long-CLIP network;

[0074] The size of the intermediate image block feature is 1x1x512;

[0075] The fused image feature is dimensionally reduced through a 1x1 convolution layer, and the dimensionally reduced image feature is taken as the query sequence. The text feature extracted by the pre-trained visual language model is dimensionally increased through a 1x1 convolution layer, and the dimensionally increased text feature is taken as the key sequence and the value sequence. The size of the query sequence is 1x2048, and the size of the key sequence and the value sequence is 1x1x2048;

[0076] The size of the text feature supervised by the image information is 1x1x2048.

[0077] Step 4, using an equidistant deformable convolution block to extract the image block to obtain an image block feature;

[0078] The image block feature is spliced with the text feature supervised by the image information to obtain a spliced and fused perception feature;

[0079] The spliced and fused perception feature is subjected to quality regression to obtain a quality prediction score;

[0080] Please refer to Figure 3 In step 4, an equidistant deformable convolution block is used to extract the image block to obtain an image block feature, and the specific steps are as follows:

[0081] The perception feature of the image block is extracted by using the equidistant deformable convolution block, and the relationship formula existing in the corresponding process is:

[0082] ;

[0083] Wherein, represents the perception feature of the image block, represents the processing through the equidistant deformable convolution block;

[0084] The perception features of all image blocks are spliced and fused to obtain an image block feature, and the relationship formula existing in the corresponding process is:

[0085] ;

[0086] Wherein, represents the image block feature, both represent the perception feature of the image block;

[0087] The image block feature is spliced with the text feature supervised by the image information to obtain a spliced and fused perception feature, and a relationship formula corresponding to a process exists as follows:

[0088] ;

[0089] wherein, represents the spliced and fused perception feature, represents processing by global average pooling;

[0090] The spliced and fused perception feature is subjected to quality regression to obtain a quality prediction score, and a relationship formula corresponding to a process exists as follows:

[0091] ;

[0092] wherein, represents the quality prediction score, represents processing by a linear layer, represents processing by a feature flattening operation.

[0093] In this step, in the process of obtaining the perception feature of the image block, the image block is first down-sampled to a size of 224x224, and then an equidistant variability convolution block is used to extract the image block feature, and the size of the perception feature of the image block is 28x28x512.

[0094] The size of the image block feature is 28x28x5120.

[0095] Further, in the execution of the above steps 1 to 4, the corresponding training method includes the following training steps:

[0096] The training data is obtained, and the training data includes a plurality of image blocks of a panoramic image and a quality description corresponding to the panoramic image. In this embodiment, 10 image blocks are used, and the pre-trained Long-CLIP parameter is obtained by repeating steps 1 and 2 with the training data as input. Then, the Long-CLIP parameter is frozen, and steps 1, 3 and 4 are repeated to obtain the predicted quality score.

[0097] The mean square error is used as the loss function of the quality score prediction, and the subjective score and the predicted quality score are used to construct the loss function, and the expression of the loss function is as follows:

[0098] ;

[0099] wherein, represents the MSE loss function, represents the number of images in the data set, represents the subjective score of the i-th image in the training data, ​a predicted quality score of an i-th image in the training data;

[0100] The MSE loss function is input into the Adam optimizer for optimization, and the Adam optimizer weight decay strategy and learning parameters are set, wherein the learning rate is set to 0.00005; the weight decay strategy, the decay rate is

[0101] The loss is minimized by updating the weights and learning parameters to improve the model performance.

[0102] Further, the model performance is evaluated by the following ways:

[0103] The average subjective score of all image data in the multi-modal panoramic image database is obtained, and the corresponding process exists a relationship formula:

[0104]

[0105] wherein, the average subjective score of the image data, the subjective score given by an i-th subject to the non-uniform distortion panoramic picture, the number of experimental persons participating in the evaluation of the quality of the non-uniform distortion panoramic image; The perception quality score of the obtained panoramic image is compared with the average subjective score of the image data to calculate the indicators of the model, wherein the indicators include the following three kinds:

[0106] The prediction monotonicity index includes the Spearman correlation coefficient, and the expression is:

[0107]

[0108]

[0109] wherein, the Spearman correlation coefficient, the difference between the subjective score of an i-th image and the objective predicted score;

[0110] The prediction accuracy index includes the Pearson correlation coefficient, and the expression is:

[0111]

[0112] wherein, the Pearson correlation coefficient, the average value of the subjective score, the average value of the objective predicted score;

[0113] The prediction error degree index includes the root mean square error, and the expression is:​​​​​​​

[0114] ;

[0115] wherein, denotes the root mean square error.

[0116] Referring to Figure 4 , the embodiment of the present application also provides a multi-modal panoramic image blind quality evaluation system based on AI-generated description, which comprises:

[0117] a data generation module, configured to:

[0118] generate a quality description text based on the distorted panoramic image;

[0119] obtain an image block containing rich visual semantic information from the distorted panoramic image by adopting a positioning and cropping manner;

[0120] a pre-training module, configured to:

[0121] pre-train a visual language model by using the distorted panoramic image and the quality description text, to obtain a pre-trained visual language model;

[0122] a feature extraction module, configured to:

[0123] extract the quality description text and the image block containing rich visual semantic information by using the pre-trained visual language model, to obtain text features supervised by image information;

[0124] a quality regression module, configured to:

[0125] extract the image block by using an isometric deformable convolution block, to obtain image block features;

[0126] splice the image block features and the text features supervised by image information, to obtain spliced and fused perception features;

[0127] perform quality regression on the spliced and fused perception features to obtain a quality prediction score.

[0128] It should be understood that the parts of the present application can be realized by hardware, software, firmware or their combination. In the above-mentioned embodiments, the plurality of steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if realized by hardware, and as in another embodiment, it can be realized by any one or their combination of the following technologies known in the art: discrete logic circuit with logic gate circuit for implementing logic function on data signal, special integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA) and the like.

[0129] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0130] The above-described embodiments only express several implementation manners of the present application, which are described in a more specific and detailed manner, but cannot be understood as a limitation on the patent scope of the present application. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. An AI-based multi-modal panoramic image blind quality evaluation method based on description generation, characterized by, The method comprises the following steps: Step 1, generating a quality description text based on a distorted panoramic image; An image block containing rich visual semantic information is obtained from the distorted panoramic image by adopting a positioning and cropping manner; Step 2, pre-training a visual language model by using the distorted panoramic image and the quality description text to obtain a pre-trained visual language model; Step 3, extracting the quality description text and the image block containing rich visual semantic information by using the pre-trained visual language model to obtain text features supervised by image information; Step 4, extracting the image block by using an equal distance deformable convolution block to obtain image block features; Splicing and fusing the image block features and the text features supervised by image information to obtain spliced and fused perception features; Performing quality regression on the spliced and fused perception features to obtain a quality prediction score; In the step 3, the quality description text and the image block containing rich visual semantic information are extracted by using the pre-trained visual language model to obtain text features supervised by image information, and the specific steps are as follows: The quality description text is labeled by the pre-trained visual language model, and then extracted by using a text encoder to obtain text features extracted by the pre-trained visual language model, and the relationship formula existing in the corresponding process is: ； wherein, represents a text feature extracted by a pre-trained visual language model, processed by a text encoder representing a pre-trained visual language model, processed by a text tagger of a pre-trained visual language model, represents a pre-trained visual language model; The perception features of the image block are extracted by using the image encoder of the pre-trained visual language model to obtain intermediate image block features, and the relationship formula existing in the corresponding process is: ； wherein, represents an intermediate image patch feature, represents processing by an image encoder of a pre-trained visual language model, represents a first image patch, represents a number of image patches; All the obtained intermediate image block features are spliced and fused to obtain fused image features, and the relationship formula existing in the corresponding process is: ； wherein, denote fused image features, denote intermediate image patch features; The text features extracted by the pre-trained visual language model and the fused image features are fused by using an attention mechanism to obtain text features supervised by image information; The text features extracted by the pre-trained visual language model and the fused image features are fused by using an attention mechanism, which specifically includes the following steps: The key sequence and the value sequence are obtained based on the text features extracted by the pre-trained visual language model, and the query sequence is obtained by the fused image features, and the relationship formula existing in the corresponding process is: ； wherein, represents processing by a convolutional layer with a size of 1x1, represents a query sequence, represents a key sequence, represents a value sequence; The query sequence, the key sequence and the value sequence are attention fused to obtain text features supervised by image information, and the relationship formula existing in the corresponding process is: ； wherein, denotes a text feature supervised by image information, denotes a scaling factor in the attention mechanism, denotes processing by the attention function; In the step 4, the image block is extracted by using the equal distance deformable convolution block to obtain the image block features, and the specific steps are as follows: The perception features of the image block are extracted by using the equal distance deformable convolution block, and the relationship formula existing in the corresponding process is: ； wherein, represents a perceptual feature of the image patch, represents processing by an equi-variant deformable convolution block; All the perception features of the image block are spliced and fused to obtain the image block features, and the relationship formula existing in the corresponding process is: ； wherein, denotes an image block feature, both denote a perceptual feature of an image block; The image block features and the text features supervised by image information are spliced to obtain spliced and fused perception features, and the relationship formula existing in the corresponding process is: ； wherein, represents the perceptual feature after splicing fusion, represents the processing by global average pooling; The spliced and fused perception features are subjected to quality regression to obtain a quality prediction score, and the relationship formula existing in the corresponding process is: ； wherein, denotes a quality prediction score, denotes processing by a linear layer, denotes processing by a feature flattening operation; In the execution of the above steps 1 to 4, the corresponding training method comprises the following training steps: Obtaining training data, the training data including a plurality of image blocks of distorted panorama images and corresponding quality descriptions, using 10 image blocks, taking the training data as input, repeating steps 1 and 2 to obtain pre-trained Long-CLIP parameters, then freezing the Long-CLIP parameters, repeating steps 1, 3 and 4 to obtain predicted quality scores; Using mean square error as the loss function of quality score prediction, constructing a loss function using subjective scores and predicted quality scores, and the expression of the loss function is: ； wherein, denotes the MSE loss function, denotes the number of images in the dataset, denotes the subjective score of the -th image in the training data, denotes the predicted quality score of the -th image in the training data; The MSE loss function is input into the Adam optimizer for optimization, and the Adam optimizer weight decay strategy and learning parameters are set, wherein the learning rate is set to 0.00005; the weight decay strategy, the decay rate is . 2.The AI-based generation description multi-modal panoramic image blind quality evaluation method according to claim 1, characterized in that, In the step 1, the quality description text is generated based on the distorted panorama image, and the corresponding process exists the following relationship: ； wherein, represents an image-text multimodal dataset, represents processing by a multimodal large language model, represents the i-th image in the input dataset, represents the i-th image in the input dataset, represents the prompt text input to the multimodal large language model, represents the number of images in the dataset, represents the image data in the dataset, represents the text data in the dataset. 3.The AI-based generation description multi-modal panoramic image blind quality evaluation method of claim 2, wherein, In the step 2, the visual language model is pre-trained using the distorted panorama image and the quality description text to obtain a pre-trained visual language model, and the process includes the following sub-steps: Each distorted panorama image and quality description text is input into the visual language model, and the feature representation is learned by comparing the differences and similarities between different image-text pairs, and the corresponding process exists the following relationship: ； in, Indicates the first Feature representation of the quality description text of an image. Indicates the first Feature representation of an image This represents a learnable temperature parameter. This represents the image-text contrastive learning loss function. This represents the text-image contrastive learning loss function. Indicates the transpose symbol. Indicates the first Feature representation of the quality description text of another image. Indicates the first Feature representation of another image, Indicates the number of samples in other images; The final contrastive learning loss expression is as follows: ； wherein, represents the final contrastive learning loss function; The visual language model is optimized by minimizing the final contrastive learning loss function, and a pre-trained visual language model is obtained.

4. An AI-based multi-modal panoramic image blind quality evaluation system generating a description, characterized by, The system applies any one of the AI-based generation description multi-modal panorama image blind quality evaluation methods according to claims 1 to 3, and the system comprises: A data generation module for: Generating quality description text based on distorted panorama image; Obtaining image blocks containing rich visual semantic information from the distorted panorama image by using positioning and cropping; A pre-training module for: Pre-training the visual language model using the distorted panorama image and the quality description text to obtain a pre-trained visual language model; A feature extraction module for: Extracting the quality description text and the image blocks containing rich visual semantic information using the pre-trained visual language model to obtain text features supervised by image information; A quality regression module for: Extracting the image blocks using the isometric deformable convolution block to obtain image block features; Splicing the image block features with the text features supervised by image information to obtain spliced and fused perception features; Performing quality regression on the spliced and fused perception features to obtain quality prediction scores.

Images