Panoramic stereo image quality evaluation method and device based on multi-level feature fusion

Abstract

The embodiments of the present invention provide a method and device for evaluating the quality of a panoramic stereoscopic image based on multi-level feature fusion. The method includes: obtaining left and right views, and then performing viewport segmentation to obtain left and right viewport images; subtracting the left viewport image from its pseudo-reference viewport image, and inputting the monocular feature extraction model to obtain the viewport monocular feature; inputting the left and right viewport images into the ResNet50 model, and respectively inputting them into the binocular sum feature extraction model and the binocular difference feature extraction model to obtain the viewport binocular sum feature and the viewport binocular difference feature; performing feature fusion on the above three features to obtain the quality evaluation score of the viewport image; and obtaining the quality evaluation score of the target based on the quality evaluation score of each viewport image of the target. In this way, the viewport characteristics and stereoscopic vision characteristics of the human eye when viewing panoramic images can be effectively combined, reducing the computational complexity while capturing rich visual information and complete stereoscopic perception information, and using multi-level feature extraction to improve the model accuracy.

Description

Technical Field

[0001] The present invention generally relates to the field of panoramic stereoscopic image quality assessment, and more specifically, to a panoramic stereoscopic image quality assessment method and device based on multi-level feature fusion. Background Art

[0002] With the development of software and hardware products, panoramic content has attracted people's attention in the fields of games, entertainment, film and television. In order to enjoy an immersive stereoscopic all-round visual feast, stereoscopic panoramic images are expected to become the main visual content. Similar to other image formats, stereoscopic panoramic images will inevitably introduce distortion during the capture, compression, and transmission processes, resulting in poor image quality perception. Stereoscopic panoramic images involve the characteristics of stereoscopic images and panoramic images. Therefore, when evaluating the quality of stereoscopic panoramic images, it is necessary to consider multiple aspects such as stereoscopic visual perception, view format, and feature fusion. It is also necessary to consider how to effectively combine this information to extract effective quality loss information. In order to solve the problem of stereoscopic panoramic image quality evaluation, the design of an effective quality evaluation model has attracted the attention of many researchers.

[0003] At present, the quality evaluation of stereoscopic panoramic images is still in its early stages. The existing indicators are all based on the research of stereoscopic image quality evaluation methods, and most of them are designed and implemented based on binocular stereo perception features, monocular panoramic vision features or a combination of the two. The existing quality evaluation models that only rely on stereoscopic visual information usually ignore the visual characteristics of panoramic images and have poor performance. Inspired by this, researchers have extracted stereoscopic visual information based on the visual perception characteristics of panoramic images. The performance of the model has been improved to a certain extent, but there are still problems such as high computational cost and insufficient extraction of deep visual semantic information. Recently, many researchers have considered the above problems and continuously proposed new models, but the mining and deep fusion of important visual features have become one of the important factors limiting the performance of current models. Therefore, it is necessary to effectively combine human panoramic vision characteristics, study deep feature extraction and fusion strategies, and construct an effective panoramic stereo image quality evaluation method. Summary of the invention

[0004] According to an embodiment of the present invention, a panoramic stereo image quality evaluation scheme based on multi-level feature fusion is provided. This scheme is universal, effectively strengthens the effective fusion of shallow binocular features and deep binocular features, reduces feature redundancy, and improves model accuracy.

[0005] The present invention provides a panoramic stereo image quality evaluation method based on multi-level feature fusion. The method comprises:

[0006] Obtaining a left view and a right view of a target distorted panoramic stereo image, performing viewport segmentation on the left view and the right view of the target distorted panoramic stereo image, and obtaining a left viewport image and a right viewport image;

[0007] Subtracting the left viewport image from the pseudo reference viewport image obtained by restoring the left viewport image to obtain viewport distortion information, inputting the viewport distortion information into a monocular feature extraction model, and outputting a viewport monocular feature; and

[0008] Input the left viewport image and the right viewport image into the ResNet50 model to obtain multi-level visual features of the two viewports; input the multi-level visual features of the two viewports into the binocular sum feature extraction model and the binocular difference feature extraction model respectively to obtain the viewport binocular sum feature and the viewport binocular difference feature;

[0009] The monocular viewport feature, the binocular viewport sum feature, and the binocular viewport difference feature are fused to obtain a quality evaluation score of the viewport image;

[0010] According to the quality evaluation score of each viewport image of the target distorted panoramic stereo image, a quality evaluation score of the target distorted panoramic stereo image is obtained.

[0011] Furthermore, the ResNet50 model includes, in sequence: a convolutional layer, a pooling layer, a first residual block, a second residual block, a third residual block and a fourth residual block;

[0012] The second residual block is used to output the first-level visual features of the left viewport and the first-level visual features of the right viewport ;

[0013] The third residual block is used to output the second-level visual features of the left viewport And the second level visual features of the right viewport ;

[0014] The fourth residual block is used to output the third-level visual features of the left viewport And the third-level visual features of the right viewport .

[0015] Furthermore, the binocular and feature extraction model includes a primary and model, a first secondary and model, and a second secondary and model;

[0016] The first level and model are the first level visual features of the left viewport and the first-level visual features of the right viewport As input, output the first level stereo and feature ;

[0017] The first secondary and the model are the second-level visual features of the left viewport , the second-level visual features of the right viewport , first-level binocular and features As input, output second-level stereo and feature ;

[0018] The second level and the model are the third level visual features of the left viewport , the third-level visual features of the right viewport , second level binocular and features As input, it outputs viewport stereo and features.

[0019] Further, the first level and model are based on the first level visual features of the left viewport and the first-level visual features of the right viewport As input, output the first level stereo and feature ,include:

[0020] The first-level visual features of the left viewport Input the first SW-MSA module, the output of the first SW-MSA module is fed into the first GELU activation function, and the output of the first GELU activation function is combined with the first-level visual features of the left viewport Add the output of the first level left viewport and features ;

[0021] The first level visual features of the right viewport Input the first LN layer, the output of the first LN layer is sent to the second GELU activation function, and the output of the second GELU activation function is combined with the first-level visual features of the right viewport Add the output of the first level right viewport and features ;

[0022] Change the first level left viewport and feature With the first level right viewport and features Input the first concatenation module, the output data of the first concatenation module passes through the second LN layer and the first fully connected layer in turn, and outputs the first-level binocular and feature .

[0023] Further, the first secondary and the model are based on the second level visual features of the left viewport , the second-level visual features of the right viewport , first-level binocular and features As input, output second-level stereo and feature ,include:

[0024] The second level visual features of the left viewport Input the second SW-MSA module, the output of the second SW-MSA module is sent to the third GELU activation function, and the output of the third GELU activation function is combined with the second-level visual features of the left viewport Add the output of the second level left viewport and features ;

[0025] The second level visual features of the right viewport Input the third LN layer, the output of the third LN layer is sent to the fourth GELU activation function, and the output of the fourth GELU activation function is combined with the second-level visual features of the right viewport Add the output of the second level right viewport and features ;

[0026] Set the secondary left viewport and feature , Second level right viewport and features and the first level binocular and feature Input the second concatenation module, the output data of the second concatenation module passes through the fourth LN layer and the second fully connected layer in turn, and outputs the second-level binocular and feature .

[0027] Further, the second level and the model are based on the third level visual features of the left viewport , the third-level visual features of the right viewport , second level binocular and features As input, output viewport stereo and features, including:

[0028] The third level visual features of the left viewport Input the third SW-MSA module, the output of the third SW-MSA module is sent to the fifth GELU activation function, the output of the fifth GELU activation function is combined with the third-level visual features of the left viewport Add the output of the third level left viewport and features ;

[0029] The third level visual features of the right viewport Input the fifth LN layer, the output of the fifth LN layer is sent to the sixth GELU activation function, and the output of the sixth GELU activation function is combined with the third-level visual features of the right viewport Add output third level right viewport and features ;

[0030] Set the tertiary left viewport and feature , tertiary right viewport and features And the second level binocular and feature The third concatenated module is input, and the output data of the third concatenated module passes through the sixth LN layer and the third fully connected layer in sequence to output the viewport binocular and features.

[0031] Furthermore, the binocular difference feature extraction model includes: a fourth concate module, a seventh LN layer, and a fourth fully connected layer;

[0032] The input data of the fourth concatenator module includes: the first-level binocular difference feature , Second level binocular difference feature and the third-level binocular difference feature ; The first level binocular difference feature is the first-level visual feature of the left viewport The first level visual features with the right viewport The result of the difference, the second-level binocular difference feature is the second-level visual feature of the left viewport The second level visual features with the right viewport The result of the difference is the third-level binocular difference feature is the third-level visual feature of the left viewport The third level visual feature with the right viewport The result of doing something wrong;

[0033] The output data of the fourth concatenated module passes through the seventh LN layer and the fourth fully connected layer in sequence, and then outputs the viewport binocular difference features.

[0034] Furthermore, the viewport monocular feature, the viewport binocular sum feature, and the viewport binocular difference feature are fused to obtain a quality evaluation score of the viewport image, including:

[0035] The viewport binocular and features and the viewport monocular features are globally pooled, and the global pooling result and the viewport binocular difference features are input into a fifth concate module; the output of the fifth concate module is used as the input of a fifth fully connected layer; the output of the fifth fully connected layer and the viewport binocular difference features are input into a sixth concate module; the output of the sixth concate module is used as the input of a sixth fully connected layer to obtain a viewport image quality evaluation score.

[0036] In a second aspect of the present invention, a panoramic stereoscopic image quality evaluation device based on multi-level feature fusion is provided. The device comprises:

[0037] An acquisition module is used to acquire a left view and a right view of the target distorted panoramic stereo image, and perform viewport segmentation on the left view and the right view of the target distorted panoramic stereo image to obtain a left viewport image and a right viewport image;

[0038] The feature extraction module comprises: a first feature extraction module and a second feature extraction module;

[0039] A first feature extraction module is used to obtain viewport distortion information by subtracting the left viewport image from the pseudo reference viewport image obtained by restoring the left viewport image, and input the viewport distortion information into a monocular feature extraction model to output a viewport monocular feature;

[0040] The second feature extraction module is used to input the left viewport image and the right viewport image into the ResNet50 model to obtain multi-level visual features of the two viewports; the multi-level visual features of the two viewports are respectively input into the binocular sum feature extraction model and the binocular difference feature extraction model to obtain the viewport binocular sum feature and the viewport binocular difference feature;

[0041] A first evaluation module is used to perform feature fusion on the viewport monocular feature, the viewport binocular sum feature, and the viewport binocular difference feature to obtain a quality evaluation score of the viewport image;

[0042] The second evaluation module is used to obtain a quality evaluation score of the target distorted panoramic stereo image according to the quality evaluation score of each viewport image of the target distorted panoramic stereo image.

[0043] In a third aspect of the present invention, an electronic device is provided. The electronic device has at least one processor; and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method of the first aspect of the present invention.

[0044] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0046] Figure 1 A flow chart of a panoramic stereoscopic image quality assessment method based on multi-level feature fusion according to an embodiment of the present invention is shown.

[0047] Figure 2 A structural diagram of a Vmamba model according to an embodiment of the present invention is shown.

[0048] Figure 3 A structural diagram of a Vmamba model VSS Block according to an embodiment of the present invention is shown.

[0049] Figure 4 A structural diagram of a ResNet50 model according to an embodiment of the present invention is shown.

[0050] Figure 5 A structural diagram of a binocular and feature extraction model according to an embodiment of the present invention is shown.

[0051] Figure 6 A structural diagram of a binocular difference feature extraction model according to an embodiment of the present invention is shown.

[0052] Figure 7 A schematic diagram of quality evaluation according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0055] Figure 1 A flow chart of a panoramic stereoscopic image quality assessment method based on multi-level feature fusion according to an embodiment of the present invention is shown.

[0056] The method includes:

[0057] S101 , obtaining a left view and a right view of a target distorted panoramic stereo image, performing viewport segmentation on the left view and the right view of the target distorted panoramic stereo image, and obtaining a left viewport image and a right viewport image.

[0058] Specifically, the viewport acquisition method FoV Selection (Field of View Selection) is used to obtain the viewport images of the left and right views. and .

[0059] Viewport segmentation of the distorted panorama left and right images is in line with human visual perception characteristics and helps to extract visual features.

[0060] S102, subtracting the left viewport image from the pseudo reference viewport image obtained by restoring the left viewport image to obtain viewport distortion information, inputting the viewport distortion information into a monocular feature extraction model, and outputting a viewport monocular feature; and

[0061] The left viewport image and the right viewport image are input into the ResNet50 model to obtain multi-level visual features of the two viewports; the multi-level visual features of the two viewports are respectively input into the binocular sum feature extraction model and the binocular difference feature extraction model to obtain the viewport binocular sum feature and the viewport binocular difference feature.

[0062] The first part is the extraction of the monocular features of the viewport:

[0063] Specifically, the process of obtaining the pseudo reference viewport image is as follows:

[0064] (1) Input the left viewport image and the text sentence “Restore image quality” into the InstructIR model (High-Quality Image Restoration Following Human Instructions), perform restoration operations on the left viewport image, and obtain a pseudo-reference viewport image. .

[0065] (2) The left viewport image of the distorted panorama With the corresponding pseudo reference viewport Subtract to get viewport distortion information , the calculation formula is as follows:

[0066]

[0067] Using a pseudo-reference viewport image to obtain distortion information can enable the method to capture rich monocular view quality distortion information without reference information, thereby improving feature representation capabilities and the generalization of the method.

[0068] In this embodiment, the monocular feature extraction module is used as follows Figure 2 The Vmamba model shown is used as a backbone network to extract monocular features. The backbone network structure includes: a Stem (Spatial Transformer Embedding Module, spatial transformer embedding model) block, a first visual state space block (VSS Block, Visual State Space Block), a first down sampling module (Down Sampling), a second visual state space block (VSS Block), a second down sampling module (Down Sampling), a third visual state space block (VSS Block), a third down sampling module (DownSampling), a fourth visual state space block (VSS Block), and a fully connected layer (FC).

[0069] Among them, Figure 3 As shown, each visual state space (VSS Block) block includes:

[0070] (1) The first LN layer (Layer Normalization): The input data is the output of the Stem block or the previous down sampling module (Down Sampling).

[0071] (2) 2D-selective-scan Block (SS2d Block): The input data is the output of the first LN.

[0072] (3) Second LN layer: The input data is the result of adding the input of the first LN layer and the output of the SS2d Block.

[0073] (4) FFN layer (Feed Forward Network): The input data is the output of the second LN layer.

[0074] The data obtained by adding the input of the second LN layer and the output of the FFN is used as the output of the VSS Block.

[0075] The second part is the extraction of the multi-level visual features:

[0076] In this embodiment, if Figure 4 As shown, the ResNet50 model includes, in sequence: a convolutional layer, a pooling layer, a first residual block, a second residual block, a third residual block and a fourth residual block.

[0077] The convolutional layer is used to extract primary features of the image.

[0078] The pooling layer is used for feature aggregation and dimensionality reduction.

[0079] The first, second, third and fourth residual blocks are used to extract deep high-level features of the image.

[0080] The second residual block is used to output the first-level visual features of the left viewport and the first-level visual features of the right viewport .

[0081] The third residual block is used to output the second-level visual features of the left viewport And the second level visual features of the right viewport .

[0082] The fourth residual block is used to output the third-level visual features of the left viewport And the third-level visual features of the right viewport .

[0083] Specifically, in this embodiment, the ResNet50 model has a total of 50 layers, namely, 1 layer of 7*7 convolutional layer with 64 channels, 1 layer of 3*3 maximum pooling layer, the first residual block (including 1 layer of 1*1 convolutional layer with 64 channels, 1 layer of 3*3 convolutional layer with 64 channels, and 1 layer of 1*1 convolutional layer with 256 channels)*3, the second residual block (including 1 layer of 1*1 convolutional layer with 128 channels, 1 layer of 3*3 convolutional layer with 128 channels, and 1 layer of 3*3 convolutional layer with 256 channels)*3. 128 convolution layer, 1 1*1 convolution layer with 512 channels)*4, the third residual block (including 1 1*1 convolution layer with 256 channels, 1 3*3 convolution layer with 256 channels, 1 1*1 convolution layer with 1024 channels)*6, the fourth residual block (including 1 1*1 convolution layer with 512 channels, 1 3*3 convolution layer with 512 channels, and 1 1*1 convolution layer with 2048 channels)*3.

[0084] By utilizing the outputs of the last three groups of residual blocks, we can effectively obtain deep monocular feature information, obtain rich monocular quality degradation information, and improve the monocular feature representation capability.

[0085] In this embodiment, if Figure 5 As shown, the binocular and feature extraction model includes: a primary and model, a first secondary and model and a second secondary and model.

[0086] The first level and model are the first level visual features of the left viewport and the first-level visual features of the right viewport As input, output the first level stereo and feature .

[0087] The first secondary and the model are the second-level visual features of the left viewport , the second-level visual features of the right viewport , first-level binocular and features As input, output second-level stereo and feature .

[0088] The second level and the model are the third level visual features of the left viewport , the third-level visual features of the right viewport , second level binocular and features As input, it outputs viewport stereo and features.

[0089] The stereo and feature extraction modules realize the gradual aggregation of features from shallow to deep, enhance the feature interaction between different layers, help to mine deep stereo and semantic information, and improve model accuracy.

[0090] Specifically, the first level and model are based on the first level visual features of the left viewport and the first-level visual features of the right viewport As input, output the first level stereo and feature ,include::

[0091] The first-level visual features of the left viewport Input the first SW-MSA module, the output of the first SW-MSA module is fed into the first GELU activation function, and the output of the first GELU activation function is combined with the first-level visual features of the left viewport Add the output of the first level left viewport and features .

[0092] The first level visual features of the right viewport Input the first LN layer, the output of the first LN layer is sent to the second GELU activation function, and the output of the second GELU activation function is combined with the first-level visual features of the right viewport Add the output of the first level right viewport and features .

[0093] Set the first level left viewport and feature With the first level right viewport and features Input the first concatenation module, the output data of the first concatenation module passes through the second LN layer and the first fully connected layer in turn, and outputs the first-level binocular and feature .

[0094] The first level and model use a sliding window to calculate the attention mechanism, which can improve the learning ability of the left viewport visual features and reduce the amount of calculation and memory consumption; the residual connection can effectively solve the gradient disappearance problem and make the deep network training more stable; the simple concatenation operation can reduce the computational complexity of the binocular and features.

[0095] Specifically, the first secondary and the model are based on the second-level visual features of the left viewport , the second-level visual features of the right viewport , first-level binocular and features As input, output second-level stereo and feature ,include:

[0096] The second level visual features of the left viewport Input the second SW-MSA module, the output of the second SW-MSA module is sent to the third GELU activation function, and the output of the third GELU activation function is combined with the second-level visual features of the left viewport Add the output of the second level left viewport and features .

[0097] The second level visual features of the right viewport Input the third LN layer, the output of the third LN layer is sent to the fourth GELU activation function, and the output of the fourth GELU activation function is combined with the second-level visual features of the right viewport Add the output of the second level right viewport and features .

[0098] Set the secondary left viewport and feature , Second level right viewport and features and the first level binocular and feature Input the second concatenation module, the output data of the second concatenation module passes through the fourth LN layer and the second fully connected layer in turn, and outputs the second-level binocular and feature .

[0099] The first and second level sum models fuse binocular sum information from different scales, and can deeply explore binocular sum features; simple concatenation operation can reduce the computational complexity of binocular sum features, and realize simple and efficient fusion of multi-level binocular sum features.

[0100] Specifically, the second level and the model are based on the third level visual features of the left viewport , the third-level visual features of the right viewport , second level binocular and features As input, output viewport stereo and features, including:

[0101] The third level visual features of the left viewport Input the third SW-MSA module, the output of the third SW-MSA module is sent to the fifth GELU activation function, the output of the fifth GELU activation function is combined with the third-level visual features of the left viewport Add the output of the third level left viewport and features .

[0102] The third level visual features of the right viewport Input the fifth LN layer, the output of the fifth LN layer is sent to the sixth GELU activation function, and the output of the sixth GELU activation function is combined with the third-level visual features of the right viewport Add output third level right viewport and features .

[0103] Set the tertiary left viewport and feature , tertiary right viewport and features And the second level binocular and feature The third concatenated module is input, and the output data of the third concatenated module passes through the sixth LN layer and the third fully connected layer in sequence to output the viewport binocular and features.

[0104] The second level sum model fuses binocular sum information from different scales and can deeply mine multi-level high-level semantic information; the simple concatenation operation can reduce the computational complexity of binocular sum features and realize the effective fusion of multi-level binocular sum features.

[0105] In this embodiment, if Figure 6 As shown, the binocular difference feature extraction model includes: a fourth concate module, a seventh LN layer, and a fourth fully connected layer.

[0106] The input data of the fourth concatenator module includes: the first-level binocular difference feature , Second level binocular difference feature and the third-level binocular difference feature .

[0107] Among them, the first-level binocular difference feature is the first-level visual feature of the left viewport The first level visual features with the right viewport The result of the difference is calculated as follows:

[0108]

[0109] Among them, the second-level binocular difference feature is the second-level visual feature of the left viewport The second level visual features with the right viewport The result of the difference is calculated as follows:

[0110]

[0111] Among them, the third-level binocular difference feature is the third-level visual feature of the left viewport The third level visual feature with the right viewport The result of the difference is calculated as follows:

[0112]

[0113] The output data of the fourth concatenated module passes through the seventh LN layer and the fourth fully connected layer in sequence, and then outputs the viewport binocular difference features.

[0114] Using multi-level monocular features to calculate binocular difference features can effectively obtain rich stereo perception quality degradation information and improve the representation ability of binocular difference features. In addition, using simple features to perform difference to obtain binocular difference features can reduce the model calculation complexity and save calculation time.

[0115] S103, performing feature fusion on the viewport monocular feature, the viewport binocular sum feature, and the viewport binocular difference feature to obtain a quality evaluation score of the viewport image.

[0116] like Figure 7 The quality assessment flow chart shown includes:

[0117] The viewport binocular and features and the viewport monocular features are globally pooled, and the global pooling result and the viewport binocular difference features are input into a fifth concate module; the output of the fifth concate module is used as the input of a fifth fully connected layer; the output of the fifth fully connected layer and the viewport binocular difference features are input into a sixth concate module; the output of the sixth concate module is used as the input of a sixth fully connected layer to obtain a viewport image quality evaluation score.

[0118] This fusion model enhances the interactive fusion of monocular features and binocular features, reduces the dimension and redundancy of features, and realizes efficient fusion of features; considering that binocular difference information is an important factor affecting the stereoscopic vision perception effect, this fusion model uses binocular difference features as the dominant feature to achieve fusion, which can enhance the proportion of disparity information in the fusion features and help improve the accuracy of the model.

[0119] S104 . Obtain a quality evaluation score of the target distorted panoramic stereo image according to the quality evaluation score of each viewport image of the target distorted panoramic stereo image.

[0120] Specifically, the average value of the quality evaluation score of each viewport image is used as the quality evaluation score of the target distorted panoramic stereo image to evaluate the quality of the target distorted panoramic stereo image.

[0121] According to the embodiments of the present invention, the present invention has the following advantages and effects compared with the prior art:

[0122] (1) Using the viewport map as the research target is more suitable for the viewing characteristics of the human eye. At the same time, the image restoration module is used to obtain the pseudo-reference left viewport map, so that this method is free from the limitation of the reference map and is more universal. Combining the distortion information to extract monocular features can reflect the monocular perception quality degradation information and improve the feature representation ability.

[0123] (2) The output of multiple residual modules of the ResNet50 model is used to extract visual features, which can effectively connect shallow visual features and deep visual features and capture rich binocular visual information.

[0124] (3) Using the binocular image processing model to construct a multi-level binocular image processing feature extraction module can effectively enhance the effective fusion of shallow binocular image processing features and deep binocular image processing features, reduce feature redundancy, and improve model accuracy.

[0125] (4) Using multi-level visual features to directly perform subtraction to obtain multi-level binocular disparity information can greatly reduce the computational complexity of disparity information. Using multi-level disparity information to directly stitch together binocular disparity information can capture rich disparity information and high-level semantic information.

[0126] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0127] The above is an introduction to a method embodiment. The following is a further explanation of the scheme of the present invention through a device embodiment having the same inventive concept as the method in the aforementioned embodiment.

[0128] The device includes:

[0129] The acquisition module is used to acquire the left view and the right view of the target distorted panoramic stereo image, and perform viewport segmentation on the left view and the right view of the target distorted panoramic stereo image to obtain the left viewport image and the right viewport image.

[0130] The feature extraction module comprises: a first feature extraction module and a second feature extraction module;

[0131] The first feature extraction module is used to obtain viewport distortion information by subtracting the left viewport image from the pseudo reference viewport image obtained by restoring the left viewport image, and input the viewport distortion information into a monocular feature extraction model to output a viewport monocular feature.

[0132] The second feature extraction module is used to input the left viewport image and the right viewport image into the ResNet50 model to obtain multi-level visual features of the two viewports; the multi-level visual features of the two viewports are respectively input into the binocular sum feature extraction model and the binocular difference feature extraction model to obtain the viewport binocular sum feature and the viewport binocular difference feature.

[0133] The first evaluation module is used to fuse the viewport monocular features, the viewport binocular sum features, and the viewport binocular difference features to obtain a quality evaluation score of the viewport image.

[0134] The second evaluation module is used to obtain a quality evaluation score of the target distorted panoramic stereo image according to the quality evaluation score of each viewport image of the target distorted panoramic stereo image.

[0135] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0136] In the technical solution of the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0137] According to an embodiment of the present invention, the present invention further provides an electronic device.

[0138] Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0139] The electronic device includes a computing unit, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device can also be stored. The computing unit, ROM, and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0140] Multiple components in an electronic device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0141] The computing unit may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing units include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit performs the various methods and processes described above, such as methods S101 to S104. For example, in some embodiments, methods S101 to S104 may be implemented as computer software programs that are tangibly contained in machine-readable media, such as storage units. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the computing unit, one or more steps of methods S101 to S104 described above may be performed. Alternatively, in other embodiments, the computing unit may be configured to perform methods S101 to S104 in any other appropriate manner (e.g., by means of firmware).

[0142] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0144] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0145] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A panoramic stereo image quality evaluation method based on multi-level feature fusion, characterized in that: include: Obtaining a left view and a right view of a target distorted panoramic stereo image, performing viewport segmentation on the left view and the right view of the target distorted panoramic stereo image, and obtaining a left viewport image and a right viewport image; Subtracting the left viewport image from the pseudo reference viewport image obtained by restoring the left viewport image, obtaining viewport distortion information, and inputting the viewport distortion information into a monocular feature extraction model to output a viewport monocular feature; as well as Input the left viewport image and the right viewport image into the ResNet50 model to obtain the multi-level visual features of the two viewports; Inputting the multi-level visual features of the two viewports into a binocular sum feature extraction model and a binocular difference feature extraction model respectively to obtain a viewport binocular sum feature and a viewport binocular difference feature; The monocular viewport feature, the binocular viewport sum feature, and the binocular viewport difference feature are fused to obtain a quality evaluation score of the viewport image; According to the quality evaluation score of each viewport image of the target distorted panoramic stereo image, a quality evaluation score of the target distorted panoramic stereo image is obtained; The ResNet50 model includes, in sequence: a convolutional layer, a pooling layer, a first residual block, a second residual block, a third residual block and a fourth residual block; The second residual block is used to output the first-level visual features of the left viewport and the first-level visual features of the right viewport ; The third residual block is used to output the second-level visual features of the left viewport And the second level visual features of the right viewport ; The fourth residual block is used to output the third-level visual features of the left viewport And the third-level visual features of the right viewport ; The binocular and feature extraction model includes a primary and model, a first and second secondary and model; The first level and model are the first level visual features of the left viewport and the first-level visual features of the right viewport As input, output the first level stereo and feature ; The first secondary and the model are the second-level visual features of the left viewport , the second-level visual features of the right viewport , first-level binocular and features As input, output second-level stereo and feature ; The second level and the model are the third level visual features of the left viewport , the third-level visual features of the right viewport , second level binocular and features As input, it outputs viewport stereo and features.

2. The method according to claim 1, characterized in that The first level and model are the first level visual features of the left viewport and the first-level visual features of the right viewport As input, output the first level stereo and feature ,include: The first level visual features of the left viewport Input the first SW-MSA module, the output of the first SW-MSA module is fed into the first GELU activation function, and the output of the first GELU activation function is combined with the first-level visual features of the left viewport Add the output of the first level left viewport and features ; The first level visual features of the right viewport Input the first LN layer, the output of the first LN layer is sent to the second GELU activation function, and the output of the second GELU activation function is combined with the first-level visual features of the right viewport Add the output of the first level right viewport and features ; Set the first level left viewport and feature With the first level right viewport and features Input the first concatenation module, the output data of the first concatenation module passes through the second LN layer and the first fully connected layer in turn, and outputs the first-level binocular and feature .

3. The method according to claim 1, characterized in that The first secondary and the model are the second-level visual features of the left viewport , the second-level visual features of the right viewport , first-level binocular and features As input, output second-level stereo and feature ,include: The second level visual features of the left viewport Input the second SW-MSA module, the output of the second SW-MSA module is sent to the third GELU activation function, and the output of the third GELU activation function is combined with the second-level visual features of the left viewport Add the output of the second level left viewport and features ; The second level visual features of the right viewport Input the third LN layer, the output of the third LN layer is sent to the fourth GELU activation function, and the output of the fourth GELU activation function is combined with the second-level visual features of the right viewport Add the output of the second level right viewport and features ; Set the secondary left viewport and feature , Second level right viewport and features and the first level binocular and feature Input the second concatenation module, the output data of the second concatenation module passes through the fourth LN layer and the second fully connected layer in turn, and outputs the second-level binocular and feature .

4. The method according to claim 1, characterized in that: The second level and the model are the third level visual features of the left viewport , the third-level visual features of the right viewport , second level binocular and features As input, output viewport stereo and features, including: The third level visual features of the left viewport Input the third SW-MSA module, the output of the third SW-MSA module is sent to the fifth GELU activation function, the output of the fifth GELU activation function is combined with the third-level visual features of the left viewport Add the output of the third level left viewport and features ; The third level visual features of the right viewport Input the fifth LN layer, the output of the fifth LN layer is sent to the sixth GELU activation function, and the output of the sixth GELU activation function is combined with the third-level visual features of the right viewport Add output third level right viewport and features ; Set the tertiary left viewport and feature , tertiary right viewport and features And the second level binocular and feature The third concatenated module is input, and the output data of the third concatenated module passes through the sixth LN layer and the third fully connected layer in sequence to output the viewport binocular and features.

5. The method according to claim 1, characterized in that The binocular difference feature extraction model includes: a fourth concate module, a seventh LN layer, and a fourth fully connected layer; The input data of the fourth concatenator module includes: the first-level binocular difference feature , Second level binocular difference feature and the third-level binocular difference feature ; The first level binocular difference feature is the first-level visual feature of the left viewport The first level visual features with the right viewport The result of the difference, the second-level binocular difference feature is the second-level visual feature of the left viewport The second level visual features with the right viewport The result of the difference is the third-level binocular difference feature is the third-level visual feature of the left viewport The third level visual feature with the right viewport The result of doing something wrong; The output data of the fourth concatenated module passes through the seventh LN layer and the fourth fully connected layer in sequence, and then outputs the viewport binocular difference features.

6. The method according to claim 1, characterized in that The viewport monocular feature, the viewport binocular sum feature, and the viewport binocular difference feature are fused to obtain a quality evaluation score of the viewport image, including: The viewport binocular and features and the viewport monocular features are globally pooled, and the global pooling result and the viewport binocular difference features are input into a fifth concate module; the output of the fifth concate module is used as the input of a fifth fully connected layer; the output of the fifth fully connected layer and the viewport binocular difference features are input into a sixth concate module; the output of the sixth concate module is used as the input of a sixth fully connected layer to obtain a viewport image quality evaluation score.

7. A panoramic stereo image quality evaluation device based on multi-level feature fusion, characterized in that: include: An acquisition module is used to acquire a left view and a right view of the target distorted panoramic stereo image, and perform viewport segmentation on the left view and the right view of the target distorted panoramic stereo image to obtain a left viewport image and a right viewport image; The feature extraction module comprises: a first feature extraction module and a second feature extraction module; A first feature extraction module is used to obtain viewport distortion information by subtracting the left viewport image from the pseudo reference viewport image obtained by restoring the left viewport image, and input the viewport distortion information into a monocular feature extraction model to output a viewport monocular feature; The second feature extraction module is used to input the left viewport image and the right viewport image into the ResNet50 model to obtain multi-level visual features of the two viewports; the multi-level visual features of the two viewports are respectively input into the binocular sum feature extraction model and the binocular difference feature extraction model to obtain the viewport binocular sum feature and the viewport binocular difference feature; A first evaluation module is used to perform feature fusion on the viewport monocular feature, the viewport binocular sum feature, and the viewport binocular difference feature to obtain a quality evaluation score of the viewport image; A second evaluation module is used to obtain a quality evaluation score of the target distorted panoramic stereo image according to the quality evaluation score of each viewport image of the target distorted panoramic stereo image; The ResNet50 model includes, in sequence: a convolutional layer, a pooling layer, a first residual block, a second residual block, a third residual block and a fourth residual block; The second residual block is used to output the first-level visual features of the left viewport and the first-level visual features of the right viewport ; The third residual block is used to output the second-level visual features of the left viewport And the second level visual features of the right viewport ; The fourth residual block is used to output the third-level visual features of the left viewport And the third-level visual features of the right viewport ; The binocular and feature extraction model includes a primary and model, a first and second secondary and model; The first level and model are the first level visual features of the left viewport and the first-level visual features of the right viewport As input, output the first level stereo and feature ; The first secondary and the model are the second-level visual features of the left viewport , the second-level visual features of the right viewport , first-level binocular and features As input, output second-level stereo and feature ; The second level and the model are the third level visual features of the left viewport , the third-level visual features of the right viewport , second level binocular and features As input, it outputs viewport stereo and features.

8. An electronic device comprising at least one processor; and a memory connected in communication with the at least one processor; characterized in that: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

Images