A panoramic stereoscopic image quality evaluation method and device

Abstract

The embodiment of the present invention provides a method and device for evaluating the quality of a panoramic stereoscopic image. The method includes: obtaining the left and right views of an image, performing viewport segmentation on the image to obtain left and right viewport images; subtracting the left viewport image from the degraded left viewport image, and inputting the image into a monocular feature extraction model to obtain a viewport monocular feature; inputting the left and right viewport images into a ResNet50 model, and then respectively inputting the binocular sum feature and binocular difference feature extraction model to obtain a viewport binocular sum feature and a viewport binocular difference feature; fusing the viewport monocular feature, the viewport binocular sum feature, and the viewport binocular difference feature to obtain a quality evaluation score; and obtaining a target quality evaluation score based on the quality evaluation score of each viewport image. In this way, the viewport characteristics and stereoscopic vision characteristics of the human eye when viewing a panoramic image can be effectively combined, reducing the computational complexity while capturing rich visual information and complete stereoscopic perception information, and using multi-level feature extraction to improve the model accuracy.

Description

Technical Field

[0001] The present invention generally relates to the field of panoramic stereoscopic image quality assessment, and more specifically, to a panoramic stereoscopic image quality assessment method and device. Background Art

[0002] With the development of virtual reality (VR) technology, panoramic images that provide immersive and realistic visual experience have attracted much attention and become one of the research hotspots in the fields of industry, education, and games. The development of 5G can use higher bandwidth to transmit VR content, and more and more panoramic images and videos are presented in stereoscopic 3D format. Unlike traditional planar images, stereoscopic panoramic images contain two views: left view and right view, and each view covers a 360×180 viewing range. With the help of head-mounted displays, users can freely explore virtual scenes similar to the real world and get experience quality in an immersive environment. However, this immersive and interactive experience places high demands on the quality of stereoscopic panoramic images and also imposes a heavy burden on storage and transmission. Distortion may be introduced at each stage of image processing, resulting in impaired quality of the captured image and reduced user experience quality. Although researchers have conducted in-depth research on image quality evaluation in recent years, research on stereoscopic panoramic image quality evaluation is still in its infancy, which limits the further development and application of panoramic image technology. Therefore, how to accurately evaluate the quality of stereoscopic panoramic images plays a vital role in practical application scenarios. At first, people directly applied planar image quality evaluation methods to evaluate the quality of stereoscopic panoramic images, without considering the large viewing angle characteristics of panoramic images, and the effect was not good. Considering that viewport images play an important role in the perception of large-scene panoramic images, researchers have proposed related stereoscopic panoramic image quality evaluation methods based on the viewport perception characteristics of panoramic images, and the performance has been improved. Subsequently, considering that stereoscopic panoramic images have the characteristics of both panoramic and stereoscopic perception, people applied stereoscopic image quality evaluation methods to the field of stereoscopic panoramic image quality evaluation. Although the performance has been improved compared to traditional planar methods, due to the lack of analysis and extraction of human visual perception characteristics, its performance still cannot meet the needs of practical applications.

[0003] The current stereo panoramic image quality evaluation methods have the following defects: (1) they simply extract the perceptual characteristics of the viewport, which lacks precision; (2) they do not effectively fuse multi-scale stereo visual features, and their performance is limited; (3) they comprehensively consider multi-scale visual features, but the research on interaction and fusion methods is not in-depth enough. In summary, in the stereo panoramic image quality evaluation method, many factors need to be considered, including viewport perception characteristics, stereo visual perception characteristics, and efficient feature interaction and fusion. Therefore, it is necessary to combine the above multiple aspects to construct a practical stereo panoramic image quality evaluation method. Summary of the invention

[0004] According to an embodiment of the present invention, a panoramic stereoscopic image quality evaluation scheme is provided. The scheme is suitable for the viewing characteristics of human eyes, conforms to the characteristics of images in actual application scenarios, has universal applicability, and can effectively improve the overall performance of the model.

[0005] The present invention provides a method for evaluating the quality of a panoramic stereoscopic image. The method comprises:

[0006] Obtaining a left view and a right view of a target distorted panoramic stereo image, performing viewport segmentation on the left view and the right view of the target distorted panoramic stereo image, and obtaining a left viewport image and a right viewport image;

[0007] Subtracting the left viewport image from the degraded left viewport image generated by the left viewport image to obtain a left viewport degradation difference image, and inputting the left viewport degradation difference image into a monocular feature extraction model to output a viewport monocular feature; and

[0008] Input the left viewport image and the right viewport image into the ResNet50 model to obtain multi-level visual features of the two viewports; input the multi-level visual features of the two viewports into the binocular sum feature extraction model and the binocular difference feature extraction model respectively to obtain the viewport binocular sum feature and the viewport binocular difference feature;

[0009] The monocular viewport feature, the binocular viewport sum feature, and the binocular viewport difference feature are fused to obtain a quality evaluation score of the viewport image;

[0010] According to the quality evaluation score of each viewport image of the target distorted panoramic stereo image, a quality evaluation score of the target distorted panoramic stereo image is obtained.

[0011] Furthermore, the ResNet50 model includes, in sequence: a convolutional layer, a pooling layer, a first residual block, a second residual block, a third residual block and a fourth residual block;

[0012] The second residual block is used to output the first-level visual features of the left viewport and the first-level visual features of the right viewport ;

[0013] The third residual block is used to output the second-level visual features of the left viewport And the second level visual features of the right viewport ;

[0014] The fourth residual block is used to output the third-level visual features of the left viewport And the third-level visual features of the right viewport .

[0015] Furthermore, the binocular and feature extraction model includes:

[0016] The first concatenation module, the input data of the first concatenation module includes: the first level binocular and feature , the second level global pooling binocular and feature And the third level global pooling binocular and feature ; The first level binocular and feature is the first-level visual feature of the left viewport The first level visual features with the right viewport The result of addition; the second level global pooling binocular and feature It is composed of the second-level binocular and feature The second-level binocular and feature is the second-level visual feature of the left viewport The second level visual features with the right viewport The result of the addition; the third level global pooling binocular and feature It is composed of the third-level binocular and feature The third-level binocular and feature is the third-level visual feature of the left viewport The third level visual feature with the right viewport The result of addition;

[0017] A first fully connected layer, wherein the input data of the first fully connected layer is the output of the first concaten module;

[0018] The second concatenation module, the input data of the second concatenation module includes: the output of the first fully connected layer and the first level binocular and feature ;

[0019] The second fully connected layer, the input data of the second fully connected layer is the output of the second concaten module, and the output of the second fully connected layer is used as the viewport binocular sum feature.

[0020] Furthermore, the binocular difference feature extraction model includes: a first-order difference model, a first-order second-order difference model, and a second-order second-order difference model;

[0021] The first-level visual features of the left viewport and the first-level visual features of the right viewport Input the first-level difference model to obtain the first-level binocular difference feature ; The first level binocular difference feature , the second level visual features of the left viewport and the second level visual features of the right viewport Input the first and second binocular difference models to obtain the second binocular difference features ; The second level binocular difference feature , the third level visual features of the left viewport and the third level visual features of the right viewport Input the second level binocular difference model and output the third level binocular difference feature , and the third-level binocular difference feature As the viewport binocular difference feature.

[0022] Furthermore, the first-level difference model is used to transform the first-level visual features of the left viewport and the first-level visual features of the right viewport After performing global pooling, the subtraction is performed, and then the subtraction result is input into the third fully connected layer for output;

[0023] The first secondary difference model is used to transform the second-level visual features of the left viewport and the second level visual features of the right viewport Perform global pooling respectively, and combine the result of global pooling with the first-level binocular difference feature The difference is then input into the fourth fully connected layer, and the output of the fourth fully connected layer is compared with the first level binocular difference feature. After inputting the third concate module, output is performed;

[0024] The second second-level difference model is used to transform the third-level visual features of the left viewport and the third-level visual features of the right viewport Perform global pooling respectively, and combine the result of global pooling with the second-level binocular difference feature The difference is then input into the fifth fully connected layer, and the output of the fifth fully connected layer is combined with the second-level binocular difference feature. After inputting the fourth concate module, output is performed.

[0025] Furthermore, the viewport monocular feature, the viewport binocular sum feature, and the viewport binocular difference feature are fused to obtain a quality evaluation score of the viewport image, including:

[0026] The viewport binocular and features are input into the first SW-MSA module, and then the output result of the first SW-MSA module is input into the first GELU activation function module; the viewport binocular difference features are input into the second SW-MSA module, and then the output result of the second SW-MSA module is input into the second GELU activation function module; the output results of the first GELU activation function module and the second GELU activation function module are fused with the monocular features, and then the fusion results are input into the LN layer, and the output result of the LN layer is input into the sixth fully connected layer to obtain the quality evaluation score of the viewport image.

[0027] In a second aspect of the present invention, a panoramic stereoscopic image quality evaluation device is provided. The device comprises:

[0028] An acquisition module is used to acquire a left view and a right view of the target distorted panoramic stereo image, and perform viewport segmentation on the left view and the right view of the target distorted panoramic stereo image to obtain a left viewport image and a right viewport image;

[0029] The feature extraction module comprises: a first feature extraction module and a second feature extraction module;

[0030] A first feature extraction module is used to make a difference between the left viewport image and the degraded left viewport image generated by the module to obtain a left viewport degraded difference image, and input the left viewport degraded difference image into a monocular feature extraction model to output a viewport monocular feature;

[0031] The second feature extraction module is used to input the left viewport image and the right viewport image into the ResNet50 model to obtain multi-level visual features of the two viewports; the multi-level visual features of the two viewports are respectively input into the binocular sum feature extraction model and the binocular difference feature extraction model to obtain the viewport binocular sum feature and the viewport binocular difference feature;

[0032] A first evaluation module is used to perform feature fusion on the viewport monocular feature, the viewport binocular sum feature, and the viewport binocular difference feature to obtain a quality evaluation score of the viewport image;

[0033] The second evaluation module is used to obtain a quality evaluation score of the target distorted panoramic stereo image according to the quality evaluation score of each viewport image of the target distorted panoramic stereo image.

[0034] In a third aspect of the present invention, an electronic device is provided. The electronic device has at least one processor; and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method of the first aspect of the present invention.

[0035] In a fourth aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the first aspect of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0037] Figure 1 A flowchart of a panoramic stereoscopic image quality evaluation method according to an embodiment of the present invention is shown;

[0038] Figure 2 A structural diagram of a Vmamba model according to an embodiment of the present invention is shown;

[0039] Figure 3 It shows a structural diagram of a Vmamba model VSS Block according to an embodiment of the present invention;

[0040] Figure 4 A structural diagram of a ResNet50 model according to an embodiment of the present invention is shown;

[0041] Figure 5 A structural diagram of a binocular and feature extraction model according to an embodiment of the present invention is shown;

[0042] Figure 6 A structural diagram of a binocular difference feature extraction model according to an embodiment of the present invention is shown;

[0043] Figure 7 A flow chart of quality evaluation according to an embodiment of the present invention is shown;

[0044] Figure 8 A block diagram of a panoramic stereoscopic image quality assessment device according to an embodiment of the present invention is shown;

[0045] Fig. 9 A block diagram of an exemplary electronic device capable of implementing embodiments of the present invention is shown.

[0046] Among them, 900 is an electronic device, 901 is a CPU, 902 is a ROM, 903 is a RAM, 904 is a bus, 905 is an I / O interface, 906 is an input unit, 907 is an output unit, 908 is a storage unit, and 909 is a communication unit. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0049] Figure 1 A flow chart of a panoramic stereoscopic image quality assessment method according to an embodiment of the present invention is shown.

[0050] The method includes:

[0051] S101 , obtaining a left view and a right view of a target distorted panoramic stereo image, performing viewport segmentation on the left view and the right view of the target distorted panoramic stereo image, and obtaining a left viewport image and a right viewport image.

[0052] Specifically, the viewport acquisition method FoV Selection (Field of View Selection) is used to obtain the left and right viewport images. and .

[0053] Viewport segmentation of the distorted panoramic left and right views is in line with human visual perception characteristics and helps to extract visual features.

[0054] S102, subtracting the left viewport image from the degraded left viewport image generated by the left viewport image to obtain a left viewport degraded difference image, and inputting the left viewport degraded difference image into a monocular feature extraction model to output a viewport monocular feature; and

[0055] The left viewport image and the right viewport image are input into the ResNet50 model to obtain multi-level visual features of the two viewports; the multi-level visual features of the two viewports are respectively input into the binocular sum feature extraction model and the binocular difference feature extraction model to obtain the viewport binocular sum feature and the viewport binocular difference feature.

[0056] The first part is the extraction of the monocular features of the viewport:

[0057] Specifically, the process of obtaining the degradation difference map is as follows:

[0058] 1) Randomly add image blur operations, down-sampling and noise to the left viewport image to generate a left degenerate viewport image with random type and random distortion degree .

[0059] 2) The left viewport image and the left degenerate viewport image Make a difference and get the left viewport degraded difference map , the calculation formula is as follows: The degraded difference map obtained by the viewport degradation difference map module can reflect the information difference of the quality degradation of the left view, which is more in line with the characteristics of random distortion of panoramic stereo images in actual application scenarios; the monocular features are extracted by using the difference map information, which enables the method to capture the monocular quality degradation information without reference information, thereby improving the feature representation capability and the generalization of the method.

[0060] In this embodiment, the monocular feature extraction module is used as follows Figure 2 The Vmamba model shown is used as a backbone network to extract monocular features. The backbone network structure includes: a Stem (Spatial Transformer Embedding Module, spatial transformer embedding model) block, a first visual state space block (VSS Block, Visual State Space Block), a first down sampling module (Down Sampling), a second visual state space block (VSS Block), a second down sampling module (Down Sampling), a third visual state space block (VSS Block), a third down sampling module (DownSampling), a fourth visual state space block (VSS Block), and a fully connected layer (FC).

[0061] Among them, Figure 3 As shown, each visual state space (VSS Block) block includes:

[0062] (1) The first LN layer (Layer Normalization): The input data is the output of the Stem block or the previous down sampling module (Down Sampling).

[0063] (2) 2D-selective-scan Block (SS2d Block): The input data is the output of the first LN.

[0064] (3) Second LN layer: The input data is the result of adding the input of the first LN layer and the output of the SS2d Block.

[0065] (4) FFN layer (Feed Forward Network): The input data is the output of the second LN layer.

[0066] The data obtained by adding the input of the second LN layer and the output of the FFN is used as the output of the VSS Block.

[0067] The second part is the extraction of the multi-level visual features:

[0068] In this embodiment, if Figure 4 As shown, the ResNet50 model includes, in sequence: a convolutional layer, a pooling layer, a first residual block, a second residual block, a third residual block and a fourth residual block.

[0069] The second residual block is used to output the first-level visual features of the left viewport and the first-level visual features of the right viewport .

[0070] The third residual block is used to output the second-level visual features of the left viewport And the second level visual features of the right viewport .

[0071] The fourth residual block is used to output the third-level visual features of the left viewport And the third-level visual features of the right viewport .

[0072] Specifically, the ResNet50 model has 50 layers, namely, 1 7*7 convolutional layer with 64 channels, 1 3*3 maximum pooling layer, the first residual block (including 1 1*1 convolutional layer with 64 channels, 1 3*3 convolutional layer with 64 channels, 1 1*1 convolutional layer with 256 channels)*3, the second residual block (including 1 1*1 convolutional layer with 128 channels, 1 3*3 convolutional layer with 128 channels)*3, convolution layer, 1 1*1 convolution layer with 512 channels)*4, the third residual block (including 1 1*1 convolution layer with 256 channels, 1 3*3 convolution layer with 256 channels, 1 1*1 convolution layer with 1024 channels)*6, the fourth residual block (including 1 1*1 convolution layer with 512 channels, 1 3*3 convolution layer with 512 channels, 1 1*1 convolution layer with 2048 channels)*3.

[0073] By utilizing the outputs of the last three groups of residual blocks, we can effectively obtain the deep information of the left and right views, improve the feature representation capability, and provide rich visual information for binocular feature extraction.

[0074] In this embodiment, if Figure 5As shown, the binocular and feature extraction model includes:

[0075] The first concatenation module, the input data of the first concatenation module includes: the first level binocular and feature , the second level global pooling binocular and feature And the third level global pooling binocular and feature ; The first level binocular and feature is the first-level visual feature of the left viewport The first level visual features with the right viewport The result of addition; the second level global pooling binocular and feature It is composed of the second-level binocular and feature The second-level binocular and feature is the second-level visual feature of the left viewport The second level visual features with the right viewport The result of the addition; the third level global pooling binocular and feature It is composed of the third-level binocular and feature The third-level binocular and feature is the third-level visual feature of the left viewport The third level visual feature with the right viewport The result of addition.

[0076] A first fully connected layer, wherein the input data of the first fully connected layer is the output of the first concaten module.

[0077] The second concatenation module, the input data of the second concatenation module includes: the output of the first fully connected layer and the first level binocular and feature .

[0078] The second fully connected layer, the input data of the second fully connected layer is the output of the second concaten module, and the output of the second fully connected layer is used as the viewport binocular sum feature.

[0079] Simple addition processing can reduce the computational complexity of binocular and information of panoramic images; at the same time, multi-level fusion of shallow information and deep information helps to obtain rich binocular and information.

[0080] In this embodiment, if Figure 6 As shown, the binocular difference feature extraction model includes: a primary difference model, a first secondary difference model and a second secondary difference model.

[0081] The first-level visual features of the left viewport and the first-level visual features of the right viewport Input the first-level difference model to obtain the first-level binocular difference feature ; The first level binocular difference feature , the second level visual features of the left viewport and the second level visual features of the right viewport Input the first and second binocular difference models to obtain the second binocular difference features ; The second level binocular difference feature , the third level visual features of the left viewport and the third level visual features of the right viewport Input the second level binocular difference model and output the third level binocular difference feature , and the third-level binocular difference feature As the viewport binocular difference feature.

[0082] The first-level difference model is used to transform the first-level visual features of the left viewport and the first-level visual features of the right viewport After global pooling, the subtraction is performed respectively, and then the subtraction result is input into the third fully connected layer for output.

[0083] The first secondary difference model is used to transform the second-level visual features of the left viewport and the second level visual features of the right viewport Perform global pooling respectively, and combine the result of global pooling with the first-level binocular difference feature The difference is then input into the fourth fully connected layer, and the output of the fourth fully connected layer is compared with the first level binocular difference feature. After inputting the third concate module, output is performed.

[0084] The second level difference model is used to transform the third level visual features of the left viewport and the third-level visual features of the right viewport Perform global pooling respectively, and combine the result of global pooling with the second-level binocular difference feature The difference is then input into the fifth fully connected layer, and the output of the fifth fully connected layer is combined with the second-level binocular difference feature. After inputting the fourth concate module, output is performed.

[0085] The binocular difference feature extraction module realizes the gradual extraction and aggregation of the difference information between the left and right views from shallow to deep, enhances the interaction between features at different layers, can mine deep binocular difference information, and improve model accuracy.

[0086] S103, performing feature fusion on the viewport monocular feature, the viewport binocular sum feature, and the viewport binocular difference feature to obtain a quality evaluation score of the viewport image.

[0087] like Figure 7 The quality assessment flow chart shown includes:

[0088] The viewport binocular and feature are input into the first SW-MSA (Shifted Windows Multi-Head Self-Attention) module, and then the output result of the first SW-MSA module is input into the first GELU (Gaussian Error Linear Units) activation function module; the viewport binocular difference feature is input into the second SW-MSA module, and then the output result of the second SW-MSA module is input into the second GELU activation function module; the output results of the first GELU activation function module and the second GELU activation function module are fused with the monocular feature, and then the fusion result is input into the LN layer, and the output result of the LN layer is input into the sixth fully connected layer to obtain the quality evaluation score of the viewport image.

[0089] Considering the important role of binocular sum features and binocular difference features in stereo perception characteristics, the effective use of the SW-MSA mechanism can enhance the model's understanding of the dependencies between different parts of the sequence while maintaining computational efficiency, obtain rich detail information and dependency mining; the effective complementarity of monocular and binocular features can improve the overall performance of the model.

[0090] S104. Obtain a quality evaluation score of the target distorted panoramic stereo image according to the quality evaluation score of each viewport image of the target distorted panoramic stereo image.

[0091] Specifically, the average value of the quality evaluation score of each viewport image is used as the quality evaluation score of the target distorted panoramic stereo image to evaluate the quality of the target distorted panoramic stereo image.

[0092] According to the embodiments of the present invention, the present invention has the following advantages and effects compared with the prior art:

[0093] (1) Using the viewport diagram as the research target is more suitable for the viewing characteristics of the human eye.

[0094] (2) The degradation difference map information is obtained by generating degraded viewport images with random types and random distortion degrees, which is more in line with the image characteristics of actual application scenarios and more universal.

[0095] (3) Using the degraded difference image information to obtain monocular features can obtain visual features that are more sensitive to monocular perception quality degradation and improve the model feature representation capability.

[0096] (4) The outputs of multiple residual modules of the ResNet50 model can capture rich stereoscopic visual information, effectively complement the monocular features, and improve the model accuracy.

[0097] (5) Using simple addition and difference operations to obtain binocular sum features and binocular difference features can greatly reduce the computational complexity of binocular vision features; at the same time, multi-scale binocular feature extraction and fusion can obtain rich stereo vision information, enhance the depth aggregation of binocular features, and improve the overall performance of the model.

[0098] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0099] The above is an introduction to a method embodiment. The following is a further explanation of the scheme of the present invention through a device embodiment having the same inventive concept as the method in the aforementioned embodiment.

[0100] like Figure 8 As shown, the device 800 includes:

[0101] The acquisition module 810 is used to acquire the left view and the right view of the target distorted panoramic stereo image, and perform viewport segmentation on the left view and the right view of the target distorted panoramic stereo image to obtain a left viewport image and a right viewport image.

[0102] The feature extraction module 820 includes: a first feature extraction module 821 and a second feature extraction module 822;

[0103] The first feature extraction module 821 is used to make a difference between the left viewport image and the degraded left viewport image generated by the module to obtain a left viewport degraded difference image, and input the left viewport degraded difference image into a monocular feature extraction model to output a viewport monocular feature;

[0104] The second feature extraction module 822 is used to input the left viewport image and the right viewport image into the ResNet50 model to obtain multi-level visual features of the two viewports; input the multi-level visual features of the two viewports into the binocular sum feature extraction model and the binocular difference feature extraction model respectively to obtain the viewport binocular sum feature and the viewport binocular difference feature.

[0105] The first evaluation module 830 is used to perform feature fusion on the viewport monocular feature, the viewport binocular sum feature, and the viewport binocular difference feature to obtain a quality evaluation score of the viewport image.

[0106] The second evaluation module 840 is used to obtain a quality evaluation score of the target distorted panoramic stereo image according to the quality evaluation score of each viewport image of the target distorted panoramic stereo image.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0108] In the technical solution of the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0109] According to an embodiment of the present invention, the present invention further provides an electronic device and a readable storage medium.

[0110] Fig. 9 A schematic block diagram of an electronic device 900 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0111] The electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0112] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0113] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as methods S101 to S104. For example, in some embodiments, methods S101 to S104 may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the methods S101 to S104 described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute methods S101 - S104 in any other appropriate manner (eg, by means of firmware).

[0114] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0116] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0118] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0119] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0121] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A panoramic stereoscopic image quality evaluation method, characterized in that: include: Obtaining a left view and a right view of a target distorted panoramic stereo image, performing viewport segmentation on the left view and the right view of the target distorted panoramic stereo image, and obtaining a left viewport image and a right viewport image; Subtracting the left viewport image from the degraded left viewport image generated by the left viewport image to obtain a left viewport degradation difference image, and inputting the left viewport degradation difference image into a monocular feature extraction model to output a viewport monocular feature; as well as Input the left viewport image and the right viewport image into the ResNet50 model to obtain the multi-level visual features of the two viewports; Inputting the multi-level visual features of the two viewports into a binocular sum feature extraction model and a binocular difference feature extraction model respectively to obtain a viewport binocular sum feature and a viewport binocular difference feature; The monocular viewport feature, the binocular viewport sum feature, and the binocular viewport difference feature are fused to obtain a quality evaluation score of the viewport image; According to the quality evaluation score of each viewport image of the target distorted panoramic stereo image, a quality evaluation score of the target distorted panoramic stereo image is obtained; The ResNet50 model includes, in sequence: a convolutional layer, a pooling layer, a first residual block, a second residual block, a third residual block and a fourth residual block; The second residual block is used to output the first-level visual features of the left viewport and the first-level visual features of the right viewport ; The third residual block is used to output the second-level visual features of the left viewport And the second level visual features of the right viewport ; The fourth residual block is used to output the third-level visual features of the left viewport And the third-level visual features of the right viewport ; The binocular and feature extraction model includes: The first concatenation module, the input data of the first concatenation module includes: the first level binocular and feature , the second level global pooling binocular and feature And the third level global pooling binocular and feature ; The first level binocular and feature is the first-level visual feature of the left viewport The first level visual features with the right viewport The result of addition; the second level global pooling binocular and feature It is composed of the second-level binocular and feature The second-level binocular and feature is the second-level visual feature of the left viewport The second level visual features with the right viewport The result of the addition; the third level global pooling binocular and feature It is composed of the third-level binocular and feature The third-level binocular and feature is the third-level visual feature of the left viewport The third level visual feature with the right viewport The result of addition; A first fully connected layer, wherein the input data of the first fully connected layer is the output of the first concaten module; The second concatenation module, the input data of the second concatenation module includes: the output of the first fully connected layer and the first level binocular and feature ; The second fully connected layer, the input data of the second fully connected layer is the output of the second concaten module, and the output of the second fully connected layer is used as the viewport binocular sum feature.

2. The method according to claim 1, characterized in that: The binocular difference feature extraction model includes: a first-order difference model, a first-order second-order difference model, and a second-order second-order difference model; The first-level visual features of the left viewport and the first-level visual features of the right viewport Input the first-level difference model to obtain the first-level binocular difference feature ; The first level binocular difference feature , the second level visual features of the left viewport and the second level visual features of the right viewport Input the first and second binocular difference models to obtain the second binocular difference features ; The second level binocular difference feature , the third level visual features of the left viewport and the third level visual features of the right viewport Input the second level binocular difference model and output the third level binocular difference feature , and the third-level binocular difference feature As the viewport binocular difference feature.

3. The method according to claim 2, characterized in that The first-level difference model is used to transform the first-level visual features of the left viewport and the first-level visual features of the right viewport After performing global pooling, the subtraction is performed, and then the subtraction result is input into the third fully connected layer for output; The first secondary difference model is used to transform the second-level visual features of the left viewport and the second level visual features of the right viewport Perform global pooling respectively, and combine the result of global pooling with the first-level binocular difference feature The difference is then input into the fourth fully connected layer, and the output of the fourth fully connected layer is compared with the first level binocular difference feature. After inputting the third concate module, output is performed; The second level difference model is used to transform the third level visual features of the left viewport and the third-level visual features of the right viewport Perform global pooling respectively, and combine the result of global pooling with the second-level binocular difference feature The difference is then input into the fifth fully connected layer, and the output of the fifth fully connected layer is combined with the second-level binocular difference feature. After inputting the fourth concate module, output is performed.

4. The method according to claim 1, characterized in that: The viewport monocular feature, the viewport binocular sum feature, and the viewport binocular difference feature are fused to obtain a quality evaluation score of the viewport image, including: The viewport binocular and features are input into the first SW-MSA module, and then the output result of the first SW-MSA module is input into the first GELU activation function module; the viewport binocular difference features are input into the second SW-MSA module, and then the output result of the second SW-MSA module is input into the second GELU activation function module; the output results of the first GELU activation function module and the second GELU activation function module are fused with the monocular features, and then the fusion results are input into the LN layer, and the output result of the LN layer is input into the sixth fully connected layer to obtain the quality evaluation score of the viewport image.

5. A panoramic stereoscopic image quality evaluation device, characterized in that: include: An acquisition module is used to acquire a left view and a right view of the target distorted panoramic stereo image, and perform viewport segmentation on the left view and the right view of the target distorted panoramic stereo image to obtain a left viewport image and a right viewport image; The feature extraction module comprises: a first feature extraction module and a second feature extraction module; A first feature extraction module is used to make a difference between the left viewport image and the degraded left viewport image generated by the module to obtain a left viewport degraded difference image, and input the left viewport degraded difference image into a monocular feature extraction model to output a viewport monocular feature; The second feature extraction module is used to input the left viewport image and the right viewport image into the ResNet50 model to obtain multi-level visual features of the two viewports; the multi-level visual features of the two viewports are respectively input into the binocular sum feature extraction model and the binocular difference feature extraction model to obtain the viewport binocular sum feature and the viewport binocular difference feature; A first evaluation module is used to perform feature fusion on the viewport monocular feature, the viewport binocular sum feature, and the viewport binocular difference feature to obtain a quality evaluation score of the viewport image; A second evaluation module is used to obtain a quality evaluation score of the target distorted panoramic stereo image according to the quality evaluation score of each viewport image of the target distorted panoramic stereo image; The ResNet50 model includes, in sequence: a convolutional layer, a pooling layer, a first residual block, a second residual block, a third residual block and a fourth residual block; The second residual block is used to output the first-level visual features of the left viewport and the first-level visual features of the right viewport ; The third residual block is used to output the second-level visual features of the left viewport And the second level visual features of the right viewport ; The fourth residual block is used to output the third-level visual features of the left viewport And the third-level visual features of the right viewport ; The binocular and feature extraction model includes: The first concatenation module, the input data of the first concatenation module includes: the first level binocular and feature , the second level global pooling binocular and feature And the third level global pooling binocular and feature ; The first level binocular and feature is the first-level visual feature of the left viewport The first level visual features with the right viewport The result of addition; the second level global pooling binocular and feature It is composed of the second-level binocular and feature The second-level binocular and feature is the second-level visual feature of the left viewport The second level visual features with the right viewport The result of the addition; the third level global pooling binocular and feature It is composed of the third-level binocular and feature The third-level binocular and feature is the third-level visual feature of the left viewport The third level visual feature with the right viewport The result of addition; A first fully connected layer, wherein the input data of the first fully connected layer is the output of the first concaten module; The second concatenation module, the input data of the second concatenation module includes: the output of the first fully connected layer and the first level binocular and feature ; The second fully connected layer, the input data of the second fully connected layer is the output of the second concaten module, and the output of the second fully connected layer is used as the viewport binocular sum feature.

6. An electronic device comprising at least one processor; and a memory connected in communication with the at least one processor; characterized in that: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-4.

Images