Panoramic image saliency prediction method and device based on fully convolutional graph neural network

By mapping panoramic images to spherical map data structures and using full convolutional map neural network for significant prediction, the problems of inaccurate prediction of panoramic images in the prior art are solved, and more efficient panoramic images are achieved Sexual prediction effect.

CN113947524BActive Publication Date: 2024-11-01SHANGHAI JIAOTONG UNIV
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202111230705.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-22
Publication Date
2024-11-01
Estimated Expiration
2041-10-22

AI Technical Summary

Technical Problem

The existing panoramic significance prediction methods are inaccurate due to non-Euclidean geometric characteristics when processing panoramic images, and the calculation complexity is high, and semantic direction information is removed, so the accuracy is limited.

Method used

Using a method based on a fully convolutional graph neural network, the plane panoramic image is mapped to the spherical graph data structure, and significant prediction is made through the U-shaped neural network structure and the expansion graph convolution and attention mechanism, and the vertex-to-vertex features are retained, and the vertex-to-vertex features are preserved. Graph pooling and anti-pooling operations are performed through a full convolutional layer.

Benefits of technology

Improves the accuracy and performance of panoramic image significance prediction, eliminates redundancy in flat panoramic images, and is superior to other advanced saliency models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113947524B_ABST
    Figure CN113947524B_ABST
Patent Text Reader

Abstract

The present invention provides a panoramic image saliency prediction method and device based on a fully convolutional graph neural network, comprising: mapping a planar panoramic image into spherical graph data; inputting the spherical graph data into a fully convolutional graph neural network for saliency prediction to obtain spherical saliency map data; transforming the spherical saliency map data onto a plane to obtain a planar panoramic saliency image. Furthermore, the fully convolutional graph neural network of the present invention is based on a residual U-shaped network structure, and uses an expanded graph convolution and an attention mechanism at the bottom. In addition, the present invention uses a new fully convolutional layer for graph pooling and de-pooling operations in a spherical graph space to retain node-to-node features. The method proposed in the present invention outperforms other more advanced saliency prediction models on large-scale data sets.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image saliency prediction, and in particular to a method and device for panoramic image saliency prediction based on a graph-based fully convolutional network. Background Art

[0002] Panoramic media content can provide viewers with an immersive viewing experience and is widely used in fields such as virtual reality and autonomous driving. Using saliency prediction algorithms, we can estimate the areas of panoramic content that are most likely to attract attention, thereby reducing the rendering level of other areas and improving transmission performance.

[0003] Saliency prediction algorithms for two-dimensional images have been extensively studied, often categorized into top-down and bottom-up frameworks. However, due to non-Euclidean geometry, these algorithms are generally not directly applicable to panoramic media content. Because panoramic visual content is distorted by equirectangular projection, their results are inaccurate.

[0004] Therefore, there are two trends in existing panoramic saliency prediction methods. One is to extend the existing 2D saliency prediction algorithms by mapping the panoramic image to multiple local planar views, such as GBVS360 and BMS360 proposed by Lebreton et al. in "P.Lebreton andA.Raake,“Gbvs360,bms360,prosal:Extending existing saliency prediction modelsfrom 2d to omnidirectional images,”Signal Processing:Image Communication,vol.69,pp.69–78,2018.”, and Salgan360 proposed by Chao et al. in "F.-Y.Chao,L.Zhang,W.Hamidouche,and O.Deforges,“Salgan360:Visual saliency prediction on360degree images with generative adversarial networks,”in IEEE InternationalConference on Multimedia&Expo Workshops,pp.01–04,IEEE,2018.”. These algorithms require repeated application of saliency prediction algorithms on multiple planes, which has high computational complexity. In addition, different regions of the same object may appear in different local windows, and there are boundary discontinuities when fusing multiple results. Recently, Xu et al. proposed a multi-stream deep reinforcement learning model SalGAIL in "M.Xu, L.Yang, X.Tao, Y.Duan, and Z.Wang, "Saliency prediction onomnidirectional image with generative adversarial imitation learning," IEEE Transactions on Image Processing, vol.30, pp.2087–2102, 2021.", which learns rewards by imitating the trajectory of a person's head. However, it still requires extracting features of 30 windows each time.

[0005] Another trend is to define other forms of convolution. Among them, Khasanova et al. designed a graph structure on the sphere in "R. Khasanova and P. Frossard, "Graph-based classification of omnidirectional images," in Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 869–878, 2017." and applied spectral domain graph convolution to learn rotation-invariant features. Cohen et al. proposed Spherical CNNs in "TS Cohen, M. Geiger, J Köhler, and M. Welling, "Spherical CNNs," in International Conference on Learning Representations, 2018." They used spherical correlation to convert feature maps to the spectral domain and used spectral domain convolution. Lv et al. (H.Lv, Q.Yang, C.Li, W.Dai, J.Zou, and H.Xiong, “Salgcn: Saliency prediction for 360-degree images based on spherical graph convolutional networks,” in Proceedings of ACM International Conference on Multimedia, pp. 682–690, 2020) proposed SalGCN, which uses a graph signal construction method based on regular icosahedron pixelation and Chebyshev polynomial filters. However, these methods have limited accuracy on panoramic images due to the removal of semantically significant directional information. Summary of the Invention

[0006] In view of the above-mentioned deficiencies in the above-mentioned prior art, the purpose of the present invention is to provide a panoramic image saliency prediction method and device based on a fully convolutional graph neural network, which can reasonably map panoramic image pixels to a spherical graph data structure for representation, thereby improving the accuracy of panoramic image saliency prediction.

[0007] A first aspect of the present invention provides a method for predicting saliency of panoramic images based on a fully convolutional graph neural network, comprising:

[0008] Mapping a planar panoramic image into spherical image data;

[0009] Inputting the spherical graph data into a fully convolutional graph neural network for saliency prediction to obtain spherical saliency graph data;

[0010] The spherical saliency map data is transformed onto a plane to obtain a planar panoramic saliency image.

[0011] Optionally, mapping the planar panoramic image into spherical image data includes:

[0012] Mapping a planar image to a spherical panoramic image;

[0013] Sampling is performed on the spherical panoramic image to obtain evenly distributed vertices, each vertex having an RGB color value from a corresponding pixel on the spherical panoramic image as its feature; edges with self-loops are constructed between each vertex and K adjacent points on the spherical panoramic image, and ultimately spherical graph data is obtained, in which each vertex has 3 features and K edges.

[0014] Optionally, the fully convolutional graph neural network structure is a U-shaped graph neural network structure, which includes:

[0015] Setting up dilated graph convolution and attention mechanism at the bottom; and / or,

[0016] Fully convolutional layers are used for graph pooling and unpooling operations in the spherical graph space to preserve vertex-to-vertex features.

[0017] Optionally, the U-shaped graph neural network structure includes an encoding part, a bottom part and a decoding part, wherein:

[0018] The encoding unit includes a first TAG convolution layer, a second TAG convolution layer, a third TAG convolution layer, a fourth TAG convolution layer, a first full convolution map pooling layer, a second full convolution map pooling layer and a third full convolution map pooling layer, wherein:

[0019] The first TAG convolution layer input is the spherical graph data,

[0020] Second TAG convolution layer: input is the output of the first TAG convolution layer;

[0021] The first fully convolutional graph pooling layer: the input is the sum of the output of the first TAG convolution layer and the output of the second TAG convolution layer;

[0022] The third TAG convolution layer: the input is the output of the first full convolutional graph pooling layer;

[0023] Second full convolutional graph pooling layer: The input is the sum of the output of the first full convolutional graph pooling layer and the output of the third TAG convolutional layer;

[0024] The fourth TAG convolution layer: the input is the output of the second full convolutional graph pooling layer;

[0025] The third full convolutional graph pooling layer: The input is the sum of the output of the second full convolutional graph pooling layer and the output of the fourth TAG convolutional layer;

[0026] The bottom portion comprises a first dilated attention block, a second dilated attention block, and a third dilated attention block connected in series, wherein: the input of the first dilated attention block is the output of the encoding unit, and the output of the bottom portion is the output of the third dilated attention block;

[0027] The decoding unit includes a fifth TAG convolution layer, a sixth TAG convolution layer, a seventh TAG convolution layer, an eighth TAG convolution layer, a ninth TAG convolution layer, a first full convolution graph depooling layer, a second full convolution graph depooling layer, and a third full convolution depooling layer, wherein:

[0028] The first fully convolutional graph depooling layer: the input is the output of the bottom layer;

[0029] The fifth TAG convolution layer: The input is the feature concatenation of the output of the first full convolutional map depooling layer and the input of the third full convolutional map pooling layer;

[0030] The second fully convolutional graph depooling layer: the input is the sum of the input of the fifth TAG convolution layer and the output of the fifth TAG convolution layer;

[0031] The sixth TAG convolution layer: The input is the feature concatenation of the output of the second full convolution graph depooling layer and the input of the second full convolution graph pooling layer;

[0032] The third full convolution graph depooling layer: the input is the sum of the input of the sixth TAG convolution layer and the output of the sixth TAG convolution layer;

[0033] The seventh TAG convolution layer: The input is the feature concatenation of the output of the third full convolution map depooling layer and the input of the first full convolution map pooling layer;

[0034] The eighth TAG convolution layer: The input is the sum of the input and output of the seventh TAG convolution layer;

[0035] Ninth TAG convolution layer: The input is the output of the eighth TAG convolution layer.

[0036] Optionally, each of the second to ninth TAG convolution layers is configured as follows:

[0037] For each vertex in the graph, the eigenvector Merge its K neighboring features into a feature matrix Among them D in is the feature dimension of the input vertex;

[0038] The feature matrix of the previous step is resampled, and the obtained feature matrix is ​​expressed as where M∈R K×K is a public learnable matrix;

[0039] The output of TAG convolution is the new vertex feature Expressed as Among them D out is the characteristic dimension of the output vertex, the matrix Contains D out anisotropic filters, b is the bias, vec(·) is the operation of concatenating the matrix into column vectors, (·) T is the matrix transpose operation.

[0040] Optionally, the first to third fully convolutional graph pooling layers, wherein the fully convolutional graph pooling is:

[0041] Assume that the number of old vertices in the input spherical graph data is N, sample the spherical position coordinates of 1 / 4N new vertices, and use the KNN algorithm to construct edges with self-loops between each new vertex and K adjacent old vertices on the spherical graph. The direction of this edge is from the old vertex to the new vertex.

[0042] The old vertex features of the spherical graph data and the constructed edge information are used as the input of the graph convolution method GCN, and the output features corresponding to 1 / 4N new vertex positions are obtained. The input feature dimension and the output feature dimension are both controllable parameters corresponding to GCN.

[0043] Optionally, each of the first to third expanded attention blocks is configured as:

[0044] For the input spherical image data, it is first processed using a dilated TAG convolution with a dilation rate of R, and then the input and output of the dilated TAG convolution are added as the input of the graph transformer, and finally the output of the graph transformer is used as the input of the TAG convolution.

[0045] Optionally, the steps of the graph transformer are:

[0046] When the head parameter H is set to 1, for each vertex feature vector The set of its neighboring point feature vectors is set as Then the output feature Expressed as in Softmax(·) is a normalized exponential function, (·) T It is a matrix transpose operation, W1, W2, W3, W4 are all learnable matrices;

[0047] When the head parameter is H, and H is not equal to 1, the above steps are performed H times to obtain H output features for feature splicing. out Dimensional features are merged into D out ×H dimensional features as output.

[0048] Optionally, each of the first to third fully convolutional image depooling layers is configured as follows:

[0049] Assume that the number of old vertices in the input spherical graph data is N, and the spherical position coordinates of 4N new vertices are sampled. The KNN algorithm is used to construct edges with self-loops between each old vertex and K adjacent new vertices on the spherical graph. The direction of this edge is from the old vertex to the new vertex.

[0050] The old vertex features of the spherical graph data and the constructed edge information are used as the input of the graph convolution method GCN, and the output features corresponding to 4N new vertex positions are obtained. The input feature dimension and the output feature dimension are both controllable parameters corresponding to GCN.

[0051] In a second aspect of the present invention, a panoramic image saliency prediction device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the panoramic image saliency prediction method based on a fully convolutional graph neural network when executing the computer program.

[0052] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0053] The panoramic image saliency prediction method and device based on a fully convolutional graph neural network proposed in the present invention is an intuitive and effective method for mapping panoramic images to a spherical graph network representation. This can eliminate redundant parts in planar panoramic images while retaining information areas, and can reasonably map panoramic image pixels to a spherical graph data structure for representation.

[0054] The present invention proposes a panoramic image saliency prediction method and device based on a fully convolutional graph neural network. The saliency prediction network is based on a residual U-shaped neural network structure and adopts dilated graph convolution and attention mechanism at the bottom to improve the performance of panoramic image saliency prediction.

[0055] The present invention proposes a panoramic image saliency prediction method and device based on a fully convolutional graph neural network. The method and device use a fully convolutional layer to perform graph pooling and unpooling operations in the spherical image space to preserve vertex-to-vertex features. The fully convolutional graph pooling and unpooling operations are introduced into the panoramic saliency prediction task, which is conducive to making the end-to-end pixel-level prediction task more accurate.

[0056] Experimental results show that the proposed method outperforms other advanced saliency models on large-scale datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0058] Figure 1 This is an overall flow chart of a panoramic image saliency prediction method based on a fully convolutional graph neural network in one embodiment of the present invention;

[0059] Figure 2 Schematic diagram of a panoramic image saliency prediction network based on a fully convolutional graph neural network in one embodiment of the present invention;

[0060] Figure 3 Schematic diagram of an expanded attention block in one embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. It should be noted that those skilled in the art may make various modifications and improvements without departing from the scope of the present invention, and these modifications and improvements fall within the scope of protection of the present invention.

[0062] Distortion caused by the non-Euclidean geometric features of panoramic images severely impacts saliency prediction. Conventional saliency prediction algorithms for planar images based on convolutional neural networks are no longer suitable for panoramic images. This paper proposes a method for panoramic image saliency prediction based on a fully convolutional graph neural network, which effectively addresses this issue.

[0063] Figure 1 This is an overall flow chart of a panoramic image saliency prediction method based on a fully convolutional graph neural network in one embodiment of the present invention. Figure 1 As shown, the panoramic image saliency prediction method based on a fully convolutional graph neural network in this embodiment includes the following steps:

[0064] Step 1: Map the planar panoramic image into spherical image data, specifically by performing I2G mapping on the planar panoramic image;

[0065] Step 2: Input the spherical graph data obtained in the previous step into the fully convolutional graph neural network for saliency prediction to obtain the spherical saliency map data;

[0066] Step 3: transforming the spherical saliency map data onto a plane to obtain a planar panoramic saliency image; wherein, performing G2I interpolation on the spherical saliency map data to transform the spherical saliency map data onto a plane.

[0067] In this embodiment, the panoramic image is mapped to a spherical graph network representation, which can eliminate redundant parts in the planar panoramic image while retaining the information area, thereby improving the performance of panoramic image saliency prediction.

[0068] In some embodiments, in order to map the planar panoramic image into spherical image data, the execution of step 3 above includes the following sub-steps:

[0069] Step 1.1: Map the planar image to a spherical panoramic image. The expression is:

[0070]

[0071] Among them, x and y are the two-dimensional coordinates of the plane panoramic image in the rectangular coordinate system, and λ is the longitude of the point (x, y) in the rectangular coordinate system of the plane panoramic image after being projected onto the spherical surface. is the latitude of the planar panoramic image after being projected onto the spherical surface, is the latitude corresponding to the horizontal central axis of the planar panoramic image, which is 0 here, and λ0 is the longitude corresponding to the central meridian of the planar panoramic image;

[0072] Step 1.2: On the spherical panoramic image, use the vertices of the spherical image that are almost evenly distributed, and the number of sampled vertices is set to the number of pixels. Each vertex of the spherical graph has the RGB color value of the corresponding pixel on the spherical panoramic image as its feature. Then, the KNN (K-Nearest Neighbor) algorithm is used to construct the edges with self-loops between each vertex and K adjacent points on the spherical graph. Through this step, a spherical graph data can be obtained as the subsequent input, in which each vertex has 3 features and K edges. In this step, sampling can be implemented using existing technologies, such as the Fibonacci grid sampling method well known in the art. Of course, other sampling methods can also be used. Each vertex has the RGB color value of the corresponding pixel on the spherical panoramic image as its feature, and the RGB value is these 3 features. The method for determining K is not specified and can be any positive number. For example, in one embodiment, K can be set to 9.

[0073] In order to perform step 2, the existing saliency prediction network is improved, wherein the fully convolutional graph neural network is based on the residual U-shaped neural network structure. In some embodiments, the dilated graph convolution and attention mechanism are used at the bottom to improve the performance of panoramic image saliency prediction. In other embodiments, the fully convolutional layer is used in the fully convolutional graph neural network to perform graph pooling and de-pooling operations in the spherical image space to retain vertex-to-vertex features. Full convolutional graph pooling and de-pooling operations are introduced in the panoramic saliency prediction task, which is conducive to making the end-to-end pixel-level prediction task more accurate. The above-mentioned dilated graph convolution and attention mechanism, full convolutional graph pooling and de-pooling operations can be selected separately, and preferably both exist at the same time. This can achieve better performance in panoramic image saliency prediction.

[0074] Specifically, in a preferred embodiment, the fully convolutional graph neural network structure is as follows Figure 2 The figure shows a four-layer U-shaped graph neural network structure, with each row representing a layer. The U-shaped network structure is a progressive relationship, with features from the previous layer derived from the features of the next layer. Pooling and unpooling layers are generally considered to connect different layers, and the number of graph nodes within each layer is the same. Specifically, the U-shaped graph neural network structure consists of three sub-components: the encoding part, the base part, and the decoding part. The detailed structure, connections, and operation of each sub-component are described below.

[0075] Specifically, in one embodiment, the encoding unit, the bottom unit, and the decoding unit may adopt the following specific structures and parameters:

[0076] (1) Coding Department

[0077] The network consists of 4 TAG convolution layers and 3 full convolution graph pooling layers, named as the first TAG convolution layer, the second TAG convolution layer, the third TAG convolution layer, the fourth TAG convolution layer, the first full convolution graph pooling layer, the second full convolution graph pooling layer, and the third full convolution graph pooling layer. The characteristics of each layer are as follows:

[0078] The first TAG convolutional layer: The input is the spherical image data obtained in step 1, the input feature dimension is 3, and the output feature dimension is 32;

[0079] Second TAG convolution layer: The input is the output of the first TAG convolution layer, the input feature dimension is 32, and the output feature dimension is 32;

[0080] First fully convolutional graph pooling layer: The input is the sum of the output of the first TAG convolution layer and the output of the second TAG convolution layer, the input feature dimension is 32, and the output feature dimension is 64;

[0081] The third TAG convolution layer: The input is the output of the first full convolutional graph pooling layer, the input feature dimension is 64, and the output feature dimension is 64;

[0082] Second full convolutional graph pooling layer: The input is the sum of the output of the first full convolutional graph pooling layer and the output of the third TAG convolutional layer. The input feature dimension is 64 and the output feature dimension is 128.

[0083] The fourth TAG convolution layer: the input is the output of the second full convolutional graph pooling layer, the input feature dimension is 128, and the output feature dimension is 128;

[0084] The third fully convolutional graph pooling layer: The input is the sum of the output of the second fully convolutional graph pooling layer and the output of the fourth TAG convolutional layer. The input feature dimension is 128 and the output feature dimension is 256.

[0085] (2) Bottom

[0086] The network contains three dilated attention blocks, named the first dilated attention block, the second dilated attention block, and the third dilated attention block. The characteristics of each block are as follows:

[0087] First dilated attention block: the input is the output of the encoding unit, the input feature dimension is 256, the output feature dimension is 256, and the dilation rate is 1;

[0088] Second dilated attention block: The input is the output of the first dilated attention block, the input feature dimension is 256, the output feature dimension is 256, and the dilation rate is 2;

[0089] The third dilated attention block: the input is the output of the second dilated attention block, the input feature dimension is 256, the output feature dimension is 256, and the dilation rate is 3.

[0090] (3) Decoding unit

[0091] The network consists of 5 TAG convolution layers and 3 full convolution graph depooling layers, named as the fifth TAG convolution layer, the sixth TAG convolution layer, the seventh TAG convolution layer, the eighth TAG convolution layer, the ninth TAG convolution layer, the first full convolution graph depooling layer, the second full convolution graph depooling layer, and the third full convolution depooling layer. The characteristics of each layer are as follows:

[0092] The first fully convolutional image depooling layer: the input is the output of the bottom layer, the input feature dimension is 256, and the output feature dimension is 128;

[0093] The fifth TAG convolutional layer takes as input the concatenation of the output of the first full convolutional graph depooling layer and the input of the third full convolutional graph pooling layer. The specific concatenation method is a dimensionality expansion method known in the art, merging the two 128-dimensional features into a 256-dimensional feature. The input feature dimension is 256, and the output feature dimension is 256.

[0094] Second full convolution graph depooling layer: the input is the sum of the input and output of the fifth TAG convolution layer, the input feature dimension is 256, and the output feature dimension is 64;

[0095] The sixth TAG convolutional layer takes as input the concatenation of the output of the second full convolutional graph unpooling layer and the input of the second full convolutional graph pooling layer. The specific concatenation method is a dimensionality expansion method known in the art, merging the two 64-dimensional features into a 128-dimensional feature. The input feature dimension is 128, and the output feature dimension is 128.

[0096] The third fully convolutional graph depooling layer: The input is the sum of the input and output of the sixth TAG convolution layer, the input feature dimension is 128, and the output feature dimension is 32;

[0097] The seventh TAG convolutional layer: The input is the concatenation of the output of the third full convolutional graph unpooling layer and the input of the first full convolutional graph pooling layer. The specific feature concatenation method is a dimensionality expansion method known in the art, merging the two 32-dimensional features into a 64-dimensional feature. The input feature dimension is 64, and the output feature dimension is 64.

[0098] The eighth TAG convolution layer: The input is the sum of the input and output of the seventh TAG convolution layer. The input feature dimension is 64, and the output feature dimension is 32.

[0099] The ninth TAG convolution layer: The input is the output of the eighth TAG convolution layer, the input feature dimension is 32, the output feature dimension is 1, and the output is the spherical saliency map data.

[0100] The above is a detailed description of the fully convolutional graph neural network structure in a preferred embodiment. Of course, the specific input and output dimensions and other parameters in the above fully convolutional graph neural network structure can be adjusted as needed and are not limited to the above-mentioned specific parameters such as 32 and 64. The embodiment of the above-mentioned fully convolutional graph neural network structure can improve the performance of panoramic image saliency prediction, retain more image features, and facilitate end-to-end pixel-level prediction tasks to be more accurate. Through the jump connection of the above-mentioned encoding and decoding parts, performance can be enhanced and the negative effects of too many neural network layers in deep learning can be reduced.

[0101] In some embodiments, for the first to ninth TAG convolution layers described above, the TAG convolution (English: Tiny Anisotropic Graph Convolution) operation of each TAG convolution layer is specifically as follows:

[0102] For each vertex in the spherical graph data, the feature vector Merge its K neighboring features into a feature matrix Among them D in is the feature dimension of the input vertex;

[0103] For the feature matrix X of the previous step i , resampled, the obtained feature matrix can be expressed as where M∈R K×K is a public learnable matrix;

[0104] The output of TAG convolution is the new vertex feature Expressed as Among them D out is the characteristic dimension of the output vertex, the matrix Contains D out anisotropic filters, b is the bias, vec(·) is the operation of concatenating the matrix into column vectors, (·) T is the matrix transpose operation.

[0105] In the above embodiment, the operation of each TAG convolution layer from the first TAG convolution layer to the ninth TAG convolution layer is the same, except that the two parameters, the feature dimension of the input vertex and the feature dimension of the output vertex, are different.

[0106] In some embodiments, the first to third fully convolutional graph pooling layers described above, wherein the fully convolutional graph pooling (English: Fully Convolutional Graph Pooling) specifically operates as follows:

[0107] (1) Assume that the number of old vertices in the input spherical graph data is N. The spherical position coordinates of 1 / 4N new vertices are obtained by Fibonacci grid sampling. The KNN algorithm is used to construct edges with self-loops between each new vertex and K adjacent old vertices on the spherical graph. The direction of this edge is from the old vertex to the new vertex.

[0108] (2) By using the old vertex features of the spherical graph data and the constructed edge information as the input of the graph convolutional network (GCN), we can obtain the output features corresponding to 1 / 4N new vertex positions. The input feature dimension and the output feature dimension are both controllable parameters corresponding to GCN.

[0109] The above embodiment illustrates the entire fully convolutional graph pooling. Different fully convolutional graph pooling layers differ only in the vertex parameter N before and after the operation in the definition. Through the fully convolutional graph pooling operation, the pooling operation in the spherical graph space can be combined with the graph depooling in the spherical graph space to preserve node-to-node features, providing conditions for improving the accuracy of the final prediction results.

[0110] like Figure 3 As shown, in some embodiments, the specific operation of each dilated attention block (English: Dilated Attention Block) of the first to third dilated attention blocks is as follows:

[0111] (1) For the input spherical image data, it is first processed using dilated TAG convolution with a dilation rate of R, and the input and output feature dimensions are both 256.

[0112] (2) The input and output of the expanded TAG convolution are added together as the input of the graph transformer. The input dimension of the graph transformer is 256, the head H is 8, and the output dimension is 2048.

[0113] (3) Finally, the output of the graph transformer is used as the input of the TAG convolution, with an input dimension of 2048 and an output dimension of 256.

[0114] In the above embodiment, in the first to third dilated attention blocks, the "attention mechanism" is embodied in the dilated TAG convolution. The "dilation" here means that compared to ordinary TAG convolution, the range of selected neighboring points is larger, hence the name "dilation". This allows a wider range of data features to be obtained compared to ordinary TAG convolution. Of course, the parameters in the above embodiment are only preferred, and other dimensions may be used in other embodiments.

[0115] In a preferred embodiment, the dilated TAG convolution (English: Dilated Tiny Anisotropic Graph Convolution) adjusts the "K neighboring points" of the TAG convolution to "randomly selecting K neighboring points from the nearest K×R neighboring points". The method for obtaining the neighboring points can adopt existing technologies, such as the KNN algorithm well known in the art.

[0116] Furthermore, the steps of the Graph Transformer are:

[0117] When the head parameter H is set to 1, for each vertex feature vector The set of its neighboring point feature vectors is set to N i ={x i,0 ,x i,1 ,…,x i,K-1}, then the output feature It can be expressed as in Softmax(·) is a normalized exponential function well known in the art, (·)T It is a matrix transpose operation, W1, W2, W3, and W4 are all learnable matrices.

[0118] When the head parameter is H, and H is not equal to 1, the above steps are performed H times to obtain H output features for feature splicing. The specific feature splicing method is the dimension expansion method known in the art. out Dimensional features are merged into D out ×H dimensional features as output.

[0119] In some embodiments, the first to third fully convolutional graph unpooling layers, wherein the fully convolutional graph unpooling (English: Fully Convolutional Graph Unpooling) specifically operates as follows:

[0120] (1) Assume that the number of old vertices in the input spherical graph data is N. The spherical position coordinates of 4N new vertices are obtained by Fibonacci grid sampling. The KNN algorithm is used to construct edges with self-loops between each old vertex and K adjacent new vertices on the spherical graph. The direction of this edge is from the old vertex to the new vertex.

[0121] (2) By using the old vertex features of the spherical graph data and the constructed edge information as the input of the graph convolutional network (GCN), we can obtain the output features corresponding to the 4N new vertex positions. The input feature dimension and the output feature dimension are both controllable parameters corresponding to GCN.

[0122] The above embodiment can realize the depooling processing in the spherical image space through the operation of full convolution graph depooling. Combined with the graph pooling in the spherical image space in the previous embodiment, it can retain the node-to-node features and provide conditions for improving the accuracy of the final prediction results.

[0123] In the preferred embodiment described above, the saliency prediction network is based on a residual U-shaped network structure, employing dilated graph convolution and an attention mechanism at the bottom. Fully convolutional layers are used to perform graph pooling and unpooling operations in the spherical graph space to preserve node-to-node features, making the entire method superior to other existing advanced saliency prediction models on large-scale datasets.

[0124] Based on the same technical concept mentioned above, another embodiment of the present invention further provides a panoramic image saliency prediction device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the panoramic image saliency prediction method based on the fully convolutional graph neural network in any of the above embodiments.

[0125] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned methods), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. In addition, the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

[0126] The aforementioned computer programs, computer instructions, etc. may be partitioned and stored in one or more memories, and the aforementioned computer programs, computer instructions, data, etc. may be called by a processor.

[0127] The processor is configured to execute the computer program stored in the memory to implement the various steps of the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0128] The processor and memory can be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor can be coupled via a bus.

[0129] According to the method steps of the above embodiment, 600 panoramic images from a public database were used for testing. The dataset used is the AOI dataset proposed by Xu et al. in "M.Xu, L.Yang, X.Tao, Y.Duan, and Z.Wang, "Saliency prediction on omnidirectional image with generative adversarial imitation learning," IEEE Transactions on Image Processing, vol.30, pp.2087–2102, 2021." This dataset is the largest panoramic saliency dataset, including 4 categories, 500 training images and 100 test images. The embodiment uses a dropout method known in the art after each convolutional layer, with a dropout ratio of 0.1. The optimizer used is AdamW proposed by Loshchilov et al. in "I.Loshchilov and F.Hutter, "Decoupled weight decay regularization," in International Conference on Learning Representations, 2019," with a learning rate of 0.00005. All experiments are performed on a single Nvidia RTX 3090 GPU.

[0130] To objectively compare the performance of the present invention with the most advanced methods, the embodiment selected four indicators: Kullback-Leibler divergence (KL), linear correlation coefficient (CC), normalized scan path saliency (NSS), and area under the receiver operating characteristic curve (AUC). These indicators are all well-known objective evaluation indicators in the field (Z. Bylinskii, T. Judd, A. Oliva, A. Torralba, and F. Durand, "What do different evaluation metrics tell us about saliency models?", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 3, pp. 740–757, 2018). The embodiment selects several advanced existing methods for comparison, including Salicon (X. Huang, C. Shen, X. Boix, and Q. Zhao, “Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 262–270, 2015.), BMS360 and GBVS360 (P. Lebreton and A. Raake, “Gbvs360, bms360, prosal: Extending existing saliency prediction models from 2d toomnidirectional images,” Signal Processing: Image Communication, vol. 69, pp. 69–78, 2018.), Zhu (Y. Zhu, G. Zhai, and X. Min, “The prediction of head and eye movement for 360 degree images,” Signal Processing: Image Communication, vol. 69, pp. 69–78, 2018.). Communication, vol.69, pp.15–25, 2018.》), Salgan360(《F.-Y.Chao, L.Zhang, W.Hamidouche, and O.Deforges, "Salgan360: Visual saliency prediction on 360degree images with generative adversarial networks," in IEEE International Conference on Multimedia&ExpoWorkshops, pp.01–04, IEEE, 2018. adversarialimitation learning," IEEE Transactions on Image Processing, vol. 30, pp. 2087–2102, 2021."). .

[0131] Table 1

[0132] AUC↑ CC↑ KL↓ NSS↑ Salicon 0.778 0.526 0.613 0.980 BMS360 0.839 0.668 0.546 1.411 GBVS360 0.799 0.588 0.563 1.147 Zhu 0.850 0.729 0.416 1.450 Salgan360 0.835 0.653 0.562 1.429 SalGAIL 0.853 0.742 0.345 1.556 SalGFCN 0.865 0.853 0.212 1.659

[0133] The corresponding performance test results are shown in Table 1, where the method proposed by the present invention is shown as SalGFCN. Bold numbers indicate the best performance, and arrows indicate the direction of higher accuracy. The table shows that the panoramic image saliency prediction method based on a fully convolutional graph neural network, provided by the above embodiment of the present invention, effectively predicts the saliency of panoramic images, outperforming all other methods.

[0134] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A panoramic image saliency prediction method based on a fully convolutional graph neural network, characterized in that: include: Mapping a planar panoramic image into spherical image data; Inputting the spherical graph data into a fully convolutional graph neural network for saliency prediction to obtain spherical saliency graph data; Transforming the spherical saliency map data onto a plane to obtain a planar panoramic saliency image; The fully convolutional graph neural network structure is a U-shaped graph neural network structure, which includes: Setting dilated graph convolution and attention mechanism at the bottom; and / or, Fully convolutional layers for graph pooling and unpooling operations in spherical graph space to preserve vertex-to-vertex features; The U-shaped graph neural network structure includes an encoding part, a bottom part and a decoding part, wherein: The encoding unit includes a first TAG convolution layer, a second TAG convolution layer, a third TAG convolution layer, a fourth TAG convolution layer, a first full convolution graph pooling layer, a second full convolution graph pooling layer and a third full convolution graph pooling layer, wherein: The first TAG convolutional layer input is the graph data of the sphere. Second TAG convolution layer: input is the output of the first TAG convolution layer; The first fully convolutional graph pooling layer: the input is the sum of the output of the first TAG convolutional layer and the output of the second TAG convolutional layer; The third TAG convolution layer: the input is the output of the first full convolutional graph pooling layer; Second full convolutional graph pooling layer: The input is the sum of the output of the first full convolutional graph pooling layer and the output of the third TAG convolutional layer; The fourth TAG convolution layer: the input is the output of the second full convolutional graph pooling layer; The third full convolutional graph pooling layer: The input is the sum of the output of the second full convolutional graph pooling layer and the output of the fourth TAG convolutional layer; The bottom portion includes a first dilated attention block, a second dilated attention block, and a third dilated attention block connected in series, wherein: the input of the first dilated attention block is the output of the encoding unit, and the output of the bottom portion is the output of the third dilated attention block; The decoding unit includes a fifth TAG convolution layer, a sixth TAG convolution layer, a seventh TAG convolution layer, an eighth TAG convolution layer, a ninth TAG convolution layer, a first full convolution graph depooling layer, a second full convolution graph depooling layer, and a third full convolution depooling layer, wherein: The first fully convolutional image unpooling layer: the input is the output of the bottom layer; The fifth TAG convolution layer: The input is the feature concatenation of the output of the first full convolution map depooling layer and the input of the third full convolution map pooling layer; The second fully convolutional graph anti-pooling layer: the input is the sum of the input of the fifth TAG convolutional layer and the output of the fifth TAG convolutional layer; The sixth TAG convolution layer: the input is the feature concatenation of the output of the second full convolution map depooling layer and the input of the second full convolution map pooling layer; The third full convolution graph anti-pooling layer: the input is the sum of the input of the sixth TAG convolution layer and the output of the sixth TAG convolution layer; The seventh TAG convolution layer: The input is the feature concatenation of the output of the third full convolution map depooling layer and the input of the first full convolution map pooling layer; The eighth TAG convolution layer: the input is the sum of the input of the seventh TAG convolution layer and the output of the seventh TAG convolution layer; Ninth TAG convolution layer: The input is the output of the eighth TAG convolution layer.

2. The method for predicting panoramic image saliency based on a fully convolutional graph neural network according to claim 1, characterized in that: The mapping of the planar panoramic image into spherical image data includes: Mapping a planar image to a spherical panoramic image; Sampling is performed on the spherical panoramic image to obtain evenly distributed vertices, each vertex having an RGB color value from a corresponding pixel on the spherical panoramic image as its feature; edges with self-loops are constructed between each vertex and K adjacent points on the spherical panoramic image, and finally spherical graph data is obtained, in which each vertex has 3 features and K edges.

3. The method for predicting panoramic image saliency based on a fully convolutional graph neural network according to claim 1, characterized in that: The second TAG convolution layer to the ninth TAG convolution layer, each TAG convolution layer is configured as follows: For each vertex in the graph, the eigenvector Merge its K neighboring point features into a feature matrix Where D in is the feature dimension of the input vertex; The feature matrix of the previous step is resampled, and the obtained feature matrix is ​​expressed as where M∈R K×K is a public learnable matrix; The output of TAG convolution is the new vertex feature Expressed as Where D out is the characteristic dimension of the output vertex, the matrix Contains D out anisotropic filter, b is the bias, vec(·) is the operation of concatenating the matrix into a column vector by column, (·) T is the matrix transpose operation.

4. The method for predicting panoramic image saliency based on a fully convolutional graph neural network according to claim 1, characterized in that: The first to third full convolutional graph pooling layers, wherein the full convolutional graph pooling is: Assume that the number of old vertices of the input spherical graph data is N, sample the spherical position coordinates of 1 / 4N new vertices, and construct the edges with self-loops between each new vertex and K adjacent old vertices on the spherical graph through the KNN algorithm. The direction of this edge is from the old vertex to the new vertex. The old vertex features of the spherical graph data and the constructed edge information are used as the input of the graph convolution method GCN, and the output features corresponding to 1 / 4N new vertex positions are obtained. The input feature dimension and the output feature dimension are both controllable parameters corresponding to GCN.

5. The method for predicting panoramic image saliency based on a fully convolutional graph neural network according to claim 1, characterized in that: The first to third dilated attention blocks, each of which is set to: For the input spherical image data, it is first processed using a dilated TAG convolution with a dilation rate of R, and then the input and output of the dilated TAG convolution are added as the input of the graph transformer, and finally the output of the graph transformer is used as the input of the TAG convolution.

6. The method for predicting panoramic image saliency based on a fully convolutional graph neural network according to claim 5, characterized in that: The steps of the graph transformer are: When the head parameter H is set to 1, for each vertex feature vector The set of its neighboring point feature vectors is set to N i ={x i,0 ,x i,1 ,…,x i,K-1 }, then the output feature Expressed as in softmax(·) is a normalized exponential function, (·) T It is a matrix transposition operation, W1, W2, W3, W4 are all learnable matrices; When the head parameter is H, and H is not equal to 1, the above steps are performed H times to obtain H output features for feature concatenation. out Dimensional features are merged into D out ×H dimensional features as output.

7. The method for predicting panoramic image saliency based on a fully convolutional graph neural network according to claim 1, characterized in that: The first to third full convolutional graph depooling layers, each of which is configured as follows: Assume that the number of old vertices of the input spherical graph data is N, sample the spherical position coordinates of 4N new vertices, and use the KNN algorithm to construct an edge with a self-loop between each old vertex and K adjacent new vertices on the spherical graph. The direction of this edge is from the old vertex to the new vertex. The old vertex features of the spherical graph data and the constructed edge information are used as the input of the graph convolution method GCN, and the output features corresponding to the 4N new vertex positions are obtained. The input feature dimension and the output feature dimension are both controllable parameters corresponding to GCN.

8. A panoramic picture saliency prediction device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Methods for the preparation of organotin compounds

    EP0002019A1

  • Remote sensing image road extraction method based on D-LinkNet

    CN111767810A

  • Panoramic image visual saliency prediction method based on convolutional neural network

    CN112488122A

  • Laser directional energy deposition area calculation method of full convolutional neural network

    CN112967266A

  • Learning-based sampling for image matting

    EP3737083A1