A method, device, and storage medium for panoramic image saliency prediction

Abstract

The present invention discloses a panoramic image saliency prediction method, device, and storage medium, including: using a natural scene panoramic image as a data set, performing projection preprocessing on the panoramic images in the data set, where the projection preprocessing includes multi-cube projection and multi-spherical rotation projection; respectively using the projection panoramic images generated by multi-cube projection and multi-spherical projection as source data to input into a global and local sub-network, performing saliency prediction training to obtain a local saliency prediction image and a global saliency prediction image; fusing the global saliency prediction image and the local saliency prediction image in proportion to obtain a balanced saliency projection that simultaneously focuses on global and local information, and performing inverse projection in the same direction and angle on the balanced saliency projection to obtain a fused saliency prediction map. The present invention simultaneously focuses on the global and local information of the panoramic image, and linearly weights and fuses the global and local feature information, which is more accurate and efficient than the convolutional neural network model.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and particularly to a panoramic image saliency prediction method, device and storage medium. Background Art

[0002] In order to further study the training of saliency prediction and measure the performance, researchers at home and abroad have made a series of saliency datasets. Datasets for traditional 2D images such as SALICON and MIT300, and panoramic image datasets such as Salient360 and VR-Saliency; a series of performance metrics have been added, namely AUC, IG, NSS, SIM, EMD, CC, KLD and their variants. In addition to optimizing the prediction model, the research directions for panoramic image saliency mainly fall into the following three points:

[0003] Equatorial bias of panoramic images: The visual habit of the human eye determines that when a person gazes at a panoramic image, they will pay more attention to the area near the equator and subconsciously ignore the upper and lower edges. For this visual mechanism, Battisti et al. assigned a saliency weight that gradually decays from the equator to the poles to the self-divided regions of the panoramic image. On this basis, Ding et al. used a Gaussian distribution model to simulate the equatorial bias and refined the saliency from the image level to the pixel level.

[0004] Data mapping of the observer's fixation point position: Some saliency prediction algorithms establish a corresponding saliency map based on the fixation point coordinate data generated when the user views the panoramic image and map it. This type of method extends the limited fixation positions of the observer to a general saliency map. Upenik et al. used the head movement trajectory to predict saliency. They determined the time spent by the observer observing a specific area of the panoramic image based on the angular velocity of the head movement. Abreu et al. distinguished the degree of the observer's visual attention based on the fixation time. However, the above methods require collecting the fixation positions and related data of the observer when viewing the panoramic image in a virtual environment, and the accuracy and timeliness of the data collection depend on the performance of the eye-tracking device used in the experiment. If the device performance fails to meet the acquisition conditions, it is difficult to guarantee the scale and quality of the data, and it is not representative. Then, there will be a large difference between the saliency map generated by the model and the true marked fixation positions. Therefore, the saliency prediction method based on data mapping of the fixation point position requires a large amount of manpower and material resources and is usually only applicable to generating true annotation maps through the original dataset.

[0005] Panoramic Projection Transformation: For projective transformation, the most commonly used method is equirectangular projection. To address the problem of distortion near the edges of images in equirectangular formats, Lebreton et al. averaged several equirectangular saliency maps with different longitudes as boundaries. However, they did not effectively handle the distortion at the upper and lower boundaries of the panorama. Startsev et al. first used the vertical midline of the equirectangular image as a cutting line, translated the left and right parts of the image separately, and generated inverse equirectangular maps. Subsequently, they fused the two saliency maps generated by the model to address the problem of discontinuous scene content at the left and right boundaries of the equirectangular image. In addition, they also predicted the saliency of the upper and lower regions of the equirectangular image separately. However, since they directly used a pre-trained 2D image saliency model to predict the equirectangular image, the model was unable to fully extract the unique features of the panorama.

[0006] The research on panoramic image saliency prediction based on deep learning has developed to date, and has achieved many results in convolutional neural networks, recurrent neural networks, long short-term memory networks, graph convolutional neural networks, encoder-decoders, and transfer learning. However, the mainstream and most effective technical route is still the convolutional neural network. It is precisely because of its strong local feature extraction ability that the prediction is effective. However, its lack of global information capture ability and its insufficient balance between global and local information lead to unstable and biased prediction results. In addition, in the data preprocessing stage, the panorama is converted into an equirectangular projection method, which separates some complete objects in the panorama, resulting in partial distortion of the saliency prediction effect, which is also a problem that needs to be solved urgently. Summary of the Invention

[0007] To overcome the aforementioned shortcomings and deficiencies of the prior art, the present invention aims to provide a method, device, and storage medium for predicting panoramic image saliency. The Transformer network architecture proposed in this invention is precisely able to better focus on the features of panoramic images at a global scale, enabling balanced extraction of global and local information, making the model's predictions in various natural scenes more realistic. Furthermore, to address the issue of insufficient data, it is necessary to appropriately project the panoramic image saliency data to meet the training requirements for data augmentation.

[0008] The purpose of the present invention is achieved through the following technical solutions:

[0009] A panoramic image saliency prediction method, comprising:

[0010] Taking a natural scene panorama as a data set, performing projection preprocessing on the panoramic images in the data set, wherein the projection preprocessing includes multi-cube projection and multi-spherical rotation projection;

[0011] The panoramic projection maps generated by multi-cube projection and multi-spherical projection are respectively used as source data to input into the global and local sub-networks, and significant prediction training is carried out to obtain local significant prediction pictures and global significant prediction pictures;

[0012] The global significant prediction picture and the local significant prediction picture are fused in proportion to obtain an equilibrium significant projection that simultaneously focuses on global and local information. The equilibrium significant projection is subjected to inverse projection in the same direction and angle to obtain a fused significant prediction map.

[0013] Furthermore, the panoramic picture generated by multi-cube projection is used as source data to input into the global and local sub-networks for significant prediction training. Specifically, local significant prediction pictures are generated through the encoder and decoder of the sub-network;

[0014] The encoder successively includes four stages:

[0015] The first stage: The equirectangular projection sequence generated after projection passes through the partition mapping module, and the entire three-channel panoramic picture is equally divided into non-overlapping picture blocks;

[0016] The second stage: The non-overlapping picture blocks output in the first stage pass through the linear embedding module - the purpose is to change the dimension of the input vector to a pre-set value, that is, the value that the Transformer module can accept; and then it is imported into two rounds of Swin Transformer Block modules, and self-attention calculations based on windows and moving windows are performed according to the moving scale within the module. Each window is called a partition, and the attention calculation result exported by the module is the attention feature information at the corresponding partition scale;

[0017] The third stage: The partition attention feature information obtained in the second stage is operated by the partition merging module, and the windows in two adjacent rows and two adjacent columns are subjected to 2-fold downsampling operation; and then through six rounds of Swin Transformer Block modules, different-scale regional self-attention feature information obtained in different rounds is fully learned;

[0018] The fourth stage: The regional self-attention feature information obtained in the previous stage is subjected to partition merging operation again, and then through two rounds of Swin Transformer Block, local fusion feature information at a higher scale is obtained.

[0019] Furthermore, the multi-cube projection is to perform spherical projection on each complete panoramic picture, and then map it to the upper, lower, left, right, front, and back six faces of a regular hexahedron to generate six straight projection maps corresponding to a 90° viewport angle.

[0020] Further, using the panoramic image after projecting the multi-cube as the source data of the local sub-network for saliency prediction includes the following:

[0021] Pass the input image through a bifurcated branch, and equally divide the panoramic image into two image patches with different sizes as the source data for the two branches;

[0022] Input the image patches of different sizes into the linear embedding module in sequence to obtain feature encodings, and add a relative position Token after each feature encoding segment to obtain two spatial feature incorporations;

[0023] Perform multi-scale cross-attention encoding on the two spatial feature incorporations respectively, and then input them into the decoder to obtain local feature information.

[0024] Further, input the saliency prediction obtained by the global sub-network and the saliency prediction of the local sub-network into a decoder with convolution-upsampling operations, output the predicted saliency map, and then obtain the saliency map of the original image through the corresponding back-projection operations respectively.

[0025] Further, the specific multi-spherical rotation projection is as follows:

[0026] Perform spherical projection on the panoramic image, and then, with the projection center as the origin, establish a three-dimensional Cartesian coordinate system perpendicular to each other in the horizontal, vertical, and longitudinal dimensions, and perform spherical rotation projection on the spherical panoramic image. The three rotation dimensions correspond to ω x , ω y , ω z axes respectively, where x, y, and z are angle sequences in [0, 360], and the lengths are K x , K y , K z , representing the rotation angles of the spherical map around the x, y, and z axes when performing the rotation operation each time;

[0027] According to the configurations of ω x , ω y , ω z , perform the corresponding spherical rotation operations in sequence to obtain K x ·K y ·K z rotated spherical maps, and finally inverse-project them into equirectangular panoramic images under the corresponding angle transformations.

[0028] Further, the prediction formula of the encoder is;

[0029]

[0030]

[0031]

[0032] Among them, when i = 1 or 2 or 4, n i = 2; when i = 3, n i = 6; and represents a linear encoding operation, WSA is a normal window self-attention module, and SW is a combination of sliding window self-attention modules bound to WSA in sequence.

[0033] Furthermore, for the back-projection, the formula is:

[0034] Y i = f ERP-S (X i )

[0035]

[0036]

[0037] Among them, x, y, and z are the three coordinate axes of the coordinate system, and f ERP-S is the mapping from equirectangular projection to spherical projection, is its inverse mapping, X i represents the i-th equirectangular projection image in the dataset, Y i is the spherical projection after its mapping, {Z i} is the projection sequence after the spherical projection Y i is rotated around the three coordinate axes, and X′ i is its corresponding equirectangular projection sequence.

[0038] A device includes: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the panoramic image saliency prediction method.

[0039] A computer-readable storage medium stores computer-executable instructions for causing a computer to execute the panoramic image saliency prediction method.

[0040] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0041] (1) The present invention utilizes the idea of transfer learning to pre-train the model on a traditional 2D saliency dataset, and then applies the pre-trained model to panoramic images for further fine-tuning;

[0042] (2) Through the sub-network with two-branch prediction, while paying attention to both the global and local information of the panoramic image, and linearly weighting and fusing the global and local feature information at the end, the prediction effect is more accurate and efficient than that of the convolutional neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic flow chart of the traditional convolutional neural network saliency prediction method;

[0044] Figure 2 is the work flow chart of the present invention;

[0045] FIG. 3(a) and FIG. 3(b) are schematic diagrams of the multi-spherical rotation projection and inverse multi-spherical rotation projection of the present invention;

[0046] FIG. 3(c) and FIG. 3(d) are schematic diagrams of the multi-cube projection and inverse multi-cube projection of the present invention;

[0047] Figure 4 is the global branch framework diagram of the panoramic image saliency prediction method of the present invention;

[0048] Figure 5 is the local branch framework diagram of the panoramic image saliency prediction method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] The present invention will be further described in detail below in conjunction with embodiments, but the embodiments of the present invention are not limited thereto.

[0050] As Figure 1 shown, it is a general flow chart of the traditional convolutional neural network saliency prediction method. It can be seen from the figure that generally only the panoramic image input is included, and feature extraction is performed through the convolutional neural network, and then the prediction result is obtained. This traditional method has a certain effect, but its bottleneck lies in that it pays too much attention to local information and not enough to global information. One of the characteristics of the panoramic image is its wide range and large angle. Therefore, not all local features are helpful for the saliency prediction of the panoramic image, that is, there is a situation of local feature redundancy in the saliency prediction of the convolutional neural network model for the panoramic image, so there is room for further improvement in the prediction accuracy.

[0051] As Figure 2 shown is the flow chart of the present invention.

[0052] Referring to Figure 2As shown, in the panoramic image saliency prediction method based on multi-angle subspace region projection enhancement of stereo-spherical in the embodiments of the present invention, first, projection preprocessing is performed on the input panoramic image, the purpose of which is to augment the dataset with insufficient scale and solve the problem of discontinuous image edges; then the preprocessed images are respectively sent to the global-local sub-network for saliency prediction, and then the saliency prediction images are fused at a specific ratio, that is, a saliency map projection that simultaneously focuses on global and local information is obtained, and after back-projection, the corresponding fused saliency map is obtained.

[0053] Furthermore, the specific ratio refers to selecting the optimal one in the form of a linear combination. For example:

[0054] a + b = 1; ①: a = 0.65, b = 0.35; ②: a = 0.6, b = 0.4;.... And so on to obtain the combination map with the best effect.

[0055] Furthermore, as Figure 3(a) - Figure 3(d) shown, the projection preprocessing includes multi-cube projection and multi-sphere projection.

[0056] Specifically, the global sub-network corresponds to multi-sphere rotation projection and its inverse projection, as shown in the following formula:

[0057] Y i = f ERP-S (X i )

[0058]

[0059]

[0060] where x, y, and z are the three coordinate axes of the coordinate system, and f ERP-S is the mapping from equirectangular projection to spherical projection, is its inverse mapping, X i represents the i-th equirectangular projection map in the dataset, Y i is the spherical projection after its mapping, {Z i} is the projection sequence after the spherical projection Y i is rotated around the three coordinate axes, and X' i is its corresponding equirectangular projection sequence.

[0061] The specific multi-sphere projection is as follows:

[0062] Perform spherical projection on the panoramic image, then use the projection center of the sphere as the origin, and establish a three-dimensional Cartesian coordinate system perpendicular to each other in the horizontal, vertical, and longitudinal dimensions, and perform spherical rotation projection on the spherical panoramic image, and the three rotation dimensions respectively correspond to ω x , ω y , ωz Axis, where x, y, and z are angle sequences in [0, 360], and the lengths are K x , K y , K z , representing the rotation angles of the spherical map around the x, y, and z axes each time a rotation operation is performed.

[0063] During specific operations, according to ω x , ω y , ω z 's configuration, the corresponding spherical rotation operations are performed in sequence to obtain K x ·K y ·K z rotated spherical maps

[0064] The rotated spherical map sequence is inversely mapped back to the original equirectangular projection format in order, and then input into the algorithm model in sequence for prediction to obtain a sequence of saliency maps corresponding to the rotation angles.

[0065] The rotated saliency map sequence is spherically projected and then rotated; during specific operations, the rotation configuration that is exactly the same as the angles in the rotation but in the opposite direction is fully executed, with the aim of fully inverse-transforming back to the original panoramic view to achieve the effect of complete content alignment;

[0066] Finally, the sequence is summed and averaged to obtain the final global fusion saliency map.

[0067] Specifically, the multi-cube projection corresponding to the local sub-network is as follows:

[0068] The panoramic view is projected onto the 6 surfaces of the cube, and each surface is a local area with a viewing angle of 90°.

[0069] A three-dimensional Cartesian coordinate system corresponding to the global sub-network is established, where x, y, and z are angle sequences in [0, 360], and the lengths are L x , L y , L z , and operations with the same configuration as the multi-spherical rotation mapping are performed to obtain L x ·L y ·L z cube map sequences with different rotation angles.

[0070] Each cube in the rotated cube map sequence is unfolded into 6 cube blocks to obtain a sequence of cube blocks corresponding to the angles, and then it is input into the local sub-network in sequence for saliency prediction to obtain a sequence of local saliency maps corresponding to the angles.

[0071] Different from the multi-spherical rotation projection, the multi-cube projection performs the same projection operation on the real annotation map, directly compares it with the local saliency map sequence of the corresponding angle, and obtains the final local fusion saliency map by summing and averaging, so that there is no need for the back-projection operation again.

[0072] Such as Figure 4 This is the flowchart of the global saliency prediction picture of the panoramic image saliency prediction method of the present invention, specifically as follows:

[0073] Through the multi-spherical rotation projection operation, the equirectangular panoramic image is spherically projected to generate a spherical mapping map. Then, with the center of the sphere as the origin, a three-dimensional Cartesian coordinate system is established in its space, and the spherical mapping map is rotated around the ω x , ω y , ω z axis, and finally inverse mapped to the equirectangular panoramic image under the corresponding angle transformation.

[0074] Coding stage: The algorithm consists of four distinct regions, specifically as Figure 4 shown:

[0075] The first region: contains an image partitioning module and 2 consecutive Swin Transformer Blocks, where the Swin Transformer Block includes an intra-window self-attention module and an inter-window self-attention module;

[0076] In the first module of the second, third, and fourth regions, the linear embedding module in the first region is replaced by a linear embedding and partition merging module respectively; and the number of Swin Transformer Blocks in the third region is 6;

[0077] Input the picture X i After passing through the image partitioning module, the entire three-channel panoramic image is equally divided into non-overlapping image blocks to increase the receptive field;

[0078] Input the projected picture sequence into the linear embedding module, encode it in sequence according to the block order, and call it a region;

[0079] Input the picture blocks into the intra-window self-attention module and the inter-window self-attention module in this region to obtain the attention feature information at the corresponding region scale.

[0080] Finally, through the partition merging operation, the windows of two adjacent rows and two columns are downsampled by a factor of 2 and merged into a larger window; then, through the corresponding number of intra-window self-attention modules and inter-window self-attention modules, the self-attention feature information at a higher scale is obtained.

[0081] Figure 5This is the process by which a local sub-network in an embodiment of the present invention obtains a local saliency prediction map, which is specifically as follows:

[0082] Through the multi-cube projection operation, each equirectangular projection map is spherically projected, and then the latter is projected onto the six faces of the cube to generate 6 straight-line mapping maps corresponding to a 90° viewing angle. The specific projection method is as follows:

[0083] Y i =f ERP-S (X i )

[0084]

[0085] X′ i =f S-ERP (Z i )

[0086]

[0087] Among them, x, y, and z are the three coordinate axes of the coordinate system, and f ERP-S is the mapping from equirectangular projection to spherical projection, is its inverse mapping, X i represents the i-th equirectangular projection map in the dataset, Y i is the spherical projection after its mapping, {Z i} is the projection sequence after the spherical projection Y i is rotated around the three coordinate axes, and X′ i is its corresponding equirectangular projection sequence C i,j represents each face after the multi-cube projection expansion of each map, and the finally output image is also represented in this form;

[0088] The input image is equally divided into image blocks of two sizes through the block mapping module, and used as the source data for two branches, and input into the local sub-network;

[0089] The image blocks of different sizes are sequentially input into the linear projection encoding module, and a relative position Token is added after each feature encoding segment to better integrate spatial features;

[0090] The two independent branches of the local sub-network are used to process the small image blocks and the large image blocks respectively, and then these Tokens are fused multiple times with attention to complement each other, so as to correctly extract the corresponding local information.

[0091] To reduce the computational load, a simple and effective Token fusion module is introduced. This module uses a single token of each branch as a query to exchange information with other branches, and the computational time and complexity required for cross-attention are linear.

[0092] Table 1 shows the performance evaluation indicators of the embodiments of the present invention. Table 1

[0093]

[0094] The above embodiments are preferred embodiments of the present invention. However, the embodiments of the present invention are not limited by the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A panoramic image saliency prediction method, characterized in that, Including: Using the natural scene panorama as the data set, and performing projection preprocessing on the panoramic images in the data set. The projection preprocessing includes multi-cube projection and multi-spherical rotation projection; Respectively taking the projection panoramas generated by multi-cube projection and multi-spherical projection as the source data and inputting them into the global and local sub-networks for saliency prediction training to obtain local saliency prediction pictures and global saliency prediction pictures. Specifically, the local saliency prediction pictures are generated through the encoders and decoders of the global and local sub-networks; Fusing the global saliency prediction picture and the local saliency prediction picture according to a ratio to obtain an equilibrium saliency projection that simultaneously focuses on global and local information, and performing inverse projection in the same direction and angle on the equilibrium saliency projection to obtain a fused saliency prediction map; Inputting the saliency prediction obtained by the global sub-network and the saliency prediction of the local sub-network into the decoder with convolution-upsampling operations to output the predicted saliency map, and then obtaining the saliency map of the original image through the corresponding back-projection operations respectively; The multi-spherical rotation projection is specifically: Perform spherical projection on the panoramic image, and then, with the center of the projection sphere as the origin, establish a three-dimensional Cartesian coordinate system that is pairwise perpendicular in the horizontal, vertical, and longitudinal dimensions, and perform spherical rotation projection on the spherical panoramic image. The three rotation dimensions respectively correspond to ω x , ω y , ω z axes, where x, y, and z are angle sequences in the range of [0, 360], and their lengths are respectively K x , K y , K z , representing the rotation angles of the spherical image around the x, y, and z axes each time the rotation operation is performed; According to ω x , ω y , ω z 's configuration, sequentially perform the corresponding spherical rotation operations to obtain K x ·K y ·K z rotated spherical diagrams, and finally inverse-map them into equirectangular panoramic images under the corresponding angular transformations; The prediction formula of the encoder is; where, when i = 1 or 2 or 4, n i = 2; when i = 3, n i = 6; and represents a linear encoding operation, WSA is a normal window self-attention module, and SW is a combined sliding window self-attention module bound to WSA in sequence; The back-projection, the formula is: Y i = f ERP-S (X i ) where x, y, and z are the three coordinate axes of the coordinate system, and f ERP-S is the mapping from equirectangular projection to spherical projection, is its inverse mapping, X i represents the i-th equirectangular projection image in the dataset, Y i is the spherical projection after its mapping, {Z i} is the projection sequence after the spherical projection Y i is rotated around the three coordinate axes, and X i ′ is its corresponding equirectangular projection sequence one by one.

2. The panoramic image saliency prediction method according to claim 1, wherein Generating local saliency prediction pictures through the encoders and decoders of the sub-networks, specifically: The encoder includes four stages in sequence: The first stage: passing the equirectangular projection sequence generated after projection through the partition mapping module to equally divide the entire three-channel panorama into non-overlapping picture blocks; The second stage: passing the non-overlapping picture blocks output in the first stage through the linear embedding module, the purpose of which is to change the dimension of the input vector to a pre-set value, that is, the value that the Transformer module can accept; and then importing it into two rounds of Swin Transformer Block modules, performing self-attention calculations based on windows and moving windows according to the moving scale within the module, and each window is called a partition, then the attention calculation result exported by the module is the attention feature information at the corresponding partition scale; The third stage: performing operations on the partition attention feature information obtained in the second stage through the partition merging module to perform 2-fold downsampling on the windows of adjacent two rows and two columns; and then passing through six rounds of Swin Transformer Block modules to fully learn the regional self-attention feature information at different scales obtained in different rounds; The fourth stage: performing the partition merging operation again on the regional self-attention feature information obtained in the previous stage, and then passing through two rounds of Swin Transformer Block to obtain the local fusion feature information at a higher scale.

3. The panoramic image saliency prediction method according to claim 1, characterized in that The multi-cube projection is to perform spherical projection on each complete panorama, and then map it to the upper, lower, left, right, front, and back six faces of a regular hexahedron to generate six straight projection maps corresponding to a 90° viewport angle.

4. The panoramic image saliency prediction method according to claim 3, wherein Performing saliency prediction on the panoramic picture after multi-cube projection as the source data of the local sub-network, including the following: Passing the input picture through a bifurcated branch to equally divide the panorama into two image blocks with different sizes as the source data of the two branches; Image patches of different sizes are sequentially input into the linear embedding module to obtain feature encodings, and a relative position token is added after each feature encoding segment to obtain two features incorporating spatial features. The two features incorporating spatial features are respectively subjected to multi-scale cross-attention encoding and then input into the decoder to obtain local feature information.

5. A device, characterized in that, It includes: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the panoramic image saliency prediction method according to any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to execute the panoramic image saliency prediction method according to any one of claims 1-4.

Images