A salient object detection method in panoramic images based on multi-projection representation

Abstract

The present invention relates to a method for detecting salient objects in panoramic images based on multi-projection representation. An end-to-end detection network with an encoder-decoder structure is constructed, using an equirectangular projection image and four corresponding cube-expanded images as inputs to the detection network. In the encoder stage, the equirectangular projection branch and the cube-expanded branch extract features using a fifty-layer deep residual network (ResNet-50) with shared parameters. In the decoder stage, a dynamic weighted fusion module adaptively fuses the equirectangular projection features and the four cube-expanded features, while a filtering and refinement module combines the encoding and decoding features to obtain a final saliency map. In this invention, the detection network combines two panoramic image representation methods, equirectangular projection and cube-expanded, using the equirectangular projection image and the four corresponding cube-expanded images as inputs. The cube-expanded images provide supplementary information to the equirectangular projection images, ensuring target integrity.

Description

Technical Field

[0001] The present invention relates to the technical field of salient object detection, and in particular to a method for detecting salient objects in panoramic images based on multi-projection representation. Background Art

[0002] With the advancement of hardware technology and the growth of virtual reality applications, the input for salient object detection is no longer limited to standard two-dimensional images. Panoramic images have emerged as a new type of salient object detection task. Compared to standard two-dimensional images, panoramic images have a 360-degree field of view, capturing information from all directions and containing more scene information. However, since panoramic images typically have a much higher resolution than standard two-dimensional images, they require higher transmission bandwidth and storage costs.

[0003] Salient object detection in panoramic images involves detecting the most interesting objects or regions within panoramic image data. By selecting regions of interest, limited computing resources can be allocated to more critical information, thereby reducing image redundancy and increasing transmission and storage efficiency. Therefore, salient object detection in panoramic images plays a key role in applications such as panoramic image encoding, editing, stitching, and quality assessment.

[0004] In recent years, the technology of salient object detection for ordinary two-dimensional images has become relatively mature, and many excellent models (salient object detection models) have emerged. For example, Chen et al. [1] proposed a progressive aggregation network that takes into account global information and fully integrates features at different levels. Although traditional salient object detection methods perform well on two-dimensional images, due to the characteristics of panoramic images, it is difficult to obtain satisfactory results by directly transplanting traditional salient object detection models to panoramic images. Therefore, it is necessary to design a special model to solve the challenges in the salient object detection task for panoramic images. Specifically, salient object detection for panoramic images faces the following two major challenges.

[0005] First, while real panoramic images are spherical, they are typically projected onto a flat surface for storage, transmission, and processing. Common projection representations of panoramic images on a flat surface include equirectangular and cubic projections. However, projection from a spherical surface to a flat surface introduces a certain degree of distortion, and this distortion is unavoidable in any projection method. This image distortion complicates salient object detection and can lead to a range of issues, including incorrect and incomplete object detection.

[0006] Secondly, since panoramic images cover all-directional viewing angles, the scenes are usually more complex, and there may be a large number of redundant objects in the background, which significantly interferes with the positioning accuracy of salient targets.

[0007] In recent years, with the rapid development and application of virtual reality technology, the research on salient object detection in panoramic images has gradually attracted people's attention. Currently, salient object detection algorithms tailored to the characteristics of panoramic images have emerged.

[0008] To solve the distortion problem, Monroy et al. [2] mapped the panoramic image into six blocks, obtained the saliency map of the blocks through a convolutional neural network, and then combined these blocks together using a special post-processing technique. Assens et al. [3] introduced a time-aware representation of saliency information into the model. Li et al. [4] built a distortion adaptive module in the network, cutting the image into many small blocks and assigning different convolution kernels to them. The network also added a multi-scale context fusion module to perceive and identify targets in the panoramic image. Huang et al. [5] used the equirectangular projection and cubic projection of the panoramic image together and proposed a feature adaptive network. The equirectangular projection image and the six cubic projection images were used as the input of the network at the same time, aiming to take advantage of the respective advantages of the two projections and combine the globality of the equirectangular projection and the low distortion of the cubic projection.

[0009] A drawback of existing technologies is that, when using the cube projection image as a supplement to the equirectangular projection image, the discontinuity of the six independent faces in the cube projection image is ignored. Using the six independent faces in the cube projection image alone may cause the integrity of the target in the panoramic image to be lost, resulting in incomplete detection and ineffective background suppression.

[0010] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the Invention

[0011] In response to the defects in the existing technology, the purpose of the present invention is to provide a panoramic image salient target detection method based on multi-projection representation. It aims to combine a set of cube unfolded images with equirectangular projection images to design a new panoramic image salient target detection network based on multi-projection representation. The detection network integrates multiple panoramic image representation methods to obtain better detection results.

[0012] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0013] A method for detecting salient objects in panoramic images based on multi-projection representation is proposed. The method is characterized by constructing an end-to-end detection network with an encoder-decoder structure, and taking an equirectangular projection image and its four corresponding cube-shaped unfolded images as inputs to the detection network.

[0014] The equirectangular projection image and the four cube expansion images correspond to two branches of the encoder, namely the equirectangular projection branch and the cube expansion branch;

[0015] In the encoder stage, the equirectangular projection branch and the cube expansion branch extract features through a fifty-layer deep residual network ResNet-50 with shared parameters;

[0016] In the decoder stage, the dynamic weighted fusion module adaptively fuses equirectangular projection features and four types of cube expansion features, and the filtering and refinement module combines the encoding and decoding features to obtain the final saliency map.

[0017] Based on the above technical solution, the dynamic weighted fusion module includes a gated inter-representation fusion unit and a weighted intra-representation fusion unit, which adaptively fuse the different projected representation features from the inter-representation and intra-representation perspectives respectively. Finally, the inter-representation fusion and intra-representation fusion are integrated into the dynamic weighted fusion module to adaptively fuse the encoder features of the two projected representations.

[0018] The filtering and refinement module suppresses redundant information between features and features through filtering strategies, while retaining detailed information related to salient targets through refinement strategies.

[0019] Based on the above technical solution, at the encoder stage, the two branches use a fifty-layer deep residual network ResNet-50 with shared parameters as the encoder. The encoder extracts features from the equirectangular projection image and the cube expansion image to learn the multi-projection representation features of the panorama. The feature extraction specifically includes:

[0020] After the two lower-level feature extractions, the cube-to-equirectangular projection conversion module is used to convert the cube unfolding features into equirectangular projection features before continuing with higher-level feature extraction.

[0021] Based on the above technical solution, at the decoder stage, the encoder features of the multi-level and multi-projection representation learned by the detection network are fed into the decoder to gradually restore the spatial resolution and generate a saliency map. Specifically,

[0022] The dynamic weighted fusion module in the network adaptively fuses equirectangular projection features and cube expansion features from both inter-representation and intra-representation perspectives to select more effective and comprehensive features.

[0023] The filtering and refinement module enhances high-level semantic features by filtering redundant information in low-level features of different projection representation features;

[0024] The detection network obtains the final saliency map under the joint supervision of the main loss function and all side output loss functions.

[0025] Based on the above technical solution, the four cube expanded images adopt a cube expansion method in which four sides are connected horizontally and three sides are connected vertically. This expansion method can extend the continuity between the cube faces in the horizontal and vertical directions, thereby maximizing the integrity of the target in the image;

[0026] Specifically include:

[0027] With the front, back, left and right faces of the cube projection image as the center, 4-3 type expansion is performed respectively to obtain four cube expansion images, and these four cube expansion images are used as the input of the cube expansion branch.

[0028] Based on the above technical solution, after the four cube expanded images and the equirectangular projection image are fed into the encoder, the cube expanded image is divided into horizontal sub-images and vertical sub-images, and low-level feature extraction is performed on each sub-image.

[0029] Through the cube-to-equirectangular projection conversion module, the cube unfolding features are converted into an equirectangular projection format, and then high-level semantic features are extracted to compensate for the slight semantic discontinuity when the cube faces are connected.

[0030] Based on the above technical solution, the dynamic weighted fusion module adaptively fuses the equirectangular projection features and the four cube expansion features, specifically including:

[0031] A gated inter-representation fusion unit is used to fuse the two projection representation features. The representations of each cube expansion feature and the equirectangular projection feature are fused, and the valuable parts of the two different projection representation features are adaptively screened to obtain the fused features guided by the cube expansion.

[0032] The gated representation fusion unit first learns an importance map P i , used to evaluate the contribution of the cube expansion feature in the fusion; then the equirectangular projection feature F E Expand features with each cube The concatenation is performed in the channel dimension, and then the SE module is used to calculate the channel attention, and then the importance map P is obtained through a bottleneck convolution layer and a sigmoid activation function. i ;

[0033] In the importance graph P iAfter that, the fusion features guided by the cube expansion can be calculated by the following formula:

[0034]

[0035] Where, represents Hadamard multiplication, F i represents the fusion feature guided by the cube expansion, which is the final output of the gated representation fusion unit;

[0036] Through the above-mentioned gated inter-representation fusion unit, the four cube expansion features are adaptively fused with the equirectangular projection features to obtain four fused features guided by cube expansion;

[0037] A weighted intra-representation fusion unit is used to achieve the intra-representation fusion of the four cube-expansion-guided fusion features;

[0038] The weighted representation fusion unit expands the original four cube features Splicing, through the SE module, obtains a weight vector α∈R 4C×1×1 ;

[0039] The weight vector is further divided into four sub-vectors α i ∈R C×1×1 , after normalization, we get the adaptive weight ω i ;

[0040] After obtaining the corresponding adaptive weights, the following representation fusion is performed:

[0041] The four fusion features F obtained by the gated representation fusion unit guided by cube expansion are i ∈R C×H×W , fused in an adaptively weighted manner, and the result is combined with the equirectangular projection feature F E Add together to get the final fusion feature F of this module f :

[0042]

[0043] Where, ω i is the corresponding weight obtained by the fusion unit in the weighted representation, represents element-wise addition, represents element-wise multiplication across the entire feature plane.

[0044] Based on the above technical solution, the filtering and refinement module implements feature screening and fusion in a step-by-step manner, which helps to suppress two types of redundant information while retaining detailed information related to salient objects. Specifically, it includes:

[0045] Taking the kth filtering and refinement module as an example, k ≥ 2, the filtering and refinement module has two feature inputs: the first is the decoder feature from the previous decoding layer The other is the encoder features of five different projected representation features from the corresponding encoding layer The last filtering and refinement module does not have the decoder features from the previous decoding layer, which are replaced here with the output of the dynamic weighted fusion module;

[0046] The filtering and refinement module adopts the following filtering strategies:

[0047] The decoder features are filtered using a modulation method. First, a 3×3 convolution is performed to reduce the number of channels. Then, two convolutional layers are used to obtain the modulation parameters a and b by performing multiplication and addition operations respectively. After this operation, the decoder features are enhanced.

[0048] The enhanced decoder feature representation is as follows:

[0049]

[0050] In the formula, δ represents the linear rectification function (RELU), α and b represent the modulation parameters of multiplication and addition respectively, represents the k+1th layer decoder feature, represents the enhanced k+1th layer decoder features;

[0051] Filtering of encoder features, in order to make the five encoder features To better adapt to the salient object detection task, the corresponding upsampled filter mask FM is obtained for the encoder feature according to the filtered decoder feature. j , the filter mask can encode the saliency attribute; then, the encoder feature is combined with the corresponding upsampled filter mask FM j Multiplication suppresses the redundancy of encoder features, thereby obtaining filtered encoder features

[0052]

[0053] Where UP represents the upsampling operation, conv 3×3 Represents a 3×3 convolution operation;

[0054] Finally, the filtered total encoder feature is obtained by addition operation

[0055]

[0056] Among them, δ represents the RELU activation function.

[0057] On the basis of the above technical solution, the filtering and refinement module further includes a refinement strategy, specifically including:

[0058] In order to emphasize the important details in the decoder features, the encoder features after filtering are Learn a detail mask DM for the decoder feature and get the refined decoder feature by the following formula

[0059]

[0060] After the filtering and refinement strategies are applied, the filtered encoder features are fused with the refined decoder features:

[0061]

[0062] Where, Represents the output of the k-th layer filtering and refinement module, which will be embedded in the filtering and refinement module of the next layer.

[0063] The present invention provides a method for detecting salient objects in panoramic images based on multi-projection representation, which has the following beneficial effects:

[0064] 1. This paper explores a method for representing panoramic images by expanding a cube, and proposes a panoramic image salient object detection network based on multi-projection representation. The detection network realizes the complementarity of multi-projection representation by designing multi-projection representation feature interaction modules in the middleware and decoding end respectively.

[0065] 2. The detection network combines the two panoramic image representation methods of equirectangular projection and cube expansion, and takes the equirectangular projection image and the corresponding four cube expansion images as common input. Among them, the cube expansion image provides supplementary information for the equirectangular projection image to ensure the integrity of the target.

[0066] 3. The present invention uses both equirectangular projection images and expanded cube images as input, combining two representation methods for panoramic images. These two representation methods have their own advantages and are complementary to a certain extent.

[0067] While equirectangular projection features provide better global information perception, they also inevitably suffer from significant distortion. On the other hand, cube-unfolded features exhibit less distortion and, by connecting the cube faces, ensure the integrity and continuity of objects in the panoramic image. Therefore, fusing these two representations before decoding yields a more comprehensive and effective panoramic image feature, leading to better decoding prediction results.

[0068] 4. The present invention introduces four cube unfolded images as supplements to the corresponding equirectangular projection images.

[0069] 5. The present invention designs a dynamic weighted fusion module, which adaptively fuses multi-projection representation features from two perspectives: inter-representation and intra-representation.

[0070] 6. The present invention designs a filtering and refinement module, which updates the encoder and decoder features through a filtering strategy, suppresses the redundant information of the features themselves, and retains the detailed information related to the salient target through a refinement strategy.

[0071] This paper explores a method for representing panoramic images using a cube-shaped expansion and proposes a network for detecting salient objects in panoramic images based on multi-projection representation. The network incorporates two modules: a weighted fusion module that adaptively fuses multi-projection representation features, and a filtering and refinement module that combines encoder and decoder features to suppress redundant information while retaining detailed information related to salient objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0073] Figure 1 A block diagram of the principle of a method for detecting salient objects in panoramic images based on multi-projection representation described in the present invention.

[0074] Figure 2 A visualization example of a panoramic image salient object detection method based on multi-projection representation described in the present invention. DETAILED DESCRIPTION

[0075] The present invention will be described in further detail below with reference to the accompanying drawings. The detailed description, which is provided for illustrative purposes only and includes various details to aid understanding of the embodiments of the present invention, should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted from the following description.

[0076] like Figure 1 As shown in the figure, the present invention provides a method for detecting salient objects in panoramic images based on multi-projection representation. An end-to-end detection network with an encoder-decoder structure is constructed, and the equirectangular projection image and the corresponding four cube unfolded images are used as the input of the detection network.

[0077] The equirectangular projection image and the four cube expansion images correspond to two branches of the encoder, namely the equirectangular projection branch and the cube expansion branch;

[0078] In the encoder stage, the equirectangular projection branch and the cube expansion branch extract features through a fifty-layer deep residual network ResNet-50 with shared parameters;

[0079] In the decoder stage, the dynamic weighted fusion module adaptively fuses equirectangular projection features and four types of cube expansion features, and the filtering and refinement module combines the encoding and decoding features to obtain the final saliency map.

[0080] Based on the above technical solution, the dynamic weighted fusion module includes a gated inter-representation fusion unit and a weighted intra-representation fusion unit, which adaptively fuse the features of different projection representations from the inter-representation and intra-representation perspectives respectively. The naming of inter-representation fusion and intra-representation fusion depends on whether the fused features come from the same projection method. Ultimately, inter-representation fusion and intra-representation fusion are integrated into the dynamic weighted fusion module to adaptively fuse the encoder features of the two projection representations.

[0081] The filtering and refinement module suppresses redundant information between features and features through filtering strategies, while retaining detailed information related to salient targets through refinement strategies.

[0082] Based on the above technical solution, at the encoder stage, the two branches use a fifty-layer deep residual network ResNet-50 with shared parameters as the encoder. The encoder extracts features from the equirectangular projection image and the cube expansion image to learn the multi-projection representation features of the panorama. The feature extraction specifically includes:

[0083] After the two lower-level feature extractions, the cube-to-equirectangular projection conversion module is used to convert the cube unfolding features into equirectangular projection features before continuing with higher-level feature extraction.

[0084] Based on the above technical solution, at the decoder stage, the encoder features of the multi-level and multi-projection representation learned by the detection network are fed into the decoder to gradually restore the spatial resolution and generate a saliency map. Specifically,

[0085] The dynamic weighted fusion module in the network adaptively fuses equirectangular projection features and cube expansion features from both inter-representation and intra-representation perspectives to select more effective and comprehensive features.

[0086] The filtering and refinement module enhances high-level semantic features by filtering redundant information in low-level features of different projection representation features;

[0087] The detection network obtains the final saliency map under the joint supervision of the main loss function and all side output loss functions.

[0088] Based on the above technical solution, the four cube expanded images adopt a cube expansion method in which four sides are connected horizontally and three sides are connected vertically. This expansion method can extend the continuity between the cube faces in the horizontal and vertical directions, thereby maximizing the integrity of the target in the image; the present invention refers to this as 4-3 expansion.

[0089] Specifically include:

[0090] With the front, back, left and right faces of the cube projection image as the center, 4-3 type expansion is performed respectively, that is, four faces are connected horizontally and three faces are connected vertically to obtain four cube expansion images, and these four cube expansion images are used as the input of the cube expansion branch.

[0091] Considering that using the six cube faces of the prior art cube projection image in isolation may cause the integrity of the target in the panoramic image to be lost, resulting in incomplete detection and ineffective background suppression, which is obviously not conducive to the detection of salient targets. To this end, the present invention connects the six cube faces of the cube projection image of the panoramic image in the following manner, thereby maximizing the continuity of the panoramic image and increasing the possibility of salient target integrity:

[0092] Carrying out 4-3 type expansion with the four horizontal cube faces as the center can increase the possibility of completing the targets in both horizontal and vertical directions.

[0093] Since the real scene in a panoramic image can only be fully displayed on a spherical surface, once converted from a spherical domain to a flat domain, boundaries are inevitably introduced, more or less destroying the integrity of the object. Panoramic images often contain large objects in the vertical direction, such as standing people and buildings. After converting the panoramic image into a cubic projection image, these objects are often separated by the boundaries between the cube faces, which is very unfavorable for the detection of salient objects. In the vertical direction, the expansion centered on the front, back, left, and right cube faces establishes four different vertical connections, thus achieving continuous expansion in the four vertical directions, which greatly increases the possibility of object integrity in the vertical direction.

[0094] Similarly, horizontally, the four faces of the cube remain connected. In each expansion centered on the four faces, each face undergoes a positional shift from the center to the edge. This means that if two faces are separated by a left or right boundary in one expansion, they remain connected in all other expansions. This allows different expansions to complement each other, enhancing the integrity of the target in the horizontal direction.

[0095] Based on the above technical solution, after the four cube expanded images and the equirectangular projection image are fed into the encoder, the cube expanded image is divided into horizontal sub-images and vertical sub-images, and low-level feature extraction is performed on each sub-image.

[0096] Through the cube-to-equirectangular projection conversion module, the cube unfolding features are converted into an equirectangular projection format, and then high-level semantic features are extracted to compensate for the slight semantic discontinuity when the cube faces are connected.

[0097] In this paper, each expanded cube image is further divided into horizontal and vertical sub-images. This is because if the 4-3 expanded cube image is directly used as input, a large area of ​​zero padding will have to be introduced, which not only wastes computing resources but also is not conducive to the localization of salient objects.

[0098] Based on the above technical solution, the dynamic weighted fusion module adaptively fuses the equirectangular projection features and the four cube expansion features, specifically including:

[0099] A gated inter-representation fusion unit is used to fuse the two projection representation features. The representations of each cube expansion feature and the equirectangular projection feature are fused, and the valuable parts of the two different projection representation features are adaptively screened to obtain the fused features guided by the cube expansion.

[0100] The equirectangular projection feature contains more global information and can maintain the integrity of the scene, but it also has serious distortion. The cube expansion feature has less distortion and more accurate detail information, which can provide structural guidance as a supplement; therefore, the gated representation fusion unit aims to combine the useful information of the two projection representation features and transform each cube expansion feature into a With the equirectangular projection feature F E ∈R C×H×W Adaptively fuse and filter out the valuable parts of the two different projection representation features;

[0101] Specifically, the gated representation fusion unit first learns an importance map P i , used to evaluate the contribution of the cube expansion feature in the fusion; then the equirectangular projection feature FE Expand features with each cube The concatenation is performed in the channel dimension, and then the SE module is used to calculate the channel attention, and then the importance map P is obtained through a bottleneck convolution layer and a sigmoid activation function. i The above process can be expressed as:

[0102]

[0103] Where, P i represents the contribution of the cube unfolding feature, F E and Represents equirectangular projection features and cube expansion features, respectively. [·,·] represents the concatenation of channel dimensions. σ represents the sigmoid activation function. conv represents the bottleneck convolution layer structure. SE represents the Squeeze-and-Excitation module.

[0104] In the importance graph P i After that, the fusion features guided by the cube expansion can be calculated by the following formula:

[0105]

[0106] Where, represents Hadamard multiplication, F i represents the fusion feature guided by the cube expansion, which is the final output of the gated representation fusion unit;

[0107] Through the above-mentioned gated inter-representation fusion unit, the four cube expansion features are adaptively fused with the equirectangular projection features to obtain four fused features guided by cube expansion;

[0108] A weighted intra-representation fusion unit is used to achieve the intra-representation fusion of the four cube-expansion-guided fusion features;

[0109] For each panoramic image, the importance of each cube expansion feature is different due to the different target distribution. The reason is that different cube expansion images have specific connection methods, and since the target distribution positions in each panoramic image are different, the importance of different cube expansion methods to the current panoramic image is different.

[0110] The weighted representation fusion unit learns dynamic importance weights corresponding to the fusion features guided by the four cube expansions, thereby selecting effective and continuous information related to the salient target;

[0111] Specifically, the weighted representation fusion unit expands the original four cube features Splicing, through the SE module, obtains a weight vector α∈R 4C×1×1 ;

[0112] The weight vector is further divided into four sub-vectors α i ∈R C×1×1 , after normalization, we get the adaptive weight ω i ;

[0113] The above process can be expressed as the following formula:

[0114]

[0115]

[0116] Where SE represents the Squeeze-and-Excitation module, Split(·) represents the operation of splitting a vector into several sub-vectors, [·,·] represents the concatenation of channel dimensions, and Sum(·) represents the sum of all elements in the vector.

[0117] After obtaining the corresponding adaptive weights, the following representation fusion is performed:

[0118] The four fusion features F obtained by the gated representation fusion unit guided by cube expansion are i ∈R C×H×W , fused in an adaptively weighted manner, and the result is combined with the equirectangular projection feature F E Add together to get the final fusion feature F of this module f :

[0119]

[0120] Where, ω i is the corresponding weight obtained by the fusion unit in the weighted representation, represents element-wise addition, represents element-wise multiplication across the entire feature plane.

[0121] The combination of encoder features and decoder features can provide more comprehensive saliency information, but it also contains a lot of redundant information, which undoubtedly affects the expressiveness of the features. Based on the source of the redundant information, they can be divided into two categories:

[0122] One type is the redundancy of the features themselves, which mainly refers to the redundant information of the encoder and decoder features in space and channels;

[0123] The other is the redundancy between features, which refers to the relative redundancy between encoder and decoder features during the feature fusion process; for this purpose, the present invention designs a filtering and refinement module.

[0124] Based on the above technical solution, the filtering and refinement module implements feature screening and fusion in a step-by-step manner, which helps to suppress two types of redundant information while retaining detailed information related to salient objects. Specifically, it includes:

[0125] Taking the kth filtering and refinement module as an example, k ≥ 2, the filtering and refinement module has two feature inputs: the first is the decoder feature from the previous decoding layer The other is the encoder features of five different projected representation features from the corresponding encoding layer The last filtering and refinement module does not have the decoder features from the previous decoding layer, which are replaced here with the output of the dynamic weighted fusion module;

[0126] For example: k = {2, 3, 4}, then when k = 4, there is no decoder feature from the previous decoding layer, where it is replaced by the output of the dynamic weighted fusion module;

[0127] The filtering and refinement module adopts the following filtering strategies:

[0128] The decoder features are filtered using a modulation method. First, a 3×3 convolution is performed to reduce the number of channels. Then, two convolutional layers are used to obtain the modulation parameters a and b by performing multiplication and addition operations respectively. After this operation, the decoder features are enhanced.

[0129] The enhanced decoder feature representation is as follows:

[0130]

[0131] In the formula, δ represents the linear rectification function (RELU), α and b represent the modulation parameters of multiplication and addition respectively, represents the k+1th layer decoder feature, represents the enhanced k+1th layer decoder features;

[0132] Filtering of encoder features, in order to make the five encoder features To better adapt to the salient object detection task, the corresponding upsampled filter mask FM is obtained for the encoder feature according to the filtered decoder feature. j , the filter mask can encode the saliency attribute; then, the encoder feature is combined with the corresponding upsampled filter mask FM j Multiplication suppresses the redundancy of encoder features, thereby obtaining filtered encoder features

[0133]

[0134] Where UP represents the upsampling operation, conv 3×3 Represents a 3×3 convolution operation;

[0135] Finally, the filtered total encoder feature is obtained by addition operation

[0136]

[0137] Among them, δ represents the RELU activation function.

[0138] The present invention employs the filtering strategy described above to update encoder and decoder features, thereby filtering out redundant information. The present invention filters decoder features using a modulation method. Furthermore, to better adapt encoder features for salient object detection, the present invention derives a filter mask for each encoder feature based on the filtered decoder features, which encodes saliency attributes. The encoder features are then multiplied by the corresponding upsampled filter mask to suppress redundant information.

[0139] On the basis of the above technical solution, the filtering and refinement module further includes a refinement strategy, specifically including:

[0140] In order to emphasize the important details in the decoder features, the encoder features after filtering are Learn a detail mask DM for the decoder feature and get the refined decoder feature by the following formula

[0141]

[0142] After the filtering and refinement strategies are applied, the filtered encoder features are fused with the refined decoder features:

[0143]

[0144] Where, Represents the output of the k-th layer filtering and refinement module, which will be embedded in the filtering and refinement module of the next layer.

[0145] The filtering and refinement module of the present invention suppresses redundant information while maintaining detailed information of significant targets through the above two strategies (filtering strategy and refinement strategy).

[0146] like Figure 2As shown in the figure, four visualization examples using the method of the present invention are given. The first row shows the four expanded cube images of Example 1, while the second to fifth rows show the visualization results of the four examples in the form of equirectangular projection images. Among them, the first column is the panoramic image, the second column is the ground truth map of salient target detection in the panoramic image, and the third column is the saliency map predicted by the present invention. As can be seen from the results, the method of the present invention achieves better visual effects in terms of target integrity and acquisition of detailed information. In conditions such as complex scenes, multiple targets, and small targets, the present invention can obtain accurate prediction results.

[0147] References:

[0148] [1] Z.Chen, Q.Xu, R.Cong, and Q.Huang, "Global context-aware progressive aggregation network for salient object detection," in AAAI, 2020, pp.10599-10606.

[0149] [2] R.Monroy, S.Lutz, T.Chalasani, and A.Smolic, "SalNet360: Saliency maps for omni-directional images with CNN," Signal Processing: Image Communication, vol.69, pp.26-34, 2018.

[0150] [3] M.AssensReina,

[0151] [4] J.Li, J.Su, C.Xia, and Y.Tian, ​​"Distortion-adaptive salient object detection in 360° omnidirectional images," IEEE J.Sel.Topics Signal Process., vol.14, no.1, pp.38-48, 2020.

[0152] [5] M.Huang, Z.Liu, G.Li,

[0153] The contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0154] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by those skilled in the art based on the contents disclosed in the present invention should be included in the protection scope recorded in the claims.

Claims

1. A method for detecting salient objects in panoramic images based on multi-projection representation, characterized in that: Construct an end-to-end detection network with an encoder-decoder structure, and use the equirectangular projection image and the corresponding four cube unfolded images as the input of the detection network; The equirectangular projection image and the four cube expansion images correspond to two branches of the encoder, namely the equirectangular projection branch and the cube expansion branch; In the encoder stage, the equirectangular projection branch and the cube expansion branch extract features through a fifty-layer deep residual network ResNet-50 with shared parameters; In the decoder stage, the dynamic weighted fusion module adaptively fuses the equirectangular projection features and the four cube expansion features, and the filtering and refinement module combines the encoding and decoding features to obtain the final saliency map; The dynamic weighted fusion module adaptively fuses the equirectangular projection features and the four cube expansion features, specifically including: A gated inter-representation fusion unit is used to fuse the two projection representation features. The representations of each cube expansion feature and the equirectangular projection feature are fused, and the valuable parts of the two different projection representation features are adaptively screened to obtain the fused features guided by the cube expansion. The gated representation fusion unit first learns an importance map P i , used to evaluate the contribution of the cube expansion feature in the fusion; then the equirectangular projection feature F E Expand features with each cube The concatenation is performed in the channel dimension, and then the SE module is used to calculate the channel attention, and then the importance map P is obtained through a bottleneck convolution layer and a sigmoid activation function. i ; In the importance graph P i After that, the fusion features guided by the cube expansion can be calculated by the following formula: Where, represents Hadamard multiplication, F i represents the fusion feature guided by the cube expansion, which is the final output of the gated representation fusion unit; Through the above-mentioned gated inter-representation fusion unit, the four cube expansion features are adaptively fused with the equirectangular projection features to obtain four fused features guided by cube expansion; A weighted intra-representation fusion unit is used to achieve the intra-representation fusion of the four cube-expansion-guided fusion features; The weighted representation fusion unit expands the original four cube features Splicing, through the SE module, obtains a weight vector α∈R 4C×1×1 ; The weight vector is further divided into four sub-vectors α i ∈R C×1×1 , after normalization, we get the adaptive weight ω i ; After obtaining the corresponding adaptive weights, the following representation fusion is performed: The four fusion features F obtained by the gated representation fusion unit guided by cube expansion are i ∈R C×H×W , fused in an adaptively weighted manner, and the result is combined with the equirectangular projection feature F E Add together to get the final fusion feature F of this module f : Where, ω i is the corresponding weight obtained by the fusion unit in the weighted representation, represents element-wise addition, represents element-wise multiplication on the entire feature plane; The filtering and refinement module implements feature filtering and fusion in a step-by-step manner, which helps to suppress two types of redundant information while retaining detailed information related to salient targets. Specifically, it includes: Taking the kth filtering and refinement module as an example, k ≥ 2, the filtering and refinement module has two feature inputs: the first is the decoder feature from the previous decoding layer The other is the encoder features of five different projected representation features from the corresponding encoding layer The last filtering and refinement module does not have the decoder features from the previous decoding layer, which are replaced here with the output of the dynamic weighted fusion module; The filtering and refinement module adopts the following filtering strategies: The decoder features are filtered using a modulation method. First, a 3×3 convolution is performed to reduce the number of channels. Then, two convolutional layers are used to obtain the modulation parameters a and b by performing multiplication and addition operations respectively. After this operation, the decoder features are enhanced. The enhanced decoder feature representation is as follows: In the formula, δ represents the linear rectification function (RELU), α and b represent the modulation parameters of multiplication and addition respectively, represents the k+1th layer decoder feature, represents the enhanced k+1th layer decoder features; Filtering of encoder features, in order to make the five encoder features To better adapt to the salient object detection task, the corresponding upsampled filter mask FM is obtained for the encoder feature according to the filtered decoder feature. j , the filter mask can encode the saliency attribute; then, the encoder feature is combined with the corresponding upsampled filter mask FM j Multiplication suppresses the redundancy of encoder features, thereby obtaining filtered encoder features Where UP represents the upsampling operation, conv 3×3 Represents a 3×3 convolution operation; Finally, the filtered total encoder feature is obtained by addition operation Among them, δ represents the RELU activation function.

2. The method for detecting salient objects in panoramic images based on multi-projection representation according to claim 1, wherein: The dynamic weighted fusion module includes a gated inter-representation fusion unit and a weighted intra-representation fusion unit, which adaptively fuse different projection representation features from the perspectives of inter-representation and intra-representation respectively; Finally, inter-representation fusion and intra-representation fusion are integrated into a dynamic weighted fusion module to adaptively fuse the encoder features of the two projected representations; The filtering and refinement module suppresses redundant information between features and features through filtering strategies, while retaining detailed information related to salient targets through refinement strategies.

3. The method for detecting salient objects in panoramic images based on multi-projection representation according to claim 1, wherein: In the encoder stage, the two branches use a fifty-layer deep residual network ResNet-50 with shared parameters as the encoder. The encoder extracts features from the equirectangular projection image and the cube-shaped expanded image to learn the multi-projection representation features of the panorama. The feature extraction specifically includes: After the two lower-level feature extractions, the cube-to-equirectangular projection conversion module is used to convert the cube unfolding features into equirectangular projection features before continuing with higher-level feature extraction.

4. The method for detecting salient objects in panoramic images based on multi-projection representation according to claim 1, wherein: In the decoder stage, the encoder features of the multi-level and multi-projection representations learned by the detection network are fed into the decoder to gradually restore the spatial resolution and generate a saliency map, specifically including: The dynamic weighted fusion module in the network adaptively fuses equirectangular projection features and cube expansion features from both inter-representation and intra-representation perspectives to select more effective and comprehensive features. The filtering and refinement module enhances high-level semantic features by filtering redundant information in low-level features of different projection representation features; The detection network obtains the final saliency map under the joint supervision of the main loss function and all side output loss functions.

5. The method for detecting salient objects in panoramic images based on multi-projection representation according to claim 1, wherein: The four cube expanded images adopt a cube expansion method in which four sides are connected horizontally and three sides are connected vertically. This expansion method can extend the continuity between the cube faces in the horizontal and vertical directions, thereby maximizing the integrity of the target in the image; Specifically include: With the front, back, left and right faces of the cube projection image as the center, 4-3 expansion is performed respectively to obtain four cube expansion images, and these four cube expansion images are used as the input of the cube expansion branch.

6. The method for detecting salient objects in panoramic images based on multi-projection representation according to claim 1, wherein: After the four cube expanded images and the equirectangular projection images are fed into the encoder, the cube expanded images are divided into horizontal sub-images and vertical sub-images, and low-level feature extraction is performed on each sub-image. Through the cube-to-equirectangular projection conversion module, the cube unfolding features are converted into an equirectangular projection format, and then high-level semantic features are extracted to compensate for the slight semantic discontinuity when the cube faces are connected.

7. The method for detecting salient objects in panoramic images based on multi-projection representation according to claim 1, wherein: The filtering and refinement module also includes a refinement strategy, specifically including: In order to emphasize the important details in the decoder features, the encoder features after filtering are Learn a detail mask DM for the decoder feature and get the refined decoder feature by the following formula After the filtering and refinement strategies are applied, the filtered encoder features are fused with the refined decoder features: Where, Represents the output of the k-th layer filtering and refinement module, which will be embedded in the filtering and refinement module of the next layer.

Images