Panoramic image salient object detection method and device, equipment and storage medium

Abstract

The present application is suitable for the field of image processing, and provides a panoramic image salient object detection method, device and equipment and storage medium, the method comprising: when receiving a salient object detection request, obtaining a panoramic image to be detected, processing the panoramic image through a pre-established salient detection model to obtain a saliency map of the panoramic image, wherein the salient detection model comprises a double-branch structure network, a hybrid projection feature fusion module and a progressive prediction module, thereby improving the detection performance of the panoramic image salient object, effectively filtering the redundant information in the panoramic image and improving the saliency of the panoramic image saliency map.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a panoramic image salient object detection method, device, equipment and storage medium. BACKGROUND

[0002] With the rapid development of technology, the popularity of panoramic cameras and the development of AR / VR, 360° panoramic images have been widely used in many fields, such as real estate, tourist attractions, exhibitions, and autonomous driving. As the most common virtual reality resource in life, panoramic images contain rich surrounding scene information and can provide viewers with a wider field of view and a more realistic scene, allowing viewers to have an immersive experience. Compared with traditional 2D images, panoramic images usually have a large resolution. How to efficiently transmit and store these massive panoramic image data has become a major challenge for the development of panoramic images. Salient object detection simulates human vision and captures human visual attention to identify the most eye-catching targets in an image. It is an important initial step for some computer vision tasks, such as image segmentation, image compression, and visual tracking. Therefore, the salient object detection algorithm for panoramic images has important research significance, and researchers are increasingly interested in salient object detection on panoramic images.

[0003] Panoramic images are usually displayed in two formats: equirectangular projection (ERP) and cubic projection (CMP). ERP uniformly samples latitude and longitude onto a rectangular plane. However, the projection from a sphere to a two-dimensional plane will cause image content distortion. CMP divides panoramic images with complete scenes into six faces, but this projection method inevitably destroys the integrity of the image. Recently, several salient object detection methods have been proposed based on different projection formats. For example, Li et al. proposed a distortion-adaptive salient object detection method to handle the distortion caused by projecting a spherical surface onto a plane and adaptively correct ERP images. Huang et al. designed a feature-adaptive salient object detection network by utilizing the advantages of ERP and CMP images. Although existing panoramic image salient object detection models consider the distortion problem of ERP format panoramic images, they usually use CMP images to alleviate the distortion of panoramic images, but they do not fully utilize the complete global characteristics of ERP images and the specific local characteristics of CMP images. For the multi-layer feature maps proposed by the feature extraction network, existing panoramic image salient object detection methods fail to fully utilize the rich spatial information of low-level features and the rich semantic information of high-level features. Moreover, for the large field of view of panoramic images, existing methods do not consider multi-scale salient objects in image content. Therefore, existing panoramic image salient object detection methods do not fully utilize the rich semantic information of high-level features and fail to consider the characteristics of salient objects of different sizes in panoramic images, which can lead to inaccurate salient object detection. SUMMARY

[0004] The present application aims to provide a panoramic image saliency object detection method, device, equipment and storage medium, aiming at solving the problem of poor panoramic image saliency object detection performance and insufficient saliency of the detected saliency object due to the fact that the prior art cannot provide an effective panoramic image saliency object detection method.

[0005] In one aspect, the present application provides a panoramic image saliency object detection method, the method comprising the following steps:

[0006] When a saliency object detection request is received, a panoramic image to be detected is acquired;

[0007] The panoramic image is processed by a pre-established saliency detection model to obtain a saliency map of the panoramic image, wherein the saliency detection model comprises a double-branch structure network, a hybrid projection feature fusion module and a progressive prediction module.

[0008] Preferably, the step of processing the panoramic image by the pre-established saliency detection model comprises:

[0009] The panoramic image is feature extracted by the double-branch structure network to obtain first and second features;

[0010] The first and second features are feature fused by the hybrid projection feature fusion module to obtain a third feature;

[0011] The first, second and third features are processed by the progressive prediction module to obtain the saliency map.

[0012] Preferably, the progressive prediction module comprises a top-level guided convolution module and a progressive refinement module.

[0013] Preferably, the step of processing the first, second and third features by the progressive prediction module comprises:

[0014] According to the first and second features, a fourth feature is obtained by the top-level guided convolution module;

[0015] According to the first, third and fourth features, the saliency map is obtained by the progressive refinement module.

[0016] In another aspect, the present application provides a panoramic image saliency object detection device, the device comprising:

[0017] An image acquisition unit is configured to acquire a panoramic image to be detected when a salient object detection request is received.

[0018] A saliency map obtaining unit is configured to obtain a saliency map of the panoramic image by processing the panoramic image through a pre-established saliency detection model, wherein the saliency detection model comprises a double-branch structure network, a hybrid projection feature fusion module and a progressive prediction module.

[0019] Preferably, the saliency map obtaining unit comprises:

[0020] A feature extraction unit is configured to extract features of the panoramic image through the double-branch structure network to obtain first features and second features;

[0021] A feature fusion unit is configured to fuse the first features and the second features through the hybrid projection feature fusion module to obtain third features; and

[0022] A feature processing unit is configured to process the first features, the second features and the third features through the progressive prediction module to obtain the saliency map.

[0023] Preferably, the progressive prediction module comprises a top-level guided convolution module and a progressive refinement module.

[0024] Preferably, the feature processing unit comprises:

[0025] A feature obtaining unit is configured to obtain fourth features through the top-level guided convolution module according to the first features and the second features; and

[0026] A saliency map obtaining subunit is configured to obtain the saliency map through the progressive refinement module according to the first features, the third features and the fourth features.

[0027] In another aspect, the present application also provides an image processing device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the panoramic image salient object detection method.

[0028] In another aspect, the present application also provides a computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the steps of the panoramic image salient object detection method.

[0029] When the salient object detection request is received, the panoramic image to be detected is acquired, and the panoramic image is processed through a pre-established saliency detection model to obtain a saliency map of the panoramic image, wherein the saliency detection model comprises a double-branch structure network, a hybrid projection feature fusion module and a progressive prediction module, so as to improve the detection performance of the salient object of the panoramic image, effectively filter the redundant information in the panoramic image, and improve the saliency of the saliency map of the panoramic image. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is an implementation flowchart of the panoramic image salient object detection method provided by the first embodiment of the present application;

[0031] Figure 2 is an implementation flowchart of the panoramic image salient object detection method provided by the second embodiment of the present application;

[0032] Figure 3 is an implementation flowchart of the panoramic image salient object detection method provided by the third embodiment of the present application;

[0033] Figure 4 is an architecture schematic diagram of the top-layer guided convolution module in the panoramic image salient object detection method provided by the third embodiment of the present application;

[0034] Figure 5 is a whole framework schematic diagram of the saliency detection model in the panoramic image salient object detection method provided by the third embodiment of the present application;

[0035] Figure 6 is a structure schematic diagram of the panoramic image salient object detection device provided by the fourth embodiment of the present application;

[0036] Figure 7 is a preferred structure schematic diagram of the panoramic image salient object detection device provided by the fourth embodiment of the present application;

[0037] Figure 8 is a structure schematic diagram of the image processing device provided by the fifth embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0039] The specific implementation of the present application is described in detail below in combination with specific embodiments:

[0040] Example One:

[0041] Figure 1 The implementation process of the method for detecting salient objects of a panoramic image provided by the embodiment of the present application is shown, only the parts related to the embodiment of the present application are shown for the convenience of description, and the details are as follows:

[0042] In step S101, when a salient object detection request is received, a panoramic image to be detected is acquired.

[0043] The embodiment of the present application is applicable to an image processing platform, device or system, for example, a personal computer, a server, etc. In the embodiment of the present application, a panoramic image is a 360° image containing rich surrounding scene information, which can provide a wider field of view and a more real scene for a viewer and enable the viewer to obtain an immersive experience. The panoramic image can be obtained by a panoramic camera or a virtual device such as AR / VR, and the panoramic image is usually displayed in an ERP format, which can also be referred to as an ERP image.

[0044] In step S102, the panoramic image is processed by a pre-established salient detection model to obtain a saliency map of the panoramic image, wherein the salient detection model includes a double-branch structure network, a hybrid projection feature fusion module and a progressive prediction module.

[0045] In the embodiment of the present application, the panoramic image is processed by the double-branch structure network, the hybrid projection feature fusion module and the progressive prediction module in the pre-established salient detection model to obtain the saliency map of the panoramic image.

[0046] In the embodiment of the present application, the specific implementation of processing the panoramic image by the pre-established salient detection model is described in the following method embodiment, which is not repeated here.

[0047] In the embodiment of the present application, when a salient object detection request is received, a panoramic image to be detected is acquired, and the panoramic image is processed by a pre-established salient detection model to obtain a saliency map of the panoramic image, wherein the salient detection model includes a double-branch structure network, a hybrid projection feature fusion module and a progressive prediction module, thereby improving the detection performance of salient objects of the panoramic image, effectively filtering the redundant information in the panoramic image and improving the saliency of the saliency map of the panoramic image.

[0048] Example Two:

[0049] Figure 2 The implementation process of the method for detecting salient objects of a panoramic image provided by the embodiment of the present application is shown, only the parts related to the embodiment of the present application are shown for the convenience of description, and the details are as follows:

[0050] The processing of the panoramic image in step S102 of the embodiment one is implemented by the following steps:

[0051] In step S201, feature extraction is performed on the panoramic image by a double-branch structure network to obtain first features and second features.

[0052] In the embodiment of the present application, since the ERP image has a serious distortion problem, only using the ERP image cannot comprehensively and accurately detect all the significant targets in the panoramic image. In order to solve this problem, a double-branch structure network (Bi-branch Net) is constructed, the ERP format panoramic image to be detected is input into the double-branch structure network of the saliency detection model, E2C operation is performed on the ERP image by the double-branch structure network to convert the panoramic image in the ERP format into a panoramic image in the CMP format, and the corresponding CMP image is obtained. Then, backbone network feature extraction is performed on the ERP image and the CMP image to obtain two groups of global features and local features. The global features are the first features, and the local features are the second features.

[0053] When the feature extraction is performed on the panoramic image by the double-branch structure network, preferably, the feature extraction on the panoramic image is implemented by the following steps:

[0054] (1) The ERP image I E ∈R B×3×H×W is converted into the corresponding CMP image I by an E2C module, wherein 6 represents six faces of front, right, back, left, up and down in a three-dimensional space, 3 represents the intensity of the red, green and blue channels corresponding to each spatial position, R represents a real number domain, B represents a batch size, H and W respectively represent the height and width of the panoramic image, and the E2C module is used for E2C operation on the panoramic image.

[0055] (2) An encoder ResNet-50 is used as a feature extractor, and the feature extractor is used to perform feature extraction on the ERP image I E and the CMP image I C respectively to generate two groups of corresponding features, i.e., the first features and the second features , wherein i = 1, 2,..., 5 is the index of the corresponding layer of the encoder.

[0056] The feature extraction on the panoramic image is implemented by the above steps (1) and (2), so that the significant targets in the panoramic image can be more comprehensively captured.

[0057] In step S202, feature fusion is performed on the first features and the second features by a hybrid projection feature fusion module to obtain third features.

[0058] In the embodiment of the present application, the first features The second feature Although the integrity of the panoramic image is destroyed, it retains more detailed features, in order to make full use of the advantages of the two features, a hybrid projection feature fusion module (FFM) is constructed, and each layer of the encoder corresponds to an FFM, that is, the index number of the FFM is 1, 2,..., 5, which can be expressed as FFM i , through the FFM corresponding to the index number i Fuse the first feature and the second feature extracted by Bi-branchNet to obtain a fusion feature with the advantages of ERP images and CMP images, that is, the third feature

[0059] When the first feature and the second feature are fused by the hybrid projection feature fusion module, the feature fusion of the first feature and the second feature is preferably realized by the following steps:

[0060] (1) A C2E module is used to project the second feature into an ERP format, and align with the first feature to obtain an aligned feature , the specific operation is , wherein the C2E module is used to convert the CMP format into the ERP format, and i represents the index of the encoder corresponding layer in the double-branch structure network;

[0061] (2) Perform element-level multiplication operation on and , and adaptively adjust the features of each channel through a squeeze and excitation (SE) module to obtain an enhanced feature , the specific operation is , wherein "x" represents element-level multiplication operation, and SE(·) represents the SE module;

[0062] (3) The enhanced feature is respectively concatenated with the first feature and the aligned feature , and fed into a GConv1 convolution group to finally obtain the corresponding enhanced features and , the specific operation is GConv k (·)=ReLU(BN(Conv k×k(·)))),wherein the GConv1 convolution group comprises a 1x1 convolution layer, a BN layer and a ReLU layer, and k is the size of the convolution kernel;

[0063] (4) connecting the enhanced ERP feature and the CMP feature together, feeding the connected feature to a 1x1 convolution layer and a ReLU layer, and finally obtaining the final fusion feature of the ERP and CMP images , and the specific operation is

[0064] The feature fusion of the first feature and the second feature is realized through the above steps (1)-(4), the complementary correlation between the ERP and CMP images is adaptively learned through the FFM, so that the third feature combines the advantages of the features in the ERP image and the CMP image, and can comprehensively represent the saliency clues of the 360° panoramic image, thereby reducing the influence of the distortion of the ERP image on the model performance.

[0065] In step S203, the first feature, the second feature and the third feature are processed by the progressive prediction module to obtain a saliency map.

[0066] In the embodiment of the application, the panoramic image generally has a large resolution, in order to detect salient objects with different scales, a progressive prediction module (PPM) is constructed here to combine salient features with different scales, and the first feature, the second feature and the third feature with different scales are processed by the PPM to obtain a saliency map of the panoramic image.

[0067] Preferably, the progressive prediction module PPM comprises a top-level guided convolution (TLGC) module and a progressive refinement module (PRM), and each layer of the encoder corresponds to a PRM, i.e. the index number of the PRM is 1, 2,..., 5, which can be expressed as PRM i , wherein the TLGC is used for multi-scale fusion processing of the topmost features of the first feature and the second feature extracted by the Bi-branchNet, to obtain high-level semantic features with multi-scale information, and the PRM is used for extracting saliency clues from the high-level semantic features obtained in the TLGC, and gradually refining through layer-by-layer PRM modules to obtain a panoramic image saliency map, so as to further refine and merge the features of each layer under the guidance of the related features, and improve the performance and accuracy of the model.

[0068] In this embodiment of the invention, the specific implementation method for processing the first feature, the second feature, and the third feature through the progressive prediction module is detailed in the following method embodiments, and will not be repeated here.

[0069] In this embodiment of the invention, features are first extracted from the panoramic image through a dual-branch network to obtain a first feature and a second feature. Then, the first feature and the second feature are fused through a hybrid projection feature fusion module to obtain a third feature. Finally, the first feature, the second feature, and the third feature are processed through a progressive prediction module to obtain a saliency map, thereby improving the performance and accuracy of the saliency detection model and enhancing the saliency of the panoramic image saliency map.

[0070] Example Three:

[0071] Figure 3 The implementation flow of the panoramic image salient object detection method provided in Embodiment 3 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below:

[0072] The processing of the first feature, the second feature, and the third feature in step S203 of Embodiment 2 is achieved through the following steps:

[0073] In step S301, a fourth feature is obtained by the top-level guided convolution module based on the first and second features.

[0074] In this embodiment of the invention, the top-level guided convolutional module TLGC can capture high-level salient features at different scales, and TLGC corresponds to an FFM module with index number 6, which can be represented as FFM. 6 This is used to receive high-level semantic features (i.e., the fourth feature) with multi-scale information from the TLGC output. The first feature is then taken from the output of the fifth layer of the encoder in the Bi-branch Net. Second feature The input is fed into TLGC. First, the C2E module is used to process the received second feature. Convert to ERP format Then, a 1×1 convolutional layer in TLGC is used to reduce the received first feature. and characteristics The number of channels is reduced, and then the reduced-channel features are fed into three branches in TLGC to obtain three saliency features with different receptive fields. The three output saliency features are then concatenated to obtain feature F. scales Finally, F scales The GConv3 convolutional group fed into TLGC yields convolutions with redundant semantic information, respectively... and Corresponding top-level guiding features That is, the fourth characteristic, abbreviated as The three branches are respectively a GConv3 convolutional group with a dilation rate of 1, a GConv3 convolutional group with a dilation rate of 3, and a GConv3 convolutional group with a dilation rate of 5. The GConv3 convolutional group includes a 3×3 convolutional layer, a BN layer, and a ReLU layer, specifically represented as GConv3(·)=ReLU(BN(Conv3· ... 3×3 (·)))), Figure 4 The architecture of the top-level guided convolution module is shown.

[0075] In step S302, a saliency map is obtained through a progressive refinement module based on the first feature, the third feature, and the fourth feature.

[0076] In this embodiment of the invention, a foreground-background attention mechanism is introduced into the Progressive Refinement Module (PRM) to refine the saliency map and refine the first feature. Third feature and the fourth feature After inputting the PRM corresponding to the index number, the PRM performs the appropriate processing to obtain the saliency map. Specifically, the saliency map is obtained through the following steps:

[0077] (1) The third feature is obtained through convolution operation. With the first feature Alignment is performed to obtain the aligned features. and

[0078] (2) and Connect them together, then use a 1×1 convolution to reduce the feature channels, and obtain the feature...

[0079] (3) Feeding the data into a 3×3 convolution yields optimized fused features.

[0080] (4) Connect features through a residual operation and To ensure the integrity of salient objects, features are obtained.

[0081] (5) Features and By connecting the components, a cascaded feature is obtained.

[0082] (6) Cascaded features The features are fed into the SE module to adaptively learn channel features and filter out redundant information, resulting in the final optimized fusion features.

[0083] (7) Use 1×1 convolution pairs for features Reduce the feature channels and output features

[0084] (8) Through each PRM i (i≤4) progressively refine the previous module PRM i+1 Specifically, when i = 5, the fourth feature will be... Enter FFM 6 and FFM 6 The generated output features The data is fed into a GConv3 convolutional group, and then a 1×1 convolution is used to reduce the number of channels, resulting in a single-channel saliency feature. and features Designated as the PRM corresponding to the fifth layer of the encoder (i.e., PRM) 5 The input is given by ), and when i = {1, 2, 3, 4}, the previous PRM layer is... i+1 Output characteristics As the current module PRM i The input features are fed into the current module PRMi, and then an upsampling operation is performed on the features. and Perform feature alignment to obtain foreground features.

[0085] (10) Using the sigmoid operation to extract foreground features Obtaining foreground saliency map

[0086] (11) According to the formula Foreground saliency map Subtracting matrix E from the background matrix yields the background saliency map of the background region. In matrix E, all elements are 1, and "-" indicates element-wise subtraction.

[0087] (12) Features and foreground salience map The input is fed into a branch containing an element-wise multiplication operation and a GConv3 convolutional group to obtain foreground saliency cues, while simultaneously transferring features... and background salient image Feeding into another branch that also contains an element-wise multiplication operation and a GConv3 convolution group yields background saliency cues;

[0088] (13)Cascade the foreground saliency cues and background saliency cues generated by the two branches into the GConv1 convolution group to reduce the number of channels, and then use 3x3 convolution to obtain salient features with foreground and background information

[0089] (14)Element-wise addition of the foreground features and the salient features obtains complete salient features The final complete salient features constitute the saliency map S of the panoramic image 360 .

[0090] The saliency map is obtained through the above steps (1) to (14), thereby improving the performance and accuracy of the model and improving the saliency of the saliency map.

[0091] Figure 5 The overall framework of the saliency detection model is shown.

[0092] In the embodiment of the application, according to the first feature and the second feature, the fourth feature is obtained through the top layer guided convolution module, and according to the first feature, the third feature and the fourth feature, the saliency map is obtained through the progressive refinement module, thereby improving the performance and accuracy of the model and improving the saliency of the saliency map.

[0093] Example Four:

[0094] Figure 6 The structure of the panoramic image saliency object detection device provided in the fourth embodiment of the application is shown. For ease of illustration, only parts related to the embodiments of the application are shown, which include:

[0095] The image acquisition unit 61 is configured to acquire a panoramic image to be detected when receiving a saliency object detection request.

[0096] The embodiments of the application are applicable to image processing platforms, devices or systems, such as personal computers, servers, etc. In the embodiments of the application, the panoramic image is a 360° image containing rich surrounding scene information, which can provide a wider field of view and a more realistic scene for the viewer and enable the viewer to have an immersive experience. The panoramic image can be obtained by a panoramic camera or AR / VR virtual device. The panoramic image is usually displayed in ERP format and can also be referred to as an ERP image.

[0097] The saliency map acquisition unit 62 is configured to process the panoramic image by using a pre-established saliency detection model to obtain a saliency map of the panoramic image, wherein the saliency detection model includes a double-branch structure network, a hybrid projection feature fusion module and a progressive prediction module.

[0098] In the embodiment of the present application, the panoramic image is processed by the double-branch structure network in the pre-established saliency detection model, the hybrid projection feature fusion module and the progressive prediction module to obtain a saliency map of the panoramic image.

[0099] As shown in Figure 7 , preferably, the saliency map obtaining unit 62 comprises:

[0100] The feature extraction unit 621 is configured to perform feature extraction on the panoramic image by the double-branch structure network to obtain first features and second features.

[0101] In the embodiment of the present application, since the ERP image has a serious distortion problem, only using the ERP image cannot comprehensively and accurately detect all the salient targets in the panoramic image. In order to solve this problem, a double-branch structure network (Bi-branchNet) is constructed. The ERP format panoramic image to be detected is input into the double-branch structure network of the saliency detection model. The ERP image is subjected to E2C operation by the double-branch structure network to convert the panoramic image in ERP format into a panoramic image in CMP format to obtain a corresponding CMP image. The ERP image and the CMP image are subjected to backbone network feature extraction to obtain two groups of global features and local features. The global features are the first features, and the local features are the second features.

[0102] Preferably, the feature extraction unit 621 comprises:

[0103] The format conversion unit 6211 is configured to convert the ERP image I E ∈R B×3×H×W into a corresponding CMP image I wherein, 6 represents the front, right, back, left, up and down six faces of the three-dimensional space, 3 represents the intensity of the red, green and blue channels corresponding to each spatial position, R represents the real number domain, B represents the batch size, H and W represent the height and width of the panoramic image respectively, and the E2C module is a module for performing E2C operation on the panoramic image; and

[0104] The feature extraction sub-unit 6212 is configured to use an encoder ResNet-50 as a feature extractor to perform feature extraction on the ERP image I E and the CMP image I C respectively by the feature extractor to generate two groups of corresponding features, i.e. the first features and the second features , wherein i = 1, 2,..., 5 is the index of the corresponding layer of the encoder.

[0105] The feature fusion unit 622 is configured to perform feature fusion on the first feature and the second feature by a hybrid projection feature fusion module to obtain a third feature.

[0106] In the embodiment of the present application, the first feature retains more complete salient object information, and the second feature destroys the integrity of the panoramic image but retains more detailed features. In order to make full use of the advantages of the two features, a hybrid projection feature fusion module (FFM) is constructed, and each layer of the encoder corresponds to an FFM, i.e., the index numbers of the FFM are 1, 2,..., 5, which can be expressed as FFM i The first feature i extracted by the Bi-branchNet and the second feature are fused by the FFM corresponding to the index number to obtain a fused feature, i.e., a third feature which has the advantages of the ERP image and the CMP image.

[0107] Preferably, the feature fusion unit 622 comprises:

[0108] The feature alignment unit 6221 is configured to project the second feature into an ERP format by a C2E module and align the second feature with the first feature to obtain aligned features , and the specific operation is wherein the C2E module is used to convert the CMP format into the ERP format, and i represents the index of the corresponding layer of the encoder.

[0109] The first enhancement unit 6222 is configured to perform an element-level multiplication operation on and and adaptively adjust the features of each channel by a Squeeze-and-Excitation (SE) module to obtain an enhanced feature , and the specific operation is wherein "x" represents the element-level multiplication operation, and SE(·) represents the SE module.

[0110] The second enhancement unit 6223 is configured to cascade the enhanced feature with the first feature and the aligned feature respectively and feed them into a GConv1 convolution group to finally obtain the corresponding enhanced features and , and the specific operation is GConv k(·)=ReLU(BN(Conv k×k (·)))),wherein the GConv1 convolution group comprises one 1x1 convolution layer, one BN layer and one ReLU layer, and k is the size of the convolution kernel; and

[0111] The feature fusion subunit 6224 is configured to connect the enhanced ERP features and CMP features together, feed the connected features to a 1x1 convolution layer and a ReLU layer, and finally obtain the final fusion features of the ERP and CMP images. Specifically, the feature fusion subunit 6224 comprises a 1x1 convolution layer and a ReLU layer.

[0112] The feature processing unit 623 is configured to process the first features, the second features and the third features through a progressive prediction module to obtain a saliency map.

[0113] In the embodiment of the present application, the panoramic image generally has a large resolution, in order to detect salient objects with different scales, a progressive prediction module (PPM) is constructed to combine salient features with different scales, and the first features, the second features and the third features with different scales are processed through the PPM to obtain a saliency map of the panoramic image.

[0114] Preferably, the progressive prediction module PPM comprises a top layer guided convolution (TLGC) module and a progressive refinement module (PRM), and each layer of the encoder corresponds to a PRM, that is, the index number of the PRM is 1, 2,..., 5, and the PRM can be represented as PRM i , wherein the TLGC is configured to perform multi-scale fusion processing on the top layer features of the first features and the second features extracted by the Bi-branch Net to obtain high-level semantic features with multi-scale information, and the PRM is configured to extract saliency clues from the high-level semantic features obtained in the TLGC and gradually refine through the PRM modules layer by layer to obtain a saliency map of the panoramic image, so as to further refine and merge the features of each layer under the guidance of the related features, and improve the performance and accuracy of the model.

[0115] Preferably, the feature processing unit 623 comprises:

[0116] The feature obtaining unit 6231 is configured to obtain fourth features through a top layer guided convolution module according to the first features and the second features.

[0117] In the embodiment of the present application, the top layer guided convolution module TLGC can capture high-level salient features with different scales, and the TLGC corresponds to an FFM module, and the index number of the FFM is 6, and the FFM can be represented as FFM 6This is used to receive high-level semantic features (i.e., the fourth feature) with multi-scale information from the TLGC output. The first feature is then taken from the output of the fifth layer of the encoder in the Bi-branch Net. Second feature The input is fed into TLGC. First, the C2E module is used to process the received second feature. Convert to ERP format Then, a 1×1 convolutional layer in TLGC is used to reduce the received first feature. and characteristics The number of channels is reduced, and then the reduced-channel features are fed into three branches in TLGC to obtain three saliency features with different receptive fields. The three output saliency features are then concatenated to obtain feature F. scales Finally, F scales The GConv3 convolutional group fed into TLGC yields convolutions with redundant semantic information, respectively... and Corresponding top-level guiding features That is, the fourth characteristic, abbreviated as These three branches are respectively a GConv3 convolutional group with a dilation rate of 1, a GConv3 convolutional group with a dilation rate of 3, and a GConv3 convolutional group with a dilation rate of 5. The GConv3 convolutional group includes a 3×3 convolutional layer, a BN layer, and a ReLU layer, specifically represented as GConv3(·)=ReLU(BN(Conv3· ... 3×3 (·)))).

[0118] The saliency map acquisition subunit 6232 is used to obtain the saliency map through a progressive refinement module based on the first feature, the third feature, and the fourth feature.

[0119] In this embodiment of the invention, a foreground-background attention mechanism is introduced into the Progressive Refinement Module (PRM) to refine the saliency map and refine the first feature. Third feature and the fourth feature After inputting the PRM corresponding to the index number, the PRM performs the appropriate processing to obtain the saliency map. Specifically, the saliency map is obtained through the following steps:

[0120] (1) The third feature is obtained through convolution operation. With the first feature Alignment is performed to obtain the aligned features. and

[0121] (2) and The channels are connected together, and then a 1×1 convolution is used to reduce the feature channels to obtain the feature...

[0122] (3) Feeding the data into a 3×3 convolution yields optimized fused features.

[0123] (4) Connect features through a residual operation and To ensure the integrity of salient objects, features are obtained.

[0124] (5) Features and By connecting the components, a cascaded feature is obtained.

[0125] (6) Cascaded features The features are fed into the SE module to adaptively learn channel features and filter out redundant information, resulting in the final optimized fusion features.

[0126] (7) Use 1×1 convolutions on features Reduce the feature channels and output features

[0127] (8) Through each PRM i (i≤4) progressively refine the previous module PRM i+1 Specifically, when i = 5, the fourth feature will be... Enter FFM 6 and FFM 6 The generated output features The data is fed into a GConv3 convolutional group, and then a 1×1 convolution is used to reduce the number of channels, resulting in a single-channel saliency feature. and features Designated as the PRM corresponding to the fifth layer of the encoder (i.e., PRM) 5 The input is given by ), and when i = {1, 2, 3, 4}, the previous PRM layer is... i+1 Output characteristics As the current module PRM i The input characteristics are input into the current module PRM. i Subsequently, an upsampling operation is used on the features. and Perform feature alignment to obtain foreground features.

[0128] (10) Using the sigmoid operation to extract foreground features Obtaining foreground saliency map

[0129] (11) According to the formula Subtracting the matrix E from the foreground saliency map , where all elements in the matrix E are 1, and "-" represents element-level subtraction operation, to obtain the background saliency map of the background region

[0130] (12) Inputting the feature and the foreground saliency map into a branch comprising an element-wise multiplication operation and a GConv3 convolution group to obtain foreground saliency clues, and inputting the feature and the background saliency map into another branch also comprising an element-wise multiplication operation and a GConv3 convolution group to obtain background saliency clues

[0131] (13) Concatenating the foreground saliency clues and the background saliency clues generated by the two branches and inputting them into a GConv1 convolution group to reduce the number of channels, and then using a 3x3 convolution to obtain saliency features with foreground and background information

[0132] (14) Performing element-wise addition on the foreground feature and the saliency feature to obtain complete saliency features , and finally the complete saliency features constitute the saliency map S 360 of the panoramic image.

[0133] In the embodiments of the present application, each unit of the panoramic image saliency object detection device can be implemented by a corresponding hardware or software unit, and each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which does not limit the present application.

[0134] Example Five:

[0135] Figure 8 The structure of the image processing device provided by the fifth embodiment of the present application is shown, and only parts related to the embodiments of the present application are shown for ease of illustration.

[0136] The image processing device 8 of the embodiment of the present application comprises a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. The processor 80 implements the steps in the panoramic image saliency object detection method embodiments when executing the computer program 82, for example Figure 1 ​​The steps S101-S102 shown. Alternatively, the processor 80 implements the functions of the units in the above-mentioned device embodiments when executing the computer program 82, for example Figure 6 The functions of the units 61-62 shown.

[0137] In the embodiment of the present application, when a salient object detection request is received, a panoramic image to be detected is acquired, and the panoramic image is processed by a pre-established salient detection model to obtain a salient map of the panoramic image, wherein the salient detection model comprises a double-branch structure network, a hybrid projection feature fusion module and a progressive prediction module, thereby improving the detection performance of the salient object of the panoramic image, effectively filtering the redundant information in the panoramic image, and improving the saliency of the salient map of the panoramic image.

[0138] The image processing device of the embodiment of the present application can be a personal computer, a server. The steps implemented when the processor 80 of the image processing device 8 executes the computer program 82 to implement the panoramic image salient object detection method can refer to the description of the above-mentioned method embodiments, which will not be described here.

[0139] Example Six:

[0140] In the embodiment of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the steps in the above-mentioned panoramic image salient object detection method embodiments when executed by a processor, for example, Figure 1 The steps S101-S102 shown. Alternatively, the computer program implements the functions of the units in the above-mentioned device embodiments when executed by a processor, for example Figure 6 The functions of the units 61-62 shown.

[0141] In the embodiment of the present application, when a salient object detection request is received, a panoramic image to be detected is acquired, and the panoramic image is processed by a pre-established salient detection model to obtain a salient map of the panoramic image, wherein the salient detection model comprises a double-branch structure network, a hybrid projection feature fusion module and a progressive prediction module, thereby improving the detection performance of the salient object of the panoramic image, effectively filtering the redundant information in the panoramic image, and improving the saliency of the salient map of the panoramic image.

[0142] The computer readable storage medium of the embodiment of the present application can include any entity or device capable of carrying computer program code, recording medium, for example, ROM / RAM, magnetic disk, optical disk, flash memory, etc.

[0143] The above merely describes preferred embodiments of the present application, and is not used to limit the present application, any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for detecting salient objects in a panoramic image, characterized in that, The method comprises the following steps: When a salient object detection request is received, a panoramic image to be detected is acquired, the panoramic image being an ERP format panoramic image; The panoramic image is processed by a pre-established salient detection model to obtain a saliency map of the panoramic image, wherein the salient detection model comprises a double-branch structure network, a hybrid projection feature fusion module and a progressive prediction module; The step of processing the panoramic image by the pre-established salient detection model comprises: The panoramic image is feature extracted by the double-branch structure network to obtain first and second features, comprising: inputting the ERP format panoramic image into the double-branch structure network, converting the ERP format panoramic image into a CMP format panoramic image by the double-branch structure network, and feature extracting the ERP format and CMP format panoramic images respectively to obtain the first and second features; The first and second features are feature fused by the hybrid projection feature fusion module to obtain a third feature; The first, second and third features are processed by the progressive prediction module to obtain the saliency map, comprising: extracting semantic features with multi-scale information of the first and second features to obtain fourth features, and obtaining the saliency map by foreground-background attention mechanism based on the first, third and fourth features.

2. The method of claim 1, wherein, The progressive prediction module comprises a top-level guided convolution module and a progressive refinement module.

3. The method of claim 2, wherein, The step of processing the first, second and third features by the progressive prediction module comprises: The fourth feature is obtained by the top-level guided convolution module according to the first and second features; The saliency map is obtained by the progressive refinement module according to the first, third and fourth features.

4. An apparatus for detecting salient objects in a panoramic image, characterized by The device comprises: An image acquisition unit configured to acquire a panoramic image to be detected when a salient object detection request is received; and A saliency map obtaining unit configured to process the panoramic image by a pre-established salient detection model to obtain a saliency map of the panoramic image, wherein the salient detection model comprises a double-branch structure network, a hybrid projection feature fusion module and a progressive prediction module; The saliency map obtaining unit comprises: A feature extraction unit configured to feature extract the panoramic image by the double-branch structure network to obtain first and second features, comprising: inputting an ERP format panoramic image into the double-branch structure network, converting the ERP format panoramic image into a CMP format panoramic image by the double-branch structure network, and feature extracting the ERP format and CMP format panoramic images respectively to obtain the first and second features; A feature fusion unit configured to feature fuse the first and second features by the hybrid projection feature fusion module to obtain a third feature; and The feature processing unit is configured to process the first feature, the second feature and the third feature by the progressive prediction module to obtain the saliency map, including: extracting semantic features with multi-scale information of the first feature and the second feature respectively to obtain a fourth feature, and obtaining the saliency map by a foreground-background attention mechanism based on the first feature, the third feature and the fourth feature.

5. The apparatus of claim 4, wherein, The progressive prediction module includes a top-level guided convolution module and a progressive refinement module.

6. The apparatus of claim 5, wherein, The feature processing unit includes: a feature obtaining unit configured to obtain a fourth feature by the top-level guided convolution module according to the first feature and the second feature; and a saliency map obtaining subunit configured to obtain the saliency map by the progressive refinement module according to the first feature, the third feature and the fourth feature.

7. An image processing apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 3.

8. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 7. The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 3.

Images