Salient object detection method based on part-object relationship based on disentangled capsule routing

Abstract

This invention discloses a method for detecting salient objects with a partial-object relationship based on disentangled capsule routing, comprising: extracting basic deep features from an input image to obtain basic deep features at five different scales; further extracting multi-receptive field deep features from the basic deep features; utilizing a pre-trained capsule network based on a disentangled routing algorithm to analyze the deep features at three deep scales to obtain corresponding partial-object relationship features; fusing the corresponding partial-object relationship features with the deep features to obtain a fused feature at three deep scales; and further fusing the deep features at the first and second scales and the fused feature at three deep scales to obtain a fused feature, and generating a saliency map based on the fused feature. This invention solves the problems of the existing techniques, such as the large number of network parameters and slow inference speed, by achieving better foreground and background segmentation and faster network inference speed. It can be used in image preprocessing in computer vision.

Description

Technical Field

[0001] The present invention relates to a salient target detection method for complex background images, and in particular to a part-target relationship salient target detection method based on disentangled capsule routing, belonging to the technical field of salient target detection in image processing. Background Art

[0002] The task of salient object detection aims to mimic the human visual attention mechanism and automatically identify the most eye-catching regions or objects in an image scene. Salient object detection can be used as a preprocessing step to improve the processing speed of various visual tasks such as segmentation, image fusion, image retrieval, and image recognition.

[0003] Early salient object detection methods detected salient objects by extracting handcrafted image features. However, due to the limited representation capabilities of handcrafted features, these methods struggled to break through performance bottlenecks. The emergence of deep learning, leveraging its powerful feature representation capabilities, has significantly improved salient object detection performance. Most deep learning-based salient object detection methods independently calculate the saliency of each image region and combine these saliency values ​​into a single saliency for the entire image to detect salient objects. This method can easily lead to incomplete detection of salient objects.

[0004] In recent years, in order to solve the above problems, researchers have tried to introduce part-object relationships into the field of salient object detection, such as "Y.Liu, Q.Zhang, D.Zhang, and J.Han, 'Employing deep part-object relationships for salient object detection,' in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 1232–1241." and "Y.Liu, D.Zhang, Q.Zhang, and J.Han, 'Part-object relational visual saliency,' IEEE Transactions on Pattern Analysis and Machine Intelligence, DOI: 10.1109 / TPAMI.2021.3053577, 2021." is to vote each capsule to all capsules to obtain part-target relationship information. This omnidirectional routing method easily leads to many network parameters and high complexity, which in turn leads to slow inference speed. At the same time, due to its high complexity, this type of method is limited to the acquisition of a single small-scale part-target relationship information, which easily leads to a decrease in the performance of significant target detection in complex scenarios. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for detecting salient objects based on part-object relationships using disentangled capsule routing. This method reduces network parameters and routing complexity, thereby accelerating the inference speed of the method for detecting salient objects based on part-object relationships.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] A method for detecting salient objects based on part-object relationships of disentangled capsule routing comprises the following steps:

[0008] Step 1: construct a salient object detection network based on part-object relationship of disentangled capsule routing, and train the salient object detection network to obtain a trained salient object detection network; the salient object detection network includes an improved VGG16 network, a dilated spatial convolutional pooling pyramid, and a capsule network based on the disentangled routing algorithm;

[0009] The improved VGG16 network includes first to fifth modules, wherein the first module includes two convolutional layers and a pooling layer connected in sequence, the second module includes two convolutional layers and a pooling layer connected in sequence, the third module includes three convolutional layers and a pooling layer connected in sequence, the fourth module includes three convolutional layers and a pooling layer connected in sequence, and the fifth module includes three convolutional layers and a pooling layer connected in sequence; the improved VGG16 network is used to extract basic depth features at five different scales on an image;

[0010] The dilated spatial convolutional pooling pyramid is used to further extract basic depth features at each scale to obtain depth features of multiple receptive fields;

[0011] The capsule network based on the disentanglement routing algorithm includes a primary capsule network layer and a convolutional capsule network layer, and the capsule network based on the disentanglement routing algorithm is used to extract part-target relationship information at three deep scales among five scales;

[0012] Step 2: Use the trained salient object detection network to detect the image to be detected and obtain the salient object detection result.

[0013] As a preferred step of the present invention, in step 1, when training the salient object detection network, the error function l used is:

[0014] l=l ce (B,G)+l iou (B,G)

[0015] Among them, l ce(B,G) represents the cross entropy error function, l iou (B,G) represents the boundary error function; the specific formula is as follows:

[0016]

[0017]

[0018] Among them, B is the predicted saliency map, G is the ground truth map, and k is the pixel index.

[0019] As a preferred step of the present invention, the specific process of step 2 is as follows:

[0020] Step 21: crop the image to be detected to 352×352, input it into the improved VGG16 network, extract the network layer features corresponding to the second convolution layer in the first module, the second convolution layer in the second module, the third convolution layer in the third module, the third convolution layer in the fourth module, and the third convolution layer in the fifth module as the basic deep features, represented as: {X i}, i = 1, 2, 3, 4, 5, X i represents the basic depth feature of the i-th scale, and the basic depth feature sizes of the 1st to 5th scales are: 352×352, 176×176, 88×88, 44×44, 22×22, and the feature channels are: 64, 128, 256, 512, 512;

[0021] Step 22: Use the dilated spatial convolutional pooling pyramid to further extract the depth features of multiple receptive fields from the basic depth features of each scale. Represents the depth feature of the i-th scale;

[0022] Step 23: Use a capsule network based on a disentanglement routing algorithm to parse the deep features of the third to fifth scales to obtain part-target relationship features of the third to fifth scales;

[0023] Step 24: For each scale from the third to the fifth scale, the corresponding part-object relationship feature and the depth feature are fused to obtain the fused features of the third to the fifth scale;

[0024] In step 25, the depth features of the first to second scales and the fused features of the third to fifth scales are fused to obtain fused features, and a saliency map is calculated using a one-layer convolutional network on the fused features.

[0025] As a preferred step of the present invention, the depth feature of the multiple receptive fields in step 22 is specifically formulated as follows:

[0026]

[0027] Among them, Concat{·} is a cascade function; Conv(·) is a convolutional network with a convolution kernel of 3×3; X i represents the basic depth feature of the i-th scale; r j is the receptive field size, and {r j}={1,3,5,7}.

[0028] As a preferred step of the present invention, the specific process of step 23 is as follows:

[0029] Step 231: Use the primary capsule network layer to construct and disentangle primary capsules

[0030] Primary capsule construction: Convert the depth feature of the i-th scale into capsule features, that is, construct the capsule's pose matrix and activation value. The pose matrix is ​​constructed as follows:

[0031]

[0032] in, and They are the posture matrix and activation value of the capsule respectively. The superscript i-1 represents the capsule constructed by the primary capsule network layer. W, H, and C represent the width, height, and number of capsule types respectively. D P 、D A are the dimensions of the pose matrix and activation value, D P =16, D A =1, Represents a convolutional network, Re represents a shape transformation operation, and when constructing the activation value, the Sigmoid function is used to transform A i-1 Map to (0,1);

[0033] Primary capsule unwrapping: Unwrap the one-dimensional capsule features in the horizontal and vertical directions from the capsule features to obtain the first layer of unwrapped horizontal capsules and vertical capsules Specifically, the horizontal capsule unwrapping method is:

[0034]

[0035] The vertical capsule unwrapping method is:

[0036]

[0037] Where T represents the transpose operation;

[0038] Step 232: Use the convolutional capsule network layer to perform the first layer voting matrix calculation, the second layer horizontal and vertical direction capsule calculation, and the matrix capsule entanglement.

[0039] Calculation of the first-layer voting matrix:

[0040] The voting matrix is ​​obtained by multiplying the posture matrix with the weight matrix learned by the network. Specifically, the horizontal posture matrix and the weight matrix learned by the network Multiply to get the horizontal voting matrix Right now:

[0041]

[0042] Similarly, the vertical attitude matrix and the weight matrix learned by the network Multiply to get the vertical voting matrix Right now:

[0043]

[0044] Calculation of capsules in the horizontal and vertical directions of the second layer:

[0045] Set the horizontal activation value of the unwrap and voting matrix Input to the maximum expectation routing algorithm EM to calculate the second layer horizontal capsule posture matrix and activation value Set the vertical activation value of the unwrap and voting matrix Input to the maximum expectation routing algorithm EM to calculate the second layer vertical capsule pose matrix and activation value

[0046] Matrix capsule entanglement: including dimension matching and capsule entanglement

[0047] (1) Dimension matching

[0048] Horizontal dimension matching:

[0049]

[0050] Vertical dimension matching:

[0051]

[0052] (2) Capsule entanglement

[0053] By the horizontal attitude matrix and the vertical attitude matrix Multiply to get the posture matrix P of the omnidirectional capsule i-2 ,Right now

[0054]

[0055] Similarly, the activation value A of the omnidirectional capsule i-2 By activating the horizontal value Vertical activation value Multiply them together and use the Sigmoid(·) function to map the activation value to (0,1), that is,

[0056]

[0057] in, Expressed as matrix multiplication.

[0058] As a preferred step of the present invention, the features after fusion of the third to fifth scales in step 24 are specifically expressed as follows:

[0059]

[0060] in, represents the features after fusion of the i-th scale, represents the depth feature of the i-th scale, Represents the part-target relationship feature of the i-th scale.

[0061] As a preferred step of the present invention, the fusion feature described in step 25 is specifically formulated as follows:

[0062]

[0063] Among them, X sal represents the fusion feature, Represent the depth features of the first and second scales respectively, Represent the features after fusion of the 3rd, 4th and 5th scales respectively.

[0064] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0065] 1. This paper untangles the omnidirectional two-dimensional capsule routing algorithm into two different one-dimensional capsule routing algorithms: horizontal and vertical. This significantly reduces the number of network parameters and computational complexity, thereby accelerating the network's inference speed. Simulations show that the present invention's salient object detection method increases detection speed by approximately five times.

[0066] 2. The present invention extracts multi-scale part-target relationship information, which can fully characterize the part-target relationship information of targets of different scales, and helps detect salient targets in complex scenes with different sizes. Simulations show that the present invention is more robust to salient targets of different sizes when detecting salient targets in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a flowchart of the implementation of the method for detecting salient targets based on the part-target relationship of disentangled capsule routing in the present invention;

[0068] Figure 2 This is a simulation diagram of the experimental results of the present invention and the prior art under the ECSSD database;

[0069] Figure 3 It is a simulation diagram of the experimental results of the present invention and the prior art in terms of network parameter quantity and inference speed, where (a) is the network parameter quantity and (b) is the inference speed. DETAILED DESCRIPTION

[0070] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be interpreted as limiting the present invention.

[0071] like Figure 1 As shown, the specific implementation scheme of the present invention is as follows:

[0072] (1) Deep feature extraction

[0073] (1a) Basic deep feature extraction

[0074] The input image is cropped to 352×352, and VGG16 is selected as the basic deep feature extraction network. In order to apply the VGG16 network to the salient object segmentation task, the following modifications are made: the last three fully connected layers of the traditional VGG16 network are removed as the improved VGG16. The improved VGG16 contains five modules: the first module includes two convolutional layers and a pooling layer connected in sequence, the second module includes two convolutional layers and a pooling layer connected in sequence, the third module includes three convolutional layers and a pooling layer connected in sequence, the fourth module includes three convolutional layers and a pooling layer connected in sequence, and the fifth module includes three convolutional layers and a pooling layer connected in sequence.

[0075] Assume that a and b in Conva_b represent the module index and the convolutional network layer index within the module respectively, extract the network layer features of Conv1_2, Conv2_2, Conv3_3, Conv4_3, and Conv5_3, and represent them as: {X i}(i=1,2,3,4,5), the depth feature sizes are: 352×352, 176×176, 88×88, 44×44, 22×22, and the feature channels are: 64, 128, 256, 512, 512 respectively.

[0076] (1b) Deep feature extraction with multiple receptive fields

[0077] In order to obtain rich contextual information under different receptive fields, the dilated spatial convolutional pooling pyramid is used to further extract the deep features of multiple receptive fields from the basic deep features of each scale, namely:

[0078]

[0079] Among them, Concat{·} is the cascade function, {r j}(j=1,2,3,4)={1,3,5,7} is the receptive field scale, Conv(·) is the convolutional network, and the convolution kernel is 3×3.

[0080] (2) Capsule Network Based on Disentanglement Routing Algorithm

[0081] (2a) Primary capsule construction

[0082] The primary capsule construction aims to convert the deep features extracted in (1) into capsule features. It is divided into two steps to construct the capsule's pose matrix and activation value respectively. The pose matrix is ​​constructed as follows:

[0083]

[0084] in and is the posture matrix and activation value of the capsule network. The superscript i-1 represents the capsule constructed by the primary capsule network layer. W, H, and C represent the width, height, and number of capsule types, respectively. D = {D P =16,D A = 1} is the dimension of the pose matrix and activation value, in particular, D P =16 represents the dimension of the posture matrix, D A =1 indicates the dimension of the activation value. Re indicates the shape transformation operation, Represents a convolutional network. In particular, when constructing the activation value, the Sigmoid function is used to convert A 1 Mapped to (0,1).

[0085] (2b) Primary capsule unwrapping

[0086] Primary capsule unwrapping aims to unwrap the one-dimensional capsule features in the horizontal and vertical directions from the capsule features to obtain the horizontal capsule of the first layer. and vertical capsules Specifically, the horizontal capsule unwrapping method is:

[0087]

[0088] Where T represents the transpose operation, and the superscript l represents the layer index, l = 1, 2, ...

[0089] Similarly, the vertical capsule unwrapping method is:

[0090]

[0091] (2c) Calculation of horizontal and vertical voting matrices for two adjacent layers of networks

[0092] The voting matrix is ​​obtained by multiplying the posture matrix with the learned weight matrix. Specifically, the horizontal posture matrix and the weight matrix learned by the network Multiply to get the horizontal voting matrix, that is:

[0093]

[0094] Similarly, the vertical attitude matrix and the weight matrix learned by the network Multiply to get the vertical voting matrix, that is:

[0095]

[0096] (2d) Higher level horizontal and vertical capsule calculation

[0097] Set the horizontal activation value of the unwrap and voting matrix Input to the maximum expectation routing algorithm EM to calculate the higher-level horizontal capsule pose matrix and activation value Set the vertical activation value of the unwrap and voting matrix Input to the maximum expectation routing algorithm EM to calculate the vertical capsule pose matrix of the next layer and activation value

[0098] (2e) Matrix Capsule Entanglement

[0099] Matrix capsule entanglement aims to extract the matrix from the pose matrix ( and ) to recover the omnidirectional pose matrix, and from the activation values ​​( and ) to recover the omnidirectional activation value. Matrix capsule entanglement is achieved by multiplying the horizontal capsule (pose matrix and activation value) and the vertical capsule (pose matrix and activation value) as follows:

[0100] (2e1) Dimension Matching

[0101] Horizontal dimension matching:

[0102]

[0103] Vertical dimension matching:

[0104]

[0105] (2e2) Capsule Entanglement

[0106] Capsule entanglement is achieved by multiplying the horizontal capsule with the vertical capsule. Specifically, the pose matrix P of the omnidirectional capsule is i-(l+1) By the horizontal attitude matrix and the vertical attitude matrix Multiply them together to get

[0107]

[0108] Similarly, the activation value A of the omnidirectional capsule i-(l+1) By activating the horizontal value Vertical activation value Multiply them together and use the Sigmoid(·) function to map the activation value to (0,1), that is,

[0109]

[0110] in, Expressed as matrix multiplication.

[0111] (3) Multi-scale part-target relationship information extraction

[0112] (3a) Multi-scale part-object relationship information extraction

[0113] Considering the lightweight nature of the capsule network based on the disentanglement routing algorithm, it can be applied to multi-scale deep features to obtain multi-scale part-target relationship information. At the same time, considering that the deep features of the three scales in the deep layer have high-level semantic information of the input image, the capsule network based on the disentanglement routing algorithm is applied to the three-layer deep features to parse the part-target relationship features at three scales. Right now

[0114]

[0115] Among them, f DCR (·) is the capsule network operation based on the disentanglement routing algorithm, are the depth features of three scales.

[0116] (3b) Multi-scale part-object relationship features and deep feature fusion

[0117] Fuse part-target relationship features in a deep-to-shallow manner at each network layer and deep features That is, the features after fusion of each network layer for

[0118]

[0119] (4) Saliency map prediction

[0120] Fusion of deep features in a deep-to-shallow manner Two-layer deep features with shallow and Right now

[0121]

[0122] For the fusion feature X sal A one-layer convolutional network is used to calculate the saliency map.

[0123] (5) Network training

[0124] The cross entropy error function and the boundary error function are used to jointly train the salient object detection network. Assume that B is the predicted salient map and G is the corresponding ground truth map. The cross entropy error function is

[0125]

[0126] The boundary error function is:

[0127]

[0128] The joint cross entropy error function and the boundary error function are:

[0129] l=l ce (B,G)+l iou (B,G)

[0130] Where k is the pixel index.

[0131] The technical effects of the present invention are further illustrated below in conjunction with simulation experiments.

[0132] 1. Simulation conditions. All experiments were implemented in Tensorflow. To avoid overfitting caused by training from scratch, Conv1_2, Conv2_2, Conv3_3, Conv4_3, and Conv5_3 of the basic network of the present invention were initialized using the corresponding network layers pre-trained by VGG16. Other weights were initialized using truncated normal random initialization (σ=0.01), and the bias was initialized to 0. The Adam optimizer was used to train the model, with an initial learning rate of 10 -5 ,β1=0.9,β2=0.999. The DUTS training dataset is selected to train the network, and horizontal flipping is used to enhance the data.

[0133] 2. Simulation content and result analysis

[0134] Simulation 1

[0135] The present invention and the existing salient object detection method based on part-object relationship information are used to conduct salient object detection experiments on the public image dataset ECSSD, and some results are compared intuitively, such as Figure 2 As shown, the original image represents the image in the database used for experimental input, and the true value image represents the manually calibrated binary image;

[0136] from Figure 2 It can be seen that compared with the existing technology, the present invention has better integrity in salient target detection, better background suppression effect, and better effect in salient target detection in complex scenes.

[0137] Simulation 2

[0138] The results of the salient target detection experiment on the public image database ECSSD were obtained by comparing the proposed method with the existing salient target detection method based on department-target relationship information. The results were objectively evaluated using recognized evaluation indicators, namely the weighted average F-measure value and the MAE value. The evaluation simulation results are shown in Table 1.

[0139] Table 1

[0140]

[0141]

[0142] As can be seen from Table 1, compared with the existing technology, the present invention has a higher weighted average F-measure value and a lower MAE value, which shows that the present invention has better completeness and consistency in salient object detection, fully demonstrating the effectiveness and superiority of the method of the present invention.

[0143] Simulation 3

[0144] The present invention is compared with the existing salient object detection method based on part-object relationship information (Prior Art 1: Y.Liu, D.Zhang, Q.Zhang, and J.Han, 'Part-object relational visual saliency', IEEE Transactions on Pattern Analysis and Machine Intelligence, DOI: 10.1109 / TPAMI.2021.3053577, 2021. Prior Art 2: Y.Liu, Q.Zhang, D.Zhang, and J.Han, 'Employing deep part-object relationships for salient object detection,' in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 1232–1241.) in a public image database ECSSD for salient object detection experiments, and the network parameter amount and inference speed are intuitively compared, as shown in FIG. Figure 3 As shown, where:

[0145] Figure 3 (a) is a comparison of network parameters of the present invention, prior art 1 and prior art 2;

[0146] Figure 3 (b) is a comparison of the network inference speed of the present invention, prior art 1 and prior art 2;

[0147] from Figure 3 It can be seen that the present invention has fewer network parameters and faster network inference speed, which shows that the present invention has lighter network complexity and more real-time network inference speed for salient target detection, fully demonstrating the effectiveness and superiority of the method of the present invention.

[0148] The above implementation is only for illustrating the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A salient object detection method based on part-object relationship of disentangled capsule routing, characterized by: The method comprises the following steps: Step 1: construct a salient object detection network based on part-object relationship of disentangled capsule routing, and train the salient object detection network to obtain a trained salient object detection network; the salient object detection network includes an improved VGG16 network, a dilated spatial convolutional pooling pyramid, and a capsule network based on the disentangled routing algorithm; The improved VGG16 network includes first to fifth modules, wherein the first module includes two convolutional layers and a pooling layer connected in sequence, the second module includes two convolutional layers and a pooling layer connected in sequence, the third module includes three convolutional layers and a pooling layer connected in sequence, the fourth module includes three convolutional layers and a pooling layer connected in sequence, and the fifth module includes three convolutional layers and a pooling layer connected in sequence; the improved VGG16 network is used to extract basic depth features at five different scales on an image; The dilated spatial convolutional pooling pyramid is used to further extract basic depth features at each scale to obtain depth features of multiple receptive fields; The capsule network based on the disentanglement routing algorithm includes a primary capsule network layer and a convolutional capsule network layer, and the capsule network based on the disentanglement routing algorithm is used to extract part-target relationship information at three deep scales among five scales; Step 2: Use the trained salient object detection network to detect the image to be detected and obtain the salient object detection results. The specific process is as follows: Step 21: crop the image to be detected to 352×352, input it into the improved VGG16 network, extract the network layer features corresponding to the second convolution layer in the first module, the second convolution layer in the second module, the third convolution layer in the third module, the third convolution layer in the fourth module, and the third convolution layer in the fifth module as the basic deep features, represented as: {X i }, i = 1, 2, 3, 4, 5, X i represents the basic depth feature of the i-th scale, and the basic depth feature sizes of the 1st to 5th scales are: 352×352, 176×176, 88×88, 44×44, 22×22, and the feature channels are: 64, 128, 256, 512, 512; Step 22: Use the dilated spatial convolutional pooling pyramid to further extract the depth features of multiple receptive fields from the basic depth features of each scale. Represents the depth feature of the i-th scale; Step 23: Use a capsule network based on a disentanglement routing algorithm to parse the deep features of the third to fifth scales to obtain part-target relationship features of the third to fifth scales; Step 24: For each scale from the third to the fifth scale, the corresponding part-object relationship feature and the depth feature are fused to obtain the fused features of the third to the fifth scale; In step 25, the depth features of the first to second scales and the fused features of the third to fifth scales are fused to obtain fused features, and a saliency map is calculated using a one-layer convolutional network on the fused features.

2. The method for detecting salient objects based on part-object relationships using disentangled capsule routing according to claim 1, wherein: In step 1, when training the salient object detection network, the error function l used is: l=l ce (B,G)+l iou (B,G) Among them, l ce (B,G) represents the cross entropy error function, l iou (B,G) represents the boundary error function; the specific formula is as follows: Among them, B is the predicted saliency map, G is the ground truth map, and k is the pixel index.

3. The method for detecting salient objects based on part-object relationships using disentangled capsule routing according to claim 1, wherein: The specific formula for the depth feature of the multiple receptive fields in step 22 is as follows: Among them, Concat{·} is a cascade function; Conv(·) is a convolutional network with a convolution kernel of 3×3; r j is the receptive field size, and {r j }={1,3,5,7}.

4. The method for detecting salient objects based on part-object relationships using disentangled capsule routing according to claim 1, wherein: The specific process of step 23 is as follows: Step 231: Use the primary capsule network layer to construct and disentangle primary capsules Primary capsule construction: Convert the depth feature of the i-th scale into capsule features, that is, construct the capsule's pose matrix and activation value. The pose matrix is ​​constructed as follows: in, and They are the posture matrix and activation value of the capsule respectively. The superscript i-1 represents the capsule constructed by the primary capsule network layer. W, H, and C represent the width, height, and number of capsule types respectively. D P 、D A are the dimensions of the pose matrix and activation value, D P =16, D A =1, Represents a convolutional network, Re represents a shape transformation operation, and when constructing the activation value, the Sigmoid function is used to transform A i-1 Map to (0,1); Primary capsule unwrapping: Unwrap the one-dimensional capsule features in the horizontal and vertical directions from the capsule features to obtain the first layer of unwrapped horizontal capsules and vertical capsules Specifically, the horizontal capsule unwrapping method is: The vertical capsule unwrapping method is: Where T represents the transpose operation; Step 232: Use the convolutional capsule network layer to perform the first layer voting matrix calculation, the second layer horizontal and vertical direction capsule calculation, and the matrix capsule entanglement. Calculation of the first-layer voting matrix: The voting matrix is ​​obtained by multiplying the posture matrix with the weight matrix learned by the network. Specifically, the horizontal posture matrix and the weight matrix learned by the network Multiply to get the horizontal voting matrix Right now: Similarly, the vertical attitude matrix and the weight matrix learned by the network Multiply to get the vertical voting matrix Right now: Calculation of capsules in the horizontal and vertical directions of the second layer: Set the horizontal activation value of the unwrap and voting matrix Input to the maximum expectation routing algorithm EM to calculate the second layer horizontal capsule posture matrix and activation value Set the vertical activation value of the unwrap and voting matrix Input to the maximum expectation routing algorithm EM to calculate the second layer vertical capsule pose matrix and activation value Matrix capsule entanglement: including dimension matching and capsule entanglement (1) Dimension matching Horizontal dimension matching: Vertical dimension matching: (2) Capsule entanglement By the horizontal attitude matrix and the vertical attitude matrix Multiply to get the posture matrix P of the omnidirectional capsule i-2 ,Right now Similarly, the activation value A of the omnidirectional capsule i-2 By activating the horizontal value Vertical activation value Multiply them together and use the Sigmoid(·) function to map the activation value to (0,1), that is, in, Expressed as matrix multiplication.

5. The method for detecting salient objects based on part-object relationships using disentangled capsule routing according to claim 1, wherein: The specific formula for the features after fusion of the third to fifth scales in step 24 is as follows: in, represents the features after fusion of the i-th scale, represents the depth feature of the i-th scale, Represents the part-target relationship feature of the i-th scale.

6. The method for detecting salient objects based on part-object relationships using disentangled capsule routing according to claim 1, wherein: The specific formula for the fusion feature described in step 25 is as follows: Among them, X sal represents the fusion feature, Represent the depth features of the first and second scales respectively, Represent the features after fusion of the 3rd, 4th and 5th scales respectively.

Images