A light field saliency object detection method and device based on implicit graph learning

Abstract

The application provides a light field saliency object detection method and device based on implicit graph learning, and the method comprises the following steps: inputting a full-focus image to be tested and a focal stack image to be tested into a pre-trained double-flow deep convolution network to obtain a full-focus feature map and a focal stack feature map; inputting the full-focus feature map and the focal stack feature map into a pre-trained contour perception model to obtain a saliency prediction result and an object contour prediction result. By inputting the full-focus image to be tested and the focal stack image into the pre-trained double-flow deep convolution network, the application promotes the mining of saliency features, and by inputting the full-focus feature map and the focal stack feature map into the pre-trained contour perception model, the application can effectively aggregate information of mixed scales, and then obtain an accurate light field saliency object detection result.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for detecting salient targets in a light field based on implicit graph learning. Background Technology

[0002] Salience detection technology is mainly based on image preprocessing methods constructed using visual attention mechanisms. In recent years, light field cameras can provide rich spatial and layout information from multi-view images and focal slices. This rich information has shown its powerful ability to promote salient object detection. Therefore, light field salient object detection (LF SOD) has attracted widespread research attention in recent years.

[0003] In light field images, a focal stack contains a series of focal slices focused at different depths, leading to blurred regions related to saliency and a lack of saliency context information. This characteristic can sever the relationships between pixels, negatively impacting saliency prediction. However, most deep learning-based light field saliency detection models simply use a single focal stack backbone for feature extraction, ignoring the internal correlation between different regions and the saliency prediction results. This fails to fully extract useful saliency features, limits the understanding of the focal stack content, and makes it difficult to obtain accurate light field saliency target detection results.

[0004] Therefore, existing technologies have shortcomings and need to be improved and developed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and apparatus for detecting salient targets in a light field based on implicit graph learning, in order to address the above-mentioned deficiencies of the prior art.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows:

[0007] A light field saliency target detection method based on implicit graph learning includes:

[0008] The fully focused image and the focus stack image to be tested are input into a pre-trained two-stream deep convolutional network to obtain the fully focused feature map and the focus stack feature map.

[0009] The full-focus feature map and the focus stack feature map are input into a pre-trained contour perception model to obtain saliency prediction results and object contour prediction results.

[0010] In one implementation, the dual-stream deep convolutional network includes a full-focus backbone and a focus stack backbone; the step of inputting the full-focus image to be tested and the focus stack image to be tested into the pre-trained dual-stream deep convolutional network to obtain the full-focus feature map and the focus stack feature map includes:

[0011] The full-focus image to be tested is input into a pre-trained full-focus backbone to obtain a full-focus feature map, and the focus stack image to be tested is input into a pre-trained focus stack backbone to obtain a focus stack feature map.

[0012] In one implementation, the fully focused feature map and the focus stack feature map are input into a pre-trained contour-aware model to obtain saliency prediction results and object contour prediction results, including:

[0013] In the pre-trained contour perception model, 2D convolutional layers and ReLU activation functions are used to process each focal stack salient feature in the focal stack feature map into a first focal stack feature, and each full focus salient feature in the full focus feature map into a first full focus feature.

[0014] A compact aggregation module is used to process each of the first focus stack features to obtain enhanced hierarchical focus stack features;

[0015] Multiple interactive feature fusion modules are used to mutually promote the fusion of each layered focus stack feature and each first full focus feature to obtain the initial fusion feature of the focus stack and the initial fusion feature of the full focus.

[0016] Based on the contour prototype representation, the initial fusion feature of the focus stack is processed into a refined fusion feature of the focus stack, and the initial fusion feature of full focus is processed into a refined fusion feature of full focus;

[0017] The saliency detection head is used to aggregate the focus stack refinement fusion feature and the full focus refinement fusion feature to obtain the saliency prediction result and the object contour prediction result.

[0018] In one implementation, the formula for calculating the hierarchical focus stack feature is:

[0019]

[0020] Among them, the This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 1. This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 2. This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 3. This represents a transformation layer used to convert the dimension of the focus stack from N×128 to 1×128 using a 3×3 convolution. The first focal stack feature is represented by l, which is the number of layers in the focal stack backbone network, and the value is 3, 4, or 5.

[0021] In one implementation, the step of employing multiple interactive feature fusion modules to mutually promote the fusion of each of the hierarchical focus stack features and each of the first full-focus features to obtain initial focus stack fusion features and initial full-focus fusion features includes:

[0022] If the input to the interactive feature fusion module is use Significance prediction And implement auxiliary deep supervision, the formula is: Wherein, the Conv 3×3 This represents a 3×3 convolutional layer, where U represents bilinear interpolation upsampling;

[0023] Object contour features are generated by predicting saliency maps. The calculation formula is: Among them, the It consists of a 3×3 convolution, a Groupnorm layer, and a PReLU activation function, used to transform the number of channels from 1 to 128;

[0024] The depth supervision is applied to the object contour prediction, and the calculation formula is as follows:

[0025] The importance of the channel response evaluation characteristics is calculated using the following formula: Where (m,n) represents the coordinate position of the feature map, w k This represents the weight of the k-th channel of the feature map F;

[0026] Through features Generate weighted response The response of different features is dynamically adjusted using an adaptive enhancement operation. The calculation formula includes:

[0027]

[0028]

[0029] Among them, the The FC represents a fully connected layer, and the U represents bilinear interpolation upsampling. This indicates the initial fusion feature of full focus.

[0030] In one implementation, the process of processing the initial focus stack fusion feature into a refined focus stack fusion feature based on the contour prototype representation, and processing the initial full focus fusion feature into a refined full focus fusion feature, includes:

[0031] The initial fusion feature of full focus is improved by embedding contour hints in the global view. The calculation formula is as follows:

[0032] Wherein, Conv represents a convolutional layer with a 3×3 kernel size, and the This indicates a fully focused, refined fusion feature.

[0033] In one implementation, the significance prediction result S P The calculation formula is:

[0034] Among them, the This indicates a fully focused, refined fusion feature, the aforementioned This indicates a focus stack refinement fusion feature;

[0035] The object contour prediction result S C The calculation formula is:

[0036] In one implementation, the training steps of the two-stream deep convolutional network include:

[0037] Obtain a training dataset, which includes: fully focused training images and focus stack training images;

[0038] The fully focused training image and the focus stack training image are input into an initial two-stream deep convolutional network, which includes a fully focused backbone and a focus stack backbone.

[0039] Output the fully focused training feature map from the fully focused backbone network. Output the focus stack training feature map from the focus stack backbone network. Where l is the number of layers of the full-focus backbone and the focus stack backbone, and the value is 2, 3, 4, or 5;

[0040] A total focus map G is constructed based on the total focus training feature map. r The G r ={A r ,F r}, where A r The F represents the adjacency matrix of the full-focus graph. r Nodes representing the full focus graph;

[0041] A focus stack map G is constructed based on the focus stack training feature map.f ={A f ,F f}, where A f The adjacency matrix representing the focus stack graph, the F f Nodes representing the focus stack graph;

[0042] By modeling a graph neural network to establish and infer the relationships between nodes in a fully focused graph, and encouraging connections between fully focused graphs to mine salient feature representations, the formula is expressed as follows:

[0043]

[0044] Wherein, the Θ r The learnable weight matrix representing the full-focused image, the The angle matrix representing the full focus image, the This represents an expression based on a full-focus image;

[0045] By modeling graph neural networks to establish and infer the relationships between nodes in a focus stack graph, and encouraging connections between focus stack graphs to mine salient feature representations, the formula is expressed as follows:

[0046]

[0047] Wherein, the Θ f The learnable weight matrix representing the focus stack graph, the The angle matrix representing the focus stack graph, the This represents a focus-based stack graph representation.

[0048] Based on the expression of the full focus map Obtain the saliency feature representation of the full focus map based on implicit graph learning. The calculation formula is: Among them, the This represents the adaptive layer corresponding to the full focus map. X represents the transformation matrix corresponding to the full-focus image. R This represents the fully focused training feature map;

[0049] Based on the expression of the focus-based stack graph Obtain the focal stack saliency feature representation based on implicit graph learning The calculation formula is: Among them, the This indicates the adaptive layer corresponding to the focus stack graph. X represents the transformation matrix corresponding to the focus stack diagram. F This represents the training feature map of the focus stack;

[0050] Using implicit graph loss function The focal stack saliency feature representation, which enables the feature map to learn through training, to approximate the full focal map saliency feature representation, is calculated using the following formula:

[0051]

[0052] Wherein, L2 represents the L2 loss function, and l represents the number of layers in the focus stack backbone and the full focus backbone;

[0053] When the number of training epochs reaches the first preset value, training is complete, and a trained two-stream deep convolutional network is obtained.

[0054] In one implementation, the training steps of the contour-aware model include:

[0055] Obtain the focus stack feature map and full-focus feature map

[0056] Two 3x3 convolutional layers and the ReLU activation function were used, and the number of channels was uniformly adjusted to 128 to obtain the first focal stack feature. and the first fully focused feature Where l = 3, 4, 5;

[0057] Compact aggregation modules are obtained through 3D depthwise separable convolutions with different dilation rates. The outputs of different paths are fused with ReLU activation function through element-wise summation to obtain enhanced hierarchical focus stack features. The calculation formula is:

[0058]

[0059] Among them, the This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 1. This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 2. This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 3. This represents a transformation layer used to convert the dimension of the focus stack from N×128 to 1×128 using a 3×3 convolution. The first focal stack feature is represented by l, which is the number of layers in the focal stack backbone network, and the value is 3, 4, or 5.

[0060] Multiple interactive feature fusion modules are used to mutually promote the fusion of each layered focus stack feature and each first full focus feature to obtain the initial fusion feature of the focus stack and the initial fusion feature of the full focus.

[0061] Based on the contour prototype representation, the initial fusion feature of the focus stack is processed into a refined fusion feature of the focus stack, and the initial fusion feature of full focus is processed into a refined fusion feature of full focus;

[0062] The focal stack refinement fusion feature and the full focus refinement fusion feature are aggregated using a saliency detection head to obtain saliency prediction results and object contour prediction results;

[0063] When the number of training epochs reaches the second preset value, training is complete, and the trained contour perception model is obtained.

[0064] In one implementation, if the input to the interactive feature fusion module is... use Significance prediction And implement auxiliary deep supervision, the formula is: Wherein, the Conv 3×3 This represents a 3×3 convolutional layer, where U represents bilinear interpolation upsampling;

[0065] Object contour features are generated by predicting saliency maps. The calculation formula is: Among them, the It consists of a 3×3 convolution, a Groupnorm layer, and a PReLU activation function, used to transform the number of channels from 1 to 128;

[0066] The depth supervision is applied to the object contour prediction, and the calculation formula is as follows:

[0067] The importance of the channel response evaluation characteristics is calculated using the following formula: Where (m,n) represents the coordinate position of the feature map, w k This represents the weight of the k-th channel of the feature map F;

[0068] Through features Generate weighted response The response of different features is dynamically adjusted using an adaptive enhancement operation. The calculation formula includes:

[0069]

[0070]

[0071] Among them, the The FC represents a fully connected layer, and the U represents bilinear interpolation upsampling. Indicates the initial fusion features of full focus;

[0072] The initial fusion feature of full focus is improved by embedding contour hints in the global view. The calculation formula is as follows:

[0073] Wherein, Conv represents a convolutional layer with a 3×3 kernel size, and the This indicates a fully focused, refined fusion feature;

[0074] The significance prediction result S P The calculation formula is:

[0075] Among them, the This indicates a fully focused, refined fusion feature, the aforementioned This indicates a focus stack refinement fusion feature;

[0076] The object contour prediction result S C The calculation formula is:

[0077] The present invention also provides a light field saliency target detection device based on implicit graph learning, comprising:

[0078] The feature map acquisition module is used to input the full-focus image to be tested and the focus stack image to be tested into a pre-trained two-stream deep convolutional network to obtain the full-focus feature map and the focus stack feature map.

[0079] The calculation module is used to input the full-focus feature map and the focus stack feature map into a pre-trained contour perception model to obtain saliency prediction results and object contour prediction results.

[0080] The present invention also provides a terminal, comprising: a memory, a processor, and a light field saliency target detection program based on implicit graph learning stored in the memory and executable on the processor, wherein when the light field saliency target detection program based on implicit graph learning is executed by the processor, the steps of the light field saliency target detection method based on implicit graph learning as described above are implemented.

[0081] The present invention also provides a computer-readable storage medium storing a computer program that can be executed to implement the steps of the implicit graph learning-based light field saliency target detection method as described above.

[0082] This invention provides a method and apparatus for salient object detection in light fields based on implicit graph learning. The method includes: inputting a full-focus image and a focus stack image to be tested into a pre-trained two-stream deep convolutional network to obtain a full-focus feature map and a focus stack feature map; and inputting the full-focus feature map and the focus stack feature map into a pre-trained contour-aware model to obtain a saliency prediction result and an object contour prediction result. This invention promotes the mining of salient features by inputting the full-focus image and focus stack image to be tested into a pre-trained two-stream deep convolutional network. Furthermore, inputting the full-focus feature map and the focus stack feature map into the pre-trained contour-aware model can effectively aggregate information of mixed scales, thereby obtaining accurate salient object detection results in light fields. Attached Figure Description

[0083] Figure 1 This is a flowchart of a preferred embodiment of the light field saliency target detection method based on implicit graph learning in this invention.

[0084] Figure 2 This is a block diagram illustrating the principles of a two-stream deep convolutional network and a contour-aware model.

[0085] Figure 3 This is a block diagram of the RFFM principle.

[0086] Figure 4 This is the test result table for the experiment.

[0087] Figure 5 This is a functional principle block diagram of a preferred embodiment of the light field saliency target detection device based on implicit graph learning in this invention.

[0088] Figure 6 This is a functional principle block diagram of the terminal in this invention. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0090] In traditional methods, the first benchmark for light field saliency detection was established, and a groundbreaking method was proposed by computing handcrafted focus features. A weighted sparse coding network was proposed, and a dictionary was used to simultaneously solve the RGB, RGB-D, and LF SOD problems. The saliency of depth contrast and color contrast was computed for saliency detection. Light field flow cues were employed to explore depth contrast information, and depth contrast was enhanced by using position priors. While earlier traditional methods have made significant progress, their reliance on handcrafted cues for light field saliency target detection limits their versatility in complex scenes.

[0091] Recently, deep learning-based networks have become mainstream and have greatly improved the accuracy of salient object detection in light fields. Based on the architecture of the inference process, most focus-stack-based methods employ a two-stream framework for salient object detection in light fields. Some methods use a late-stage fusion network to extract features from both the focus stack backbone and the full-focus backbone, then combine them with a ConvLSTM in the final step. Existing methods either follow a separate two-stream approach, using a memory-oriented decoder for intermediate fusion, or propose a lightweight refinement and integration module to aggregate features extracted from both backbones. Similarly, some design a local graph decoder with a recursive fusion scheme for information fusion. Furthermore, some transfer focus information to a single full-focus network to improve computational efficiency. However, incomplete or even incorrect responses from the focus stack hinder accurate performance in salient object detection in light fields.

[0092] In summary, existing focus stack-based SOD models only use the skeleton of a single focus stack to extract features. They are negatively affected by blurred saliency-related regions and contours within the focus stack, leading to insufficient exploration of informative features. Furthermore, most methods rarely consider the quality of object contours, resulting in coarse saliency contours.

[0093] This invention can effectively prevent misleading by ambiguous regions and effectively mine and utilize the feature information of the focus stack.

[0094] Please see Figure 1 , Figure 1 This is a flowchart of the light field saliency target detection method based on implicit graph learning in this invention. For example... Figure 1 As shown in the embodiment of the present invention, the light field saliency target detection method based on implicit graph learning includes the following steps:

[0095] Step S100: Input the full-focus image to be tested and the focus stack image to be tested into a pre-trained two-stream deep convolutional network to obtain the full-focus feature map and the focus stack feature map.

[0096] Specifically, the dual-stream deep convolutional network can employ backbones such as ResNet34, ResNet18, ResNet50, ResNet101, and Transformer. The dual-stream deep convolutional network is used to extract salient feature information from the fully focused image and the focused stack image in layers.

[0097] In one implementation, the dual-stream deep convolutional network includes a full-focus backbone and a focus stack backbone; step S100 specifically includes: inputting the full-focus image to be tested into the pre-trained full-focus backbone to obtain a full-focus feature map, and inputting the focus stack image to be tested into the pre-trained focus stack backbone to obtain a focus stack feature map.

[0098] Specifically, the outputs of the fully focused backbone and the focused stack backbone are respectively expressed as: l represents the number of layers in the fully focused backbone and the focused stacked backbone, with values ​​of 2, 3, 4, and 5. The pre-trained two-stream deep convolutional network utilizes an implicit graph representation learning strategy. Unlike typical LF SOD models that use independent backbones for feature extraction, this method facilitates the focused stacked network in mining representative salient-related features and enhances saliency perception. More importantly, the implicit graph representation learning strategy only needs to be used during training and does not require additional computation or parameters during inference, thus achieving better performance in light field salient object detection.

[0099] In one embodiment, such as Figure 2 As shown, the training steps of the dual-stream deep convolutional network include:

[0100] A1. Obtain the training dataset, which includes: full-focus training images and focus stack training images.

[0101] A2. Input the fully focused training image and the focus stack training image into the initial two-stream deep convolutional network, which includes a fully focused backbone and a focus stack backbone.

[0102] A3. Output the fully focused training feature map from the fully focused backbone network. Output the focus stack training feature map from the focus stack backbone network. Where l represents the number of layers in the fully focused backbone and the focused stack backbone, with values ​​of 2, 3, 4, and 5.

[0103] A4. Construct a full-focus map G based on the full-focus training feature map. r The G r ={A r ,F r}, where A rThe F represents the adjacency matrix of the full-focus graph. r The nodes represent the full focus map; a focus stack map G is constructed based on the focus stack training feature map. f ={Af,Ff}, where A f The adjacency matrix representing the focus stack graph, the F f This represents a node in the focus stack graph.

[0104] Here, the adjacency matrix A represents the similarity between pairs of nodes. Representing a self-connected adjacency matrix, by We obtain, where I is the identity matrix.

[0105] A5. By modeling a graph neural network to establish and infer the relationships between nodes in a fully focused graph, and encouraging connections between fully focused graphs to mine salient feature representations, the formula is expressed as:

[0106]

[0107] Wherein, the Θ r The learnable weight matrix representing the full-focused image, the The angle matrix representing the full focus image, the This represents an expression based on a full-focus image.

[0108] By modeling graph neural networks to establish and infer the relationships between nodes in a focus stack graph, and encouraging connections between focus stack graphs to mine salient feature representations, the formula is expressed as follows:

[0109]

[0110] Wherein, the Θ f The learnable weight matrix representing the focus stack graph, the The angle matrix representing the focus stack graph, the This represents a focus-based stack graph representation.

[0111] Specifically, diag represents a diagonal matrix.

[0112] A6. Based on the expression of the full focus map Obtain the saliency feature representation of the full focus map based on implicit graph learning. The calculation formula is: Among them, the This represents the adaptive layer corresponding to the full focus map. X represents the transformation matrix corresponding to the full-focus image. R This represents the fully focused training feature map. It is based on the representation of the focus stack graph. Obtain the focal stack saliency feature representation based on implicit graph learning The calculation formula is: Among them, the This indicates the adaptive layer corresponding to the focus stack graph. X represents the transformation matrix corresponding to the focus stack diagram. F This represents the training feature map of the focus stack.

[0113] Specifically, to achieve a more stable training process conducive to optimization, an adaptive layer and a residual connection are proposed. The transformation matrix projects the features of the graph space to the original feature space through graph backprojection, and the adaptive layer is implemented through a combination of a 1×1 convolution and a ReLU activation function. The reasons for designing this adaptive layer are as follows: 1) It adjusts the number of channels in the full-focus network to match the number of channels in the focal stack network for distance calculation. 2) This operation can promote stable improvement in network training.

[0114] A7. Using the implicit graph loss function The focal stack saliency feature representation, which enables the feature map to learn through training, to approximate the full focal map saliency feature representation, is calculated using the following formula:

[0115]

[0116] Here, L2 represents the L2 loss function, and l represents the number of layers in the focus stack backbone and the fully focused backbone. It should be noted that the layers are cut off here. The backpropagation gradient flow is used because only the focus stack network is needed to mine important salient features.

[0117] A8. When the number of training epochs reaches the first preset value, training is complete, and the trained two-stream deep convolutional network is obtained.

[0118] The implicit graph representation learning strategy designed in this invention fully models and infers the relationships between contextual content, mitigating the effects of ambiguity and interference while forming accurate and discriminative feature representations. Implicit graph representation learning can also be implemented using other distillation strategies, not limited to graph distillation, such as pixel-wise distillation and non-local-wise distillation.

[0119] After step S100, the following step is: Step S200, inputting the full-focus feature map and the focus stack feature map into the pre-trained contour perception model to obtain the saliency prediction result and the object contour prediction result.

[0120] Specifically, for accurate salient object detection in the light field, fusion between full-focus features and focus stack features (i.e., multi-modal features) is crucial. Previous methods rarely considered utilizing object contour cues, which is helpful for better SOD (Saliency Detection). The lack of explicit utilization of contour cues can result in coarse saliency maps. Due to the fuzzy nature of focus slices, the contours of focus slices and full-focus salient objects often differ in quality. Direct fusion and contour supervision may limit the utilization of complementarity and saliency performance. Therefore, the decoder in this embodiment not only effectively integrates multimodal features but also explicitly utilizes object contour information.

[0121] In one implementation, step S200 specifically includes:

[0122] Step S210: In the pre-trained contour perception model, 2D convolutional layers and ReLU activation function are used to process each focal stack salient feature in the focal stack feature map into a first focal stack feature, and each full focus salient feature in the full focus feature map into a first full focus feature.

[0123] Step S220: Process each of the first focus stack features using a compact aggregation module to obtain enhanced hierarchical focus stack features;

[0124] Step S230: Use multiple interactive feature fusion modules to mutually promote the fusion of each layered focus stack feature and each first full focus feature to obtain the initial fusion feature of the focus stack and the initial fusion feature of the full focus.

[0125] Step S240: Based on the contour prototype expression, process the initial fusion feature of the focus stack into a refined fusion feature of the focus stack, and process the initial fusion feature of full focus into a refined fusion feature of full focus;

[0126] Step S250: Use the saliency detection head to aggregate the focus stack refinement fusion feature and the full focus refinement fusion feature to obtain the saliency prediction result and the object contour prediction result.

[0127] Specifically, this embodiment proposes a mutually reinforcing and interactive contour-aware fusion strategy, reciprocal contour-aware fusion (RCF). First, for more effective focal slice feature representation, multi-scale and multi-slice (i.e., mixed-scale) feature integration is crucial. To this end, a Compact Aggregation Module (CAM) is designed, employing 3D-dilated depthwise separable convolutions instead of ordinary 2D convolutions. By designing a multi-path learning scheme, CAM can effectively aggregate mixed-scale information to enhance the feature representation of the focal stack. Second, since multimodal features exhibit different useful representations, an interactive feature fusion module (RFFM) is proposed. Instead of using concatenation or summation for multimodal feature fusion, this invention proposes adaptively adjusting the weights of different features to fully utilize complementarity and generate representative saliency features. Third, to further improve the saliency detection results for objects with sharp contours, a contour-hint injection mechanism is developed to accurately embed contour cues into the above fusion process. Guided by contour cues, the network is induced to understand global image content, thereby filtering out unnecessary background interference and better locating salient areas in complex scenes.

[0128] In one embodiment, in step S220, the calculation formula for the hierarchical focus stack feature is:

[0129]

[0130] Among them, the This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 1. This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 2. This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 3. This represents a transformation layer used to convert the dimension of the focus stack from N×128 to 1×128 using a 3×3 convolution. This represents the first focal stack feature, where l is the number of layers in the focal stack backbone, taking values ​​of 3, 4, or 5. In this way, Enhanced hierarchical focus stack features can be obtained by feeding in CAMs of different layers.

[0131] Specifically, for the focus stack feature map and full-focus feature map XR 5 First, two 3x3 convolutional layers and the ReLU activation function are used, and the number of channels is uniformly adjusted to 128 to obtain the first focal stack feature. and the first fully focused feature Where l = 3, 4, 5. For example... Figure 2 As shown, the proposed CAM consists of three parallel paths, implemented through 3D depthwise separable convolutions with different dilation rates. The outputs of different paths are fused using element-wise summation and a ReLU activation function. This yields enhanced hierarchical focus stack features.

[0132] In one implementation, multimodal, multi-level feature fusion is achieved gradually using several RFFMs to obtain more refined salient feature representations. For example... Figure 3 As shown, in step S230, if the input of the interactive feature fusion module is... use Significance prediction And implement auxiliary deep supervision, the formula is: Wherein, the Conv 3×3 This represents a 3×3 convolutional layer, where U represents bilinear interpolation upsampling.

[0133] Object contour features are generated by predicting saliency maps. The calculation formula is: Among them, the It consists of a 3×3 convolution, a Groupnorm layer, and a PReLU activation function, used to transform the number of channels from 1 to 128. This forces the generated contours to be closer to salient objects.

[0134] The depth supervision is applied to the object contour prediction, and the calculation formula is as follows: This helps generate more accurate contours, and depth-assisted supervision helps network training, thus achieving better optimization.

[0135] In RFFM, a reciprocal fusion process is designed to adaptively fuse complementary features, thereby enhancing the representation of useful salient features. Specifically, the importance of features is evaluated by calculating the channel responses, using the following formula: Where (m,n) represents the coordinate position of the feature map, w k This represents the weight of the k-th channel of the feature map F.

[0136] Assume the input of RFFM is Through features Generate weighted response The response to different features is dynamically adjusted using an adaptive augmentation operation, which can be expressed as follows:

[0137]

[0138]

[0139] Among them, the The term represents a concatenation operation, FC represents a fully connected layer, and U represents bilinear interpolation upsampling. This indicates the initial fusion feature of full focus.

[0140] Thus, this embodiment can generate fully focused initial fusion features. Adaptively fuse multimodal complementary saliency features and object contour features.

[0141] In one embodiment, to further ensure the accuracy of this fusion process, this embodiment enhances the initial fusion feature based on a contour prototype representation. It refines the initial fusion feature by embedding contour hints from the global view. Step S240 specifically includes: refining the fully focused initial fusion feature by embedding contour hints from the global view, such as... Figure 3 As shown in (c), this process can be calculated as: Wherein, Conv represents a convolutional layer with a 3×3 kernel size, and the This represents the full-focus refinement fusion feature. In this way, the full-focus refinement fusion feature can be obtained. Similarly, as... Figure 2 As shown, by implementing several RFFMs, reciprocal refined features can be gradually generated, and the salient feature representation is gradually enhanced and refined.

[0142] In one implementation, a saliency detection head is used to aggregate and refine fused features. and This is used to obtain the final saliency prediction result and object contour prediction result. The saliency prediction result S... P The calculation formula is: Among them, the This indicates a fully focused, refined fusion feature, the aforementioned This represents the focus stack refinement and fusion feature. The object contour prediction result S... C The calculation formula is:

[0143] With the combination of the IGL method and RCF strategy proposed in this embodiment, the network of this embodiment can further mine important focal stack features and make full use of complementary saliency features and object contour information to promote light field saliency detection.

[0144] In one embodiment, the training steps of the contour-aware model include:

[0145] Obtain the focus stack feature map and full-focus feature map

[0146] Two 3x3 convolutional layers and the ReLU activation function were used, and the number of channels was uniformly adjusted to 128 to obtain the first focal stack feature. and the first fully focused feature Where l = 3, 4, 5;

[0147] Compact aggregation modules are obtained through 3D depthwise separable convolutions with different dilation rates. The outputs of different paths are fused with ReLU activation function through element-wise summation to obtain enhanced hierarchical focus stack features. The calculation formula is:

[0148]

[0149] Among them, the This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 1. This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 2. This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 3. This represents a transformation layer used to convert the dimension of the focus stack from N×128 to 1×128 using a 3×3 convolution. The first focal stack feature is represented by l, which is the number of layers in the focal stack backbone network, and the value is 3, 4, or 5.

[0150] Multiple interactive feature fusion modules are used to mutually promote the fusion of each layered focus stack feature and each first full focus feature to obtain the initial fusion feature of the focus stack and the initial fusion feature of the full focus.

[0151] Based on the contour prototype representation, the initial fusion feature of the focus stack is processed into a refined fusion feature of the focus stack, and the initial fusion feature of full focus is processed into a refined fusion feature of full focus;

[0152] The focal stack refinement fusion feature and the full focus refinement fusion feature are aggregated using a saliency detection head to obtain saliency prediction results and object contour prediction results;

[0153] When the number of training epochs reaches the second preset value, training is complete, and the trained contour perception model is obtained.

[0154] This embodiment designs a mutually reinforcing and interactive contour-aware fusion (RCF) strategy to effectively aggregate complementary features and explicitly utilize contour information, thereby further improving the performance and robustness of light field saliency target detection.

[0155] In one embodiment, if the input to the interactive feature fusion module is use Significance prediction And implement auxiliary deep supervision, the formula is: Wherein, the Conv 3×3 This represents a 3×3 convolutional layer, where U represents bilinear interpolation upsampling;

[0156] Object contour features are generated by predicting saliency maps. The calculation formula is: Among them, the It consists of a 3×3 convolution, a Groupnorm layer, and a PReLU activation function, used to transform the number of channels from 1 to 128;

[0157] The depth supervision is applied to the object contour prediction, and the calculation formula is as follows:

[0158] The importance of the channel response evaluation characteristics is calculated using the following formula: Where (m,n) represents the coordinate position of the feature map, w k This represents the weight of the k-th channel of the feature map F;

[0159] Through features Generate weighted response The response of different features is dynamically adjusted using an adaptive enhancement operation. The calculation formula includes:

[0160]

[0161]

[0162] Among them, the The FC represents a fully connected layer, and the U represents bilinear interpolation upsampling. Indicates the initial fusion features of full focus;

[0163] The initial fusion feature of full focus is improved by embedding contour hints in the global view. The calculation formula is as follows:

[0164] Wherein, Conv represents a convolutional layer with a 3×3 kernel size, and the This indicates a fully focused, refined fusion feature;

[0165] The significance prediction result S P The calculation formula is:

[0166] Among them, the This indicates a fully focused, refined fusion feature, the aforementioned This indicates a focus stack refinement fusion feature;

[0167] The object contour prediction result S C The calculation formula is:

[0168] The implicit graph representation learning in this embodiment enhances feature extraction from the focus stack feature backbone; and the mutually reinforcing and interactive contour perception strategy enhances feature fusion.

[0169] This invention was experimentally tested on three publicly available light field SOD image datasets: HFUT, DUTLF, and LFSD. Four commonly used evaluation metrics—maximum E-measure, S-measure, maximum F-measure, and MAE—were used for comparison. The experimental results are as follows: Figure 4 As shown in the figure. The experimental results demonstrate that the light field saliency target detection method based on implicit graph learning proposed in this embodiment has significant advantages.

[0170] Furthermore, such as Figure 5 As shown, based on the above-mentioned method for detecting salient targets in a light field based on implicit graph learning, this invention also provides a corresponding device for detecting salient targets in a light field based on implicit graph learning, comprising:

[0171] The feature map acquisition module 100 is used to input the full-focus image to be tested and the focus stack image to be tested into a pre-trained two-stream deep convolutional network to obtain the full-focus feature map and the focus stack feature map.

[0172] The calculation module 200 is used to input the full-focus feature map and the focus stack feature map into a pre-trained contour perception model to obtain saliency prediction results and object contour prediction results.

[0173] like Figure 6As shown, the present invention also provides a terminal, including: a memory 20, a processor 10, and an implicit graph learning-based light field saliency target detection program 30 stored in the memory 20 and executable on the processor 10. When the implicit graph learning-based light field saliency target detection program 30 is executed by the processor 10, it implements the steps of the implicit graph learning-based light field saliency target detection method as described above.

[0174] The present invention also provides a computer-readable storage medium storing a computer program that can be executed to implement the steps of the implicit graph learning-based light field saliency target detection method as described above.

[0175] In summary, this invention discloses a method and apparatus for salient object detection in light fields based on implicit graph learning. The method includes: inputting a full-focus image and a focus stack image to be tested into a pre-trained two-stream deep convolutional network to obtain a full-focus feature map and a focus stack feature map; and inputting the full-focus feature map and the focus stack feature map into a pre-trained contour-aware model to obtain a saliency prediction result and an object contour prediction result. This invention promotes the mining of salient features by inputting the full-focus image and focus stack image to be tested into a pre-trained two-stream deep convolutional network. Furthermore, inputting the full-focus feature map and the focus stack feature map into a pre-trained contour-aware model can effectively aggregate information at mixed scales, thereby obtaining accurate salient object detection results in light fields.

[0176] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for detecting salient targets in a light field based on implicit graph learning, characterized in that, include: The fully focused image and the focus stack image to be tested are input into a pre-trained two-stream deep convolutional network to obtain the fully focused feature map and the focus stack feature map. The full-focus feature map and the focus stack feature map are input into a pre-trained contour perception model to obtain saliency prediction results and object contour prediction results. The training steps for the dual-stream deep convolutional network include: Obtain a training dataset, which includes: fully focused training images and focus stack training images; The fully focused training image and the focus stack training image are input into an initial two-stream deep convolutional network, which includes a fully focused backbone and a focus stack backbone. Output the fully focused training feature map from the fully focused backbone network. Output the focus stack training feature map from the focus stack backbone network. ,in, The number of layers for the fully focused backbone and the focused stack backbone, with values ​​of 2, 3, 4, and 5; A total focus map is constructed based on the total focus training feature map. The , wherein The adjacency matrix represents the full focus graph. Nodes representing the full focus graph; A focus stack map is constructed based on the focus stack training feature map. , wherein The adjacency matrix represents the focus stack graph. Nodes representing the focus stack graph; By modeling a graph neural network to establish and infer the relationships between nodes in a fully focused graph, and by mining salient feature representations through the connections between fully focused graphs, the formula is expressed as follows: ； Among them, the The learnable weight matrix representing the full-focused image, the The angle matrix representing the full focus image, the This represents an expression based on a full-focus image; The relationships between nodes in the focus stack graph are established and inferred by modeling a graph neural network, and salient feature representations are mined through the connections between the focus stack graphs. The formula is expressed as follows: ； Among them, the The learnable weight matrix representing the focus stack graph, the The angle matrix representing the focus stack graph, the This represents a focus-based stack graph representation. Based on the expression of the full focus map Obtain the saliency feature representation of the full focus map based on implicit graph learning. The calculation formula is: ; wherein, the This represents the adaptive layer corresponding to the full focus map. This represents the transformation matrix corresponding to the full focus image. This represents the fully focused training feature map; According to the expression based on the focus stack graph Obtain the focal stack saliency feature representation based on implicit graph learning The calculation formula is: ; wherein, the This indicates the adaptive layer corresponding to the focus stack graph. The transformation matrix corresponding to the focus stack diagram, the This represents the training feature map of the focus stack; Using implicit graph loss function The focal stack saliency feature representation, which enables the feature map to learn through training, to approximate the full focal map saliency feature representation, is calculated using the following formula: ； Among them, the represent loss function, the Indicates the number of layers in the focused stack backbone and the fully focused backbone; When the number of training epochs reaches the first preset value, training is complete, and a trained two-stream deep convolutional network is obtained.

2. The light field saliency target detection method based on implicit graph learning according to claim 1, characterized in that, The dual-stream deep convolutional network includes a fully focused backbone and a focus stack backbone; The process of inputting the full-focus image and the focus stack image to be tested into a pre-trained two-stream deep convolutional network to obtain the full-focus feature map and the focus stack feature map includes: The full-focus image to be tested is input into a pre-trained full-focus backbone to obtain a full-focus feature map, and the focus stack image to be tested is input into a pre-trained focus stack backbone to obtain a focus stack feature map.

3. The light field saliency target detection method based on implicit graph learning according to claim 1, characterized in that, The fully focused feature map and the focus stack feature map are input into a pre-trained contour-aware model to obtain saliency prediction results and object contour prediction results, including: In the pre-trained contour perception model, 2D convolutional layers and ReLU activation functions are used to process each focal stack salient feature in the focal stack feature map into a first focal stack feature, and each full focus salient feature in the full focus feature map into a first full focus feature. A compact aggregation module is used to process each of the first focus stack features to obtain enhanced hierarchical focus stack features; Multiple interactive feature fusion modules are used to mutually promote the fusion of each layered focus stack feature and each first full focus feature to obtain the initial fusion feature of the focus stack and the initial fusion feature of the full focus. Based on the contour prototype representation, the initial fusion feature of the focus stack is processed into a refined fusion feature of the focus stack, and the initial fusion feature of full focus is processed into a refined fusion feature of full focus; The saliency detection head is used to aggregate the focus stack refinement fusion feature and the full focus refinement fusion feature to obtain the saliency prediction result and the object contour prediction result.

4. The light field saliency target detection method based on implicit graph learning according to claim 3, characterized in that, The calculation formula for the hierarchical focus stack feature is as follows: ； Among them, the This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 1. This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 2. This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 3. This represents a transformation layer used to convert the dimension of the focus stack from N×128 to 1×128 using a 3×3 convolution. Indicates the first focus stack feature, the This represents the number of layers in the focus stack backbone, with values ​​of 3, 4, or 5.

5. The light field saliency target detection method based on implicit graph learning according to claim 3, characterized in that, The method employs multiple interactive feature fusion modules to mutually promote the fusion of each hierarchical focus stack feature and each of the first full-focus features, obtaining initial focus stack fusion features and initial full-focus fusion features, including: If the input to the interactive feature fusion module is ,use Significance prediction And implement auxiliary deep supervision, the formula is: ; wherein, the This represents a 3×3 convolutional layer. This indicates bilinear interpolation upsampling; Object contour features are generated by predicting saliency maps. The calculation formula is: ; wherein, the It consists of a 3×3 convolution, a Groupnorm layer, and a PReLU activation function, used to transform the number of channels from 1 to 128; The depth supervision is applied to the object contour prediction, and the calculation formula is as follows: ; The importance of the channel response evaluation characteristics is calculated using the following formula: ;in Represents the coordinate position of the feature map. This represents the weight of the k-th channel of the feature map F; Through features Generate weighted response It dynamically adjusts the responses of different features using an adaptive enhancement operation, and the calculation formula includes: ； ； Among them, the Represents a serial operation, the Representing a fully connected layer, the Indicates bilinear interpolation upsampling, the This indicates the initial fusion feature of full focus.

6. The light field saliency target detection method based on implicit graph learning according to claim 5, characterized in that, The process of processing the initial focus stack fusion feature into a refined focus stack fusion feature based on the contour prototype representation, and processing the initial full focus fusion feature into a refined full focus fusion feature, includes: The initial fusion feature of full focus is improved by embedding contour hints in the global view. The calculation formula is as follows: ; Among them, the Representing a convolutional layer with a 3×3 kernel size, the This indicates a fully focused, refined fusion feature.

7. The light field saliency target detection method based on implicit graph learning according to claim 6, characterized in that, The significance prediction results The calculation formula is: ; Among them, the This indicates a fully focused, refined fusion feature, the aforementioned This indicates a focus stack refinement fusion feature; The object contour prediction result The calculation formula is: .

8. The light field saliency target detection method based on implicit graph learning according to claim 1, characterized in that, The training steps of the contour-aware model include: Obtain the focus stack feature map and full-focus feature map ; Two 3x3 convolutional layers and the ReLU activation function were used, and the number of channels was uniformly adjusted to 128 to obtain the first focal stack feature. and the first fully focused feature ,in ; Compact aggregation modules are obtained through 3D depthwise separable convolutions with different dilation rates. The outputs of different paths are fused with ReLU activation function through element-wise summation to obtain enhanced hierarchical focus stack features. The calculation formula is: ； Among them, the This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 1. This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 2. This represents a depthwise separable convolution with a 3×3×3 3D dilation and a hole rate of 3. This represents a transformation layer used to convert the dimension of the focus stack from N×128 to 1×128 using a 3×3 convolution. Indicates the first focus stack feature, the This represents the number of layers in the focus stack backbone, with values ​​of 3, 4, or 5. Multiple interactive feature fusion modules are used to mutually promote the fusion of each layered focus stack feature and each first full focus feature to obtain the initial fusion feature of the focus stack and the initial fusion feature of the full focus. Based on the contour prototype representation, the initial fusion feature of the focus stack is processed into a refined fusion feature of the focus stack, and the initial fusion feature of full focus is processed into a refined fusion feature of full focus; The focal stack refinement fusion feature and the full focus refinement fusion feature are aggregated using a saliency detection head to obtain saliency prediction results and object contour prediction results; When the number of training epochs reaches the second preset value, training is complete, and the trained contour perception model is obtained.

9. The light field saliency target detection method based on implicit graph learning according to claim 8, characterized in that, If the input to the interactive feature fusion module is ,use Significance prediction And implement auxiliary deep supervision, the formula is: ; wherein, the This represents a 3×3 convolutional layer. This indicates bilinear interpolation upsampling; Object contour features are generated by predicting saliency maps. The calculation formula is: ; wherein, the It consists of a 3×3 convolution, a Groupnorm layer, and a PReLU activation function, used to transform the number of channels from 1 to 128; The depth supervision is applied to the object contour prediction, and the calculation formula is as follows: ; The importance of the channel response evaluation characteristics is calculated using the following formula: ;in Represents the coordinate position of the feature map. This represents the weight of the k-th channel of the feature map F; Through features Generate weighted response It dynamically adjusts the responses of different features using an adaptive enhancement operation, and the calculation formula includes: ； ； Among them, the Represents a serial operation, the Representing a fully connected layer, the Indicates bilinear interpolation upsampling, the Indicates the initial fusion features of full focus; The initial fusion feature of full focus is improved by embedding contour hints in the global view. The calculation formula is as follows: ; Among them, the Representing a convolutional layer with a 3×3 kernel size, the This indicates a fully focused, refined fusion feature; The significance prediction results The calculation formula is: ; Among them, the This indicates a fully focused, refined fusion feature, the aforementioned This indicates a focus stack refinement fusion feature; The object contour prediction result The calculation formula is: .

10. A light field saliency target detection device based on implicit graph learning, characterized in that, include: The feature map acquisition module is used to input the full-focus image to be tested and the focus stack image to be tested into a pre-trained two-stream deep convolutional network to obtain the full-focus feature map and the focus stack feature map. The calculation module is used to input the full-focus feature map and the focus stack feature map into a pre-trained contour perception model to obtain saliency prediction results and object contour prediction results; The training steps for the dual-stream deep convolutional network include: Obtain a training dataset, which includes: fully focused training images and focus stack training images; The fully focused training image and the focus stack training image are input into an initial two-stream deep convolutional network, which includes a fully focused backbone and a focus stack backbone. Output the fully focused training feature map from the fully focused backbone network. Output the focus stack training feature map from the focus stack backbone network. ,in, The number of layers for the fully focused backbone and the focused stack backbone, with values ​​of 2, 3, 4, and 5; A total focus map is constructed based on the total focus training feature map. The , wherein The adjacency matrix represents the full focus graph. Nodes representing the full focus graph; A focus stack map is constructed based on the focus stack training feature map. , wherein The adjacency matrix represents the focus stack graph. Nodes representing the focus stack graph; By modeling a graph neural network to establish and infer the relationships between nodes in a fully focused graph, and by mining salient feature representations through the connections between fully focused graphs, the formula is expressed as follows: ； Among them, the The learnable weight matrix representing the full-focused image, the The angle matrix representing the full focus image, the This represents an expression based on a full-focus image; The relationships between nodes in the focus stack graph are established and inferred by modeling a graph neural network, and salient feature representations are mined through the connections between the focus stack graphs. The formula is expressed as follows: ； Among them, the The learnable weight matrix representing the focus stack graph, the The angle matrix representing the focus stack graph, the This represents a focus-based stack graph representation. Based on the expression of the full focus map Obtain the saliency feature representation of the full focus map based on implicit graph learning. The calculation formula is: ; wherein, the This represents the adaptive layer corresponding to the full focus map. This represents the transformation matrix corresponding to the full focus image. This represents the fully focused training feature map; According to the expression based on the focus stack graph Obtain the focal stack saliency feature representation based on implicit graph learning The calculation formula is: ; wherein, the This indicates the adaptive layer corresponding to the focus stack graph. The transformation matrix corresponding to the focus stack diagram, the This represents the training feature map of the focus stack; Using implicit graph loss function The focal stack saliency feature representation, which enables the feature map to learn through training, to approximate the full focal map saliency feature representation, is calculated using the following formula: ； Among them, the represent loss function, the Indicates the number of layers in the focused stack backbone and the fully focused backbone; When the number of training epochs reaches the first preset value, training is complete, and a trained two-stream deep convolutional network is obtained.

11. A terminal, characterized in that, include: The system includes a memory, a processor, and a light field saliency target detection program based on implicit graph learning, which is stored in the memory and can run on the processor. When the processor executes the implicit graph learning-based light field saliency target detection program, it implements the steps of the light field saliency target detection method based on implicit graph learning as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed to implement the steps of the light field saliency target detection method based on implicit graph learning as described in any one of claims 1 to 9.

Images