Light field salient target detection method based on edge perception and hierarchical fusion

Abstract

The invention relates to the technical field of light field image salient target detection, in particular to a light field salient target detection method based on edge perception and hierarchical fusion, and the method comprises the steps: carrying out the multi-scale feature extraction of a focus stack image and a full-focus image through a backbone network; performing edge fusion enhancement on the focus stack features of four layers of different scales through an SEPM module and an EFM module; fusing high-level multi-modal semantic information from the global and local aspects by using an LHFM module; fusing low-layer space information and refining a salient target by using an LLFM module; and aggregating multi-scale information of a high layer and a low layer, and decoding the multi-scale information into an accurate saliency prediction image by using a detection head. According to the method, edge perception and a lightweight hierarchical fusion strategy are combined, the model parameter quantity and the calculation complexity are remarkably reduced while the high detection performance is kept, and the optimal balance between the performance and the efficiency is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of salient object detection in light field images, and in particular to a light field salient object detection method based on edge perception and layered fusion. Background Art

[0002] Multimodal salient object detection (SOD) aims to fuse multi-source visual information to accurately locate the most attractive objects or regions in an image. It has been widely used in many visual tasks, including object detection, semantic segmentation, depth estimation, and image quality assessment. In recent years, two research directions have emerged: RGB-D depth salient object detection (RGB-D SOD) and light field salient object detection (LF SOD), using red, green, and blue (RGB) images, depth images, and light field images as input modalities. Existing RGB-D SOD models have gradually diverged into two development directions: heavyweight and lightweight. Heavyweight RGB-D SOD algorithms achieve high performance through high computational cost and large model size, but this design severely limits their edge deployment and real-time applications. Lightweight RGB-D SOD methods focus on balancing speed and accuracy, achieving high detection accuracy with low parameters and computational complexity, enabling faster inference efficiency and reduced hardware dependence, and holding great promise for mobile applications and real-time tasks.

[0003] Light field images, rich in focus, disparity, and angular information, demonstrate unique potential for salient object detection. In contrast, existing research on light field salient object detection, a higher-dimensional multimodal vision task, remains dominated by heavyweight architectures. Specifically, existing methods commonly employ heavyweight two-stream encoders (such as ResNet, VGG, and PVT) and complex multimodal fusion modules, which rely on significant computational resources to extract and fuse multimodal light field information. Consequently, these LF SOD methods suffer from large model parameters and high computational complexity, making them difficult to apply to resource-constrained or low-power scenarios. Furthermore, light field images inherently contain a significant amount of redundant information, particularly in focus stack images, resulting in duplication and waste in feature computation. Existing methods generally lack design mechanisms to model and compress this redundant structure, resulting in slow progress in lightweight approaches. Therefore, exploring lightweight light field salient object detection methods is of great practical significance.

[0004] Due to the continuous loss of high-frequency information during feature extraction and downsampling, existing methods, while achieving superior performance in overall saliency prediction, still suffer from common problems such as blurred edges of salient objects and missing structural details. These methods perform particularly poorly against complex backgrounds or when the object boundary is close to the background. Edge information, as a direct representation of salient object structure, helps enhance the model's structural perception and improve the accuracy of salient object localization and segmentation in complex backgrounds. Furthermore, edge maps are inherently sparse and highly guided. Introducing an edge supervision mechanism not only does not significantly increase model complexity, but can actually improve the model's training efficiency and generalization capabilities, making it suitable for lightweight network design.

[0005] In summary, how to effectively utilize edge information and multimodal light field information to achieve higher detection performance while significantly reducing the number of model parameters and computational complexity has become an urgent problem that technicians in this field need to solve. Summary of the Invention

[0006] In view of the above-mentioned shortcomings of the existing technology, the technical problem to be solved by the present invention is: how to provide a light field salient target detection method based on edge perception and hierarchical fusion, combining edge perception and lightweight hierarchical fusion strategy, while maintaining high detection performance, significantly reducing the number of model parameters and computational complexity, and achieving the best balance between performance and efficiency.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A light field salient object detection method based on edge perception and hierarchical fusion, comprising:

[0009] S1: Acquire a focus stack image and a fully focused image containing the target to be detected;

[0010] S2: Input the focus stack image and the all-focus image into the trained object detection model and output the corresponding salient object detection image;

[0011] The processing steps of the object detection model include:

[0012] S201: performing step-by-step feature extraction on the focus stack image and the all-focus image using K feature extraction modules through the backbone network to obtain K-scale focus stack feature maps and all-focus feature maps;

[0013] S202: performing edge prediction on the focal stack feature maps of K scales through K SEPM modules to obtain edge feature maps of K scales; fusing the edge feature maps of the K scales in sequence to obtain an edge fusion feature map;

[0014] S203: fusing the edge fusion feature map with the K-scale focal stack feature map step by step through the cascaded K EFM modules to obtain the K-scale focal-edge fusion feature map;

[0015] S204: Dividing the focus-edge fusion feature maps and the full focus feature map of K scales into high-level features and low-level features according to the scales of the feature maps;

[0016] S205: performing deep semantic fusion and complementation on the focus-edge fusion feature maps and the full focus feature maps of N scales belonging to high-level features through N LHFM modules, thereby obtaining high-level fusion feature maps of N scales;

[0017] S206: performing deep semantic fusion and detail enhancement on the focus-edge fusion feature maps and the full focus feature map of M scales belonging to the low-level features through M LLFM modules, thereby obtaining low-level fusion feature maps of M scales;

[0018] S207: The decoder uses K-1 cascaded decoding modules to decode the minimum-scale high-level fusion feature map step by step, and during the decoding process, the high-level fusion feature map or the low-level fusion feature map of the corresponding scale is input into the decoding module of the corresponding scale for decoding, to obtain the final multi-scale fusion feature map;

[0019] S208: Input the multi-scale fusion feature map into the detection head for processing to obtain a corresponding salient object detection image.

[0020] Preferably, in step S201, a lightweight MobileViT-S with the pooling layer and the last fully connected layer deleted is selected as the backbone network.

[0021] Preferably, in step S202, the SEPM module includes a cascaded MPCR module, a PDB module, a Sobel convolution layer and a PBR module;

[0022] The MPCR module is used to extract features in different receptive fields in parallel, and obtain local details and global structural information at the same time;

[0023] The PDB module consists of a cascaded central differential convolution module, an angular differential convolution module, a radial differential convolution module, and a 3×3 convolution layer;

[0024] The PBR module consists of a cascade of point-by-point convolutional layers, batch normalization layers, and ReLU functions.

[0025] Preferably, in step S202, the processing steps of the MPCR module include:

[0026] S2021: Input the input feature map into the point-by-point convolution layer for processing, and split the obtained feature map into four feature sub-maps according to the channel;

[0027] S2022: Input the four feature sub-graphs into the four branches for processing respectively, and concatenate the obtained four feature graphs along the channel dimension to obtain a restored feature graph; each branch includes a cascaded partial convolution layer with expansion and a point-by-point convolution layer;

[0028] S2023: Input the restored feature map into the point-by-point convolution layer for processing, and obtain the feature map as the output of the MPCR module.

[0029] Preferably, in step S203, the processing steps of the EFM module include:

[0030] S2031: Get the input edge fusion feature map e final and the focal stack feature map of the i-th scale

[0031] S2032: Edge fusion feature map e final Use bilinear interpolation to the focal stack feature map Same spatial resolution;

[0032] The formula is:

[0033] e(i)=U(e final );

[0034] Where: e(i) represents the edge fusion feature map after interpolation;

[0035] S2033: The interpolated edge fusion feature map e(i) and the focal stack feature map are fused using a bidirectional dense fusion strategy Perform fusion to obtain a preliminary fusion feature map;

[0036] The formula is:

[0037]

[0038] Where: f e (i) represents the preliminary fusion feature map; Represents element-wise multiplication;

[0039] S2034: After the preliminary fusion feature map is processed step by step through the cascaded partial convolution layer, PBR module and channel attention enhancement module, it is element-wise multiplied with the output of the PBR module to obtain a focus-edge fusion feature map; wherein the channel attention enhancement module includes a cascaded pooling layer, a point-by-point convolution layer and a sigmoid function;

[0040] The formula is:

[0041]

[0042] Where: represents the focus-edge fusion feature map of the i-th scale; PConv() represents partial convolution; CAE() represents the channel attention enhancement operation; PBR() represents the PBR module.

[0043] Preferably, in step S205, the processing steps of the LHFM module include:

[0044] S2051: Get the focus-edge fusion feature map of the i-th scale of the input and the all-focus feature map F a (i);

[0045] S2052: Fusion of focus-edge feature maps After the input is processed by the cascaded 3×3 partial convolution layer, PBR module and point-by-point convolution layer, it is added element-by-element with the output of the 3×3 partial convolution layer to obtain the focus-edge refinement feature map F fs-H (i);

[0046] The formula is:

[0047]

[0048] Where: F fs-H (i) represents the focus-edge refinement feature map; PIRB ( ) represents the combined refinement operation of the inverted residual structure consisting of a 3×3 partial convolutional layer, a PBR module, and a point-by-point convolutional layer;

[0049] S2053: The fully focused feature map F after the Batch dimension expansion operation a (i) After the input is processed by the cascaded 3×3 partial convolution layer, PBR module and point-by-point convolution layer, it is added element-by-element with the output of the 3×3 partial convolution layer to obtain the fully focused refined feature map F a-H (i);

[0050] The formula is:

[0051] F a-H (i)=PIRB(replicate(F a (i)));

[0052] Where: F a-H (i) represents the fully focused refined feature map; replicate() represents the expansion operation of the Batch dimension;

[0053] S2054: Focus point-edge refinement feature map F fs-H (i) and the fully focused refined feature map F a-H(i) After element-by-element addition, the input is fed into the partial convolution attention module to implement a lightweight convolution operation on some channels, retaining the necessary local receptive field enhancement, and obtaining a high-level local fusion feature map F H-local (i);

[0054] S2055: Refine the focus-edge feature map F fs-H (i) and the fully focused refined feature map F a-H (i) Input them together into the proxy attention mechanism module to realize the focus-edge refinement feature map F fs-H (i) and the fully focused refined feature map F a-H (i) The high-level features of the global attention interaction are used to obtain the global branch fusion feature map F H-global (i);

[0055] S2056: high-level local fusion feature map F H-local (i) and the global branch fusion feature map F H-global (i) After the splicing operation along the channel dimension, the image is input into the PBR module for processing to obtain a high-level fusion feature map;

[0056] The formula is:

[0057] F H (i) = PBR(cat(F H-global (i),F H-local (i)));

[0058] Where: F H (i) represents the high-level fusion feature map of the i-th scale output by the LHFM module; cat( ) represents the splicing operation along the channel dimension; PBR( ) represents the PBR module.

[0059] Preferably, in step S2054, the partial convolutional attention module includes a cascaded PCA module, a PBR module, a point-by-point convolution layer, a batch normalization layer, and a PSA module; wherein the feature map obtained by element-wise multiplication of the original input of the partial convolutional attention module and the output of the PSA module is used as the output of the partial convolutional attention module;

[0060] The processing logic of the PCA module is as follows: the input feature map is split into two feature sub-maps; the first feature sub-map is sequentially input into the point-by-point convolution layer and the H-Sigmoid function for processing, the resulting feature map is multiplied pixel by pixel with the first feature sub-map itself, and then processed through a batch normalization layer to obtain the processing result of the first feature sub-map; the second sub-map is input into the point-by-point convolution layer for processing, the resulting feature map is concatenated with the processing result of the first feature sub-map along the channel dimension to obtain the output of the PCA module;

[0061] The processing logic of the PSA module is as follows: the input feature map is split into two feature sub-maps; after the results of std and mean processing of the first sub-map are spliced, the spliced ​​feature map is input into the point-by-point convolution layer and the H-Sigmoid function for processing to obtain the feature map, which is multiplied pixel by pixel with the first feature sub-map itself, and then processed through a batch normalization layer to obtain the processing result of the first feature sub-map; the second feature sub-map is input into the 3×3 convolution layer for processing to obtain the feature map, which is spliced ​​with the processing result of the first feature sub-map along the channel dimension to obtain the output of the PSA module.

[0062] Preferably, in step S2055, the calculation formula of the proxy attention mechanism module is as follows:

[0063] F H-global (i) = Atten s (Q,Q A ,Atten s (Q A ,K,V))+DWConv(V);

[0064] Q=Linear(F fs-H (i));

[0065] Q A =Pool(Q);

[0066] K=Linear(F a-H (i));

[0067] V=Linear(F a-H (i));

[0068] Where: F H-global (i) Represents the global branch fusion feature map output by the proxy attention mechanism module; Atten s ( ) represents the Softmax attention operation; DWConv( ) represents the depth-wise separable convolutional layer; Linear( ) represents linear mapping of features; Pool( ) represents adaptive global average pooling; F fs-H (i) F a-H (i) represent the input focus-edge refinement feature map and full focus refinement feature map, respectively.

[0069] Preferably, in step S206, the processing steps of the LLFM module include:

[0070] S2061: Obtain the focus-edge fusion feature map of the i-th scale of the input and the all-focus feature map F a (i);

[0071] S2062: The fully focused feature map F after the Batch dimension expansion operation a (i) Input the feature map obtained by the PBR module and the focus-edge fusion feature map The feature map obtained by inputting the PBR module for processing is spliced ​​along the channel dimension to obtain the preliminary fusion feature map F L-f (i);

[0072] The formula is:

[0073]

[0074] Where: cat( ) represents the splicing operation along the channel;

[0075] S2063: Initially fusion feature map F L-f (i) Input the feature map obtained by processing the 3×3 depth-separable convolution layer and the preliminary fusion feature map F L-f (i) Input the feature map obtained by point-by-point convolution layer and the preliminary fusion feature map F L-f (i) The feature maps obtained by inputting the PSA module layer for processing are added element by element and then input into the PBR module for processing to obtain the low-level fusion feature map;

[0076] The formula is:

[0077] F L (i) = PBR(DS(DS(F L-f (i)))+PW(F L-f (i))+PSA(F L-f (i)));

[0078] Where: F L (i) represents the low-level fused feature map of the i-th scale output by the LLFM module; DS( ) represents depthwise separable convolution; PSA( ) represents partial spatial attention mechanism; PW( ) represents point-wise convolution; PBR( ) represents PBR module.

[0079] Preferably, in step S207, the decoding module of the decoder is a compact pyramid refinement module;

[0080] The calculation formula of the decoder is expressed as:

[0081] S final =CPR(F L (1),CPR(F L (2),CPR(F H (3),F H (4))));

[0082] Where: Sfinal Represents the final multi-scale fusion feature map; CPR() represents the CPR fusion operation; F H (i) represents the high-level fusion feature map of the i-th scale; F L (i) represents the low-level fusion feature map of the i-th scale.

[0083] Compared with the prior art, the light field salient object detection method based on edge perception and layered fusion in this invention has the following advantages:

[0084] Beneficial effects:

[0085] The present invention aims to solve the problems of high-frequency information loss of salient targets and lack of lightweight design in existing light field salient target detection methods, and proposes a light field salient target detection method based on edge perception and hierarchical fusion. On the one hand, the existing light field salient target detection methods generally focus on multimodal feature fusion, ignoring the high-frequency edge information of salient targets, resulting in edge blur and structure loss in complex backgrounds or low-contrast scenes. The present invention combines the complete depth and structural characteristics of the focal stack image, introduces a slice-guided edge prediction module (SEPM module) and an edge fusion module (EFM module), respectively predicts rough edge information from the focal stack image, and fuses it with the focal stack features, guiding the network to focus on the structural details of the salient target, thereby effectively improving the detection accuracy of the edge area and the integrity of the target contour. On the other hand, most of the existing light field salient target detection methods rely on complex multimodal fusion modules, resulting in a significant increase in the number of model parameters and computational complexity, making it difficult to apply to resource-constrained scenarios. Based on the different characteristics of high-level and low-level features, this paper constructs a lightweight hierarchical fusion strategy, and designs a lightweight low-level fusion module (LLFM module) and a high-level fusion module (LHFM module) respectively, realizing multimodal feature fusion from shallow spatial details to deep semantics. While maintaining the lightweight of the network, this strategy effectively alleviates the problems of information redundancy and feature misalignment, and enhances the model's ability to perceive salient objects of different scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to make the purpose, technical solutions and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0087] Figure 1 This is the network architecture diagram of the target detection model.

[0088] Figure 2 Schematic diagram of the slice-guided edge prediction module and edge fusion module.

[0089] Figure 3 Schematic diagram of the lightweight high-level fusion module.

[0090] Figure 4 Schematic diagram of a partial convolutional attention block.

[0091] Figure 5 Schematic diagram of the agent attention mechanism.

[0092] Figure 6 Schematic diagram of the lightweight low-level fusion module.

[0093] Figure 7 The figure shows the prediction results of the model of the present invention and other methods in challenging scenarios. DETAILED DESCRIPTION

[0094] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0095] It should be noted that similar reference numerals and letters denote similar items in the following figures. Therefore, once an item is defined in one figure, it does not require further definition or explanation in subsequent figures. In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" indicate positions or relationships based on the positions or relationships shown in the figures, or the positions or relationships in which the inventive product is typically placed when in use. These terms are intended solely to facilitate the description of the present invention and simplify the description. They do not indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation, and are therefore not to be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third," etc., are used solely to distinguish descriptions and are not to be construed as indicating or implying relative importance. Furthermore, terms such as "horizontal" and "vertical" do not imply that a component is absolutely horizontal or overhanging, but rather may be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but rather may be slightly tilted. In the description of the present invention, it should also be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections or electrical connections; they may refer to direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0096] The following is a further detailed description through specific implementation methods:

[0097] Example:

[0098] This embodiment discloses a light field salient object detection method based on edge perception and layered fusion.

[0099] like Figure 1 As shown in FIG, the light field salient object detection method based on edge perception and hierarchical fusion includes:

[0100] S1: Acquire a (light field multimodal) focus stack image and a fully focused image containing the target to be detected;

[0101] S2: Input the focus stack image and the all-focus image into the trained object detection model and output the corresponding salient object detection image;

[0102] The processing steps of the object detection model include:

[0103] S201: Using the backbone network, K (set to 4) feature extraction modules are used to extract features from the focus stack image and the fully focused image step by step, obtaining focus stack feature maps and fully focused feature maps of K scales; each feature extraction module outputs a focus stack feature map or fully focused feature map of the corresponding scale;

[0104] S202: performing edge prediction on the K-scale focal stack feature maps respectively through K focal slice-guided edge prediction modules (SEPMs) to obtain K-scale edge feature maps; fusing the K-scale edge feature maps in sequence to obtain an edge fusion feature map;

[0105] S203: The edge fusion feature map is sequentially fused with the focal stack feature maps of K scales through a cascade of K edge fusion modules (EFMs) to obtain focal-edge fusion feature maps of K scales; each EFM module outputs a focal-edge fusion feature map of the corresponding scale;

[0106] S204: Dividing the focus-edge fusion feature maps and the full focus feature map of K scales into high-level features and low-level features according to the scales of the feature maps;

[0107] S205: performing deep semantic fusion and complementation on the focus-edge fusion feature maps and the full focus feature maps of the N scales belonging to the high-level features through N (set to 2) lightweight high-level fusion modules (LHFM) modules, respectively, to obtain high-level fusion feature maps of N scales; each LHFM module outputs a high-level fusion feature map of the corresponding scale;

[0108] S206: performing deep semantic fusion and detail enhancement on the focus-edge fusion feature maps and the full focus feature maps of the M scales belonging to the low-level features through M (set to 2) lightweight low-level fusion modules (LLFM) to obtain low-level fusion feature maps of M scales; each LLFM module outputs a low-level fusion feature map of the corresponding scale;

[0109] Where N+M=K;

[0110] S207: The decoder uses K-1 cascaded decoding modules to decode the minimum-scale high-level fusion feature map step by step, and during the decoding process, the high-level fusion feature map or the low-level fusion feature map of the corresponding scale is input into the decoding module of the corresponding scale for decoding, to obtain the final multi-scale fusion feature map;

[0111] S208: Input the multi-scale fusion feature map into the detection head for processing to obtain a corresponding salient object detection image.

[0112] The present invention discloses a light field salient target detection method based on edge perception and hierarchical fusion. The method inputs a multimodal focal stack image and a fully focused image into a trained light field salient target detection model and outputs a saliency prediction image. The training steps include: extracting multi-scale features through a lightweight backbone network; implementing edge fusion enhancement of four layers of focal stack features at different scales through a slice-guided edge prediction module and an edge fusion module; utilizing a lightweight high-level fusion module to fuse high-level multimodal semantic information globally and locally; utilizing a lightweight low-level fusion module to fuse low-level spatial information and refine salient targets; utilizing a compact pyramid refinement module to aggregate high-level and low-level multi-scale information, and utilizing a lightweight saliency prediction head to decode the information into an accurate saliency prediction image; upsampling the saliency prediction image to the same resolution as the true image, and calculating the loss with the true image to train the network.

[0113] The present invention aims to solve the problems of high-frequency information loss of salient targets and lack of lightweight design in existing light field salient target detection methods, and proposes a light field salient target detection method based on edge perception and hierarchical fusion. On the one hand, the existing light field salient target detection methods generally focus on multimodal feature fusion, ignoring the high-frequency edge information of salient targets, resulting in edge blur and structure loss in complex backgrounds or low-contrast scenes. The present invention combines the complete depth and structural characteristics of the focal stack image, introduces a slice-guided edge prediction module (SEPM module) and an edge fusion module (EFM module), respectively predicts rough edge information from the focal stack image, and fuses it with the focal stack features, guiding the network to focus on the structural details of the salient target, thereby effectively improving the detection accuracy of the edge area and the integrity of the target contour. On the other hand, most of the existing light field salient target detection methods rely on complex multimodal fusion modules, resulting in a significant increase in the number of model parameters and computational complexity, making it difficult to apply to resource-constrained scenarios. Based on the different characteristics of high-level and low-level features, this paper constructs a lightweight hierarchical fusion strategy, and designs a lightweight low-level fusion module (LLFM module) and a high-level fusion module (LHFM module) respectively, realizing multimodal feature fusion from shallow spatial details to deep semantics. While maintaining the lightweight of the network, this strategy effectively alleviates the problems of information redundancy and feature misalignment, and enhances the model's ability to perceive salient objects of different scales.

[0114] In order to better introduce the technical solution of the present invention, this embodiment is described through the following parts.

[0115] 1. Dataset

[0116] In this example, the three public light field datasets used for training and testing the object detection model are DUTLF-FS, HFUT, and LFSD. To ensure fairness, a unified setting is adopted: 1000 samples from DUTLF-FS and 100 samples from HFUT are selected for training, and the remaining samples are all used for testing. All light field samples consist of focus stack images, all-focus images, and corresponding ground-truth images, with a uniform resolution of 256×256. The data is augmented by random rotation, cropping, and flipping before being fed into the network for training.

[0117] 2. Backbone Network

[0118] To adapt to the task of lightweight salient object detection, this paper selects the lightweight MobileViT-S with the pooling layer and the last fully connected layer deleted as the backbone network, extracts the focus stack and full focus features respectively, and obtains feature maps of four different scales, namely 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image.

[0119] 3. SEPM module and EFM module

[0120] In order to solve the problem of loss of high-frequency information of salient targets in the process of feature extraction and downsampling of light field salient target detection, the present invention combines the characteristics of more comprehensive depth representation of focal stack images without being affected by image quality, and designs SEPM modules and EFM modules, such as Figure 2 The SEPM module and EFM module not only effectively preserve the edge details of salient targets, but also overcome the problem of traditional convolution suppressing high-frequency signals, providing more accurate boundary guidance for subsequent saliency segmentation.

[0121] 1. SEPM module

[0122] like Figure 2 As shown in the figure, the SEPM module includes a cascaded multi-dilated partial convolution refinement module (MPCR) module, a pixel difference convolution block (PDB) module, a Sobel convolution layer and a PBR (PWConv+Batch Normalization+ReLU) module;

[0123] The MPCR module is used to extract features under different receptive fields in parallel, and simultaneously obtain local details (small expansion rate) and global structure (large expansion rate) information;

[0124] The PDB module consists of a cascaded Central Difference Convolution (CDConv) module, an Angular Difference Convolution (ADConv) module, a Radial Difference Convolution (ADConv) module, and a 3×3 convolution layer.

[0125] The PBR module consists of a cascade of point-by-point convolutional layers, batch normalization layers, and ReLU functions.

[0126] The calculation formula of the SEPM module is expressed as:

[0127]

[0128] e pred (i) = PBR(Sobel(e enhance (i)));

[0129] e final =σ(e pred (1)+U(e pred (2))+U(e pred (3))+U(e pred (4))); Different feature maps need to be sampled to the same level as e by bilinear interpolation. pred (1) Only the same resolution can be added.

[0130] Where: e final Represents the edge fusion feature map output by the SEPM module; e pred (i) represents the edge feature map of the i-th scale; Sobel ( ) represents Sobel convolution; PBR ( ) represents PBR module; σ ( ) represents Sigmoid activation; PDB ( ) represents pixel difference convolution module; MPCR ( ) represents multi-expansion partial convolution refinement module; represents the focal stack feature map of the i-th scale of the input; e enhance (i) represents the edge enhancement feature map; U( ) represents the bilinear interpolation sampling; n represents the number of focal stack images (from 1 to 12); i represents the number of multi-scale feature layers (from 1 to 4).

[0131] In order to avoid excessive computational costs, the four-layer dual-stream features of MobileViT-S are compressed to 32 channels through a simple pointwise convolution (PWConv) operation.

[0132] The focus slice guided edge prediction module (SEPM module) consists of edge enhancement and edge prediction parts. In the edge enhancement part, it is often difficult to capture global or medium-to-near range structural information by relying solely on convolution of a small receptive field, and large kernel convolution will bring huge computational overhead. Therefore, the present invention proposes a multi-expansion partial convolution refinement module (MPCR module) to extract features under different receptive fields in parallel, so as to obtain local details (small expansion rate) and global structure (large expansion rate) information in one module at the same time, which greatly improves the ability of edge features to describe cross-scale structures. The standard 3×3 convolution operation tends to smooth low-frequency components, and will gradually weaken high-frequency information such as edges after stacking multiple layers. The computational cost brought by the extensive use of this convolution operation is huge, and it is difficult to apply to lightweight network design. Therefore, the present invention adopts a pixel difference convolution module (PDB module) for edge enhancement, wherein the pixel difference convolution block consists of a center difference convolution module, an angle difference convolution module, a radial difference convolution module and a 3×3 convolution. The three difference convolutions are convolved from the center, angle and radial directions respectively, and can capture gradient components in different directions. Compared with the 3×3 convolution group, the pixel difference convolution module has a stronger edge perception ability and avoids the huge parameter and computational overhead brought by the large kernel convolution, providing a clearer and more discriminative feature representation for subsequent edge prediction.

[0133] Combine Figure 2 As shown, the processing steps of the MPCR module include:

[0134] S2021: Input the input feature map into the point-by-point convolution layer for processing, and split the obtained feature map into four feature sub-maps according to the channel;

[0135] S2022: Input the four feature sub-graphs into the four branches for processing respectively, and concatenate the obtained four feature graphs along the channel dimension to obtain a restored feature graph; each branch includes a cascaded dilated partial convolution layer and a point-by-point convolution layer, and the dilation rates of the dilated partial convolution layers in the four branches are d1, d2, d3, and d4, respectively;

[0136] S2023: Input the restored feature map into the point-by-point convolution layer for processing, and obtain the feature map as the output of the MPCR module.

[0137] In edge prediction, the present invention uses the Sobel convolution layer to accurately locate pixel grayscale mutations through horizontal and vertical filtering at extremely low computational cost, thereby quickly obtaining a preliminary gradient map of the target contour. However, its fixed kernel weight cannot adapt to the complex background and diversity of multi-scale salient targets in light field images. To this end, the PBR module that follows it performs channel mapping on the gradient map extracted by the Sobel convolution layer through point-by-point convolution, combines batch normalization (BN) to stabilize the feature distribution, and enhances the gradient response and suppresses noise through the ReLU activation function, thereby achieving adaptive enhancement of diverse edge details while maintaining lightweight. In order to make full use of structural information at different levels, the edge features from four scales are first upsampled to a uniform resolution, then multi-scale fusion is achieved by element-by-element addition, and finally the aggregated features are input into the edge detection head to generate the final coarse edge prediction map. This design can capture the complete edge structure information of salient targets while ensuring a lightweight network architecture.

[0138] 2. EFM module

[0139] Simply superimposing edge features onto the semantic features of the backbone network through element-by-element addition or multiplication often results in the edge information being diluted or mixed with background noise during the deep convolution process, thus failing to effectively improve the boundary accuracy of saliency segmentation. To this end, this paper proposes an edge fusion module (EFM module), which aims to deeply fuse the fine-grained edge features obtained from the SEPM module with the focus features extracted from the light field backbone through explicit feature interaction and channel attention mechanism, enabling the network to perceive the edge texture information of salient targets.

[0140] In order to achieve deep coupling of edge and focus stack features, the EFM module first performs edge feature extraction on the output of the SEPM module. final Bilinear interpolation is used to the focal stack features of the backbone network The module then introduces a bidirectional dense fusion strategy: the interpolated edge features and focus features are element-wise multiplied and fed back to each other via a residual connection to explicitly strengthen the mutual guidance between the two different feature information. Finally, the two enhanced features are element-wise added to achieve a preliminary fusion of edge details and global semantics.

[0141] To further refine the fusion results while maintaining lightweight, inspired by the partial convolution mechanism (PConv), this paper uses a combination of PConv and PBR to achieve the refinement of fusion features. Among them, PConv only performs conventional 3×3 convolution on some channels, and the remaining channels are directly spliced ​​with the convolution channels, and then refined by PBR. This significantly reduces the number of parameters and computational complexity while ensuring detection accuracy. Finally, the module adaptively assigns weights to each channel through the channel attention mechanism, further strengthening the complementary information across channels, and outputs the final result of focus-edge fusion features.

[0142] Combine Figure 2 As shown, the processing steps of the EFM module include:

[0143] S2031: Get the input edge fusion feature map e final and the focal stack feature map of the i-th scale

[0144] S2032: Edge fusion feature map e final Use bilinear interpolation to the focal stack feature map Same spatial resolution;

[0145] The formula is:

[0146] e(i)=U(e final );

[0147] Where: e(i) represents the edge fusion feature map after interpolation;

[0148] S2033: The interpolated edge fusion feature map e(i) and the focal stack feature map are fused using a bidirectional dense fusion strategy Perform fusion to obtain a preliminary fusion feature map;

[0149] The formula is:

[0150]

[0151] Where: f e (i) represents the preliminary fusion feature map; Represents element-wise multiplication;

[0152] S2034: After the preliminary fusion feature map is processed step by step through the cascaded partial convolution layer, PBR module and channel attention enhancement module, it is element-wise multiplied with the output of the PBR module to obtain a focus-edge fusion feature map; wherein the channel attention enhancement module includes a cascaded pooling layer, a point-by-point convolution layer and a Sigmiod function; the PBR module includes a cascaded point-by-point convolution layer, a batch normalization layer and a ReLU function;

[0153] The formula is:

[0154]

[0155] Where: represents the focus-edge fusion feature map of the i-th scale; PConv( ) represents partial convolution; CAE( ) represents the channel attention enhancement operation; PBR( ) represents the PBR module.

[0156] 4. LHFM module

[0157] In multimodal feature fusion, features at different levels have significant differences in resolution, receptive field and semantic information. Existing lightweight RGB-D SOD methods usually only fuse deep semantics, because deep features contain most of the information of salient targets and can effectively control computational costs while ensuring accuracy. However, deep semantics will lose a lot of high-frequency information (such as edges, textures, etc.) during the downsampling process of feature extraction. Such methods ignore the fine depiction of high-frequency details by shallow features, which often leads to texture loss or blurred edges in saliency maps. In contrast, although heavyweight networks can integrate features of all levels through high-computational-cost fusion modules, their huge number of parameters and computational complexity make them difficult to deploy in resource-constrained environments and lightweight networks. In order to balance accuracy and efficiency, the present invention proposes a hierarchical fusion strategy: customizing fusion modules based on the contributions of high-level and low-level features in the task of salient target detection.

[0158] High-level features contain rich semantic information and are key information for locating salient targets. There are two major challenges in fusing the high-level features of the focus stack branch and the full focus branch: both high-level features carry global semantic information, and fusion without differentiation processing is likely to cause feature redundancy; conventional modules (such as multi-head self-attention or a large number of convolution combination fusion) will significantly increase the number of model parameters and computational complexity (Floating-point Operations Per Second, FLOPs), which does not meet the lightweight design goal. To this end, the present invention proposes a lightweight high-level fusion module (LHFM module), such as Figure 3The LHFM module integrates the partial convolutional attention mechanism (a combination of partial spatial and partial channel attention) and the agent attention mechanism, aiming to achieve efficient fusion and complementarity of deep semantics from both local and global perspectives while ensuring low computational overhead.

[0159] like Figure 3 As shown, the processing steps of the LHFM module include:

[0160] S2051: Get the focus-edge fusion feature map of the i-th scale of the input and the all-focus feature map F a (i);

[0161] S2052: Fusion of focus-edge feature maps After the input is processed by the cascaded 3×3 partial convolution layer, PBR module and point-by-point convolution layer, it is added element-by-element with the output of the 3×3 partial convolution layer to obtain the focus-edge refinement feature map F fs-H (i);

[0162] Among them, the PBR module includes a cascade of point-by-point convolutional layers, batch normalization layers, and ReLU functions;

[0163] The formula is:

[0164]

[0165] Where: F fs-H (i) represents the focus-edge refinement feature map; PIRB ( ) represents the combined refinement operation of the inverted residual structure consisting of a 3×3 partial convolution layer, a PBR module, and a point-by-point convolution layer. The channel expansion factor of the inverted residual structure is set to 2;

[0166] S2053: The fully focused feature map F after the Batch dimension expansion operation a (i) After the input is processed by the cascaded 3×3 partial convolution layer, PBR module and point-by-point convolution layer, it is added element-by-element with the output of the 3×3 partial convolution layer to obtain the fully focused refined feature map F a-H (i);

[0167] The formula is:

[0168] F a-H (i)=PIRB(replicate(F a (i)));

[0169] Where: F a-H (i) represents the fully focused refined feature map; replicate( ) represents the expansion operation of the Batch dimension;

[0170] S2054: Focus point-edge refinement feature map F fs-H (i) and the fully focused refined feature map F a-H (i) After element-by-element addition, the input is fed into the partial convolution attention module to implement a lightweight convolution operation on some channels, retaining the necessary local receptive field enhancement, and obtaining a high-level local fusion feature map F H-local (i);

[0171] S2055: Refine the focus-edge feature map F fs-H (i) and the fully focused refined feature map F a-H (i) Input them together into the proxy attention mechanism module to realize the focus-edge refinement feature map F fs-H (i) and the fully focused refined feature map F a-H (i) The high-level features of the global attention interaction are used to obtain the global branch fusion feature map F H-global (i);

[0172] S2056: high-level local fusion feature map F H-local (i) and the global branch fusion feature map F H-global (i) After the splicing operation along the channel dimension, the image is input into the PBR module for processing to obtain a high-level fusion feature map;

[0173] The formula is:

[0174] F H (i) = PBR(cat(F H-global (i),F H-local (i)));

[0175] Where: F H (i) represents the high-level fusion feature map of the i-th scale output by the LHFM module; cat( ) represents the concatenation operation along the channel dimension.

[0176] Specifically, given the focal feature and the full focusing feature F a (i) Since multimodal features differ in the Batch dimension, the present invention expands the fully focused features by 12 times in the Batch dimension to align the multimodal features, and reduces the number of channels of the dual-stream features by half to avoid excessively high computational costs. Before fusion, the multimodal features are respectively subjected to lightweight PConv and inverted residual structures to refine the dual-stream features. The inverted residual structure first uses PBR to increase the dimension in the channel dimension, then uses PWConv to reduce the dimension to the original number of channels, and finally sets the residual connection to perform element-by-element addition with the result of PConv. This structural design improves the feature expression capability of the lightweight module without significantly increasing the computational cost.

[0177] 1. Partial Convolution Attention Module

[0178] In the local branch fusion, the present invention introduces a partial convolution attention module (Partial Block), such as Figure 4 As shown in the figure, the partial convolutional attention module consists of partial channel attention (PCA) and partial spatial attention (PSA). Its core idea is to apply lightweight convolution operations to some channels to retain the necessary local receptive field enhancement, while the remaining channels are guided and enhanced using the less computationally expensive spatial or channel attention mechanism. Compared with traditional 3×3 convolution, this structural design can significantly reduce computational overhead while effectively enhancing the network's ability to distinguish foreground from background. Specifically, the focus features and fully focused features are first initially fused through element-by-element addition. The fused features are then fed into the partial convolutional attention block: the PCA module is responsible for screening semantically key channels in the channel dimension to improve channel attention response; while the PSA module highlights high-frequency structural details such as edges and textures of salient objects in the spatial dimension.

[0179] like Figure 4 As shown in the figure, the partial convolution attention module includes a cascaded PCA module, a PBR module, a point-by-point convolution layer, a batch normalization layer, and a PSA module; the feature map obtained by element-wise multiplication of the original input of the partial convolution attention module and the output of the PSA module is used as the output of the partial convolution attention module;

[0180] The processing logic of the PCA module is as follows: the input feature map is split into two feature sub-maps; the first feature sub-map is sequentially input into the point-by-point convolution layer and the H-Sigmoid function for processing, the resulting feature map is multiplied pixel by pixel with the first feature sub-map itself, and then processed through a batch normalization layer to obtain the processing result of the first feature sub-map; the second sub-map is input into the point-by-point convolution layer for processing, the resulting feature map is concatenated with the processing result of the first feature sub-map along the channel dimension to obtain the output of the PCA module;

[0181] The processing logic of the PSA module is as follows: the input feature map is split into two feature sub-maps; after the results of std and mean processing of the first sub-map are spliced, the spliced ​​feature map is input into the point-by-point convolution layer and the H-Sigmoid function for processing to obtain the feature map, which is multiplied pixel by pixel with the first feature sub-map itself, and then processed through a batch normalization layer to obtain the processing result of the first feature sub-map; the second feature sub-map is input into the 3×3 convolution layer for processing to obtain the feature map, which is spliced ​​with the processing result of the first feature sub-map along the channel dimension to obtain the output of the PSA module.

[0182] 2. Agent Attention Mechanism Module

[0183] In high-level feature fusion, it is still difficult to fully grasp the overall semantics of salient targets by relying only on the partial convolutional attention module of the local branch to supplement the edge and texture details. In many heavyweight networks, the self-attention mechanism with strong global context modeling ability is often used to design the fusion module, but it is difficult to deploy it in lightweight networks due to its quadratic computational complexity. Based on this, the present invention introduces a lightweight agent attention mechanism (Agent Attention) module with strong global modeling ability to design the global fusion module, such as Figure 5 As shown, it aims to perform global attention interaction on the high-level features output by the focus stack stream and the full focus stream, and avoid excessive computational cost.

[0184] Specifically, before fusion, the dual-stream features are transformed from feature form (R 12×C×H×W ) is converted into Token form (R 12 ×N×C ), where N = H × W, H, W and C represent the height, width and number of channels of the image respectively. Different from traditional self-attention, this method designs cross-attention fusion. The focus stack flow captures rich depth and focus clues on different focal planes. When used as a query Q, it can accurately guide the model to focus on the salient target position; the full focus flow has global clarity and consistent semantic expression. As the key K and value V, it can ensure that stable and complete contextual information is extracted from the global field of view. In order to avoid additional computational costs, the present invention adopts a simple pooling operation to generate the core proxy Token (Q A ). It can be expressed as:

[0185] Q=Linear(F fs-H (i));

[0186] Q A =Pool(Q);

[0187] K=Linear(F a-H (i));

[0188] V=Linear(F a-H (i));

[0189] Where Linear() represents linear mapping of features, and Pool() represents adaptive global average pooling.

[0190] Q after average pooling A ∈R 12×n×C Compared to Q∈R 12×N×CFor only a small part of the Token value, where N>n. The complexity of the attention calculation of Q and K in the traditional Softmax attention is O(N 2 C), and Q A The complexity of the attention calculation with K is only O(NnC), which has linear complexity while maintaining the ability to model global context. The proxy attention mechanism module includes two Softmax attention calculations. In the first stage, Q A As the query Q, perform a Softmax attention calculation with K and V. In the second stage, Q A As K A , the output of the first Sofmax attention is V A , and Q performs a Softmax attention calculation, so both Softmax calculations have only linear computational complexity. Although linear complexity attention calculation is friendly to lightweight networks, its feature diversity is limited. The fully focused feature has complete spatial information. Therefore, the present invention restores feature diversity by performing a residual connection between V obtained by the fully focused feature mapping and the final Softmax result through depthwise convolution (DWConv). The entire process can be expressed as:

[0191] F H-global (i) = Atten s (Q,QA,Atten s (Q A ,K,V))+DWConv(V);

[0192] Among them F H-global (i) represents the global branch fusion result, Atten s ( ) represents the Softmax attention operation, and DWConv( ) represents the depthwise convolution.

[0193] 5. LLFM module

[0194] Low-level features have rich local spatial details and primary semantic information, which are used to restore high-frequency information such as texture and contour of salient targets lost during feature extraction and downsampling. Compared with deep-level features, their contribution is relatively low, so the present invention adopts a more lightweight module for fusion. If the dual-stream features are directly added or spliced, irrelevant noise of the low-frequency background will often be introduced or high-frequency salient target details will be suppressed. Although the traditional 3×3 convolution and spatial attention mechanism can fuse spatial information, it will bring significant computational overhead and it is difficult to meet the lightweight requirements. For this reason, the present invention proposes a lightweight low-level fusion module (LLFM module) at a lower level, such as Figure 6As shown in the figure, by combining depthwise separable convolution with partial spatial attention, efficient fusion of multimodal shallow features and enhancement of high-frequency details are achieved.

[0195] Specifically, given the two-stream features and F a (i), the same as high-level fusion, batch expansion operation and channel dimensionality reduction operation are performed, and then the dual-stream features are spliced ​​according to the channel dimension as the preliminary fusion result. It is expensive to use full-channel 3×3 convolution in high-resolution shallow features, so the present invention uses two sets of lightweight depth-wise separable convolutions (DWConv+PWConv) to replace 3×3 convolution to achieve local detail enhancement. The spatial attention mechanism works very well for features rich in low-level spatial information, but in order to further reduce the computational complexity, the present invention adopts a partial spatial attention mechanism (PSA), which significantly reduces the computational complexity of the model while achieving spatial detail enhancement. Finally, the two-way branch results are residually connected with the preliminary fusion result, and the PBR module is used for refinement to obtain the final fusion result of LLFM.

[0196] like Figure 6 As shown, the processing steps of the LLFM module include:

[0197] S2061: Obtain the focus-edge fusion feature map of the i-th scale of the input and the all-focus feature map F a (i);

[0198] S2062: The fully focused feature map F after the Batch dimension expansion operation a (i) Input the feature map obtained by the PBR module and the focus-edge fusion feature map The feature map obtained by inputting the PBR module for processing is spliced ​​along the channel dimension to obtain the preliminary fusion feature map F L-f (i);

[0199] The formula is:

[0200]

[0201] Where: cat( ) represents the splicing operation along the channel;

[0202] S2063: Initially fusion feature map F L-f (i) Input the feature map obtained by processing the 3×3 depth-separable convolution layer and the preliminary fusion feature map F L-f (i) Input the feature map obtained by point-by-point convolution layer and the preliminary fusion feature map F L-f(i) The feature maps obtained by inputting the PSA module layer for processing are added element by element and then input into the PBR module for processing to obtain the low-level fusion feature map;

[0203] The formula is:

[0204] F L (i) = PBR(DS(DS(F L-f (i)))+PW(F L-f (i))+PSA(F L-f (i)));

[0205] Where: F L (i) represents the low-level fused feature map of the i-th scale output by the LLFM module; DS( ) represents depthwise separable convolution; PSA( ) represents partial spatial attention mechanism; PW( ) represents point-wise convolution; PBR( ) represents PBR module.

[0206] 6. Decoder

[0207] In order to ensure the efficiency of fusing multi-scale information, the Compact Pyramid Refinement (CPR) module is adopted as the decoding module, which uses the channel attention mechanism and depth-wise separable convolution for multi-scale learning and aggregates multi-scale features from top to bottom.

[0208] The calculation formula of the decoder is expressed as:

[0209] S final =CPR(F L (1),CPR(F L (2),CPR(F H (3),F H (4))));

[0210] Where: S final Represents the final multi-scale fusion feature map; CPR () represents the CPR fusion operation; F H (i) represents the high-level fusion feature map of the i-th scale; F L (i) represents the low-level fusion feature map of the i-th scale.

[0211] Finally, the multi-scale fusion feature map is fed into the lightweight saliency detection head composed of PBR to generate the final salient object detection image. The formula is expressed as:

[0212] S pred =PBR(S final );

[0213] Where: S pred Represents a salient object detection image.

[0214] 7. Loss Function

[0215] In order to learn accurate salient target information, this paper introduces a hybrid structure loss consisting of binary cross entropy loss and intersection-over-union loss for training supervision. The overall loss function can be expressed as:

[0216] L total =L s (S pred ,G S )+L edge ;

[0217] Among them L total Represents the overall loss function of the network, L s ( ) represents a hybrid structure loss function consisting of binary cross entropy loss and intersection-over-union loss, G S Represents the ground truth saliency map.

[0218] Dice loss focuses on the overlap between the predicted and true edge regions, thereby efficiently mining foreground edges and ensuring the overall coherence of the contour. It can effectively alleviate the class imbalance problem caused by too many background pixels and drive the model to prioritize learning the complete contour structure. Since edge pixels are extremely sparse in the entire image and easily dominated by the background, this paper only uses Dice loss to calculate the edge loss at 1 / 4 of the input image size. Its formula can be expressed as:

[0219] L edge =L Dice (e final ,G edge );

[0220] Among them L edge represents the marginal loss, L Dice represents the Dice loss, G edge Represents the edge truth map.

[0221] 8. Experimental Description

[0222] 1. Experimental Dataset and Evaluation Metrics

[0223] To verify the effectiveness of the algorithm, the proposed network is evaluated on three extensive public light field datasets: DUTLF-FS, HFUT, and LFSD. DUTLF-FS contains 1462 light field samples of indoor and outdoor scenes, HFUT and LFSD contain 255 and 100 samples of challenging scenes, such as occlusion, small objects, and multiple objects, respectively. All samples are composed of fully focused images, multiple focus stack images (the number is uniformly set to 12), and corresponding ground truth images. To ensure fairness and repeatability, the public training settings are followed: the training set consists of 1000 samples in DUTLF-FS and 100 samples in HFUT, and all other samples are used as test sets.

[0224] This experiment uses four commonly used significance detection indicators as evaluation indicators:

[0225] 1) Mean absolute error (MAE) is the average of the absolute values ​​of the differences between the predicted values ​​and the true values;

[0226] 2) F-measure (F β ) is defined as an adaptive value that measures precision and recall;

[0227] 3) S-measure (S) evaluates the structural similarity between the predicted value and the true value;

[0228] 3) E-measure (E β ) takes into account both local and global similarities between predicted and true values.

[0229] Among them, F β , S and E β The larger the value, the better the model performance, and the smaller the MAE, the better. This paper also reports two indicators of the model's computational cost: model parameters (Params) and computational complexity (Giga Floating-point Operations Per Second, GFLOPs). The smaller the values ​​of these two indicators, the lighter the model.

[0230] 2. Experimental implementation details

[0231] The proposed network is implemented in the PyTorch framework. The encoder is initialized using the pretrained lightweight MobileViT-S backbone network. To ensure image consistency, the fully focused image, focus stack image, and ground truth image are resized to 256×256, and random rotation, cropping, and flipping are used for data augmentation. The proposed model is trained and tested on a single NVIDIA RTX 4090 GPU. The AdamW optimizer with a learning rate of 5e-5 and a weight decay of 1e-4 is used to optimize the network. The training epochs are 150 and the batch size is 2.

[0232] 3. Experimental comparison

[0233] For comprehensive comparison, the proposed network is compared with seven state-of-the-art heavyweight (MoLF, SANet, DLGLRG, MEANet, GFRNet, LFTransNet (LFTrans), and SAGNet) and three lightweight (ERNet, PANet, and LF-Tracy) light field salient object detection methods on three public light field datasets (DUTLF-FS, HFUT, and LFSD). To ensure fairness in the comparison, all results are obtained from official releases or tested with officially released prediction maps or models, and all evaluation criteria remain consistent. According to the existing comparison scheme, the light field method uses fully focused images and focus stack images as input.

[0234] Table 1 Quantitative comparison of different methods on three light field datasets

[0235]

[0236] Using mean absolute error (MAE), adaptive F-measure (F β ), S-measure(S), and adaptive E-measure(E β ) is used as the evaluation index, ↑(↓) indicates that the higher (lower) the value, the better. The best results are marked in bold, and the suboptimal results are marked in Underline mark.

[0237] Table 1 shows the quantitative comparison results of the proposed method with the most advanced heavyweight and lightweight methods. In terms of model size, the parameters of the network proposed in this paper are only 12.2M and the GFLOPs are only 65.4, which are the lowest among the existing light field salient object detection networks. Compared with the existing lightweight methods, the F βIt achieved the top two results on all three datasets, and the number of parameters and GFLOPs of LF-Tracy, the best lightweight network, were more than twice that of the proposed method. Specifically, the HFUT and LFSD datasets contain a large number of challenging scenes, and are therefore the two most difficult benchmarks for light field salient target detection. The method proposed in this invention has an overall performance comparable to that of LF-Tracy on HFUT, with only a slight gap, while on LFSD, it surpasses all lightweight and even most heavyweight methods with its excellent multimodal fusion and edge perception capabilities. On the conventional DUTLF-FS dataset, the method proposed in this invention lags behind LF-Tracy by only a very small margin and achieves second place. Compared with heavyweight methods, the method proposed in this invention maintains highly competitive detection performance while significantly reducing the number of parameters and GFLOPs. Specifically, on DUTLF-FS, the heavyweight LFTrans achieved an F of 0.937 with 57.2M parameters and 155.1 GFLOPs. β , while the proposed method achieves a score of 0.931 with only 12.2M parameters and 65.4 GFLOPs, which reduces the parameters and GFLOPs by about 78.67% and 57.83% respectively. β Only down 0.64%; on HFUT F β Only 0.77% decrease, even on LFSD F β Overall, the proposed method significantly reduces the number of model parameters and computational complexity while still achieving or even exceeding the accuracy of some heavyweight methods, fully demonstrating the excellent performance of the proposed lightweight hierarchical multimodal fusion strategy and edge-aware design under lightweight conditions.

[0238] Figure 7 The following are the qualitative comparison results of the proposed method with eight state-of-the-art light field salient object detection methods on three datasets, where the scenes contain complex textures (first, third, fifth, sixth, and eighth rows), background interference (second row), transparent salient objects (fourth row), and low light conditions (seventh row). Figure 7 The results show that the proposed method is very close to the performance of existing heavyweight networks in handling complex scenes and suppressing noise. Specifically, the drone blades in the first row and the plant leaves in the eighth row are very prone to false detection and missed detection. The proposed method effectively retains the sharp edges and fine details of the salient objects through partial convolutional attention and edge perception capabilities. Figure 7In the second row, the background trees are highly interfering, but our proposed method not only fully captures the overall shape of the salient target but also meticulously restores the subtle edges, while other methods suffer from varying degrees of missed detection or blurred edges. In the more challenging fourth row, featuring a glass cup, the salient target has a subtle color difference from the surroundings, but our proposed method still manages to accurately outline the cup and suppress reflective artifacts thanks to its edge perception module. Overall, these qualitative results further demonstrate that our proposed method, while lightweight, can achieve stable and relatively accurate salient target segmentation in a variety of complex scenarios through layered multimodal fusion and powerful edge perception.

[0239] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the technical solutions. Those skilled in the art should understand that modifications or equivalent replacements of the technical solutions of the present invention that do not depart from the purpose and scope of the technical solutions of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A light field salient object detection method based on edge perception and hierarchical fusion, characterized in that: include: S1: Acquire a focus stack image and a fully focused image containing the target to be detected; S2: Input the focus stack image and the all-focus image into the trained object detection model and output the corresponding salient object detection image; The processing steps of the object detection model include: S201: performing step-by-step feature extraction on the focus stack image and the all-focus image using K feature extraction modules through the backbone network to obtain K-scale focus stack feature maps and all-focus feature maps; S202: performing edge prediction on the focal stack feature maps of K scales through K SEPM modules to obtain edge feature maps of K scales; fusing the edge feature maps of the K scales in sequence to obtain an edge fusion feature map; S203: fusing the edge fusion feature map with the K-scale focal stack feature map step by step through the cascaded K EFM modules to obtain the K-scale focal-edge fusion feature map; S204: Divide the K-scale focus-edge fusion feature maps and the full focus feature map into high-level features and low-level features; S205: performing deep semantic fusion and complementation on the focus-edge fusion feature maps and the full focus feature maps of N scales belonging to high-level features through N LHFM modules, thereby obtaining high-level fusion feature maps of N scales; S206: performing deep semantic fusion and detail enhancement on the focus-edge fusion feature maps and the full focus feature map of M scales belonging to the low-level features through M LLFM modules, thereby obtaining low-level fusion feature maps of M scales; S207: The decoder uses K-1 cascaded decoding modules to decode the minimum-scale high-level fusion feature map step by step, and during the decoding process, the high-level fusion feature map or the low-level fusion feature map of the corresponding scale is input into the decoding module of the corresponding scale for decoding, to obtain the final multi-scale fusion feature map; S208: Input the multi-scale fusion feature map into the detection head for processing to obtain a corresponding salient object detection image.

2. The light field salient object detection method based on edge perception and layered fusion according to claim 1, characterized in that: In step S201, the lightweight MobileViT-S with the pooling layer and the last fully connected layer deleted is selected as the backbone network.

3. The light field salient object detection method based on edge perception and layered fusion according to claim 1, characterized in that: In step S202, the SEPM module includes a cascaded MPCR module, a PDB module, a Sobel convolution layer, and a PBR module; The MPCR module is used to extract features in different receptive fields in parallel, and obtain local details and global structural information at the same time; The PDB module consists of a cascaded central differential convolution module, an angular differential convolution module, a radial differential convolution module, and a 3×3 convolution layer; The PBR module consists of a cascade of point-by-point convolutional layers, batch normalization layers, and ReLU functions.

4. The light field salient object detection method based on edge perception and layered fusion according to claim 3, characterized in that: In step S202, the processing steps of the MPCR module include: S2021: Input the input feature map into the point-by-point convolution layer for processing, and split the obtained feature map into four feature sub-maps according to the channel; S2022: Input the four feature sub-graphs into the four branches for processing respectively, and concatenate the obtained four feature graphs along the channel dimension to obtain a restored feature graph; each branch includes a cascaded partial convolution layer with expansion and a point-by-point convolution layer; S2023: Input the restored feature map into the point-by-point convolution layer for processing, and obtain the feature map as the output of the MPCR module.

5. The method for detecting salient objects in a light field based on edge perception and layered fusion according to claim 1, wherein: In step S203, the processing steps of the EFM module include: S2031: Get the input edge fusion feature map e final and the focal stack feature map of the i-th scale S2032: Edge fusion feature map e final Use bilinear interpolation to the focal stack feature map Same spatial resolution; The formula is: e(i)=U(e final ); Where: e(i) represents the edge fusion feature map after interpolation; S2033: The interpolated edge fusion feature map e(i) and the focal stack feature map are fused using a bidirectional dense fusion strategy Perform fusion to obtain a preliminary fusion feature map; The formula is: Where: f e (i) represents the preliminary fusion feature map; Represents element-wise multiplication; S2034: After the preliminary fusion feature map is processed step by step through the cascaded partial convolution layer, PBR module and channel attention enhancement module, it is element-wise multiplied with the output of the PBR module to obtain a focus-edge fusion feature map; wherein the channel attention enhancement module includes a cascaded pooling layer, a point-by-point convolution layer and a sigmoid function; The formula is: Where: represents the focus-edge fusion feature map of the i-th scale; PConv( ) represents partial convolution; CAE( ) represents the channel attention enhancement operation; PBR( ) represents the PBR module.

6. The light field salient object detection method based on edge perception and layered fusion according to claim 1, characterized in that: In step S205, the processing steps of the LHFM module include: S2051: Get the focus-edge fusion feature map of the i-th scale of the input and the all-focus feature map F a (i); S2052: Fusion of focus-edge feature maps After the input is processed by the cascaded 3×3 partial convolution layer, PBR module and point-by-point convolution layer, it is added element-by-element with the output of the 3×3 partial convolution layer to obtain the focus-edge refinement feature map F fs-H (i); The formula is: Where: F fs-H (i) represents the focus-edge refinement feature map; PIRB() represents the combined refinement operation of the inverted residual structure consisting of a 3×3 partial convolution layer, a PBR module, and a point-by-point convolution layer; S2053: The fully focused feature map F after the Batch dimension expansion operation a (i) After the input is processed by the cascaded 3×3 partial convolution layer, PBR module and point-by-point convolution layer, it is added element-by-element with the output of the 3×3 partial convolution layer to obtain the fully focused refined feature map F a-H (i); The formula is: F a-H (i)=PIRB(replicate(F a (i))); Where: F a-H (i) represents the fully focused refined feature map; replicate( ) represents the expansion operation of the Batch dimension; S2054: Focus point-edge refinement feature map F fs-H (i) and the fully focused refined feature map F a-H (i) After element-by-element addition, the input is fed into the partial convolution attention module to implement a lightweight convolution operation on some channels, retaining the necessary local receptive field enhancement, and obtaining a high-level local fusion feature map F H-local (i); S2055: Refine the focus-edge feature map F fs-H (i) and the fully focused refined feature map F a-H (i) Input them together into the proxy attention mechanism module to realize the focus-edge refinement feature map F fs-H (i) and the fully focused refined feature map F a-H (i) The high-level features of the global attention interaction are used to obtain the global branch fusion feature map F H-global (i); S2056: high-level local fusion feature map F H-local (i) and the global branch fusion feature map F H-global (i) After the splicing operation along the channel dimension, the image is input into the PBR module for processing to obtain a high-level fusion feature map; The formula is: F H (i)=PBR(cat(F H-global (i),F H-local (i))); Where: F H (i) represents the high-level fusion feature map of the i-th scale output by the LHFM module; cat() represents the splicing operation along the channel dimension; PBR() represents the PBR module.

7. The method for detecting salient objects in a light field based on edge perception and layered fusion according to claim 6, wherein: In step S2054, the partial convolutional attention module includes a cascaded PCA module, a PBR module, a point-by-point convolution layer, a batch normalization layer, and a PSA module; wherein the feature map obtained by element-wise multiplication of the original input of the partial convolutional attention module and the output of the PSA module is used as the output of the partial convolutional attention module; The processing logic of the PCA module is as follows: the input feature map is split into two feature sub-maps; the first feature sub-map is sequentially input into the point-by-point convolution layer and the H-Sigmoid function for processing, the resulting feature map is multiplied pixel by pixel with the first feature sub-map itself, and then processed through a batch normalization layer to obtain the processing result of the first feature sub-map; the second sub-map is input into the point-by-point convolution layer for processing, the resulting feature map is concatenated with the processing result of the first feature sub-map along the channel dimension to obtain the output of the PCA module; The processing logic of the PSA module is as follows: the input feature map is split into two feature sub-maps; after the results of std and mean processing of the first sub-map are spliced, the spliced ​​feature map is input into the point-by-point convolution layer and the H-Sigmoid function for processing to obtain the feature map, which is multiplied pixel by pixel with the first feature sub-map itself, and then processed through a batch normalization layer to obtain the processing result of the first feature sub-map; the second feature sub-map is input into the 3×3 convolution layer for processing to obtain the feature map, which is spliced ​​with the processing result of the first feature sub-map along the channel dimension to obtain the output of the PSA module.

8. The method for detecting salient objects in a light field based on edge perception and layered fusion according to claim 6, wherein: In step S2055, the calculation formula of the proxy attention mechanism module is as follows: F H-global (i)=Atten s (Q,Q A ,Atten s (Q A ,K,V))+DWConv(V); Q=Linear(F fs-H (i)); Q A =Pool(Q); K=Linear(F a-H (i)); V=Linear(F a-H (i)); Where: F H-global (i) Represents the global branch fusion feature map output by the proxy attention mechanism module; Atten s ( ) represents the Softmax attention operation; DWConv( ) represents the depth-wise separable convolutional layer; Linear( ) represents linear mapping of features; Pool( ) represents adaptive global average pooling; F fs-H (i) F a-H (i) represent the input focus-edge refinement feature map and full focus refinement feature map, respectively.

9. The method for detecting salient objects in a light field based on edge perception and layered fusion according to claim 1, wherein: In step S206, the processing steps of the LLFM module include: S2061: Obtain the focus-edge fusion feature map of the i-th scale of the input and the all-focus feature map F a (i); S2062: The fully focused feature map F after the Batch dimension expansion operation a (i) Input the feature map obtained by the PBR module and the focus-edge fusion feature map The feature map obtained by inputting the PBR module for processing is spliced ​​along the channel dimension to obtain the preliminary fusion feature map F L-f (i); The formula is: Where: cat( ) represents the splicing operation along the channel; S2063: Initially fusion feature map F L-f (i) Input the feature map obtained by processing the 3×3 depth-separable convolution layer and the preliminary fusion feature map F L-f (i) Input the feature map obtained by point-by-point convolution layer and the preliminary fusion feature map F L-f (i) The feature maps obtained by inputting the PSA module layer for processing are added element by element and then input into the PBR module for processing to obtain the low-level fusion feature map; The formula is: F L (i)=PBR(DS(DS(F L-f (i)))+PW(F L-f (i))+PSA(F L-f (i))); Where: F L (i) represents the low-level fused feature map of the i-th scale output by the LLFM module; DS( ) represents depthwise separable convolution; PSA( ) represents partial spatial attention mechanism; PW( ) represents point-wise convolution; PBR() represents PBR module.

10. The method for detecting salient objects in a light field based on edge perception and layered fusion according to claim 1, wherein: In step S207, the decoding module of the decoder is a compact pyramid refinement module; The calculation formula of the decoder is expressed as: S final =CPR(F L (1),CPR(F L (2),CPR(F H (3),F H (4)))); Where: S final Represents the final multi-scale fusion feature map; CPR() represents the CPR fusion operation; F H (i) represents the high-level fusion feature map of the i-th scale; F L (i) represents the low-level fusion feature map of the i-th scale.

Images