Light field salient object detection method based on view selection and multi-feature fusion

Abstract

The application discloses a light field salient object detection method based on view angle selection and multi-feature fusion, and comprises the following steps: 1, acquiring a light field salient object detection dataset; 2, a multi-view image selection scheme; 3, constructing a deep convolutional neural network based on view angle selection and multi-feature fusion, taking an RGB image, a multi-view image and a depth image as input, training the deep neural network, and obtaining a light field salient object detection model; 4, using the trained model to realize salient object detection on a to-be-detected image. The application can effectively improve the accuracy of salient object detection of a scene in a complex and changeable environment, thereby providing important prior information for related computer vision tasks.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, image processing and analysis, in particular to a light field salient object detection method based on view selection and multi-feature fusion. BACKGROUND

[0002] Visual saliency is a kind of attention mechanism of human visual system. When we observe a scene, there is often a salient region that can attract our attention, at the same time, we will naturally ignore the non-salient regions, so that we can process a large amount of image information in a short time. Salient object detection (SOD) aims to quickly and effectively extract the most eye-catching objects or regions in a scene. Accurate salient object detection results can provide important prior information for related computer vision tasks, such as image segmentation, visual tracking, autonomous driving and video compression.

[0003] According to the type of input data, the existing salient object detection task is mainly divided into three categories: (1) salient object detection based on RGB image; (2) salient object detection based on RGB-D image; (3) salient object detection based on light field. Traditional methods based on RGB image usually obtain saliency map according to low-level features such as color, shape, background and texture of the image. Such methods can achieve ideal results in simple scenes, but the detection results are poor for some challenging scenes (such as target and background similar, low contrast, complex background and occlusion scene, etc.). Although the method based on deep learning can further obtain high-level features of the image, the lack of depth information in the RGB image itself limits its further development. The method based on RGB-D image inputs RGB image and depth map at the same time, and the accurate depth map can provide clear scene depth and structure information for salient object detection task, but such method requires high quality of depth map, and is limited by the quality of depth map obtained by depth camera, so it is difficult to obtain ideal results. Light field can record light rays from different directions of the same scene, and can provide intensity information and multi-angle geometric information of light rays at the same time, which provides more information support for obtaining high-quality saliency map.

[0004] At present, the light field based salient object detection methods are roughly divided into two categories, namely a feature based method and a learning based method. The feature based method is to estimate a saliency map by using color, depth, background priori and the like information on the basis of light field focal stack images and full focus images. This kind of method has low detection accuracy because only a few limited features are considered. The learning based method trains a salient object detection model by a certain amount of training samples. The learning based method depends on the powerful learning ability of a deep neural network and fuses a plurality of features, so that the detection accuracy is greatly improved compared with the feature based method. However, these learning based methods still have the following problems:

[0005] 1. Most of the focal stack based methods only fuse the features of focal slices and full focus images, and the relationship between different focal regions is not mined, so that the rich light field information cannot be fully utilized;

[0006] 2. For the focal stack based method, the local blur in different focal slices is not conducive to obtaining a saliency map with sharp edges by this kind of method, and when the depth range of the scene is narrow, it is difficult to obtain an ideal effect by this kind of method;

[0007] 3. For the multi-view image based method, if all the view images are directly taken as inputs, a large amount of data redundancy and great calculation cost will be caused; if only a part of continuous view images are selected as inputs, it is difficult to maximize the use of disparity information. SUMMARY

[0008] The present application provides a light field salient object detection method based on view selection and multi-feature fusion to solve the problems in the prior art, so as to realize high-precision salient object detection in challenging scenes such as multiple targets and complex backgrounds, and provide important prior information for related computer vision tasks.

[0009] To solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0010] The light field salient object detection method based on view selection and multi-feature fusion provided by the present application has the following characteristics:

[0011] Step 1, obtain a light field salient object detection data set Ω = {R, M, Z}; wherein R represents a RGB image set, M represents a multi-view image set with each image in the RGB image set R as a center image, and Z represents a depth image set corresponding to each image in the RGB image set R; let any mth image in the RGB image set be r m , then take the image r mThe multi-view images of the center image are denoted as {view m，1 , view m，2 ,..., view m,n ,..., view m，N}, wherein view m，n represents the nth view image of the center image r m ; N represents the number of multi-view images; and the depth image corresponding to the image r m is denoted as z m ;

[0012] Step 2, multi-view image selection;

[0013] From the multi-view images {view m , view m，1 ,..., view m，2 ,..., view m,n} of the image r m，N , a plurality of images at view angles with an interval of 45° are selected to form a group of view images, denoted as {list θ | θ = 1, 2, 3, 4}; wherein list θ represents the θth group of view images;

[0014] After the θth group of view images list θ is fused and reduced in the channel dimension, the θth group of fused images list′ θ is obtained;

[0015] Step 3, constructing a light field saliency target detection model based on view selection and multi-feature fusion, including an encoder, a channel dimension reduction module, three disparity extraction modules, a decoder and a fusion-feedback module;

[0016] Step 3.1, the encoder comprises L basic blocks, and each basic block processes the θth group of fused images list′ θ using a convolution layer to obtain L feature maps at the θth view angle wherein, represents the ith feature map at the θth view angle;

[0017] The image r m in the RGB image set R and the corresponding depth image z m are input into the encoder for processing to obtain L edge features {E i |i = 1, 2,..., N} and L depth features {D i |i = 1, 2,..., N}; wherein E i represents the ith edge feature, and Di represents the i-th depth feature;

[0018] Step 3.2, the channel dimension reduction module is sequentially composed of two convolutional layers, a batch normalization layer and a ReLU activation layer, and L feature maps under the i-th view angle are processed to obtain L dimension-reduced feature maps wherein, represents the i-th feature map after dimension reduction under the i-th view angle;

[0019] Step 3.3, three disparity extraction modules sequentially process to obtain the i-th total disparity information under four view angles so as to obtain the total disparity flow

[0020] Step 3.4, the decoder is composed of L decoding modules; each decoding module is sequentially composed of a convolutional layer, a batch normalization layer and a ReLU activation layer;

[0021] When i = L, the i-th total disparity information is input into the i-th decoding module for processing to obtain the i-th decoding feature

[0022] When i = L-1, L-2, …, 1, the i+1-th decoding feature and the i-th total disparity information are input into the i-th decoding module for processing to obtain the i-th decoding feature so that the first decoding module outputs the final multi-view image feature

[0023] The RGB image features {E i |i = 1, 2, …, L} and the depth image features {D i |i = 1, 2, …, L} are input into the decoder for processing to obtain the decoded RGB image features and the depth image features

[0024] Step 3.5, the fusion-feedback module is composed of a channel fusion module and a feedback module;

[0025] Step 3.5.1, the channel fusion module obtains a rough predicted saliency map using formula (4)

[0026]

[0027] ​In formula (4), cat represents a channel fusion operation, and represents an element-level multiplication;

[0028] Step 3.5.2, the feedback module is sequentially composed of L sub-feedback modules, each of which is sequentially composed of a convolutional layer, a batch normalization layer, a ReLU activation layer and a max-pooling layer;

[0029] The rough predicted saliency map By sequentially processing through the L sub-feedback modules, and after generating a scale of features at each sub-feedback module, the corresponding feedback is input into the L decoders, thereby generating an accurate predicted saliency map X pre ;

[0030] Step 4, training a light field saliency target detection model based on view selection and multi-feature fusion;

[0031] Step 4.1, establishing a loss function;

[0032] Step 4.1.1, a spatial loss function L s , an edge loss function L e and a depth loss function L d are respectively established by formula (5), formula (6) and formula (7):

[0033]

[0034]

[0035]

[0036] In formula (5), formula (6) and formula (7), E gt is the true edge map of the image r m , X gt is the true saliency map of the image r m ;

[0037] Step 4.1.2, a fusion feature loss function is established by formula (8):

[0038]

[0039] Step 4.1.3, a total loss function is established by formula (9):

[0040] L=L s +L e +L d +L cat (9)

[0041] Step 4.2, based on the dataset Ω = {R, M, Z}, the deep neural network model based on view selection and multi-feature fusion is trained by using a stochastic gradient descent algorithm, and a total loss function L is calculated to update the network parameters until the total loss function converges, thereby obtaining an optimal light field saliency target detection model for saliency target detection of a light field image.

[0042] The light field saliency target detection method based on view selection and multi-feature fusion according to the present application is also characterized in that the step 3.3 is performed according to the following steps:

[0043] Step 3.3.1, the θth sub-disparity extraction module performs element-level multiplication calculation on the i th feature map under the θth view angle and the i th feature map under the θ+2th view angle to obtain a set of i th common features under a set of mutually orthogonal view angles

[0044] Step 3.3.2, the θth sub-disparity extraction module uses formula (1) and formula (2) to obtain the i th disparity information under the θth view angle and the i th disparity information under the θ+2th view angle

[0045]

[0046]

[0047] Step 3.3.3, the θth sub-disparity extraction module uses formula (3) to obtain a set of i th sub-disparity information under a set of mutually orthogonal view angles

[0048]

[0049] In formula (3), α is the attenuation coefficient;

[0050] Step 3.3.4, input the two sets of i th sub-disparity information into the third sub-disparity extraction module for processing to obtain the i th total disparity information under the four view angles and thereby obtain the total disparity flow

[0051] The electronic device according to the present application comprises a memory and a processor, wherein the memory is configured to store a program supporting the processor to execute the light field saliency target detection method, and the processor is configured to execute the program stored in the memory.

[0052] The application discloses a computer readable storage medium, and a computer program is stored on the computer readable storage medium.

[0053] Compared with the prior art, the application has the advantages that:

[0054] 1、The application constructs a deep neural network based on view selection and multi-feature fusion, uses label data to supervise the deep neural network to learn, and thus obtains a robust light field saliency target detection feature model, so as to solve the problem that a traditional method acquires a saliency map according to low-level features such as color, shape, background and texture of an image, ignores a lot of feature information, and has low detection precision.

[0055] 2、The deep neural network based on view selection and multi-feature fusion constructed by the application is compared with a view selection method adopted by a current mainstream method, a new view selection scheme is proposed according to geometric features of a light field, and thus a larger disparity amount can be obtained while a smaller number of views is used under the premise that angle distribution is balanced, and multi-view image viewing information is more efficiently utilized.

[0056] 3、The deep neural network based on view selection and multi-feature fusion constructed by the application designs a disparity extraction module, extracts common features and differentiated features in a multi-view image queue in different orthogonal directions, and completes modeling according to the correlation of the extracted features, so that the saliency object in the image is positioned, and differentiated information of different views is efficiently aggregated, equivalent to realizing small-range dynamic observation of a target in a scene, and improving the precision of saliency target detection in a complex and changeable scene.

[0057] 4、The deep neural network based on view selection and multi-feature fusion constructed by the application designs an effective fusion-feedback module for realizing feature interaction between branches through the synergistic complementarity of light field space, depth and edge feature information, so that a more accurate saliency map can be generated. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A flowchart of light field saliency target detection based on a view selection and multi-feature fusion network of the application;

[0059] Figure 2 A comparison diagram of a multi-view selection scheme of the application and other methods;

[0060] Figure 3 A deep neural network structure diagram of light field saliency target detection based on view selection and multi-feature fusion of the application;

[0061] Figure 4 This is a graph showing the detection results of the present invention and other salient target detection methods on the test set. Detailed Implementation

[0062] In this embodiment, as Figure 1 As shown, a light field saliency target detection method based on viewpoint selection and multi-feature fusion is performed according to the following steps:

[0063] Step 1: Obtain the light field saliency target detection dataset Ω = {R, M, Z}; where R represents the RGB image set, M represents the multi-view image set centered on each image in the RGB image set R, and Z represents the depth image set corresponding to each image in the RGB image set R; let any m-th image in the RGB image set be denoted as r. m Then, with image r m A multi-view image centered on a single image is denoted as {view}. m，1 view m，2 , ...,view m,n , ...,view m，N}, where view m，n Represented by image r m The nth viewpoint image of the center image; N represents the number of multi-view images; image r m The corresponding depth image is denoted as z. m In this embodiment, N = 81.

[0064] In this embodiment, three datasets—DUTLF-V2, DUTLF-MV, and Lytro-Illum—were used for model training and evaluation. DUTLF-V2 is currently the largest dataset for salient object detection in light fields, providing 4204 light field scenes, including challenging scenarios with multiple objects, complex backgrounds, and occlusion. Of these, 2957 scenes were used for training, and 1247 for testing. Each scene provides multiple viewpoints, focus stacks, depth maps, and label maps. The provided multi-view images have an angle and spatial resolution of 9×9 and 400×600, respectively. DUTLF-MV contains 1580 scenes, with 1100 used for training and 480 for testing. The spatial resolution of the images is 590×400. Lytro-Illum contains 640 scenes; this dataset does not have separate training and testing sets, and in this embodiment, all of it was used for testing. The spatial resolution of the images is 540×375.

[0065] Step 2: Selecting images from multiple perspectives;

[0066] From image r m In multi-view images {view m，1 viewm，2 ,..., view m,n ,..., view m，N} are respectively composed of a group of view images, denoted as {list θ | θ = 1, 2, 3, 4}; wherein, list θ represents the θth group of view images;

[0067] In the embodiment, the multi-view selection scheme is compared with, as shown in Figure 2 , the scheme shown in (a) in Figure 2 selects only the multi-view images in the horizontal direction, Figure 2 , the scheme shown in (b) selects the multi-view images in the horizontal and vertical directions, Figure 2 , the scheme shown in (c) selects the multi-view images in the horizontal, vertical and cross directions, and the embodiment adopts the discrete view selection scheme shown in (d) in Figure 2 , the four view angles are 0°, 45°, 90° and 135°, which can balance the total amount of parallax and distribution bias with a smaller number of views; the parallax conditions of different view selection schemes are shown in Table 1. Since the baseline between the multi-view images is narrow, the absolute value of the total amount of parallax has little significance for the extraction of spatial features of the salient object detection task, and excessive reliance on the average parallax amount of a single view will cause angle bias of the view angle. According to the statistical results in Table 1, the multi-view selection scheme of the embodiment balances the angle distribution, uses a smaller number of views, and obtains a larger amount of parallax.

[0068] Table 1

[0069]

[0070] The θth group of view images list θ After the fusion and dimension reduction operation in the channel dimension, the θth group of fused images list' θ is obtained; in the embodiment, the channel number of the images in the four view angle directions is changed to 64 after the dimension reduction operation.

[0071] Step 3, as shown in Figure 3 , a light field salient object detection model based on view selection and multi-feature fusion is constructed, including an encoder, a channel dimension reduction module, three parallax extraction modules, a decoder and a fusion-feedback module.

[0072] Step 3.1, the encoder includes L basic blocks, each basic block processes the θth group of fused images list' θ using a convolution layer to obtain L feature maps under the θth view angle, wherein, represents the i-th feature map under the θ-th view angle; in this embodiment, the number of basic blocks contained in the encoder L = 4.

[0073] The images r in the RGB image set R m and the corresponding depth images are denoted as z m are input into the encoder respectively for processing, obtaining L edge features {E i | i = 1, 2,... N} and L depth features {D i | i = 1, 2,... N}; wherein, E i represents the i-th edge feature, and D i represents the i-th depth feature.

[0074] Step 3.2, the channel dimension reduction module is sequentially composed of two convolutional layers, a batch normalization layer and a ReLU activation layer, and after processing the L feature maps under the θ-th view angle, L feature maps after dimension reduction are obtained. wherein, E represents the i-th feature map after dimension reduction under the θ-th view angle; in this embodiment, the channel dimension after dimension reduction is 64.

[0075] Step 3.3, the three disparity extraction modules sequentially process to obtain the i-th total disparity information under the four view angles , thereby obtaining the total disparity flow

[0076] Step 3.3.1, the θ-th sub-disparity extraction module performs element-level multiplication calculation on the i-th feature map under the θ-th view angle and the i-th feature map under the θ+2-th view angle, to obtain a set of i-th common features under a set of mutually orthogonal view angles

[0077] Step 3.3.2, the θ-th sub-disparity extraction module uses formula (1) and formula (2) to obtain the i-th disparity information under the θ-th view angle and the i-th disparity information under the θ+2-th view angle

[0078]

[0079]

[0080] Step 3.3.3, the θ-th sub-disparity extraction module uses formula (3) to obtain a set of i-th sub-disparity information under a set of mutually orthogonal view angles

[0081]

[0082] In formula (3), a is an attenuation coefficient; in this embodiment, since element-level multiplication will cause an increase in feature values, an attenuation coefficient a = 0.4 is introduced.

[0083] Step 3.3.4, the two groups of the i-th sub-parallax information After being input into the third sub-parallax extraction module for processing, the i-th total parallax information under four view angles is obtained Thus, the total parallax flow is obtained

[0084] Step 3.4, the decoder is composed of L decoding modules; each decoding module is sequentially composed of a convolution layer, a batch normalization layer and a ReLU activation layer; in this embodiment, the number of decoding modules L = 4;

[0085] When i = L, the i-th total parallax information is input into the i-th decoding module for processing to obtain the i-th decoding feature

[0086] When i = L-1, L-2,..., 1, the i+1-th decoding feature and the i-th total parallax information are input into the i-th decoding module for processing to obtain the i-th decoding feature Thus, the i-th decoding module outputs the final multi-view image feature from the first

[0087] The RGB image features {E i |i = 1, 2,..., L} and the depth image features {D i |i = 1, 2,..., L} are input into the decoder for processing to obtain the decoded RGB image features and the depth image features

[0088] Step 3.5, the fusion-feedback module is composed of a channel fusion module and a feedback module;

[0089] Step 3.5.1, the channel fusion module obtains a rough predicted saliency map by formula (4)

[0090]

[0091] In formula (4), cat represents a channel fusion operation, and represents an element-level multiplication.

[0092] Step 3.5.2, the feedback module is sequentially composed of L sub-feedback modules, each of which is sequentially composed of a convolutional layer, a batch normalization layer, a ReLU activation layer and a max-pooling layer; in this embodiment, the number of sub-feedback modules L = 4.

[0093] Coarse prediction saliency map By sequentially processing through the L sub-feedback modules, and after generating a scale of features at each sub-feedback module, the corresponding feedback is input into the L decoders, thereby generating an accurate prediction saliency map X pre ;

[0094] Step 4, training a light field saliency target detection model based on view selection and multi-feature fusion;

[0095] Step 4.1, establishing a loss function;

[0096] Step 4.1.1, the spatial loss function L s , the edge loss function L e and the depth loss function L d are respectively established by equations (5), (6) and (7):

[0097]

[0098]

[0099]

[0100] In equations (5), (6) and (7), E gt is the true edge map of the image r m , X gt is the true saliency map of the image r m .

[0101] Step 4.1.2, the fusion feature loss function is established by equation (8):

[0102]

[0103] Step 4.1.3, the total loss function is established by equation (9):

[0104] L = L s + L e + L d + L cat 9

[0105] In this embodiment, in the training stage, the network is trained for 60 epochs, the initial learning rate is set to 0.02, the momentum factor is set to 0.9, and the weight decay is set to 5e -4 .

[0106] Step 4.2, based on the dataset Ω = {R, M, Z}, a deep neural network model based on view selection and multi-feature fusion is trained using a stochastic gradient descent algorithm, and a total loss function L is calculated to update the network parameters until the total loss function converges, thereby obtaining an optimal light field saliency target detection model for saliency target detection of light field images.

[0107] In this embodiment, an electronic device includes a memory for storing a program supporting the processor to execute the above method, and a processor configured to execute the program stored in the memory.

[0108] In this embodiment, a computer readable storage medium has a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0109] Table 2 is a comparison result of the light field saliency target detection method based on view selection and multi-feature fusion of the present application with 12 learning-based saliency target detection methods, respectively using "S-measure", "F-measure", "E-measure", and "MAE" as evaluation criteria, and using the test set of the light field saliency target detection dataset DUTLF-V2, DUTLF-MV and Lytro-Illum. "S-measure" is usually used to measure the similarity of the predicted saliency map and the real saliency map in the spatial structure, and the closer the value is to 1, the better the saliency target detection effect. "F-measure" is the weighted harmonic mean of precision and recall, and the closer the value is to 1, the better the saliency target detection effect. "E-measure" is a measure that considers the local pixel similarity and global pixel statistics between the predicted saliency map and the real saliency map, and the closer the value is to 1, the better the saliency target detection effect. "MAE" is an overlap evaluation index, which describes the probability of correct salient pixels being assigned as non-salient pixels, and the closer the value is to 0, the better the saliency target detection effect. According to the quantitative analysis of Table 2, it can be seen that in the test on the largest light field dataset DUTLF-V2, the present application has achieved the best results in all evaluation indicators; in the test on the dataset DUTLF-MV, the average indicators have achieved the second best result only inferior to OBGNet, it is worth noting that the data amount used by the model of the present application is only 1 / 6 of OBGNet; in the test on the dataset Lytro-Illum, the present application has also achieved the best result ranked first in all evaluation indicators.

[0110] Table 2

[0111]

[0112] Figure 4 The detection results of the present application based on view selection and multi-feature fusion light field saliency target detection method and the current other saliency target detection methods on three data sets. It contains simple, complex scene, multi-target, dark light and high light and other challenging scenes. Multi-target small object scene (row 3), high light and dark light scene (row 4 and row 10), similar scene of saliency target and background (row 6), similar target interference scene in background (row 8), clutter information interference scene (row 1, row 5 and row 9). Ours is the light field saliency target detection method of the present application, which can be directly seen that the method of the present application has obvious advantages in saliency target positioning and segmentation and edge details.

Claims

1. A method for light field salient object detection based on view selection and multi-feature fusion, characterized in that, is performed according to the following steps: Step 1, obtain a light field saliency target detection dataset Ω = {R, M, Z}; wherein R represents a set of RGB images, M represents a set of multi-view images with each image in the set of RGB images R as a center image, and Z represents a set of depth images corresponding to each image in the set of RGB images R; let any m-th image in the set of RGB images be denoted as r m , then the multi-view images with the image r m as a center image are denoted as {view m,1 , view m,2 , …, view m,n , …, view m,N}, wherein view m,n represents an n-th view image with the image r m as a center image; N represents the number of multi-view images; and the depth image corresponding to the image r m is denoted as z m . Step 2, multi-view image selection; From the image r m , a plurality of view images {view m,1 , view m,2 , …, view m,n , …, view m,N} are selected, and a plurality of groups of view images are formed by selecting a plurality of images at view point angles with an interval of 45°, and each group of view images is recorded as {list θ | θ = 1, 2, 3, 4}; wherein, list θ represents the θth group of view images; θ-th group of view images list θ After the fusion and dimension reduction operations in the channel dimension, the θ-th group of fused images list' θ ; Step 3, constructing a light field saliency object detection model based on view selection and multi-feature fusion, including: an encoder, a channel dimension reduction module, three disparity extraction modules, a decoder and a fusion-feedback module; Step 3.1, the encoder includes L basic blocks, each basic block uses a convolutional layer to process the θth group of fusion images list θ After processing, L feature maps under the θth view angle are obtained Wherein, represents the ith feature map under the θth view angle An image r in the RGB image set R is denoted as m and its corresponding depth image is denoted as z m are input into the encoder respectively for processing, obtaining L edge features {E i | i = 1, 2, … N} and L depth features {D i | i = 1, 2, … N}; wherein E i represents the i-th edge feature, and D i represents the i-th depth feature. Step 3.2, the channel dimension reduction module is sequentially composed of two convolutional layers, a batch normalization layer and a ReLU activation layer, and processes L feature maps under the θth view angle After processing, L feature maps after dimension reduction are obtained wherein, represents the i-th feature map after dimension reduction under the θth view angle Step 3.3, 3 disparity extraction modules process in turn to obtain the i-th total disparity information under 4 view angles Thus, the total disparity stream is obtained Step 3.4, the decoder is composed of L decoding modules; each decoding module is composed of a convolution layer, a batch normalization layer and a ReLU activation layer in turn; When i = L, the i-th total disparity information is input into the i-th decoding module for processing to obtain the i-th decoding feature when i = L - 1, L - 2, …, 1, the (i + 1)th decoding feature is decoded from the (i + 1)th decoded feature and the ith total disparity information input into the ith decoding module for processing to obtain the ith decoding feature so that the final multi-view image feature is output by the first decoding module from the ith decoding module The RGB image features {E i The depth image features {D i The decoded RGB image features {E and the decoded depth image features {D Step 3.5, the fusion-feedback module is composed of a channel fusion module and a feedback module; Step 3.5.1, the channel fusion module utilizes formula (4) to obtain a rough prediction saliency map In formula (4), cat represents a channel fusion operation, and represents an element-level multiplication; Step 3.5.2, the feedback module is composed of L sub-feedback modules in turn, and each sub-feedback module is composed of a convolution layer, a batch normalization layer, a ReLU activation layer and a maximum pooling layer in turn; the coarse prediction saliency map By sequentially processing through L sub-feedback modules and generating a scale of features at each sub-feedback module, the corresponding feedback is input into the L decoders to generate an accurate prediction saliency map X pre ; Step 4, training the light field saliency object detection model based on view selection and multi-feature fusion; Step 4.1, establishing a loss function; Step 4.1.

1. Establishing the space loss function L by formula (5), formula (6) and formula (7) respectively s , the edge loss function L e and the depth loss function L d : In formula (5), formula (6), and formula (7), E gt is a true edge map of the image r m gt is a true saliency map of the image r m ;​ Step 4.1.2, the fusion feature loss function is established by formula (8): Step 4.1.3, the total loss function is established by formula (9): L = L s + L e + L d + L caf (9) Step 4.2, based on the data set Ω={R, M, Z}, the deep neural network model based on view selection and multi-feature fusion is trained using a stochastic gradient descent algorithm, and the total loss function L is calculated to update the network parameters until the total loss function converges, thereby obtaining an optimal light field saliency object detection model for saliency object detection of light field images. 2.The method of claim 1, wherein, The step 3.3 is performed according to the following steps: Step 3.3.1, the θth sub-parallax extraction module performs element-level multiplication calculation on the i th feature map under the θth view angle and the i th feature map under the θ+2th view angle to obtain a set of i th common features under mutually orthogonal view angles ​ Step 3.3.2, the θth sub-parallax extraction module obtains the i th parallax information under the θth view angle and the i th parallax information under the θ+2th view angle respectively by using formula (1) and formula (2) and θ+2th view angle Step 3.3.3, the θth sub-parallax extraction module obtains a set of mutually orthogonal i-th sub-parallax information under the view angle of formula (3) In formula (3), α is the attenuation coefficient; Step 3.3.4, the two groups of the i-th sub-parallax information After inputting into the third sub-parallax extraction module for processing, the i-th total parallax information under four view angles is obtained Thus, the total parallax stream is obtained 3. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the light field saliency object detection method of claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to perform the steps of the light field saliency object detection method of claim 1 or 2.

Images