An extremely lightweight real-time salient object detection method

Abstract

The application provides an extremely lightweight real-time salient object detection method, and relates to the technical field of object detection.The application first proposes a novel wavelet transform module WTM, which realizes feature down-sampling by fusing wavelet transform theory and CNN.An encoder is constructed by cascading WTM, high-quality multi-level coding is realized using limited parameters, and pre-training operation is not needed.Then a novel wavelet transform fusion module WTFM is proposed, which can realize feature up-sampling and fusion operation through wavelet transform, CNN and a residual mechanism.An decoder is constructed by cascading WTFM, high-quality decoding is realized, and finally high-quality training of the model can be realized only by using a conventional BCE+IOU hybrid loss function.The application can be easily embedded into other computer vision tasks to improve their performance due to its extremely lightweight degree, good accuracy and high real-time serial data processing, and is no longer limited by computing resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly relates to a real-time salient target detection method with extremely light weight. BACKGROUND

[0002] Salient target detection has attracted extensive attention due to its ability to quickly capture salient targets in complex scenes similar to human vision. In recent years, with the vigorous development of artificial intelligence, computer vision has developed rapidly and begun to flood into human life and production practice. In practical applications, the image content obtained by the camera sensor is not all useful information, and often useless information occupies the dominant position of the whole image information. Therefore, the salient target detection algorithm is an algorithm that simulates the human visual system and gives the computer the ability to efficiently capture useful data from the hundreds of millions of bits of data transmitted by the retina, similar to the human brain. It can be embedded in many computer vision tasks, not only can greatly reduce the demand for computing resources, but also can effectively improve the performance of the task. Today they are widely used in video processing, target detection, semantic segmentation, simultaneous localization and mapping, and robot navigation. Unfortunately, these methods often constantly refresh the detection performance through complex network architecture, advanced feature fusion mechanism, efficient loss function and the introduction of edge features, but ignore the huge computing cost, making it difficult to deploy on devices with limited computing resources. Although some excellent lightweight salient target detection algorithms have emerged, the three of light weight, accuracy and real-time performance have not reached a good balance.

[0003] The document “Y. Liu, Y.-C. Gu, X.-Y. Zhang, W. Wang and M.-M. Cheng. Lightweight Salient Object Detection via Hierarchical Visual Perception Learning. IEEE Transactions on Cybernetics, vol. 51, no. 9, pp. 4439-4449. 2021.” proposes a HVP module to mimic primate visual cortex for hierarchical visual perception learning. Subsequently, the document “Y. Liu, X.-Y. Zhang, J.-W. Bian, L. Zhang and M.-M. Cheng. SAMNet: Stereoscopically Attentive Multi-scale Network for Lightweight Salient Object Detection. IEEE Transactions on Image Processing, vol. 30, pp. 3804-3814, 2021.” proposes a SAM module that enables small networks to encode high-level features and low-level details. The document “Y. Liu, X.-Y. Zhang, J.-W. Bian, L. Zhang and M.-M. Cheng. EDN: Salient Object Detection via Extremely-Downsampled Network. IEEE Transactions on Image Processing, vol. 31, pp. 3125-3136, 2022.” proposes an extremely-downsampled network that employs an extreme technique to effectively learn the global view of the entire image. The document uses scale-related pyramid convolutions to construct a decoder for recovering object details from the aforementioned extreme downsampling. The document “M.-M. Cheng, S.-H. Gao, A. Borji, Y.-Q. Tan, Z. Lin and M. Wang. A Highly Efficient Model to Study the Semantics of Salient Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 8006-8021, 2022.” proposes a generalized OctConv that leverages intra- and cross-stage multi-scale features while reducing representational redundancy through a dynamic weight decay scheme.Meanwhile, a very lightweight model CSNet* is constructed using gOctConvs in the literature, which is the most lightweight salient object detection model to date.

[0004] The documents “Y. Liu, Y.-C. Gu, X.-Y. Zhang, W. Wang and M.-M. Cheng. Lightweight Salient Object Detection via Hierarchical Visual Perception Learning. IEEE Transactions on Cybernetics, vol. 51, no. 9, pp. 4439-4449. 2021.”, “Y. Liu, X.-Y. Zhang, J.-W. Bian, L. Zhang and M.-M. Cheng. SAMNet: Stereoscopically Attentive Multi-scale Network for Lightweight Salient Object Detection. IEEE Transactions on Image Processing, vol. 30, pp. 3804-3814, 2021.” and “Y. Liu, X.-Y. Zhang, J.-W. Bian, L. Zhang and M.-M. Cheng. EDN: Salient Object Detection via Extremely-Downsampled Network. IEEE Transactions on Image Processing, vol. 31, pp. 3125-3136, 2022.” proposed three representative lightweight salient object detection models, although the detection performance has certain competitiveness with heavyweight salient detection models, but their serial (in the form of image frame by frame to the model) processing data real-time is weak, often difficult to meet the application scenarios with high real-time serial processing. The document “M.-M. Cheng, S.-H. Gao, A. Borji, Y.-Q. Tan, Z. Lin and M. Wang. A Highly Efficient Model to Study the Semantics of Salient Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 8006-8021, 2022.” proposed an extremely lightweight model CSNet and its variant CSNet*, although the computing resources required by the model are very low, but its detection performance and real-time of serial processing data are not high.In summary, the current study cannot well balance the relationship among lightweight, accuracy and real-time. SUMMARY

[0005] In view of the deficiencies of the prior art, the application provides an extremely lightweight real-time salient object detection method. An extremely lightweight real-time salient object detection model is proposed to achieve a better balance among lightweight, accuracy and real-time, and meanwhile, the salient object detection model can be easily embedded into various computer vision tasks due to its extremely lightweight characteristics, effectively improving their task performance.

[0006] An extremely lightweight real-time salient object detection method comprises the following steps:

[0007] Step 1: Establishing a wavelet transform module WTM, and designing an extremely lightweight salient object detection encoder by using wavelet transform theory;

[0008] The salient object detection encoder is obtained by stacking the wavelet transform module WTM; the salient object detection encoder realizes the output of five levels of different resolution features, and the resolutions from high to low are represented as f out1 , f out2 , f out3 , f out4 and f out5 . With the 2 times downsampling function of the wavelet transform module WTM, multi-level features are generated, and the output of the WTM is divided into two parts: (1) the output f out is used as the input of the next wavelet transform module WTM to continue feature extraction; (2) the sub-band mappings X LL , X LH , X HL and X HH enhance the decoding ability of the decoder through the residual mechanism; the establishment process of the wavelet transform module WTM comprises the following steps:

[0009] Step S1: The input X is decomposed into 4 sub-band maps by 2D DWT, and the sub-band mapping X LL is a low-frequency component and represents the basic features of the input X. The sub-band mappings X LH , X HL and X HH are high-frequency components and represent the detailed features of the input X;

[0010] Step S2: The low-frequency component X LL and the fused high-frequency component X high are further learned and extracted by convolution to drive useful information for tasks, and the fused low-frequency feature f low and the high-frequency feature f high are represented as:

[0011] f low = DSConv(X LL )

[0012] f high = DSConv(Concat(X LH , X HL , X HH ))

[0013] where DSConv represents a depth separable convolution, and Concat represents a feature concatenation in the channel dimension.

[0014] Step S3: f low and f high are integrated by convolution, in which the low-frequency and high-frequency features complement each other to form a down-sampled feature map; and f out is output.

[0015] f out = DSConv(f low + f high )

[0016] Step 2: Establish a wavelet transform fusion module WTFM, and design an extremely lightweight salient object detection decoder using wavelet transform theory;

[0017] The salient object detection decoder is obtained by stacking the wavelet transform module WTFM;

[0018] The establishment process of the wavelet transform fusion module WTFM includes the following steps:

[0019] Step D1: After the input X in is up-sampled, a discrete wavelet transform DWT is performed, and the outputs of the high-frequency feature g LL and the low-frequency features g HL , g HL and g HH of the discrete wavelet transform DWT are respectively represented as:

[0020] g LL , g HL , g HL , g HH = DWT[Up(DSConv(X in ))]

[0021] Step D2: Enhance the current features at the frequency level through a residual mechanism; and the enhanced features E LL , E HL , E HL and E HH are represented as:

[0022] E i = DSConv(g i + X i ), i = {LL, HL, LH, HH}

[0023] Step D3: low-frequency E LL component and high-frequency E HL , E HL and E HH fusion components are processed by inverse discrete wavelet transform IDWT to obtain up-sampling features g out :

[0024] g out = IDWT(E LL .E HL .E LH , E HH )

[0025] Step 3: establishing a salient object detection model;

[0026] The salient object detection model is constructed by cascading a wavelet transform module WTM and a wavelet transform fusion module WTFM to build an encoder and a decoder, and then establishing an "encoder-decoder" architecture.

[0027] Step 4: image acquisition using a camera to complete real-time salient object detection;

[0028] The acquired RGB image is transmitted frame by frame to the salient object detection model, and the feature interaction fusion in the frequency domain and the time domain is completed through discrete wavelet transform and convolution operation, that is, WTM and WTFM, and the mask image of the salient object is output, realizing the detection and segmentation of the salient object at the pixel level.

[0029] The beneficial effects produced by the above technical solutions are:

[0030] The application provides an extremely lightweight real-time saliency target detection method. In view of the problem that the current lightweight saliency model has poor real-time performance in serial processing of data while the performance is reduced compared with a heavyweight model, the application provides an extremely lightweight saliency target detection algorithm, and realizes the balance of lightweight, accuracy and real-time performance. First, a novel wavelet transform module WTM is provided, which realizes feature down-sampling by fusing wavelet transform theory and CNN. An encoder is constructed by cascading the WTM, and high-quality multi-level encoding is realized by using limited parameters, and pre-training operation is not required. Then, a novel wavelet transform fusion module WTFM is provided, which can realize feature up-sampling and fusion operation by wavelet transform, CNN and a residual mechanism. A decoder is constructed by cascading the WTFM, high-quality decoding is realized, and finally, high-quality training of the model can be realized only by using a conventional BCE+IOU hybrid loss function. The application can be easily embedded into other computer vision tasks to improve their performance due to its extremely lightweight degree, good accuracy and high real-time performance of serial data processing, and is no longer limited by computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A network flowchart in the embodiment of the application;

[0032] Figure 2 A wavelet transform module (WTM) structure diagram in the embodiment of the application;

[0033] Figure 3 A wavelet transform fusion module (WTFM) structure diagram in the embodiment of the application. DETAILED DESCRIPTION

[0034] The specific embodiments of the application will be further described in detail below with reference to the accompanying drawings and embodiments. The following embodiments are used to illustrate the application, but are not used to limit the scope of the application.

[0035] An extremely lightweight real-time saliency target detection method, Figure 1 is the main flowchart of the technical scheme of the application. As shown in Figure 1 , the following steps are included:

[0036] Step 1: Establish a wavelet transform module WTM, and design an extremely lightweight saliency target detection encoder by using wavelet transform theory;

[0037] The wavelet transform inherits the advantages of the Fourier transform while overcoming its shortcomings. The wavelet transform can provide a "time-frequency" window that varies with frequency, which dominates in the analysis of (non-) stationary signals. Due to the reversibility of the wavelet transform, the wavelet transform is widely used in signal processing, image processing and other fields. Therefore, the present embodiment uses Haar wavelet transform to realize an efficient feature encoder. As Figure 1 shown;

[0038] The salient object detection encoder is obtained by stacking the wavelet transform module WTM; as Figure 1 shown, with the help of the wavelet transform module WTM, a salient object detection encoder similar to ResNet and VGG is proposed, which realizes the output of five levels of different resolution features, and the resolutions from high to low are represented as f out1 , f out2 , f out3 , f out4 and f out5 . The architecture of the encoder is very efficient, and the wavelet transform module WTM generates multi-level features with 2 times downsampling function. The output of WTM is divided into two parts: (1) the output f out is used as the input of the next wavelet transform module WTM to continue feature extraction; (2) the sub-band mapping X LL , X LH , X HL and X HH enhance the decoding ability of the decoder through the residual mechanism. The encoder is designed for extremely lightweight salient object detection method, which does not need to pre-train the model, effectively reduces the redundancy of multi-level features, and can more accurately capture the deep meaning of the salient object.

[0039] The encoder needs to encode the input features and generate multi-level features, and the present application is inspired by the decomposition process of the discrete wavelet transform DWT, and a wavelet transform module WTM is proposed, which combines the discrete wavelet transform DWT with convolution operation to realize feature downsampling. As Figure 2 shown, the establishment process of the wavelet transform module WTM includes the following steps:

[0040] Step S1: the input X is decomposed into 4 sub-band maps by 2D DWT, and the sub-band mapping X LL is a low-frequency component and represents the basic features of the input X. The sub-band mappings X LH , X HL and X HH are high-frequency components and represent the detailed features (such as edge contours, etc.) of the input X.

[0041] Step S2: the low-frequency component X LL and the fused high-frequency component X highare further learned and extracted by convolution respectively to drive useful information for the task, and the fused low-frequency feature f low and high-frequency feature f high are respectively represented as:

[0042] f low = DSConv(X LL )

[0043] f high = DSConv(Concat(X LH , X HL , X HH ))

[0044] where DSConv represents a depth separable convolution, and Concat represents a feature concatenation in the channel dimension.

[0045] Step S3: f low and f high are integrated by convolution, in which the low-frequency and high-frequency features complement each other, thereby forming a down-sampled feature map (with complete high-frequency and low-frequency information). The output f out is represented as:

[0046] f out = DSConv(f low + f high )

[0047] Step 2: Establish a wavelet transform fusion module WTFM, and design an extremely lightweight salient object detection decoder using wavelet transform theory;

[0048] The salient object detection decoder is obtained by stacking a wavelet transform module WTFM;

[0049] Since the wavelet transform is reversible, the inverse discrete wavelet transform IDWT is the inverse operation of DWT. Since the decoder needs to decode the multi-level features generated by the encoder, a wavelet transform fusion module WTFM is proposed to realize high-quality up-sampling and fusion operations. For this purpose, the decoder is obtained by stacking a wavelet transform module WTFM, and an efficient decoder is built by cascading WTFM, as shown in Figure 3 The establishment process of the wavelet transform fusion module WTFM includes the following steps:

[0050] Step D1: Perform up-sampling operation on the input X in (with complete frequency characteristics) and then perform discrete wavelet transform DWT, to realize low-frequency and high-frequency feature decomposition of the input feature. This operation not only realizes matching with multi-frequency feature input by using residual structure, but also improves the anti-interference ability of the network through up-sampling operation. The high-frequency feature gLL and low frequency feature g HL , g HL and g HH The outputs are denoted as:

[0051] g LL , g HL , g HL , g HH = DWT [Up (DSConv (X in ))

[0052] Step D2: The upsampling operation in the encoder stage, decoder stage and the first stage will introduce some uncertainty (such as interpolation inaccuracy), so the previous feature is enhanced in the frequency level by the residual mechanism; the enhanced feature E LL , E HL , E HL and E HH are denoted as:

[0053] E i = DSConv (g i + X i ), i = {LL, HL, LH, HH}

[0054] Step D3: Thanks to the invertibility of wavelet transform, the low frequency E LL component and the fusion component of high frequency E HL , E HL and E HH are processed by the inverse discrete wavelet transform IDWT to obtain the upsampled feature g out :

[0055] g out = IDWT (E LL .E HL .E LH , E HH )

[0056] As shown in Figure 1 , the present patent proposes a decoder suitable for the encoder, which adopts a decoder structure asymmetric with the encoder structure. This asymmetric decoding structure can not only maintain the network performance, but also control the network parameters (use one less WTFM to decode the feature of f out4 ). Using WTFM and residual mechanism, the proposed decoder can well decode the encoded feature with limited information generated by the encoder, and output the expected saliency map.

[0057] Due to the superiority of the overall network architecture design, the present patent only uses the mixed loss function of binary cross entropy (BCE) and intersection union (IOU) to optimize the network at the pixel and object levels to obtain superior performance.

[0058] Step 3: Establishing a salient object detection model;

[0059] The salient object detection model is constructed by cascading a wavelet transform module WTM and a wavelet transform fusion module WTFM to build an encoder and a decoder, and then establishing an "encoder-decoder" architecture.

[0060] Step 4: Image acquisition using a camera to complete real-time salient object detection;

[0061] The collected RGB images are transmitted frame by frame to the salient object detection model, and the feature interaction and fusion in the frequency domain and time domain are completed through discrete wavelet transform and convolution operation, i.e. WTM and WTFM, and the mask image of the salient object is output, realizing the detection and segmentation of dynamic objects at the pixel level.

[0062] The salient object detection model processes visual information in real time in the field of unmanned driving, efficiently detects and segments dynamic objects (people and vehicles, etc.) in the surrounding environment at the pixel level, and also achieves high detection accuracy for small targets, effectively improving the accuracy and robustness of the surrounding environment perception of unmanned vehicles. Moreover, the model can be easily added to devices with limited resources, achieving the effect of plug and play.

[0063] In order to verify that the method can realize a better trade-off among light weight, accuracy and real-time performance, the present application tests on five data sets of DUTS-TE, DUT-OMRON, ECSSD, PASCAL-S and HKU-IS respectively, and the experimental equipment platform is NVIDIA RTX3090 GPU and Intel(R) Xeon(R) Platinum 8260C CPU@2.30GHz. The model of the present application has the light weight index performance of 76K model parameters, 0.38G Floating-Point Operations (FLOPs) and GPU: 70 (CPU: 19) FPS, and the m-Fmeasure (accuracy and recall combined weight) indexes of the present application on the five data sets of DUTS-TE, DUT-OMRON, ECSSD, PASCAL-S and HKU-IS are 0.696, 0.669, 0.858, 0.746 and 0.850 respectively, and the MAE (Mean Absolute Error) indexes are 0.075, 0.083, 0.061, 0.102 and 0.051 respectively. It can be seen that the algorithm of the present application exhibits good detection performance in different complexity environments, and at the same time, the model parameters of 76K and the FLOPs index of 0.38G make the model can be easily embedded into other visual tasks, and also do not increase the calculation burden. At the same time, the high real-time performance on GPU and CPU also guarantees the efficiency of the model.

[0064] The above description is only the preferred embodiments of the present disclosure and the explanation of the applied technical principles. Those skilled in the art should understand that the scope of the application involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with the technical features disclosed in the embodiments of the present disclosure (but not limited to) having similar functions to form technical solutions.

Claims

1. An extremely lightweight real-time salient target detection method, characterized in that, Includes the following steps: Step 1: Establish the wavelet transform module WTM and design an extremely lightweight salient target detection encoder using wavelet transform theory; Step 2: Establish the wavelet transform fusion module WTFM and design an extremely lightweight salient target detection decoder using wavelet transform theory; Step 3: Establish a salient target detection model; Step 4: Use a camera to acquire images and complete real-time salient target detection; The salient target detection encoder described in step 1 is obtained by stacking wavelet transform modules (WTM). The salient object detection encoder outputs features at five different resolution levels, denoted as f from highest to lowest. out1 f out2 f out3 f out4 and f out5 Multi-level features were generated using the 2x downsampling function of the wavelet transform module WTM. The output of WTM is divided into two parts: (1) Output f out Used as input to the next wavelet transform module WTM to continue feature extraction; (2) Subband mapping X LL X LH X HL and X HH The decoder's decoding capability was enhanced through a residual mechanism; The process of establishing the wavelet transform module (WTM) in step 1 includes the following steps: Step S1: Input X is decomposed into 4 sub-band maps by 2D DWT, and the sub-band maps are X. LL The low-frequency components represent the fundamental characteristics of the input X; subband mapping X LH X HL and X HH It represents high-frequency components and indicates the detailed features of the input X; Step S2: Low-frequency component X LL and the fused high-frequency component X high The low-frequency features f are further learned and extracted through convolution to drive the task. low and high-frequency features f high They are represented as follows: ； ； Where DSConv represents depthwise separable convolution, and Concat represents channel-dimensional feature concatenation; Step S3: f low and f high Through convolutional ensemble, low-frequency and high-frequency features complement each other, thus forming a downsampled feature map; the output f out Represented as: ； The salient target detection decoder described in step 2 is obtained by stacking wavelet transform modules (WTFM). The establishment process of the wavelet transform fusion module WTFM in step 2 includes the following steps: Step D1: For input X in After upsampling, Discrete Wavelet Transform (DWT) is performed. The high-frequency characteristics g of the Discrete Wavelet Transform (DWT) are... LL and low-frequency features g HL g HL and g HH The outputs are represented as follows: ； Step D2: Enhance the current feature at the frequency level using a residual mechanism; Enhance Feature E LL E HL E HL and E HH Represented as: ， ； Step D3: Low frequency E LL Components and high-frequency E HL E HL and E HH The fused components are processed by Discrete Wavelet Inverse Transform (IDWT) to obtain the upsampled feature g. out : ； The salient target detection model described in step 3 is to construct an encoder and a decoder by using a cascaded wavelet transform module WTM and a cascaded wavelet transform fusion module WTFM, and then establish an "encoder-decoder" architecture. Step 4 specifically involves transmitting the acquired RGB images frame by frame to the salient target detection model. Through discrete wavelet transform and convolution operations, namely WTM and WTFM, the features are interactively fused in the frequency and time domains, and a mask map of the salient target is output, thus realizing the detection and segmentation of dynamic targets at the pixel level.

Images