A shadow detection method based on random attention mechanism

Abstract

The present invention discloses a shadow detection method based on a random attention mechanism, comprising the following steps: constructing a CNN convolutional neural network to obtain feature maps of different resolutions, i.e., the original skeleton; and convolving the original image using different convolution kernels to obtain feature maps of different resolutions. The present invention adopts a random attention mechanism to randomly extract representative multi-level and multi-layer features in the original skeleton network, and then constructs a symmetrical feature pyramid network from high to low and then from low to high using the skeleton structure and the feature connection structure, and randomly takes multi-level and multi-layer high-level features from the first half of the U-shaped structure to fuse as any feature map of the latter part, and finally constructs a fusion module to fuse the features of the same level in the low to high parts of each feature network, and obtains the final multi-layer feature pyramid and uses it for detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a shadow detection method based on a random attention mechanism. Background Art

[0002] In daily life, shadows caused by blocked light sources are common. Obtaining the position of the shadow can, on the one hand, obtain the direction of the light, the position parameters of the object, and the approximate shape of the object; on the other hand, the shadow will cause the content of the detected target image to be incomplete, block important information, and affect the accuracy of target recognition and target tracking. Therefore, how to accurately detect the position and outline of the shadow is a prerequisite for computer vision research.

[0003] At present, shadow detection can be divided into two categories: one is the traditional shadow detection method, and the other is the shadow detection method based on deep learning. The traditional shadow detection method mainly uses physical models such as color space to set certain feature parameters, such as color, brightness, direction, etc. This method is overly dependent on the selection of artificial parameters and is not suitable for more complex images. Another method, with the rise of convolutional neural networks, many methods for target detection have emerged, and the detection performance has been greatly improved compared to traditional methods.

[0004] So far, there have been many breakthroughs in shadow detection methods using convolutional neural networks. For example, the multi-level feature pyramid shadow detection method solves the problem of small shadows with complex contours, such as the shadows of pedestrians captured by traffic cameras.

[0005] The above shadow detection methods have a significant performance advantage over traditional shadow detection methods from the perspective of data-driven training process. However, these methods have problems with incomplete detection of distant shadow targets or loss of high-level features. The first half of the U-shaped module gradually reduces the resolution of the input feature map, and the semantic information of high-level features is seriously lost. These problems cannot be solved at the same time, which affects the improvement of shadow detection performance based on deep learning. In addition, when using the feature pyramid shadow detection method, the memory and computational complexity will be greatly increased, thereby greatly reducing the efficiency of this type of target detector. The method using the hybrid attention mechanism will also lead to the problem of excessive computation. Summary of the invention

[0006] The purpose of the present invention is to provide a shadow detection method based on a random attention mechanism, which adopts a random attention mechanism to solve the problems raised in the above background.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A shadow detection method based on random attention mechanism includes the following steps:

[0009] S1: Construct a CNN convolutional neural network to obtain feature maps of different resolutions, i.e., the original skeleton. The original image is convolved with different convolution kernels to obtain feature maps of different resolutions. Low-level features are suitable for describing targets with simple appearances and rich spatial information, while high-level features are suitable for describing targets with complex appearances and rich semantic information.

[0010] S2: Construct a feature connection structure to extract representative multi-level and multi-layer features from the original skeleton network;

[0011] S3: Construct a U-shaped module of a multi-level feature pyramid from high to low and then from low to high. Use the obtained basic feature map to obtain two symmetrical feature pyramids with resolutions from high to low and then from low to high. Adjust the symmetrical corresponding layers to have the same feature size and the same number of channels, and use the output of one U-shaped module as the input of another U-shaped module.

[0012] S4: Construct a random attention mechanism to randomly select multi-level and multi-layer high-level feature fusion as any feature of the next part; when constructing the U-shaped module, randomly select multi-level and multi-layer feature maps from the feature maps with high to low resolution in the first half, and fuse them into one feature map through the feature connection structure, and use it as the feature map of any resolution in the lower half (decoding layer) of the U-shaped structure;

[0013] S5: Construct a fusion module to fuse the features of the same level in each feature network from low to high parts to obtain the final multi-layer feature pyramid; use the feature map of the second half (decoding layer) of each U-shaped module to (combine) the feature layers of equivalent scale (size) to form a feature pyramid for final object detection, where each feature map consists of multiple levels of features. This feature pyramid is much deeper than the feature map in the backbone structure and is also more representative. Each feature map contains decoder layers from multiple U-shaped modules;

[0014] S6: Use multi-layer feature pyramid to make predictions.

[0015] Preferably, the random mixed attention mechanism described in step S4:

[0016] (41) Randomly select multi-level and multi-layer feature maps in the first half of the U-shaped model, perform average fusion, obtain a fused shadow map, and use the shadow map as any feature of the second half of the U-shaped model;

[0017] (42) Multi-level and multi-layer feature random extraction;

[0018] (43) The extracted multiple features are connected through the feature structure to obtain a new feature map:

[0019] Using feature maps of multiple levels as input, an upsampling operation is added before the concatenation operation to adjust the size;

[0020] (44) The fused features are up- (down-)sampled and used as a feature of the latter part of the U-shaped module.

[0021] Preferably, the specific implementation method of step S4 is:

[0022] When the U-shaped model is obtained, the size of each resolution feature map has been obtained. Therefore, after obtaining the feature map of the fusion of the multi-layer and multi-level feature maps in step S4, it can be upsampled to make it the same as the size of the random insertion position:

[0023] .

[0024] in, (*) indicates upsampling to random position feature map size.

[0025] Preferably, the specific implementation method of step S3 is:

[0026] Construct a feature connection structure, randomly select multiple layers of multi-level feature maps from the feature maps with high to low resolution in the first half of the U-shaped model, and fuse them into one feature map through the feature connection structure, which is used as a feature map in the second half of the U-shaped model:

[0027] .

[0028] Among them, Up (*) indicates upsampling operation;

[0029] Cat (*) indicates a cascade operation.

[0030] The shadow detection method based on random attention mechanism provided by the present invention has the following beneficial effects:

[0031] The present invention adopts a random attention mechanism to randomly extract representative multi-level and multi-layer features in the original skeleton network, and then uses the skeleton structure and the feature connection structure to construct a symmetrical feature pyramid network (U-shaped structure) from high to low and then from low to high, and randomly takes multi-level and multi-layer high-level features from the first half of the U-shaped structure to fuse as any feature map of the latter part, and finally constructs a fusion module to fuse the features of the same level in each feature network from low to high parts to obtain the final multi-layer feature pyramid and use it for detection. The present invention makes full use of the semantic information of high-resolution features and the spatial information of low-resolution features, and further solves the detection performance of farther and smaller shadows through multi-level random fusion, thereby improving the recognition rate of shadow detection and the adaptability of the detection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The module composition and flow chart of the shadow detection method based on the random attention mechanism according to the embodiment of the present invention;

[0033] Figure 2 Schematic diagram of a shadow detection method based on a random attention mechanism according to an embodiment of the present invention;

[0034] Figure 3 Schematic diagram of the random attention mechanism module structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0036] Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention. Example

[0037] like Figure 1-3 As shown, the shadow detection method based on the random attention mechanism provided by the embodiment of the present invention includes the following steps:

[0038] S1: Construct a CNN convolutional neural network to obtain feature maps of different resolutions, i.e., the original skeleton. The original image is convolved with different convolution kernels to obtain feature maps of different resolutions. Low-level features are suitable for describing targets with simple appearances and rich spatial information, while high-level features are suitable for describing targets with complex appearances and rich semantic information.

[0039] S2: Construct a feature connection structure to extract representative multi-level and multi-layer features from the original skeleton network;

[0040] S3: Construct a U-shaped module of a multi-level feature pyramid from high to low and then from low to high. Use the obtained basic feature map to obtain two symmetrical feature pyramids with resolutions from high to low and then from low to high. Adjust the symmetrical corresponding layers to have the same feature size and the same number of channels, and use the output of one U-shaped module as the input of another U-shaped module.

[0041] The specific steps are:

[0042] When the U-shaped model is obtained, the size of each resolution feature map has been obtained. Therefore, after obtaining the feature map of the fusion of the multi-layer and multi-level feature maps in step 2, it can be up- or down-sampled to make it the same size as the random insertion position:

[0043] ;

[0044] in, (*) indicates upsampling to random position feature map size.

[0045] Construct a feature connection structure, randomly select multiple layers of multi-level feature maps from the feature maps with high to low resolution in the first half of the U-shaped model, and fuse them into one feature map through the feature connection structure, which is used as a feature map in the second half of the U-shaped model:

[0046] .

[0047] Among them, Up (*) indicates upsampling operation;

[0048] Cat (*) indicates a cascade operation.

[0049] S4: Construct a random attention mechanism to randomly select multi-level and multi-layer high-level feature fusion as any feature of the next part; when constructing the U-shaped module, randomly select multi-level and multi-layer feature maps from the feature maps with high to low resolution in the first half, and fuse them into one feature map through the feature connection structure, and use it as the feature map of any resolution in the lower half (decoding layer) of the U-shaped structure;

[0050] The random mixed attention mechanism specifically includes the following steps:

[0051] (41) Randomly select multi-level and multi-layer feature maps in the first half of the U-shaped model, perform average fusion, obtain a fused shadow map, and use the shadow map as any feature of the second half of the U-shaped model;

[0052] (42) Multi-level and multi-layer feature random extraction;

[0053] (43) The extracted multiple features are connected through the feature structure to obtain a new feature map:

[0054] Using feature maps of multiple levels as input, an upsampling operation is added before the concatenation operation to adjust the size;

[0055] (44) The fused features are up- (down-)sampled and used as a feature of the latter part of the U-shaped module.

[0056] S5: Construct a fusion module to fuse the features of the same level in each feature network from low to high parts to obtain the final multi-layer feature pyramid; use the feature map of the second half (decoding layer) of each U-shaped module to (combine) the feature layers of equivalent scale (size) to form a feature pyramid for final object detection, where each feature map consists of multiple levels of features. This feature pyramid is much deeper than the feature map in the backbone structure and is also more representative. Each feature map contains decoder layers from multiple U-shaped modules;

[0057] S6: Use multi-layer feature pyramid to make predictions.

[0058] More briefly, the shadow detection method based on the random attention mechanism provided by the embodiment of the present invention comprises the following steps: obtaining feature maps of different resolutions, that is, the original skeleton. In this example, a CNN convolutional neural network is used. The original image (detection sample) is convolved with different convolution kernels to obtain feature maps of different resolutions;

[0059] Extract the representative multi-level and multi-layer features in the original skeleton network and use them as the first-layer input of the model. Use the feature connection structure to extract the multi-level and multi-layer feature graphs in the previous step and connect them. The obtained feature graph has rich feature information.

[0060] A U-shaped module that builds a multi-level feature pyramid from high to low and then from low to high;

[0061] Through the obtained basic feature map, two symmetrical feature pyramids with high to low resolution and low to high resolution are obtained, and the corresponding symmetrical layers are adjusted to have the same feature size and the same number of channels. The output of one U-shaped module is used as the input of another U-shaped module;

[0062] Randomly extract multi-level and multi-layer feature maps for fusion and construct a random attention mechanism;

[0063] When constructing a U-shaped module, randomly select multiple layers of multi-level feature maps from the feature maps with high to low resolution in the first half of the module, fuse them into one feature map through a feature connection structure, and use it as a feature map of any resolution in the lower half (decoding layer) of the U-shaped module;

[0064] The formula used for feature connection in this step is:

[0065] ,

[0066] This is followed by a feature fusion step;

[0067] Using the feature maps of the second half (decoding layer) of each U-shaped module, the feature layers of equivalent scale (size) are combined to form a feature pyramid for final object detection, where each feature map consists of multiple levels of features. This feature pyramid is much deeper and more representative than the feature maps in the backbone structure, and each feature map contains decoder layers from multiple U-shaped modules;

[0068] Finally, a multi-layer feature pyramid is used for prediction.

[0069] The present invention randomly extracts representative multi-level and multi-layer features in the original skeleton network through a random attention mechanism, and then uses the skeleton structure and the feature connection structure to construct a symmetrical feature pyramid network (U-shaped structure) from high to low and then from low to high. In the U-shaped structure, multi-level and multi-layer high-level features are randomly taken from the first half to be fused as any feature map of the latter part. Finally, a fusion module is constructed to fuse the features of the same level in each feature network from low to high parts to obtain the final multi-layer feature pyramid and use it for detection. The detection performance of farther and smaller shadows is further solved through multi-level random fusion, and the recognition rate of shadow detection and the adaptability of the detection system are improved.

[0070] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A shadow detection method based on random attention mechanism, characterized in that: The following steps are involved: S1: Construct a CNN convolutional neural network to obtain feature maps of different resolutions, i.e., the original skeleton. The original image is convolved with different convolution kernels to obtain feature maps of different resolutions. Low-level features are suitable for describing targets with simple appearances and rich spatial information, while high-level features are suitable for describing targets with complex appearances and rich semantic information. S2: Construct a feature connection structure to extract representative multi-level and multi-layer features from the original skeleton network; S3: Construct a U-shaped module of a multi-level feature pyramid from high to low and then from low to high; obtain two symmetrical feature pyramids with resolutions from high to low and then from low to high through the obtained basic feature map, and adjust the symmetrical corresponding layers to have the same feature size and the same number of channels, and use the output of one U-shaped module as the input of another U-shaped module; S4: Construct a random attention mechanism to randomly select multi-level and multi-layer high-level feature fusion as any feature of the next part; when constructing a U-shaped module, randomly select multi-level and multi-layer feature maps from the feature maps with high to low resolution in the first half, and fuse them into one feature map through the feature connection structure, and use it as the feature map of any resolution in the lower half of the U-shaped module; S5: Construct a fusion module to fuse the features of the same level in each feature network from low to high parts to obtain the final multi-layer feature pyramid; use the feature maps of the second half of each U-shaped module to combine the feature layers of equivalent scales to form a feature pyramid for final object detection, where each feature map consists of multiple levels of features. This feature pyramid is much deeper than the feature maps in the backbone structure and is also more representative. Each feature map contains decoder layers from multiple U-shaped modules; S6: Use multi-layer feature pyramid to make predictions.

2. The shadow detection method based on random attention mechanism according to claim 1, characterized in that: The construction of the random hybrid attention mechanism described in step S4 includes the following steps: (41) Randomly select multi-level and multi-layer feature maps in the first half of the U-shaped model, perform average fusion, obtain a fused shadow map, and use the shadow map as any feature of the second half of the U-shaped model; (42) Multi-level and multi-layer feature random extraction; (43) The extracted multiple features are connected through the feature connection structure to obtain a new feature map: Using feature maps of multiple levels as input, an upsampling operation is added before the concatenation operation to adjust the size; (44) The fused features are upsampled or downsampled and used as a feature in the latter part of the U-shaped module.

3. The shadow detection method based on random attention mechanism according to claim 1, characterized in that: Specific implementation method of step S3: Construct a feature connection structure, randomly select multiple layers of multi-level feature maps from the feature maps with high to low resolution in the first half of the U-shaped model, and fuse them into one feature map through the feature connection structure, which is used as a feature map in the second half of the U-shaped model: F1=Up(F L1 ) F2=Up(F L2 ) F=Cat(F H ,Cat(F L1 , F L2 )); Among them, Up(*) indicates upsampling operation; Cat(*) indicates a cascade operation.

Images