A single-spectrum night pedestrian detection method and system based on foreground-background contrast attention

Abstract

The present application relates to a kind of single-spectrum night pedestrian detection method and system based on foreground-background contrast attention.The method comprises: the image to be detected is input into the backbone network to extract different scale features containing different semantic information and detail information;Different scale features extracted by the backbone network are input into the feature fusion network, the different scale features are fused by the feature fusion network, and the feature in the foreground feature and background information two dimensions is adaptively adjusted by foreground-background contrast attention, i.e.strengthen the attention degree of foreground feature, weaken the attention degree of background information, so that the network can focus more on foreground feature in night scene;The multi-scale features after fusion are sent to the detection head, and the detection head maps the multi-scale features after fusion into a prediction box, so as to obtain pedestrian detection result.The present application can effectively improve the accuracy of night pedestrian detection.

Description

Technical Field

[0001] This invention belongs to the field of computer vision research technology, and more specifically, it relates to a single-spectrum nighttime pedestrian detection method and system based on foreground-background contrast attention. Background Technology

[0002] Nighttime pedestrian detection has many applications closely related to this invention, such as video surveillance, driver assistance systems, and intelligent robots. While multispectral pedestrian detection technology has achieved satisfactory results in these scenarios, the need to integrate expensive sensors into specific devices makes the research on monospectral nighttime pedestrian detection using only RGB images of significant importance for both research and practice in this field.

[0003] In the era of deep learning, thanks to the rapid rise of object detection, a large number of commendable detectors have emerged. Based on whether they use infrared images, detectors can be divided into multispectral detectors and monospectral detectors. Compared to color cameras, which are poor at acquiring useful information at night, thermal imaging cameras, unaffected by lighting conditions, can overcome some of the limitations of color cameras. To compensate for this lack of useful information, multispectral detectors use both color and thermal images as input. Recently developed nighttime pedestrian detection networks all employ multimodal data. However, the inexpensive and easy-to-use nature of data acquisition with visible light cameras has led to the recent proposal of monospectral nighttime pedestrian datasets, which has contributed to the development of this field.

[0004] Attention mechanisms have proven helpful for a variety of computer vision tasks. A successful example is SENet, which learns weights for each channel through global average pooling and multi-layer fully connected networks, applying them to the input feature map to enhance the network's expressive power. CBAM further advances this idea by introducing spatial information encoding through convolutions with large kernels. Subsequent works have extended this idea by employing different spatial attention mechanisms or designing advanced attention blocks. Self-attention networks have recently become very popular due to their ability to establish spatial or channel attention; they all utilize nonlocal mechanisms to capture different types of spatial information.

[0005] However, none of them effectively utilized background information. Research by psychologists and neuroscientists has shown that background information plays a crucial role in target recognition. Furthermore, for single-spectrum nighttime pedestrian detection, foreground targets and background information are often difficult to distinguish in the detection scene, and are severely affected by artificial illumination. The aforementioned attention mechanisms only consider using foreground target features to adaptively adjust the network, which introduces background information to some extent, thus impairing the network's detection performance in nighttime scenes. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a single-spectrum nighttime pedestrian detection method and system based on Foreground-Background Contrast Attention (FBCA). The model of this invention incorporates a Foreground-Background Attention Feature Enhancement Module (FBCsp) on the backbone network VGG16_BN. Through FBCA enhancement embedded in the FBCsp, foreground features of different scales are obtained from the backbone network, reducing the attention to background information and effectively improving the accuracy of nighttime pedestrian detection.

[0007] The model of this invention mainly consists of the following three key components: 1) This invention proposes a novel attention method: foreground-background contrastive attention. By treating features as a combination of foreground feature channels and background information channels, features are mapped into attention vectors of foreground features and background information, effectively adaptively adjusting foreground features and background information. 2) This invention designs a feature enhancement module based on foreground-background contrastive attention, which can better apply foreground-background attention to correct the attention to input features. Extensive experimental results show that the model of this invention can perform well in nighttime monospectral pedestrian detection. Both qualitatively and quantitatively, the method of this invention outperforms most state-of-the-art nighttime monospectral pedestrian detection algorithms, effectively solving the aforementioned technical problems.

[0008] To address the aforementioned technical problems, this invention provides a single-spectrum nighttime pedestrian detection method and system based on foreground-background contrast attention. The purpose and effectiveness of this method and system are achieved through the following specific technical means:

[0009] A single-spectral nighttime pedestrian detection method based on foreground-background contrast attention includes the following steps:

[0010] The input image to be detected (RGB image) is sent to the backbone network to extract features at different scales containing different semantic and detailed information;

[0011] Features of different scales extracted by the backbone network are input into the feature fusion network (Neck network). The feature fusion network fuses features of different scales and adaptively adjusts the features between the foreground features and background information through foreground-background contrast attention. That is, it strengthens the attention to foreground features and weakens the attention to background information, so that the network can focus more on foreground features in night scenes.

[0012] The fused multi-scale features are fed into the detection head, which maps the features into prediction boxes to obtain pedestrian detection results.

[0013] A single-spectral nighttime pedestrian detection system based on foreground-background contrast attention comprises a backbone network, a feature fusion network based on foreground-background contrast attention, and a detection head. The backbone network extracts features of different scales containing different semantic and detail information from the input image to be detected. The feature fusion network fuses features of different scales and adaptively adjusts the features between foreground features and background information through foreground-background contrast attention, that is, strengthening the attention to foreground features and weakening the attention to background information, so that the network can focus more on foreground features in nighttime scenes. The detection head maps the fused multi-scale features into prediction boxes to obtain pedestrian detection results.

[0014] Furthermore, the backbone network is pre-trained with VGG16_BN, extracting image features at three different scales from shallow to deep layers. Preferably, the backbone network uses VGG16_BN pre-trained on the ImageNet Image dataset.

[0015] Furthermore, the feature fusion network based on foreground-background contrast attention is designed using an FBCsp module based on foreground-background contrast attention. The proposed FBCsp module applies foreground-background contrast attention at two positions (positions 1 and 3, which experiments show offer the best performance) to make the network focus on foreground features while reducing attention to background information. This FBCsp module can correct the degree of attention the network gives to feature channels after multiple feature layers have undergone a concat operation. Therefore, the method of this invention can adjust both foreground features and background information, adaptively correcting the foreground and background feature weights of the feature layers. This allows the branch structure of the FBCsp module to learn more accurate features, resulting in a more spatially accurate representation and ultimately a feature map with rich global contextual information.

[0016] Furthermore, in the FBCsp module, the fused features are adjusted for their foreground and background information; that is, for the fused features... Where C represents the channel, and H and W represent the height and width of the spatial scale, respectively. First, a convolutional kernel is used to compress the channel dimension to one dimension, and then the sigmoid function is used to map the foreground region.

[0017]

[0018] in, CBLR() represents the Conv, BN, and LeakyReLU activation functions, respectively; that is, CBLR() represents these three operations. σ() is the Sigmoid function. Similarly, the background region of the feature layer can be described as:

[0019]

[0020] Next, the foreground activation map and background activation image Ultimately, the feature layer F can be decomposed into foreground feature representation and background feature representation, i.e., v f and v b For a given feature layer, v f and v b It can be described as:

[0021]

[0022]

[0023] in, and F t Flattened, that is and T and T represent matrix multiplication and transpose, respectively.

[0024] Then, a simple gating mechanism with sigmoid activation is chosen to generate foreground-background contrast attention.

[0025]

[0026]

[0027] Where δ represents the LeakyReLU activation function, c f c represents the degree of attention paid to foreground features of pedestrians at night. b This vector represents the degree of attention paid to nighttime background information. and r is the compression ratio, which controls the size of the parameter. Here, c f The focus is on the foreground feature channel, c b The focus is on the background information channel. Then, to further amplify the degree of difference between foreground and background features, c... f c b By performing vector difference, we obtain the foreground-background contrast attention d. w :

[0028] d w =c f -c b

[0029] Finally, the output of the foreground-background contrast attention method of this invention can be written as:

[0030] F′=F·d w

[0031] Furthermore, the detection head used is the detection head proposed by YOLOv6.

[0032] Furthermore, the detection head employs a simpler anchor-free detection method. Anchor-based detectors require clustering analysis before training to determine the optimal anchor set, which increases the detector's complexity. Additionally, in some edge applications, the need to transfer numerous detection results between hardware components introduces further latency. Anchor-free methods, due to their strong generalization ability and simpler decoding logic, have become widely used in recent years.

[0033] The present invention has at least the following beneficial effects:

[0034] This invention provides an end-to-end trainable single-spectral nighttime pedestrian detection network (FBCNet) based on foreground-background contrastive attention, which can effectively detect pedestrians in nighttime scenes. The method is also applicable to daytime pedestrian detection, improving its performance. This invention proposes a novel foreground-background contrastive attention mechanism. The foreground-background contrastive attention module abstracts the feature map into a foreground vector that has a long-range dependency with foreground target features and a background vector that has a long-range dependency with background information. Then, a vector difference operation is performed on the foreground and background vectors to further expand the attention to foreground and background information, achieving correlation modeling of foreground and background information in the input feature map, resulting in a feature map with rich global contextual information. This invention designs an effective attention feature enhancement module (FBCsp) for feature fusion of features extracted from the backbone network at different scales and adaptive adjustment of the degree of attention to the foreground and background. Attached Figure Description

[0035] Figure 1 This is a diagram of the network model of the present invention; where Backbone represents the backbone network, Neck represents the feature fusion network, Head represents the detection head, F represents the FBCsp module (Fore-Background Cross Stage Partial Module), cls and reg represent the target category and bounding box coordinate information output by the detection head, CBS represents conv+BN+Silu, FBCA represents Fore-Background contrast attention, and BottleRep represents the residual structure (see Appendix). Figure 3 Concat stands for concatenate, which concatenates features according to channel dimensions.

[0036] Figure 2This is a diagram of the foreground-background contrast attention model of the present invention;

[0037] Figure 3 This is a schematic diagram of the attention feature enhancement module, i.e., the FBCsp module, of the present invention;

[0038] Figure 4 Comparison of ablation effects of the attention feature enhancement module of the present invention. Detailed Implementation

[0039] The embodiments of the present invention will be described in further detail below through examples. These examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0040] In the description of this invention, unless otherwise stated, "a plurality of" means two or more; the terms "coaxial," "bottom," "one end," "top," "middle," "other end," "upper," "side," "top," "inner," "front," "center," "both ends," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0041] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "setting," "connection," "fixing," "screw connection," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Unless otherwise explicitly limited, those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0042] Example:

[0043] This invention provides a single-spectrum nighttime pedestrian detection system based on foreground-background contrast attention, such as... Figure 1 As shown, the detection network used in this method includes a backbone network, a feature fusion network (Neck), and a detection head.

[0044] The backbone network is used to extract features from nighttime images. For a nighttime pedestrian image with input HxW, it outputs three features F3, F4 and F5 with different semantic information, and their spatial scales are downsampled to 1 / 8, 1 / 16 and 1 / 32 respectively.

[0045] The feature fusion network includes a feature fusion module based on foreground-background contrast attention, namely the FBCsp module (FBCspBlock). Before passing through the FBCsp module of the Neck network, nighttime pedestrian features at different spatial scales must be upsampled or downsampled to achieve the same spatial dimension. The FBCsp module concatenates the adjusted features, which contain different semantic information, using a concat operation, and then passes them through Conv... 1x1 The model compresses channels while fusing channel information. Due to varying illumination levels in nighttime scenes, the backbone network learns insufficient semantic information about blurred and occluded pedestrian targets. The FBCA embedded in the FBCsp module treats compressed nighttime pedestrian features as a combination of channels focusing on different nighttime pedestrian features or background information. FBCA utilizes the global spatial information of nighttime pedestrian features to map features into nighttime pedestrian feature vectors and background vectors representing the importance of each channel. By calculating the vector difference between the nighttime pedestrian feature vectors and background vectors, the model focuses on the contrastive learning of differences between nighttime pedestrian features and background information. For problems such as noise, uneven illumination, shadows, and motion blur caused by changes in nighttime illumination, this contrastive learning allows the model to further distinguish the semantic information of nighttime images. Combined with the newly designed Neck network, which further enhances semantic information exchange, the model can fully enhance semantic information within the Neck network. The Neck network has three outputs, namely F3... ′ F4 ′ and F5 ′ .

[0046] The detection head uses a decoupled head structure from YOLOv6. Finally, the detection head maps the output of the Neck network to a bounding box.

[0047] As described above, the backbone network uses VGG16_BN pre-trained on the ImageNet Image dataset and extracts features at three different scales. The FBCsp module can correct the network's focus on feature channels after multiple feature layers perform a concat operation. Therefore, the method of this invention can adjust in two dimensions: foreground features and background information. It can adaptively correct the foreground and background feature weights of the feature layers, enabling the branch structure of the FBCsp module to learn more accurate features, thus making the learned representation more accurate in space and further obtaining a feature map with rich global contextual information.

[0048] As described above, this invention proposes a novel attention method: foreground-background contrastive attention. By abstracting features into a combination of foreground features and background information, a foreground-background attention vector is generated, effectively and adaptively adjusting the foreground features and background information.

[0049] As described above, this invention designs a feature enhancement module based on foreground-background contrast attention, namely the FBCsp module, which can better correct the attention to input features. Extensive experimental results demonstrate that the model of this invention can perform well in nighttime monospectral pedestrian detection. Both qualitatively and quantitatively, the method of this invention outperforms most state-of-the-art nighttime monospectral pedestrian detection algorithms, effectively solving the aforementioned technical problems.

[0050] This embodiment presents a single-spectrum nighttime pedestrian detection method based on foreground-background contrast attention. The steps of the method are as follows:

[0051] First, the input RGB image is fed into the backbone network to extract features at three different scales containing different semantic and detail information.

[0052] Second, the multi-scale features extracted by the backbone network are input into the feature fusion network. The FBCsp module in the feature fusion network fuses the features of different scales and adaptively adjusts the features between the foreground features and background information through foreground-background contrast attention. That is, it strengthens the attention to foreground features and weakens the attention to background information, so that the network can focus more on foreground features in night scenes.

[0053] Third, the fused multi-scale features are fed into the detection head, which maps the features into prediction boxes.

[0054] In step two, the foreground-background contrast attention model in the FBCsp module is as follows: Figure 2 As shown, F represents the feature, k() represents the convolution kernel that compresses the channels to one dimension, s() represents the Sigmoid function, Flatt. represents the flattening operation, T represents the transpose, and l() represents the combination of a fully connected layer and the Sigmoid function. For the fused features, this invention adjusts their foreground and background information; that is, for the fused features... This invention first uses a convolutional kernel to compress the channel dimension to one dimension, and then uses the sigmoid function to map the foreground and background regions:

[0055]

[0056] in, CBLR() represents the Conv, BN, and LeakyReLU activation functions, respectively. σ() is the Sigmoid function. Similarly, the background region of the feature layer can be described as:

[0057]

[0058] Next, the foreground and background activation maps can ultimately decompose the feature layer F into foreground and background feature representations, i.e., v f , and v b For a given feature layer, v f and v b It can be described as:

[0059]

[0060]

[0061] in, With F T Flattened, that is and T and T represent matrix multiplication and transpose, respectively.

[0062] Finally, this invention chooses to use a simple gating mechanism with sigmoid activation to generate foreground-background contrast attention.

[0063]

[0064]

[0065] Where δ represents the LeakyReLU activation function, and r is the compression ratio that controls the block size. Here, c f The focus is on the foreground feature channel, c b The focus is on the background information channel. Then, to further expand the degree of difference between foreground and background features, this invention addresses c... f c b Divide the vectors:

[0066] d w =c f -c b

[0067] The output of the foreground-background contrast attention in this invention can be written as:

[0068] F′=F·d w

[0069] Figure 3This is a schematic diagram of the attention feature enhancement module, FBCsp, of the present invention, and is an example of using FBCA. The FBCsp module has two branches. When a feature enters a branch, it needs to be compressed in terms of channel dimension using a 1x1 convolution to reduce computational cost and feature fusion. Then, the features of one branch are adaptively adjusted using FBCA and then enter the residual structure BootleRep. The features of the two branches are then concatenated along the channel dimension, and then 1x1 convolution is used to reduce dimensionality to compress channel and fusion information. Finally, FBCA is used again to adaptively adjust the importance of each channel of the feature.

[0070] Figure 4 The images show a comparison of the ablation effects of the attention feature enhancement module of this invention. The first row of images shows the effect of the baseline model, and the second row shows the effect of the method proposed in this invention. It can be seen that, in the first image, the method of this invention can detect pedestrians in severely limited illumination. In the second and third images, the method of this invention can detect small-target pedestrians in nighttime scenes.

[0071] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0072] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0073] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A single-spectral nighttime pedestrian detection method based on foreground-background contrast attention, characterized in that, Includes the following steps: The image to be detected is input into the backbone network to extract features at different scales containing different semantic and detail information; Features of different scales extracted by the backbone network are input into the feature fusion network. The feature fusion network fuses features of different scales and adaptively adjusts the features between the foreground features and background information through foreground-background contrast attention. That is, it strengthens the attention to foreground features and weakens the attention to background information, so that the network can focus more on foreground features in night scenes. The fused multi-scale features are fed into the detection head, which maps the fused multi-scale features into prediction boxes, thereby obtaining the pedestrian detection results. The feature fusion network includes an FBCsp module based on foreground-background contrast attention. The FBCsp module applies foreground-background contrast attention, enabling the network to focus on foreground feature information while weakening the focus on background information. The FBCsp module can correct the degree of attention the network pays to feature channels after multiple feature layers perform a concat operation. It can adjust in two dimensions: foreground features and background information, to adaptively correct the foreground and background feature weights of the feature layers, making the learned representation more accurate in space and obtaining a feature map with rich global context information. The processing procedure of the FBCsp module include: For features Where C represents the channel, and H and W represent the height and width of the spatial scale, respectively, a convolutional kernel is first used to compress the channel dimension to one dimension, and then the sigmoid function is used to map the foreground region: in, express BN and Activation function, i.e., using represent BN and Three operations, The Sigmoid function is used; the background region of the feature layer is described as follows: Then, and Feature layer It is decomposed into foreground feature representation and background feature representation, that is and ; Then, a simple gating mechanism with Sigmoid activation is chosen to generate foreground-background contrast attention. : in, express Activation function This represents the degree of focus on foreground features of pedestrians at night. This vector represents the degree of attention paid to nighttime background information. , , It is the compression ratio that controls the size of the parameter; Finally, focus on the contrast between the foreground and background. Acting on the original features ,get ; The FBCsp module has two branches. When a feature enters a branch, it first needs to be compressed using a 1x1 convolution to reduce the computational cost and facilitate feature fusion. Then, the features of one branch are adaptively adjusted using foreground-background contrast attention, and then enter the residual structure BootleRep. The features of the two branches are then concatenated according to the channel dimension, and after concatenation, they are reduced in dimensionality using a 1x1 convolution to compress the channels and fuse information. Finally, foreground-background contrast attention is used to adaptively adjust the importance of each channel of the feature.

2. The method according to claim 1, characterized in that, The backbone network is a VGG16_BN network pre-trained on the ImageNet Image dataset, which extracts image features at three different scales from shallow to deep layers.

3. The method according to claim 1, characterized in that, The detection head used is the detection head from YOLOv6.

4. The method according to claim 1, characterized in that, The detection head uses an anchor-free detection method for pedestrian detection.

5. A single-spectrum nighttime pedestrian detection system based on foreground-background contrast attention using the method described in any one of claims 1 to 4, characterized in that, The system comprises a backbone network, a feature fusion network based on foreground-background contrast attention, and a detection head. The backbone network extracts features of different scales containing different semantic and detail information from the input image to be detected. The feature fusion network fuses features of different scales and adaptively adjusts the features between foreground features and background information through foreground-background contrast attention, i.e., strengthening the attention to foreground features and weakening the attention to background information, so that the network can focus more on foreground features in nighttime scenes. The detection head maps the fused multi-scale features into prediction boxes to obtain pedestrian detection results.

6. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 4.

Images