Non-motor vehicle illegal behavior efficient monitoring method and system based on deep learning

Abstract

The invention relates to image data processing, and discloses a non-motor vehicle illegal behavior efficient monitoring method and system based on deep learning, so as to improve the overall performance of recognition. The method comprises the following steps: carrying out feature extraction in a down-sampling process step by step by using a bcheck module of MobileNetV3; in a Neck network layer, a DFSM unit is added in a branch for feature fusion of a C3 module; the DFSM unit comprises an AFA Conv subunit and a CSA Conv subunit; based on an AFA Conv subunit, introducing double attention of a space and a channel as a time domain to enable a local modeling part to have global information; the DFSM is added on the basis of a C3 module, the importance of time domain and frequency domain features in a learning feature graph can be effectively enhanced, key features of a target are better captured by enhancing features with higher distinction degree, and the performance and robustness of the model are improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method and system for efficiently monitoring non-motor vehicle violations based on deep learning. Background Art

[0002] Non-motor vehicles refer to vehicles that are driven by human or animal power and travel on the road, as well as vehicles such as electric bicycles and motorized wheelchairs for the disabled that are driven by power devices but have a designed maximum speed, empty vehicle weight, and external dimensions that meet relevant national standards. Non-motor vehicles include bicycles, tricycles, etc.

[0003] At present, there are few studies on non-motor vehicle violation detection at home and abroad. Although for surveillance videos, it is almost inexhaustible. However, there are too many small samples in the surveillance images, and the surveillance coverage is wide and the vertical height of the surveillance is high, so the images generally contain more samples, such as cars, pedestrians, etc. Therefore, for surveillance images, samples close to the edge of the image will become very small due to the long distance from the surveillance. These samples are small targets. For most of the images in the data set, there are small targets, so small target detection is a difficult point. Effective detection of these small targets can significantly improve the effect of the detection model. For real street images, because there are generally green belts around urban roads, some trees and other vegetation will be planted. Therefore, these trees will block non-motor vehicles, which will reduce the accuracy of network recognition.

[0004] In the existing non-motor vehicle violation monitoring, the target monitoring method uses the YOLOv5 algorithm. Among them, the YOLOv5 backbone Backbone feature extraction network adopts a more complex C3 structure, which brings a large number of parameters, slow detection speed, and limited application. In some real application scenarios such as mobile or embedded devices, such a large and complex model is difficult to be applied. First, the model is too large and faces the problem of insufficient memory. Secondly, these scenarios require low latency, or fast response speed. Therefore, it is most important to study small and efficient CNN models in these fields. Summary of the invention

[0005] The purpose of the present invention is to disclose an efficient monitoring method and system for non-motor vehicle violations based on deep learning to improve the overall recognition performance.

[0006] To achieve the above purpose, the present invention discloses a method for efficiently monitoring non-motor vehicle violations based on deep learning, which includes: Use the bneck module of MobileNetV3 to extract features in the downsampling process step by step; In the Neck network layer, a DFSM (Dual attention in the Frequency and Spatial domains Module) unit is added to a branch of the C3 module for feature fusion; the DFSM unit includes an AFA (Adaptive Frequency Attention) Conv subunit and a CSA (Channel and Spatial Attention) Conv subunit; The data processing process of the AFA Conv subunit is as follows: based on the Fourier transform layer, the spatial domain information of the input feature is converted into frequency domain information and then divided into two outputs, the first output is sequentially output after passing through the first 1x1 convolution layer, the first pooling layer, the first 7x7 convolution layer, and the first activation layer; the first pooling layer performs maximum pooling and average pooling on the features after the output of the first 1x1 convolution layer with two branches respectively, and then fuses them into the input features of the first 7x7 convolution layer; the second output is calculated by point multiplication with the first output after being processed by the first activation layer, and then switched back to the spatial domain through the inverse Fourier transform layer; The data processing process of the CSA Conv subunit is specifically as follows: the input features are first processed by the second 1x1 convolution layer and then input into the second 7x7 convolution layer and then divided into the third and fourth outputs. The third output is processed by two branches based on the channel attention mechanism and the spatial attention mechanism respectively and then added to the fourth output and then input into the second activation layer for activation processing. The output processed by the second activation layer is multiplied with the input features and then output; The DFSM unit is provided with a fifth and a sixth branch. The fifth branch processes the input features in parallel with the CSA Conv subunit and the AFA Conv subunit and then performs feature fusion. The fused features are then processed by GroupNorm normalization, 1x1 convolution and Gelu activation function in sequence, and then added to the features processed by 1x1 depth convolution of the sixth branch and output.

[0007] Preferably, before the first bneck module of the MobileNetV3, the input image is sequentially processed through 3x3 convolution feature extraction, BN normalization processing and h-switch activation function processing.

[0008] Preferably, a total of 15 Bneck modules, namely Bneck module 1 to Bneck module 15, are used, and the convolution kernel sizes of each Bneck module are as follows: 3x3, 3x3, 3x3, 5x5, 5x5, 5x5, 3x3, 3x3, 3x3, 3x3, 3x3, 3x3, 5x5, 5x5, 5x5, and 1x1.

[0009] Preferably, a skip connection is established between two different bneck modules.

[0010] Preferably, a SE structure for adaptively weighting the features of each channel is configured in some bneck modules.

[0011] Preferably, each bneck module adopts a unified Gelu activation function.

[0012] To achieve the above-mentioned purpose, the present invention also discloses an efficient monitoring system for non-motor vehicle violations based on deep learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method when executing the computer program.

[0013] The present invention has the following beneficial effects: 1. Based on the data processing mechanism of the AFA Conv subunit, dual guidance of frequency and space is realized; it can significantly enhance the boundary information of non-motor vehicles while reducing the uneven information inside non-motor vehicles.

[0014] 2. Based on the data processing mechanism of the CSA Conv subunit, spatial attention can make the neural network pay more attention to the pixel areas in the image that are decisive for classification and ignore insignificant areas. Channel attention is used to process the distribution relationship of feature map channels. At the same time, allocating attention to two dimensions enhances the effect of the attention mechanism on improving model performance. For example: Specifically for non-motor vehicle detection, when the non-motor vehicle is larger than the receptive field, the pixels falling into the non-motor vehicle cannot obtain enough information to determine whether it is a detection target. Therefore, we introduce dual attention of space and channel as the time domain to enable the local modeling part to have global information.

[0015] 3. Adding DFSM on the basis of the C3 module can effectively enhance the importance of time domain and frequency domain features in the learning feature graph. By strengthening more discriminative features to better capture the key features of the target, the performance and robustness of the model are improved, and the detection accuracy is further improved.

[0016] 4. Based on the technology disclosed in the present invention, non-motor vehicle detection and identification of whether a helmet is worn can be completed, and a foundation can be laid for further identification of other violations based on the association between different frames according to the DeepSORT algorithm.

[0017] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 It is an architectural block diagram of the backbone network layer of the bneck module based on MobileNet V3 disclosed in an embodiment of the present invention.

[0019] Figure 2 It is an architectural block diagram of the Neck network layer disclosed in an embodiment of the present invention.

[0020] Figure 3 yes Figure 2 The structural block diagram of C3-DFSM.

[0021] Figure 4 It is a structural block diagram of the DFSM unit disclosed in an embodiment of the present invention.

[0022] Figure 5 It is a data processing flow chart of the AFA Conv subunit disclosed in the embodiment of the present invention.

[0023] Figure 6 It is a data processing flow chart of the CSA Conv subunit disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0025] Example 1 This embodiment discloses an efficient monitoring method for non-motor vehicle violations based on deep learning.

[0026] The network model of this embodiment includes an input layer, a backbone network layer based on the bneck module of MobileNet V3, a Neck network layer and a Head network layer; wherein the bneck module is a block containing a convolutional layer, which is used to implement channel-separable convolution + SE (Squeeze-and-Excite) channel attention mechanism + residual connection. The internal structure and working mechanism of the module are existing technologies and will not be elaborated on.

[0027] In the network model of this embodiment, before the first bneck module based on the MobileNetV3 feature extraction network, a convolution layer is first used to perform 3x3 convolution feature extraction, BN normalization processing and h-switch activation function processing on the input image in sequence. Among them, h-swish calculation speed is faster than swish, but slower than ReLU, easier to quantize, and there is no difference in accuracy; the expression of h-switch activation function is: .

[0028] Assume that the input image size is W×W, the convolution kernel size is F×F, the step size is S, the pixel value of the padding is P, and the output image size is N×N. The convolution calculation formula is usually N = (W - F + 2P) / S +1. N in the above formula is a natural number rounded down, and the redundant pixels are not included in the calculation.

[0029] With the help of the aforementioned 3x3 convolution, the number of convolution kernels for the 224x224x3 input image can be reduced from 32 to 16. MobileNetV3 is a lightweight network with advantages such as fewer parameters, less computation, and shorter inference time.

[0030] In this embodiment, if Figure 1 As shown in the figure, the backbone network layer based on the bneck module of MobileNet V3 can use 15 bneck modules from bneck module 1 to bneck module 15 to extract features in the downsampling process step by step. The convolution kernel size of each bneck module is in the following order: 3x3, 3x3, 3x3, 5x5, 5x5, 5x5, 3x3, 3x3, 3x3, 3x3, 3x3, 3x3, 5x5, 5x5, 5x5, 1x1. After the input feature map enters the Block module, the number of channels is first expanded by 1x1 convolution; the purpose is to increase the dimension of feature representation and provide richer feature information for subsequent depth-separable convolution. 3x3 or 5x5 convolution, dilation channel, using stride, by decomposing the standard convolution into two parts: depth convolution (convolution for each input channel separately) and point-by-point convolution (1x1 convolution, fusion of channel information), the amount of parameters and calculation is significantly reduced. The final 1x1 convolution is used to convert the dilated channel to the output channel.

[0031] As an option, a skip connection can be established between two different bneck modules, and a SE structure that adaptively weights the features of each channel can be configured in some bneck modules. In addition, each bneck module can use a unified Gelu activation function.

[0032] In this embodiment, in the Neck network layer, such as Figure 3 As shown in Figure 1, a DFSM unit is added to a branch of the C3 module for feature fusion. Figure 4 As shown, the DFSM unit includes an AFA Conv subunit and a CSA Conv subunit.

[0033] like Figure 5 As shown in the figure, the data processing process of the AFA Conv subunit is as follows: based on the Fourier transform layer, the spatial domain information of the input features is converted into frequency domain information and then divided into two outputs. The first output is sequentially output after passing through the first 1x1 convolution layer, the first pooling layer, the first 7x7 convolution layer, and the first activation layer; the first pooling layer performs maximum pooling and average pooling on the features after the output of the first 1x1 convolution layer with two branches respectively, and then fuses them into the input features of the first 7x7 convolution layer; the second output is calculated by point multiplication with the first output after being processed by the first activation layer, and then switched back to the spatial domain through the inverse Fourier transform layer.

[0034] like Figure 6 As shown in the figure, the data processing process of the CSA Conv subunit is as follows: the input features are first processed by the second 1x1 convolution layer and then input into the second 7x7 convolution layer and then divided into the third and fourth outputs. The third output is processed by two branches based on the channel attention mechanism and the spatial attention mechanism respectively, and then added to the fourth output (the addition operation is to add the corresponding elements, which will not be repeated later), and then input into the second activation layer for activation processing. The output after the second activation layer is processed and the input feature is multiplied and then output.

[0035] like Figure 4 As shown, the DFSM unit has a fifth and a sixth branch. The fifth branch processes the input features in parallel with the CSA Conv subunit and the AFAConv subunit and then performs feature fusion. The fused features are then processed by GroupNorm normalization, 1x1 convolution and Gelu activation function in sequence, and then added to the features processed by the sixth branch by 1x1 depth convolution before output.

[0036] In this embodiment, there is a difference between global and local modeling of target detection in the time domain. Their difference is explained by the limited receptive field of the CNN-based network. The receptive field can be defined as the area around a pixel that contributes to the output of that pixel. Due to the inherent limitations of the convolution operator, the receptive field of the network is limited, especially in the early layers. Therefore, the existing CNN-based methods cannot accurately model the long-range dependencies of the image. Specifically for non-motor vehicle detection, when the non-motor vehicle is larger than the receptive field, the pixels falling into the non-motor vehicle cannot obtain enough information to determine whether it is a detection target. Therefore, this embodiment enables the local modeling part to have global information by introducing the CSA Conv module.

[0037] The attention mechanism in CSA Conv consists of channel attention mechanism and spatial attention mechanism. Spatial attention can make the neural network pay more attention to the pixel areas in the image that are decisive for classification and ignore insignificant areas. Channel attention is used to process the distribution relationship of feature map channels. At the same time, allocating attention to two dimensions enhances the effect of attention mechanism on improving model performance.

[0038] For a given image, low-frequency information is related to slowly changing grayscale components, such as the color and brightness of the image. High-frequency information is related to rapidly changing grayscale components, which are usually manifested as edge contours and speckle noise of objects. After performing a fast Fourier transform (FFT) on the input image, the high-frequency and low-frequency information in the image are adaptively adjusted, and then the input image is reconstructed by inverse FFT. This method can enhance the boundary information of non-motor vehicles while reducing the uneven information inside non-motor vehicles. Therefore, this embodiment introduces the AFA Conv module to achieve such an effect.

[0039] Inspired by the CBAM (Convolutional Block Attention Module, an attention module that combines channel and spatial attention) attention mechanism, mask generation is based on the spatial attention mechanism in CBAM. This approach focuses on identifying the importance of various frequency components rather than distinguishing individual channels. The weight mask shared by all channels ensures that the entire network pays consistent attention to the same spatial location (i.e., frequency component). The differential features are first max-pooled and average-pooled along the channel dimension. Then, a frequency domain feature attention map is generated by a 7×7 convolution followed by a Sigmoid function. Finally, it is multiplied with the original feature map.

[0040] AFA Conv uses Fourier transform (FT) to convert from the spatial domain to the frequency domain for feature extraction, and vice versa, that is, inverse Fourier transform (iFT). It combines the attention mechanism to dynamically and adaptively focus on the information in the key specific frequency domain to achieve feature enhancement. Among them, Fourier transform is a basic technology in frequency domain analysis, which can convert the signal into a domain that is easier to analyze its global statistical characteristics.

[0041] In this embodiment, Figure 2As shown in the figure, the Neck network adopts the feature pyramid structure of FPN. The feature pyramid can balance speed and accuracy, and can be used to obtain more robust semantic information. The mechanism is that there are targets of different sizes in the image, and different targets have different features. Simple targets can be distinguished by shallow features; complex targets can be distinguished by deep features; in the process of signal transmission, the processed low-level features and the processed high-level features are accumulated. The purpose is that the low-level features can provide more accurate location information, and multiple downsampling and upsampling operations make the positioning information of the deep network have errors. Therefore, the combination of the two constructs a deeper feature pyramid, which integrates multiple layers of feature information and outputs it at different features.

[0042] In this embodiment, the detection head module can adopt a multi-level feature fusion method. First, the feature maps of different levels (or scales) output by the backbone network are sequentially passed through a Conv module to reduce the number of channels and scale the feature maps. Then, the feature maps of different levels are fused to obtain richer feature information, thereby improving the detection performance. Head is mainly used for image detection and target classification. The loss function of the output boundary box can be CIoU (Complete Intersection over Union), which can greatly improve the detection accuracy of the algorithm. Furthermore, in the processing of the loss function, SoftNMS (Soft Non-Maximum Suppression) can be used in this embodiment to cleverly solve the problem that when two targets are very close, the low confidence will be suppressed by the high confidence box, and the original score is replaced by a slightly lower score.

[0043] Based on the network model described above, the target object of the non-motor vehicle can be anchored and the classification processing such as whether the driver is wearing a helmet can be identified. Moreover, based on the features extracted by the detection frame, the driving trajectory of the non-motor vehicle can be further tracked based on the association between different frames according to the DeepSORT algorithm to identify other violations. In the process of tracking the driving trajectory, the Mahalanobis distance and the minimum cosine distance formula can be combined to determine the similarity between the two frames of potential association facing each potential tracking object. Furthermore, when there is a new target, the motion features and the minimum cosine distance of the lost target and the new target are analyzed by the Mahalanobis distance formula, the color features between the two targets are analyzed and the target is re-identified, and it is determined whether the new target is a target that has been lost before. If so, the lost ID number is replaced with the new ID number to ensure that the target can keep the same ID number before and after encountering occlusion, which is convenient for subsequent processing of the target data.

[0044] Example 2 Corresponding to the above-mentioned embodiment, this embodiment discloses an efficient monitoring system for non-motor vehicle violations based on deep learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the above-mentioned method is implemented when the processor executes the computer program.

[0045] In summary, the high-efficiency monitoring method and system for non-motor vehicle violations based on deep learning disclosed in the present invention have at least the following beneficial effects: 1. Based on the data processing mechanism of the AFA Conv subunit, dual guidance of frequency and space is realized; it can significantly enhance the boundary information of non-motor vehicles while reducing the uneven information inside non-motor vehicles.

[0046] 2. Based on the data processing mechanism of the CSA Conv subunit, spatial attention can make the neural network pay more attention to the pixel areas in the image that are decisive for classification and ignore insignificant areas. Channel attention is used to process the distribution relationship of feature map channels. At the same time, allocating attention to two dimensions enhances the effect of the attention mechanism on improving model performance. For example: Specifically for non-motor vehicle detection, when the non-motor vehicle is larger than the receptive field, the pixels falling into the non-motor vehicle cannot obtain enough information to determine whether it is a detection target. Therefore, we introduce dual attention of space and channel as the time domain to enable the local modeling part to have global information.

[0047] 3. Adding DFSM on the basis of the C3 module can effectively enhance the importance of time domain and frequency domain features in the learning feature graph. By strengthening more discriminative features to better capture the key features of the target, the performance and robustness of the model are improved, and the detection accuracy is further improved.

[0048] 4. Based on the technology disclosed in the present invention, non-motor vehicle detection and identification of whether a helmet is worn can be completed, and a foundation can be laid for further identification of other violations based on the association between different frames according to the DeepSORT algorithm.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An efficient monitoring method for non-motor vehicle violations based on deep learning, characterized in that: include: Use the bneck module of MobileNetV3 to extract features in the downsampling process step by step; In the Neck network layer, a DFSM unit is added to a branch of the C3 module for feature fusion; the DFSM unit includes an AFA Conv subunit and a CSA Conv subunit; The data processing process of the AFA Conv subunit is as follows: based on the Fourier transform layer, the spatial domain information of the input feature is converted into frequency domain information and then divided into two outputs, the first output is sequentially output after passing through the first 1x1 convolution layer, the first pooling layer, the first 7x7 convolution layer, and the first activation layer; the first pooling layer performs maximum pooling and average pooling on the features after the output of the first 1x1 convolution layer with two branches respectively, and then fuses them into the input features of the first 7x7 convolution layer; the second output is calculated by point multiplication with the first output after being processed by the first activation layer, and then switched back to the spatial domain through the inverse Fourier transform layer; The data processing process of the CSA Conv subunit is specifically as follows: the input features are first processed by the second 1x1 convolution layer and then input into the second 7x7 convolution layer and then divided into the third and fourth outputs. The third output is processed by two branches based on the channel attention mechanism and the spatial attention mechanism respectively and then added to the fourth output and then input into the second activation layer for activation processing. The output processed by the second activation layer is multiplied with the input features and then output; The DFSM unit is provided with a fifth and a sixth branch. The fifth branch processes the input features in parallel with the CSA Conv subunit and the AFAConv subunit and then performs feature fusion. The fused features are then processed by GroupNorm normalization, 1x1 convolution and Gelu activation function in sequence, and then added to the features processed by 1x1 depth convolution of the sixth branch and output.

2. The method according to claim 1, characterized in that Before the first bneck module of MobileNetV3, the input image is processed by 3x3 convolution to extract features, BN normalization and h-switch activation function.

3. The method according to claim 1 or 2, characterized in that: During the downsampling process, 15 bneck modules from bneck module 1 to bneck module 15 are used, and the convolution kernel sizes of each bneck module are as follows: 3x3, 3x3, 3x3, 5x5, 5x5, 5x5, 3x3, 3x3, 3x3, 3x3, 3x3, 3x3, 5x5, 5x5, 5x5, and 1x1.

4. The method according to claim 3, characterized in that A skip connection is established between two different bneck modules.

5. The method according to claim 3, characterized in that: Some bneck modules are configured with a SE structure that adaptively weights the features of each channel.

6. The method according to claim 3, characterized in that Each bneck module uses a unified Gelu activation function.

7. An efficient monitoring system for non-motor vehicle violations based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

Images