Accident detection method, device and equipment based on unmanned aerial vehicle, and storage medium

Abstract

The invention is suitable for the technical field of unmanned aerial vehicles, and provides a traffic event detection method, device and equipment based on an unmanned aerial vehicle, and a storage medium, and the traffic event detection method comprises the steps: obtaining a traffic event detection image through the unmanned aerial vehicle; carrying out defogging processing on the traffic event detection image to obtain a to-be-processed image; inputting the to-be-processed image into the lightweight convolutional neural network model to obtain target identification information; traffic event detection is carried out according to the target identification information, interference of fog on the image is reduced by determining the fog area in the image collected by the unmanned aerial vehicle and carrying out defogging processing on the fog area, the traffic event detection accuracy is improved, the defogging processing area is limited to improve the defogging efficiency, and the defogging efficiency is improved by combining a lightweight convolutional neural network model. Therefore, the traffic incident detection efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of drone technology, and in particular to a drone-based traffic incident detection method, device, equipment and storage medium. Background Art

[0002] With the rapid advancement of urbanization, urban road traffic safety issues are becoming increasingly serious, and frequent traffic incidents have posed a serious threat to people's lives and property safety. Against this background, the rapid development of drone technology has brought new possibilities to the field of traffic incident detection. Drones have many advantages such as flexibility, wide monitoring range, and remote control. They can easily fly over obstacles and obtain more comprehensive traffic conditions information, providing strong support for addressing traffic safety challenges.

[0003] When drones take images in the air, they are often affected by weather factors such as clouds and fog, which causes the quality of the images to deteriorate and affects the accuracy of traffic incident detection. The interference factors of clouds and fog in the images affect the efficiency of traffic incident detection. Summary of the invention

[0004] The present application provides a method, device, equipment and storage medium for detecting traffic incidents based on drones. By determining the foggy area in the image collected by the drone, the foggy area is defogged to reduce the interference of fog on the image, and the accuracy of traffic incident detection is improved. The defogging area is limited to improve the defogging efficiency, and a lightweight convolutional neural network model is combined to improve the efficiency of traffic incident detection.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a detection method is provided, the detection method comprising: acquiring a traffic event detection image, the traffic event detection image being acquired by an image acquisition device of a drone; dividing the traffic event detection image into a first preset number of sub-image blocks, and determining a plurality of fog image blocks in the first preset number of sub-image blocks according to pixel values ​​of the sub-image blocks; determining a fog image area in the traffic event detection image according to the plurality of fog image blocks, the fog image area being formed by the plurality of fog image blocks, and a neighborhood of each of the fog image blocks in the fog image area including at least one adjacent fog image block, the fog image blocks in the fog image area being greater than or equal to a second preset number, the second preset number being determined based on the first preset number; performing defogging processing on the fog image area to obtain an image to be processed; inputting the image to be processed into a lightweight convolutional neural network model to obtain target recognition information; and performing traffic event detection according to the target recognition information.

[0007] In some embodiments, dividing the traffic event detection image into a first preset number of sub-image blocks, and determining a plurality of fog image blocks in the first preset number of sub-image blocks according to pixel values ​​of the sub-image blocks, comprises:

[0008] Obtaining RGB channel pixel values ​​of the traffic event detection image;

[0009] Determine a fog channel image according to the RGB channel pixel values, wherein the pixel at each pixel position in the fog channel image is the minimum value of the three channel pixel values ​​at the corresponding position in the traffic event detection image;

[0010] Dividing the fog channel image into a first preset number of sub-image blocks;

[0011] Performing histogram calculation on each of the sub-image blocks to obtain pixel value distribution information of the sub-image block;

[0012] A plurality of fog image blocks in a first preset number of the sub-image blocks are determined according to the pixel value distribution information.

[0013] In some embodiments, determining a plurality of fog image blocks in a first preset number of sub-image blocks according to the pixel value distribution information includes: obtaining a distribution ratio according to the pixel distribution information, wherein the distribution ratio is a ratio of an area in the sub-image block where the pixel value is greater than a pixel value threshold to an area occupied by the sub-image block;

[0014] If the distribution ratio is greater than the ratio threshold, it is determined that the sub-image block is a foggy image block.

[0015] In some embodiments, the determining of the fog image area in the traffic event detection image according to the plurality of fog image blocks, and performing defogging on the fog image area to obtain the image to be processed includes: determining the category of the sub-image block according to the pixel distribution information, wherein the fog concentration of the sub-image blocks of different categories is different; determining the fog image area according to the category, and determining the defogging strategy corresponding to the fog image area according to the category;

[0016] The corresponding foggy image area is defogged according to the defogging strategy.

[0017] In some embodiments, the lightweight convolutional neural network model includes an input network layer, a backbone network layer, a neck network layer and a prediction network layer. The input network layer is used to process the image to be processed to obtain a target image; the backbone network layer uses GhostNet as a backbone feature extraction network to extract features of the target image and obtain a feature map; the neck network layer processes the feature map to obtain target features; the prediction network layer identifies the target features to obtain target identification information.

[0018] In some embodiments, the integrated attention mechanism SENet is set as the output end of the backbone feature extraction network of the lightweight convolutional neural network model.

[0019] In some embodiments, the lightweight convolutional neural network model is constructed through depthwise separable convolution.

[0020] In a second aspect, a traffic incident detection device is provided, comprising:

[0021] An acquisition module, used to acquire a traffic event detection image, wherein the traffic event detection image is acquired by an image acquisition device of a drone;

[0022] a segmentation determination module, configured to segment the traffic event detection image into a first preset number of sub-image blocks, and determine a plurality of fog image blocks in the first preset number of sub-image blocks according to pixel values ​​of the sub-image blocks; and determine a fog image region in the traffic event detection image according to the plurality of fog image blocks, wherein the fog image region is formed by the plurality of fog image blocks, and a neighborhood of each fog image block in the fog image region includes at least one adjacent fog image block, and the number of fog image blocks in the fog image region is greater than or equal to a second preset number, and the second preset number is determined based on the first preset number;

[0023] A defogging module, used for performing defogging processing on the foggy image area to obtain an image to be processed;

[0024] A recognition module, used for inputting the image to be processed into a lightweight convolutional neural network model to obtain target recognition information;

[0025] A detection module is used to detect traffic incidents based on the target recognition information.

[0026] In a third aspect, a traffic event detection device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the detection method in any optional implementation of the first aspect are implemented.

[0027] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the detection method as described in any one of the first aspects are implemented.

[0028] The detection method, device, equipment and storage medium provided in the embodiments of the present application have the following effective effects:

[0029] The traffic incident detection method provided in the present application determines the foggy area in the image collected by the drone, defogs the foggy area, reduces the interference of fog on the image, improves the accuracy of traffic incident detection, limits the defogging area to improve the defogging efficiency, and combines with a lightweight convolutional neural network model to improve the efficiency of traffic incident detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flow chart of a method for detecting traffic incidents based on a drone provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of a fog image area provided in this application;

[0032] Figure 3 A flow chart of a method for detecting traffic incidents based on a drone provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of determining a fog channel image provided by an embodiment of the present application;

[0034] Figure 5 A schematic diagram of a sub-image block histogram calculation provided in an embodiment of the present application.

[0035] Figure 6 A network structure diagram of a lightweight convolutional neural network provided in an embodiment of the present application;

[0036] Figure 7 It is a structure diagram of GhostNet;

[0037] Figure 8 It is a network structure diagram of deep separable convolution;

[0038] Fig. 9 A schematic diagram of a defogging effect of an embodiment of the present application;

[0039] Fig.10 A schematic diagram of the structure of a traffic incident detection device provided in an embodiment of the present application;

[0040] Fig.11 A schematic diagram of the structure of a traffic incident detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] It should be noted that the terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. In the description of the embodiments of the present application, unless otherwise specified, "multiple" refers to two or more than two, and "at least one", "one or more" refers to one, two or more. The terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, it is defined that the "first" and "second" features can explicitly or implicitly include one or more of the features.

[0042] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0043] To facilitate understanding, the terms involved in the embodiments of the present application are first briefly introduced.

[0044] An embodiment of the present application provides a traffic incident detection method based on a drone. The detection method determines the fog area in the image collected by the drone, defogs the fog area, reduces the interference of fog on the image, improves the accuracy of traffic incident detection, limits the defogging area to improve the defogging efficiency, and combines a lightweight convolutional neural network model to improve the efficiency of traffic incident detection.

[0045] See also Figure 1 , Figure 1 A flow chart of a traffic incident detection method based on a drone provided in an embodiment of the present application, the detection method comprises the following steps:

[0046] S101. Acquire a traffic event detection image, where the traffic event detection image is acquired by an image acquisition device of a drone.

[0047] During the drone cruising process, an image of the target area, namely a traffic event detection image, is obtained through an image acquisition device (such as a camera) installed on the drone, so as to determine whether there is an accident in the target area and the type of accident through the traffic event detection image, wherein the accident may be a traffic accident.

[0048] S102: Segment the traffic event detection image into a first preset number of sub-image blocks, and determine a plurality of fog image blocks in the first preset number of sub-image blocks according to pixel values ​​of the sub-image blocks.

[0049] Optionally, after acquiring the traffic event detection image, the traffic event detection image is divided into n×n sub-image blocks, and the sub-image blocks are numbered k, where i represents the row number and j represents the column number. Then each sub-image block can be represented as:

[0050] k = (i-1) × n + j;

[0051] Optionally, the sub-image block is a square image, that is, the length and width of the sub-image block are the same.

[0052] It is easy to understand that the higher the pixel value (for example, 255), the more likely it is an area affected by fog, and the smaller the pixel value (for example, 0), the more likely it is an area not affected by fog. In this way, the pixel value of each sub-image block can be used to determine whether the sub-image block is affected by fog; if it is affected by fog, the sub-image block is a foggy image block.

[0053] S103. Determine a fog image area in the traffic event detection image based on multiple fog image blocks, where the fog image area is formed by multiple fog image blocks, and a neighborhood of each fog image block in the fog image area includes at least one adjacent fog image block, and the number of fog image blocks in the fog image area is greater than or equal to a second preset number, which is determined based on the first preset number.

[0054] It is easy to understand that in the traffic event detection image, the pixel values ​​of some targets are close to the pixel values ​​of the fog-affected area, such as interference caused by a small range of smoke, flying sand, and white buildings for some reason, for example, the pixel value of the white lane line is close to the pixel value of the fog-affected area. However, the area occupied by the white vehicle line in the image is limited, for example, it is non-continuous and disconnected, while the fog-affected area is a continuous area or multiple areas. Therefore, based on this feature, after determining the fog image block, the fog image area is determined, where the fog image area is the area formed by the fog image blocks that meet specific rules. Since the fog interference area is continuous, the specific rule is that the neighborhood of each fog image block in the fog image area includes at least one fog image block.

[0055] See also Figure 2 , Figure 2 A schematic diagram of a fog image area provided in this application, Figure 2 In the traffic incident detection image in , the fog image area is around the non-fog image area, and the fog image area is formed by a combination of multiple adjacent fog image blocks. If there is a fog image block in the neighborhood of each fog image block, the two fog image blocks are said to be connected, and the fog image area can also be called a connected fog block area.

[0056] It is easy to understand that, since the fog image area is a continuous area, the number of fog image blocks in the fog image area satisfies a certain number. For example, in the present application, the number of fog image blocks is greater than the second preset number.

[0057] In the present application, the second preset number is determined based on the first preset number, and the second preset number is smaller than the first preset number. Optionally, the second preset number may be a specific proportion of the first preset number.

[0058] Optionally, the second preset number is 30% of the first preset number. For example, if the traffic event detection image is divided into 100 pieces, the first preset number is 100 and the second preset number is 30. If the number of connected fog image blocks is 33 and greater than 30, the connected fog image blocks can form a fog image area.

[0059] Optionally, the effect of fog on the traffic incident detection image is different in different areas or at different times in the same area, and the method further includes: obtaining weather information of the area corresponding to the traffic incident detection image; and determining the specific ratio according to the weather information. The specific ratios corresponding to different weather information can be pre-stored, for example, the better the weather, the smaller the specific ratio; the worse the weather, the larger the specific ratio.

[0060] S104: Defogging the foggy image area to obtain an image to be processed.

[0061] Optionally, the fog image area in the traffic event detection image may be defogged using an atmospheric scattering model.

[0062] Optionally, the atmospheric scattering model may be:

[0063] I(t)=J(x)t(x)+A(1-t(x));

[0064] Among them, I(t) is the traffic incident detection image affected by fog, J(x) is the image not affected by fog, A is the atmospheric light value, and t(x) is the transmittance.

[0065] Considering the transmittance t(x) and the atmospheric light value A as constants, we can obtain:

[0066]

[0067] By minimizing both sides of the above equation, we can get:

[0068]

[0069] Among them, A c is the atmospheric light value of the image c channel. Combining the above formula with the atmospheric scattering model formula, we can get:

[0070] Then introduce a constant ω to avoid image color distortion, and get:

[0071] Where ω is 0.95, the transmittance t(x) can be obtained.

[0072] In image processing, the atmospheric light value A is generally equal to the pixel value with the largest value in the image. However, the images taken by video surveillance often contain sky areas. If the pixel value with the largest value in the image is used as the atmospheric light value, there will be a large error between the atmospheric light value and the actual atmospheric light value of the real scene. To avoid this, the grayscale values ​​of the low-value pixels in the first 15% of the transmittance map are regarded as atmospheric light values.

[0073] Obtain the transmittance t(x) and the atmospheric illumination value A, and substitute them into the atmospheric model formula to obtain:

[0074] That is, an image that is not affected by fog is obtained, that is, the image to be processed.

[0075] S105: Input the image to be processed into a lightweight convolutional neural network model to obtain target recognition information.

[0076] Optionally, the lightweight convolutional neural network model has the advantages of small parameters and fast detection speed, which can effectively improve the speed of image recognition.

[0077] S106: Perform traffic incident detection based on the target recognition information.

[0078] Optionally, the recognition information includes target category information, target frame position information and target confidence, so that traffic incident detection can be performed based on the target and target position recognized in the image.

[0079] In this way, the image quality of traffic event detection images is improved by dehazing the images, the efficiency of dehazing is improved by limiting the dehazing area, and the efficiency of image detection is improved by using a lightweight convolutional neural network model. The two are combined to improve the efficiency of traffic event detection.

[0080] In some embodiments, see Figure 3 , Figure 3 A flow chart of a method for detecting traffic incidents based on a drone provided in an embodiment of the present application is provided. Figure 3 In S102, the following are included:

[0081] S301. Obtain pixel values ​​of RGB channels of a traffic event detection image.

[0082] Among them, RGB represents the colors of the three channels of red, green, and blue, that is, the RGB three-channel pixel value of each pixel position in the traffic event detection image is obtained.

[0083] S302, determining a fog channel image according to RGB channel pixel values, wherein the pixel at each pixel position in the fog channel image is the minimum value of the three channel pixel values ​​at the corresponding position in the traffic event detection image;

[0084] Among them, the fog channel image is a single channel image.

[0085] Exemplarily, the pixel values ​​of the three channels at each pixel position in the traffic event detection image are obtained, and the minimum value of the pixel values ​​of the three channels is used as the pixel value of the pixel position:

[0086]

[0087] Among them, J fog is the fog channel image, Ω(x) represents the neighborhood of the target pixel (or pixel position), J c is a channel in the haze-free image J(x) obtained after dehazing.

[0088] See also Figure 4 , Figure 4 (A) is the traffic incident detection image, (B) is the fog channel image, and the fog-free area in the fog channel image is black, so the J fog (x)→0; foggy areas are white or gray.

[0089] S303: Divide the fog channel image into a first preset number of sub-image blocks.

[0090] S304: Perform histogram calculation on each sub-image block to obtain pixel value distribution information of the sub-image block.

[0091] See also Figure 5 , Figure 5 A schematic diagram of sub-image block histogram calculation. Figure 5 The horizontal axis is the pixel value, and the vertical axis is the frequency point. Figure 5 The distribution of pixel values ​​of each sub-image block can be obtained.

[0092] S305 . Determine a plurality of fog image blocks in a first preset number of sub-image blocks according to pixel value distribution information.

[0093] Optionally, whether the sub-image block is a foggy image block is determined according to distribution information of pixel values ​​of each sub-image block, so that a corresponding foggy image block is selected from a plurality of sub-image blocks.

[0094] In this way, by converting the traffic incident detection image into a fog channel image, the fog channel image only contains pixels of a single channel and contains less information, so as to improve the efficiency of the defogging process. Then the fog channel image is divided into multiple sub-image blocks, and the fog image block is selected from the multiple sub-image blocks. The processed object is further limited by the fog image block to improve the efficiency of the defogging process.

[0095] In some embodiments, S304 includes: obtaining a distribution ratio based on pixel distribution information, the distribution ratio being the ratio of an area in a sub-image block whose pixel values ​​are greater than a pixel value threshold to an area occupied by the sub-image block; if the distribution ratio is greater than or equal to the ratio threshold, determining that the sub-image block is a foggy image block.

[0096] In this way, in order to determine whether the sub-image block is a foggy image block, the ratio of the area with pixel values ​​greater than the pixel value threshold to the area occupied by the sub-image block in each sub-image block can be determined according to the pixel distribution information; if the distribution ratio is greater than or equal to the ratio threshold, the sub-image block is determined to be a foggy image block; if the distribution ratio is less than the ratio threshold, the sub-image block is determined to be a non-foggy image block.

[0097] Exemplarily, if the pixel value threshold is 180 and the ratio threshold is 0.55; if the area in the sub-image block with a pixel value greater than 180 occupies 0.5 of the total area of ​​the sub-image block, that is, less than 0.55, then the sub-image block is determined to be a non-fog image block; if the area in the sub-image block with a pixel value greater than 180 occupies 0.75 of the total area of ​​the sub-image block, that is, greater than 0.55, then the sub-image block is determined to be a foggy image block.

[0098] In some embodiments, the method of determining the fog image area in the traffic event detection image according to the plurality of fog image blocks, performing defogging on the fog image area, and obtaining the image to be processed includes: determining the category of the sub-image block according to the pixel distribution information, and the fog concentration of the sub-image blocks of different categories is different; determining the fog image area according to the category, and determining the defogging strategy corresponding to the fog image area according to the category; and defogging the corresponding fog image area according to the defogging strategy. It is easy to understand that the degree of influence of fog on the traffic event detection image is different with different fog concentrations. For example, thin fog has little influence on the image quality; thick fog has a greater influence on the image quality. Then, the sub-image blocks can be divided into corresponding categories according to different fog concentrations. For example, the categories may include no influence, slight influence, moderate influence and severe influence. Then, different categories can adopt different defogging strategies for defogging. Different defogging strategies include whether to perform defogging; for example, if the category is no influence, there is no need to perform defogging on the area; if the area needs to be defogged, the defogging strategy also includes defogging methods, and different types correspond to different defogging methods. Different dehazing methods have different characteristics. For example, the method based on histogram equalization is simple to calculate, does not require redundant information, and can quickly realize single image dehazing. The method based on physical models has high computational complexity and can handle more physical factors, such as light propagation distance, atmospheric concentration, etc., and is suitable for complex scenes. The method based on deep learning can automatically learn image features and haze distribution laws, has strong generalization ability, and may have overfitting problems, that is, the model performs well on the training set but not well on the test set. Therefore, the corresponding dehazing strategy can be selected according to different types, and the image can be dehazed according to the corresponding dehazing strategy.

[0099] See also Figure 6 , Figure 6 A network structure diagram of a lightweight convolutional neural network provided in an embodiment of the present application, Figure 6 The lightweight convolutional neural network includes input layer, backbone network layer, neck network layer and prediction network side.

[0100] The input layer is used to read the image and automatically adjust the image and anchor box to the optimal size.

[0101] Optionally, the input layer is responsible for reading the image into the YOLO network and performing appropriate image processing. For example, for mosaic data enhancement to achieve data set diversification, the anchor box can be adaptively trained according to different data sets to calculate the best anchor box value, and the image size can be adaptively adjusted to improve the neural network operation speed.

[0102] Among them, the backbone network layer (Backbone) is used to extract features in the image and generate feature maps.

[0103] Optionally, the backbone network layer is mainly used to extract features from the image and generate corresponding image feature maps, which facilitates tasks based on image features. The main network structures are the Focus structure and the Cross Stage Partial Network (CSP) structure. The Focus structure is mainly used to slice the image matrix data, and the CSP structure is mainly used to optimize the gradient duplication problem of the backbone network layer, and upsample the calculated gradient change value and map it to the feature map, which enhances the learning ability of the convolutional network, reduces the amount of parameter calculation while ensuring accuracy, and improves the operation speed of the YOLO network.

[0104] Among them, the neck network layer (Neck) is used to learn the features in the image and pass the processed features to the prediction network layer. The neck (Neck) network layer adopts the FPN+PAN structure. The FPN structure provides strong semantic features from the deep network to the shallow network, and the PAN structure provides strong positioning features from the shallow network to the deep network, which enhances the YOLO network's technology for extracting image features.

[0105] Among them, the prediction network layer (Prediction) is used to regress the target bounding box and predict the target type according to the image features. The prediction network layer is also the output layer, which mainly includes the bounding box (bbox) loss function and non-maximum suppression (NMS). The bounding box (bbox) loss function uses GIoU Loss to achieve bounding box regression of the predicted target. Non-maximum suppression (NMS) mainly selects and sets values ​​for multiple target boxes, and takes the one with the highest confidence as the final target box.

[0106] Since the traffic event detection images of this application are taken by drones, and the resolution of the images collected by drones is high, the targets in the images are small and large in number, and the demand for computing resources is large, in order to improve the accuracy and real-time performance of the lightweight convolutional neural network, GhostNet is used to replace the backbone network layer (Backbone) of the lightweight convolutional neural network.

[0107] GhostNet has the advantages of maintaining similar recognition performance and reducing convolution operations, and surpasses MobileNet, SSD and other networks in terms of accuracy and efficiency. It can effectively reduce network parameters and improve network performance.

[0108] Figure 7This is a structural diagram of GhostNet. GhostNet includes Conv2D, GhostBNx, Feat1, Feat2 and Feat3. Conv2D is used to perform two-dimensional convolution operations on the input feature map. GhostBNx represents Ghost Bottle Neck. x represents the number of layers. It is the basic unit of GhostNet. Feat1, Feat2 and Feat3 represent feature maps of three scales respectively. Feat1 obtains feature maps at a lower level in the network. It is used to capture local details and edge information. The initial output of the Conv2D or GhostBNx module is used as its input. The feature map processed by the Ghost module is usually resized to adapt to the input of the subsequent layer. Feat2 is used to obtain the feature map of the middle level. The feature map contains medium-resolution features and captures some more complex texture and shape information. Feat1 or the feature map of the previous layer is used as the input of Feat2. The output of Feat2 is the feature map processed by the Ghost module, which has medium spatial resolution and rich semantic information. Feat3 is a high-level feature map with low spatial resolution but contains higher-level semantic information, such as object categories or complex structures. The feature map of Feat2 or the previous layer is used as the input of Feat3. The output of Feat3 is the feature map processed by the Ghost module.

[0109] Among them, Ghost Bottle Neck includes two Ghost modules; Ghost module uses linear transformation and normal convolution. By performing normal convolution operation on the input image X, the feature map Y is obtained, and then linear operation is performed on Y to obtain the feature map Y^'.

[0110] The first Ghost module of Ghost Bottle Neck is used as an expansion layer to increase the number of channels, while the second Ghost module uses an operation to reduce the number of channels and connects the input and output of these two Ghost modules. Ghost Bottle Neck with Stride 2 is connected to a depth-separable convolutional layer to reduce network parameters.

[0111] The attention mechanism is a structure that enhances the network's attention to important features, thereby improving network performance. This application uses the channel attention mechanism as the output of the backbone feature extraction network in the backbone network layer. By strengthening the network's attention to the relationship between channels to assign feature weights to each channel, the lightweight convolutional neural network model's ability to extract small target features is improved.

[0112] Among them, the channel attention mechanism is SENet.

[0113] In some embodiments, the enhanced feature network in the lightweight convolutional neural network model and the ordinary convolution with a step size of 1 in the prediction network are replaced by deep separable convolutions, that is, a lightweight convolutional neural network model is constructed by deep separable convolutions to effectively reduce the number of convolution parameters and the amount of calculation.

[0114] See also Figure 8 , Figure 8 It is a network structure diagram of a deep separable convolution. Figure 8 In the , the depthwise separable convolution is the core part of MobileNetV1 in the lightweight convolutional neural network model. The depthwise separable convolution includes 3×3DepthwiseConv, BN, ReLU, and 1×1Conv. Among them, the feature map generated by the depthwise convolution (3×3Depthwise Conv) is standardized by BatchNormalization, which helps to reduce the internal covariate shift. The standardized feature map will pass through the ReLU activation function to introduce nonlinear characteristics and retain positive features, thereby improving the learning ability of the model. The feature map processed by the depthwise convolution, BN, and ReLU will be passed as input to the pointwise convolution (1×1Conv). The pointwise convolution linearly transforms the features at each position through a 1×1 convolution kernel, adjusts the depth of the feature map, and integrates the information of each channel. After the depthwise separable convolution operation, it passes through the BN (Batch Normalization) layer and the ReLU (Rectified Linear Unit) layer. The BN layer has the functions of accelerating the training and convergence of the network, controlling gradient explosion, preventing gradient disappearance, and preventing overfitting, while the ReLU layer is the Relu activation function.

[0115] Among them, the depthwise separable convolution includes depthwise convolution and pointwise convolution.

[0116] For example, if the input image size is 5×5×3, the normal convolution kernel is 3×3×4, the depth convolution kernel is 3×3×3, and the point-by-point convolution kernel is 1×1×3×4. Then the parameter quantity under the traditional convolution operation is "Par" "a" _ "norm" = 2700;

[0117] The number of parameters under the depthwise separable convolution operation is "Par" "a" _ "dsc" = 975, so the depthwise separable convolution operation can effectively reduce the number of network parameters.

[0118] Optionally, the YOLOv5-GSD model used in the present invention is implemented in Python and built using the Pytorch framework. A self-built drone aerial photography data set is used; the data set is obtained by capturing frames of the video recorded by the drone's automatic cruise, with a total of 3,000 road images. Among them, the training set accounts for 70% and the test set accounts for 30%. The LabelImg annotation software is used to annotate the images, including pedestrians, cars, trucks, buses, motorcycles, non-motor vehicles, etc., and the annotation generates a json file for subsequent model training and testing.

[0119] Please refer to Table 1, which is a comparison diagram of the lightweight convolutional neural network model (ie, YOLOv5-GSD) provided in this application and the existing YOLOv5.

[0120] Table 1

[0121] Network Model Parameter quantity fps mAP YOLOv5-GSD 37,453,300 30.61 56.03％ YOLOv5 44,296,900 23.75 53.11％

[0122] As shown in Table 1, compared with the existing YOLOv5, the YOLOv5-GSD of this application has a 15.4% decrease in parameters, a 1.86 increase in the detection frame rate per second (fps), and a 2.92% increase in the average detection accuracy (mAP) during the detection process, which can provide target recognition information for subsequent traffic event detection.

[0123] See also Fig. 9 , is a schematic diagram of a defogging effect of an embodiment of the present application, Fig. 9 (A) is the traffic incident detection image before defogging, and (B) is the image to be processed, that is, the image after defogging. Compared with (A), (B) shows the influence of defogging.

[0124] See also Fig.10 , shows a schematic diagram of the structure of a traffic event detection device provided in an embodiment of the present application; for ease of explanation, only the parts related to the embodiment of the present application are shown.

[0125] The detection device may specifically include the following modules:

[0126] An acquisition module, used to acquire a traffic event detection image, wherein the traffic event detection image is acquired by an image acquisition device of a drone;

[0127] a segmentation determination module, configured to segment the traffic event detection image into a first preset number of sub-image blocks, and determine a plurality of fog image blocks in the first preset number of sub-image blocks according to pixel values ​​of the sub-image blocks; and determine a fog image region in the traffic event detection image according to the plurality of fog image blocks, wherein the fog image region is formed by the plurality of fog image blocks, and a neighborhood of each fog image block in the fog image region includes at least one adjacent fog image block, and the number of fog image blocks in the fog image region is greater than or equal to a second preset number, and the second preset number is determined based on the first preset number;

[0128] A defogging module, used for performing defogging processing on the foggy image area to obtain an image to be processed;

[0129] A recognition module, used for inputting the image to be processed into a lightweight convolutional neural network model to obtain target recognition information;

[0130] A detection module is used to detect traffic incidents based on the target recognition information.

[0131] In some embodiments, the traffic event detection image is divided into a first preset number of sub-image blocks, and a plurality of fog image blocks in the first preset number of sub-image blocks are determined according to the pixel values ​​of the sub-image blocks, including: obtaining RGB channel pixel values ​​of the traffic event detection image; determining a fog channel image according to the RGB channel pixel values, wherein the pixel at each pixel position in the fog channel image is the minimum value of the three channel pixel values ​​at the corresponding position in the traffic event detection image; dividing the fog channel image into a first preset number of sub-image blocks; performing histogram calculation on each of the sub-image blocks to obtain pixel value distribution information of the sub-image block; and determining a plurality of fog image blocks in the first preset number of sub-image blocks according to the pixel value distribution information.

[0132] In some embodiments, determining a plurality of foggy image blocks in a first preset number of sub-image blocks according to the pixel value distribution information includes: obtaining a distribution ratio according to the pixel distribution information, wherein the distribution ratio is a ratio of an area in the sub-image block where pixel values ​​are greater than a pixel value threshold to an area occupied by the sub-image block; if the distribution ratio is greater than the ratio threshold, determining that the sub-image block is a foggy image block.

[0133] In some embodiments, the fog image area in the traffic event detection image is determined based on multiple fog image blocks, and the fog image area is defogged to obtain an image to be processed, including: determining the category of the sub-image block based on the pixel distribution information, and the fog concentration of the sub-image blocks of different categories is different; determining the fog image area according to the category, and determining the defogging strategy corresponding to the fog image area according to the category; and defogging the corresponding fog image area according to the defogging strategy.

[0134] In some embodiments, the lightweight convolutional neural network model includes an input network layer, a backbone network layer, a neck network layer and a prediction network layer. The input network layer is used to process the image to be processed to obtain a target image; the backbone network layer uses GhostNet as a backbone feature extraction network to extract features of the target image and obtain a feature map; the neck network layer processes the feature map to obtain target features; the prediction network layer identifies the target features to obtain target identification information.

[0135] In some embodiments, the integrated attention mechanism SENet is set as the output end of the backbone feature extraction network of the lightweight convolutional neural network model.

[0136] In some embodiments, the lightweight convolutional neural network model is constructed through depthwise separable convolution.

[0137] See also Fig.11 , is a schematic diagram of the structure of a traffic incident detection device provided in an embodiment of the present application. As shown in the figure, the traffic incident detection device 110 provided in this embodiment may include: a processor 1140, a memory 1141, and a computer program 1142 stored in the memory 1141 and executable on the processor 1140, such as a program corresponding to the detection method. When the processor 1140 executes the computer program 1142, the steps applied in the detection method embodiment are implemented, such as Figure 1 Steps shown.

[0138] Exemplarily, the computer program 1142 may be divided into one or more modules / units, one or more modules / units are stored in the memory 1141, and executed by the processor 1140 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 1142 in the traffic event detection device 110.

[0139] Those skilled in the art will understand that Fig.11This is merely an example of the traffic event detection device 110 and does not constitute a limitation on the detection device 20 , which may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0140] The processor 1140 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0141] The memory 1141 may be an internal storage unit of the traffic event detection device 110, such as a hard disk or memory of the traffic event detection device 110. The memory 1141 may also be an external storage device of the traffic event detection device 110, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, or a flash card, etc., equipped on an electronic device. Further, the memory 1141 may also include both an internal storage unit of the traffic event detection device 110 and an external storage device.

[0142] The memory 1141 is used to store computer programs and other programs and data required by the electronic device. The memory 1141 can also be used to temporarily store data that has been output or is to be output.

[0143] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional units as needed, that is, the internal structure of the power supply control device can be divided into different functional units to complete all or part of the functions described above. The functional units in the embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0144] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0145] An embodiment of the present application provides a computer program product. When the computer program product is run on a detection device, the detection device implements the steps in the above-mentioned various method embodiments.

[0146] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0147] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0148] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0149] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A traffic incident detection method based on drone, characterized in that: The method comprises: Acquire a traffic event detection image, where the traffic event detection image is acquired by an image acquisition device of a drone; Segmenting the traffic event detection image into a first preset number of sub-image blocks, and determining a plurality of fog image blocks in the first preset number of sub-image blocks according to pixel values ​​of the sub-image blocks; Determine a fog image area in the traffic event detection image according to the plurality of fog image blocks, wherein the fog image area is formed by the plurality of fog image blocks, and a neighborhood of each of the fog image blocks in the fog image area includes at least one adjacent fog image block, and the number of fog image blocks in the fog image area is greater than or equal to a second preset number, and the second preset number is determined based on the first preset number; Defogging the foggy image area to obtain an image to be processed; Inputting the image to be processed into a lightweight convolutional neural network model to obtain target recognition information; Traffic incident detection is performed according to the target recognition information.

2. The method according to claim 1, characterized in that: The step of dividing the traffic event detection image into a first preset number of sub-image blocks, and determining a plurality of fog image blocks in the first preset number of sub-image blocks according to pixel values ​​of the sub-image blocks, comprises: Obtaining RGB channel pixel values ​​of the traffic event detection image; Determine a fog channel image according to the RGB channel pixel values, wherein the pixel at each pixel position in the fog channel image is the minimum value of the three channel pixel values ​​at the corresponding position in the traffic event detection image; Dividing the fog channel image into a first preset number of sub-image blocks; Performing histogram calculation on each of the sub-image blocks to obtain pixel value distribution information of the sub-image block; A plurality of fog image blocks in a first preset number of the sub-image blocks are determined according to the pixel value distribution information.

3. The method according to claim 2, characterized in that The step of determining a plurality of fog image blocks in a first preset number of sub-image blocks according to the pixel value distribution information comprises: Obtaining a distribution ratio according to the pixel distribution information, wherein the distribution ratio is a ratio of an area in the sub-image block where the pixel value is greater than a pixel value threshold to an area occupied by the sub-image block; If the distribution ratio is greater than the ratio threshold, it is determined that the sub-image block is a foggy image block.

4. The method according to claim 2, characterized in that: The step of determining a fog image area in the traffic event detection image according to the plurality of fog image blocks, and performing defogging processing on the fog image area to obtain an image to be processed includes: Determining the category of the sub-image block according to the pixel distribution information, where sub-image blocks of different categories have different fog densities; Determine the foggy image area according to the category, and determine a defogging strategy corresponding to the foggy image area according to the category; The corresponding foggy image area is defogged according to the defogging strategy.

5. The method according to claim 1, characterized in that The lightweight convolutional neural network model includes an input network layer, a backbone network layer, a neck network layer and a prediction network layer. The input network layer is used to process the image to be processed to obtain a target image; the backbone network layer uses GhostNet as a backbone feature extraction network to extract features of the target image and obtain a feature map; The neck network layer processes the feature map to obtain target features; The prediction network layer identifies the target features to obtain target identification information.

6. The method according to claim 5, characterized in that The integrated attention mechanism SENet is set as the output end of the backbone feature extraction network of the lightweight convolutional neural network model.

7. The method according to claim 5 or 6, characterized in that: The lightweight convolutional neural network model is constructed through depth-wise separable convolution.

8. A traffic incident detection device, characterized in that: include: An acquisition module, used to acquire a traffic event detection image, wherein the traffic event detection image is acquired by an image acquisition device of a drone; a segmentation determination module, configured to segment the traffic event detection image into a first preset number of sub-image blocks, and determine a plurality of fog image blocks in the first preset number of sub-image blocks according to pixel values ​​of the sub-image blocks; and determine a fog image region in the traffic event detection image according to the plurality of fog image blocks, wherein the fog image region is formed by the plurality of fog image blocks, and a neighborhood of each fog image block in the fog image region includes at least one adjacent fog image block, and the number of fog image blocks in the fog image region is greater than or equal to a second preset number, and the second preset number is determined based on the first preset number; A defogging module, used for performing defogging processing on the foggy image area to obtain an image to be processed; A recognition module, used for inputting the image to be processed into a lightweight convolutional neural network model to obtain target recognition information; A detection module is used to detect traffic incidents based on the target recognition information.

9. A traffic incident detection device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the detection method according to any one of claims 1 to 7 are implemented.

Images