Electric vehicle rider helmet wearing monitoring system based on unmanned aerial vehicle aerial photography

Abstract

This invention discloses a helmet-wearing monitoring system for electric bicycle riders based on drone aerial photography. The system includes a gimbal mounted on the drone, with a camera integrated on the gimbal. The drone performs low-speed aerial photography over a designated road, transmitting the captured road images to a processing unit. The processing unit detects whether any electric bicycle riders are not wearing helmets in the road images; if so, the drone issues a helmet-wearing reminder. This invention improves the YOLOv7 network structure to enhance the helmet detection accuracy for electric bicycle riders, thereby addressing the issue of riders not wearing helmets.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and more specifically, this invention relates to an electric vehicle rider helmet wearing monitoring system based on drone aerial photography. Background Technology

[0002] In actual road driving, to obtain limited right-of-way, electric vehicles frequently interact with other road users, which can easily lead to collisions and traffic accidents. Helmets, as protective equipment for riders' heads, can effectively reduce the severity of head injuries in traffic accidents, thus ensuring rider safety. However, current checks on helmet use by electric vehicle riders mainly rely on manual methods, making continuous enforcement difficult. The phenomenon of not wearing helmets as required by law often rebounds rapidly after a period of strict enforcement. Therefore, ensuring that electric vehicle riders wear helmets is a problem that urgently needs to be addressed.

[0003] Using drones to monitor electric scooter helmets is a feasible solution to improve helmet use among riders. However, this solution faces several challenges, including:

[0004] Drone aerial photography mostly detects small targets. When the network model uses convolutional neural networks to extract features from small targets, the feature information of small targets will be severely lost during the continuous convolution process, which will lead to a deviation in the detection effect of small targets in aerial photography and ultimately affect the detection accuracy of electric vehicle helmets. Summary of the Invention

[0005] This invention provides a helmet-wearing monitoring system for electric vehicle riders based on drone aerial photography, aiming to improve the above-mentioned problems.

[0006] This invention is implemented as follows: a helmet-wearing monitoring system for electric vehicle riders based on drone aerial photography, characterized in that the system comprises:

[0007] A gimbal mounted on a drone integrates a camera. The drone performs low-speed aerial photography over a designated road and transmits the captured road images to a processing unit.

[0008] The processing unit detects whether there are any electric bike riders without helmets in the road images. If so, the drone issues a reminder to wear a helmet.

[0009] Furthermore, the YOLOv7 network outputs the detection boxes for electric vehicles, riders, and helmets in the road image, along with the confidence scores for each detection box.

[0010] Furthermore, the specific methods for identifying electric bike riders who are not wearing helmets are as follows:

[0011] The system reads the detection box and its confidence level of the electric vehicle in the road image. It checks whether the confidence level of the electric vehicle's detection box reaches a set confidence threshold. If the detection result is yes, it checks whether there is a rider's detection box within the electric vehicle's detection box. If there is, it checks whether the confidence level of the rider's detection box is greater than the confidence threshold. If the detection result is yes, it checks whether there is a helmet's detection box within the rider's detection box. If the detection result is no, it is determined that the rider is not wearing a helmet. If the detection result is yes, it checks whether the confidence level of the helmet's detection box is greater than the confidence threshold. If the detection result is yes, it is determined that the rider on the electric vehicle is wearing a helmet. If the detection result is no, it is determined that the rider on the electric vehicle is not wearing a helmet.

[0012] Furthermore, the YOLOv7 network outputs the detection boxes of electric bike riders and helmets in the road image, as well as the confidence scores of each detection box.

[0013] Furthermore, the specific methods for identifying electric bike riders who are not wearing helmets are as follows:

[0014] Treating the electric scooter and its rider as a target object, the system reads the detection box containing the target object and its confidence level in the road image. It then checks whether the confidence level of the target object's detection box reaches a set confidence threshold. If the result is yes, it checks whether a helmet detection box exists within the target object's detection box. If the result is no, it is determined that the rider is not wearing a helmet. If the result is yes, it checks whether the confidence level of the helmet detection box is greater than a confidence threshold. If the result is yes, it is determined that the rider on the electric scooter is wearing a helmet. If the result is no, it is determined that the rider on the electric scooter is not wearing a helmet.

[0015] Furthermore, the YOLOv7 network consists of an Input layer, a Backbone layer, a Neck layer, and a Head layer connected in sequence; the last and penultimate ELAN modules in the Backbone layer are followed by IECA modules.

[0016] Furthermore, the IECA module includes:

[0017] The MaxPool and AvgPool layers are connected in parallel. The output of the MaxPool layer is connected to the convolutional layer for output, and the input is fused into the Concat fusion layer. The output of the AvgPool layer is connected to the convolutional layer for output, and the input is fused into the Concat fusion layer. The Concat fusion layer is connected to the Sigmoid function and multiplied with the input of the IECA module for output.

[0018] Furthermore, a fusion module Fusion is set in the Ncek layer. Two adjacent ELAN modules in the Backbone layer are connected to a fusion module Fusion through two CBS modules respectively, which are used to fuse the feature image information output by the Backbone layer.

[0019] Furthermore, the Fusion module includes:

[0020] The Concat fusion layer 1 is connected to the input terminal. The Concat fusion layer 1 is connected to two parallel convolutional layers Conv. One of the convolutional layers Conv is connected in series with three convolutional groups consisting of deformable convolutional layer DConv and convolutional layer Conv. The output terminal of the other convolutional layer Conv and each convolutional group is connected to the Concat fusion layer 2. The output terminal of the Concat fusion layer 2 is the output terminal of the Fusion module.

[0021] Furthermore, the Head layer uses the KLD loss function to calculate the regression loss of the detection box and outputs a (x,y,w,h,θ) rotated bounding box, where (x,y) represent the coordinates of the center point of the rotated bounding box, w and h represent the long side and short side of the rotated bounding box, respectively, and θ is the angle between the positive x-axis and the long side w.

[0022] This invention improves the detection accuracy of electric bicycle helmets by modifying the YOLOv7 network structure, thereby addressing the issue of electric bicycle riders not wearing helmets. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the structure of a drone-based aerial photography system for monitoring the helmet wearing of electric vehicle riders, provided in an embodiment of the present invention.

[0024] Figure 2 A schematic diagram of the structure of the improved YOLOv7 network is provided for embodiments of the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of the IECA module provided in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of the Fusion module provided in an embodiment of the present invention;

[0027] Figure 5 A flowchart of a method for monitoring helmet wearing by electric vehicle riders using drone aerial photography, provided in an embodiment of the present invention. Detailed Implementation

[0028] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0029] Figure 1 This is a schematic diagram of a drone-based aerial photography system for monitoring helmet wearing on electric bicycle riders, provided in an embodiment of the present invention. For ease of explanation, only the parts relevant to this embodiment are shown. The system includes:

[0030] A gimbal mounted on a drone integrates a camera. The drone performs low-speed aerial photography over a designated road and transmits the captured road images to a processing unit.

[0031] The processing unit detects whether there are any electric bike riders not wearing helmets in the road images. If so, the drone issues a reminder to wear a helmet.

[0032] In this embodiment of the invention, the camera inputs real-time aerial images of the road into an improved YOLOv7 network. The improved YOLOv7 network outputs bounding boxes for electric vehicles, riders, and helmets in the road images, along with the confidence scores of each bounding box. Based on the bounding boxes in the road images, electric vehicle riders not wearing helmets are identified. The specific identification method is as follows:

[0033] Method 1: Read the detection box and confidence score of the electric vehicle in the road image. Check if the confidence score of the electric vehicle detection box reaches the set confidence threshold. If the detection result is yes, it is determined that there is an electric vehicle in the road image. Then, check if there is a rider detection box within the electric vehicle detection box. If there is, determine if the confidence score of the rider detection box is greater than the confidence threshold. If the detection result is yes, it is determined that there is a rider on the electric vehicle. Then, check if there is a helmet detection box within the rider detection box. If the detection result is no, it is determined that the rider is not wearing a helmet. If the detection result is yes, check if the confidence score of the helmet detection box is greater than the confidence threshold. If the detection result is yes, it is determined that the rider on the electric vehicle is wearing a helmet. If the detection result is no, it is determined that the rider on the electric vehicle is not wearing a helmet.

[0034] In this embodiment of the invention, during the process of constructing samples, it is necessary to label the collected sample images. If the rider on the electric vehicle in the sample image is wearing a helmet, the detection box of the electric vehicle in the sample image is set as the smallest detection box containing the electric vehicle, the rider and the helmet they are wearing, the detection box of the rider is set as the smallest detection box containing the rider and the helmet they are wearing, and the helmet detection box is set as the smallest detection box containing the helmet. If the rider on the electric vehicle in the sample image is not wearing a helmet, the detection box of the electric vehicle in the sample image is set as the smallest detection box containing the electric vehicle and the rider, the detection box of the rider is set as the smallest detection box containing the rider, and no helmet detection box is set.

[0035] The camera inputs real-time aerial images of the road into an improved YOLOv7 network. The improved YOLOv7 network outputs bounding boxes for electric bike riders and helmets in the road images, along with the confidence scores of each bounding box. Based on these bounding boxes, electric bike riders without helmets are identified. The specific identification method is as follows:

[0036] Method 2: Treat the electric bike and its rider as a target object. Read the detection box containing the target object and its confidence level in the road image. Check if the confidence level of the detection box containing the target object reaches a set confidence threshold. If the detection result is yes, it is determined that there is a rider riding an electric bike in the road image. Then, check if there is a helmet detection box within the detection box containing the target object. If the detection result is no, it is determined that the rider is not wearing a helmet. If the detection result is yes, check if the confidence level of the helmet detection box is greater than the confidence threshold. If the detection result is yes, it is determined that the rider on the electric bike is wearing a helmet. If the detection result is no, it is determined that the rider on the electric bike is not wearing a helmet.

[0037] In this embodiment of the invention, during the process of constructing samples, it is necessary to label the collected sample images. If the rider on the electric vehicle in the sample image is wearing a helmet, the detection box of the target object in the sample image is set as the smallest detection box that includes the electric vehicle, the rider and the helmet they are wearing, and the helmet detection box is set as the smallest detection box that includes the helmet. If the rider on the electric vehicle in the sample image is not wearing a helmet, the detection box of the target object in the sample image is set as the smallest detection box that includes the electric vehicle and the rider, and no helmet detection box is set.

[0038] In this embodiment of the invention, combined with Figure 2 The improved YOLOv7 network structure is described in detail. The YOLOv7 network structure consists of an Input terminal, a Backbone layer, a Neck layer, and a Head layer connected in sequence. An IECA module is added after the ELAN module in the deep structure of the Backbone layer.

[0039] The IECA module is set after the last and second-to-last ELAN module in the Backbone layer. The structure of the IECA module is as follows: Figure 3 As shown, the IECA module includes: a MaxPool layer and an AvgPool layer connected in parallel; the output of the MaxPool layer is connected to the convolutional layer for output, and the input is fused into the Concat fusion layer; the output of the AvgPool layer is connected to the convolutional layer for output, and the input is fused into the Concat fusion layer; the Concat fusion layer connects the Sigmoid function and multiplies it with the input of the IECA module for output.

[0040] The IECA module is placed in the last layer and the fourth-to-last layer of the Backbone layer. It uses global average pooling and max pooling operations to aggregate the spatial information of the feature map. The calculation is as follows:

[0041] M(F)=σ(Conv(AvgPool(F))+Conv(MaxPool(F)))

[0042] Here, σ represents the sigmoid non-linear activation function. First, global average pooling and max pooling operations are used to aggregate the spatial information of the feature maps, followed by optimization using convolution. Max pooling and average pooling can increase the receptive field, maintain image shift invariance, and reduce network parameters. Then, the sigmoid activation function is applied to transform the features of channels with higher importance, compensating for certain information in the depthwise convolution. Finally, M(F) is multiplied by the initial input, allowing the network to focus more on regions of interest in the image, helping the backbone feature network extract important features, thus fully acquiring image feature information.

[0043] An IECA module was added to the backbone feature extraction part of the improved YOLOv7 model. Since the helmet target in drone aerial photography is a small target with few pixels in the image, this module helps the network model quickly and effectively obtain the helmet's characteristic information in the image region, indicating where the target appears in the image. At the same time, small targets have limited detail, and repeated convolutions during feature extraction can lead to the loss of spatial information about small targets. IECA utilizes the characteristic of acquiring target spatial information in the image to improve the target spatial structure, thereby enabling the acquisition of spatial dependencies. The IECA module can fully utilize the spatial information of small targets captured by aerial photography, accurately acquiring regions of interest in the feature map, helping the model better extract feature information of small targets like drone-captured helmets.

[0044] In this embodiment of the invention, a fusion module Fusion is provided in the Ncek layer, and two adjacent ELAN modules in the Backbone layer are respectively connected to a fusion module Fusion through two CBS modules, for fusing the feature image information output by the Backbone layer. Figure 4 The Fusion module is described in detail. It includes a Concat fusion layer 1 connected to the input. The Concat fusion layer 1 is connected to two parallel convolutional layers Conv. One of the convolutional layers Conv is connected in series with three convolutional groups consisting of a deformable convolutional layer DConv and a convolutional layer Conv. The output of the other convolutional layer Conv and each convolutional group is connected to the Concat fusion layer 2. The output of the Concat fusion layer 2 is the output of the Fusion module.

[0045] The improved YOLOv7 network employs a Fusion module as a new feature fusion module. The Fusion module integrates feature maps from three adjacent layers and upgrades the Concat fusion layer by introducing deformable convolutions and efficient layer aggregation networks. This enhances the detection performance without significantly increasing computational burden, addressing the issue of insufficient detail information for small targets. The network model can then fuse more feature information from small targets, achieving higher detection accuracy. Simultaneously, it helps the network aggregate high-level and low-level semantic features, providing more accurate recognition. In this context, fusing more accurate feature information from small targets is crucial for detecting small targets in drone aerial photography of helmet-mounted targets.

[0046] In this embodiment of the invention, the KLD loss function is used to design the bounding box regression loss in the Head layer.

[0047] Existing object detection methods typically use horizontal detection, using four parameters (x, y, w, h) to represent the horizontal bounding box. These parameters represent the center coordinates (x, y), width (w), and height (h) of the bounding box. However, horizontal bounding boxes cannot effectively detect targets in different directions and are subject to background interference, thus having certain limitations. Rotational object detection, in addition to obtaining the horizontal bounding box, adds prediction information about the rotation angle, enabling the network to acquire more comprehensive feature information about the target. The rotational bounding box is determined by five parameters (x, y, w, h, θ), where (x, y) represent the center coordinates of the rotational bounding box, w and h represent the long and short sides of the rotational bounding box, respectively, and θ is the angle between the positive x-axis and the long side w, ranging from -90° to 90°. θ is used to introduce the target's angular information into the network.

[0048] When adding angular parameters to the network, the regression of rotated bounding boxes can lead to rotation sensitivity errors, causing instability during model training and affecting detection accuracy. Therefore, the invention employs the KLD (Kullback-Leibler Divergence) loss function to calculate the bounding box regression loss. The KLD loss function dynamically adjusts weights based on changes in the target's scale, reducing the impact of small angular errors on detection accuracy and achieving high-precision rotation detection. This helps the network more accurately detect electric vehicles captured by drones, avoiding interference from background information.

[0049] Figure 5 The flowchart of the method for monitoring helmet wearing by electric vehicle riders based on drone aerial photography provided in this embodiment of the invention includes the following steps:

[0050] S1. Real-time capture of road surface images;

[0051] S2. Detect whether there are electric bike riders not wearing helmets in the road image. If so, the drone will issue a reminder to wear a helmet.

[0052] In this embodiment of the invention, the method for detecting electric bicycle riders who are not wearing helmets is as follows:

[0053] The captured road image is input into the improved YOLOv7 network, which outputs the detection boxes for electric bikes, riders, and helmets in the road image, along with the confidence scores of each box. The network reads the detection box containing the electric bike and its confidence score, and checks if the confidence score of the electric bike's detection box reaches a set confidence threshold. If yes, an electric bike is identified in the road image. Then, it checks if a rider's detection box exists within the electric bike's detection box. If so, it checks if the confidence score of the rider's detection box is greater than the confidence threshold. If yes, a rider is identified on the electric bike. Then, it checks if a helmet's detection box exists within the rider's detection box. If no, the rider is not wearing a helmet. If yes, it checks if the confidence score of the helmet's detection box is greater than the confidence threshold. If yes, the rider on the electric bike is wearing a helmet; if no, the rider on the electric bike is not wearing a helmet.

[0054] In this embodiment of the invention, during the process of constructing samples, it is necessary to label the collected sample images. If the rider on the electric vehicle in the sample image is wearing a helmet, the detection box of the electric vehicle in the sample image is set as the smallest detection box containing the electric vehicle, the rider and the helmet they are wearing, the detection box of the rider is set as the smallest detection box containing the rider and the helmet they are wearing, and the helmet detection box is set as the smallest detection box containing the helmet. If the rider on the electric vehicle in the sample image is not wearing a helmet, the detection box of the electric vehicle in the sample image is set as the smallest detection box containing the electric vehicle and the rider, the detection box of the rider is set as the smallest detection box containing the rider, and no helmet detection box is set.

[0055] Treating the electric scooter and its rider as target objects, the improved YOLOv7 network inputs captured road images. It outputs bounding boxes for the target objects and helmets in the road images, along with the confidence scores of each box. The network reads the bounding boxes containing the target objects and their confidence scores, and checks if the confidence score of the bounding boxes reaches a set confidence threshold. If yes, it determines that a rider on an electric scooter exists in the road image. It then checks if a helmet bounding box exists within the target object's bounding box. If no, it determines that the rider is not wearing a helmet. If yes, it checks if the confidence score of the helmet bounding box is greater than a confidence threshold. If yes, it determines that the rider on the electric scooter is wearing a helmet; otherwise, it determines that the rider on the electric scooter is not wearing a helmet.

[0056] In this embodiment of the invention, during the process of constructing samples, it is necessary to label the collected sample images. If the rider on the electric vehicle in the sample image is wearing a helmet, the detection box of the target object in the sample image is set as the smallest detection box that includes the electric vehicle, the rider and the helmet they are wearing, and the helmet detection box is set as the smallest detection box that includes the helmet. If the rider on the electric vehicle in the sample image is not wearing a helmet, the detection box of the target object in the sample image is set as the smallest detection box that includes the electric vehicle and the rider, and no helmet detection box is set.

[0057] The present invention has been described by way of example. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other situations without modification, are all within the protection scope of the present invention.

Claims

1. A helmet-wearing monitoring system for electric vehicle riders based on drone aerial photography, characterized in that, The system includes: A gimbal mounted on a drone integrates a camera. The drone performs low-speed aerial photography over a designated road and transmits the captured road images to a processing unit. The processing unit detects whether there are electric bike riders without helmets in the road image. If so, the drone issues a reminder to wear a helmet. The YOLOv7 network outputs the bounding boxes for electric vehicles, riders, and helmets in the road image, along with the confidence scores for each bounding box. A YOLOv7 network consists of an Input layer, a Backbone layer, a Neck layer, and a Head layer connected in sequence. An IECA module is located after the last and second-to-last ELAN modules in the Backbone layer. The IECA module includes: The MaxPool and AvgPool layers are connected in parallel. The output of the MaxPool layer is connected to the convolutional layer for output, and then input to the Concat fusion layer for fusion. The output of the AvgPool layer is connected to the convolutional layer for output, and then input to the Concat fusion layer for fusion. The Concat fusion layer is connected to the Sigmoid function and multiplied with the input of the IECA module for output. The Ncek layer includes a fusion module called Fusion. Two adjacent ELAN modules in the Backbone layer are each connected to the Fusion module via two CBS modules to fuse the feature image information output from the Backbone layer. The Fusion module includes: The Concat fusion layer 1 is connected to the input terminal. The Concat fusion layer 1 is connected to two parallel convolutional layers Conv. One of the convolutional layers Conv is connected in series with three convolutional groups consisting of deformable convolutional layer DConv and convolutional layer Conv. The output terminal of the other convolutional layer Conv and each convolutional group is connected to the Concat fusion layer 2. The output terminal of the Concat fusion layer 2 is the output terminal of the Fusion module.

2. The electric vehicle rider helmet wearing monitoring system based on UAV aerial photography as described in claim 1, characterized in that, The specific methods for identifying electric bike riders who are not wearing helmets are as follows: The system reads the detection box and its confidence level of the electric vehicle in the road image. It checks whether the confidence level of the electric vehicle's detection box reaches a set confidence threshold. If the detection result is yes, it checks whether there is a rider's detection box within the electric vehicle's detection box. If there is, it checks whether the confidence level of the rider's detection box is greater than the confidence threshold. If the detection result is yes, it checks whether there is a helmet's detection box within the rider's detection box. If the detection result is no, it is determined that the rider is not wearing a helmet. If the detection result is yes, it checks whether the confidence level of the helmet's detection box is greater than the confidence threshold. If the detection result is yes, it is determined that the rider on the electric vehicle is wearing a helmet. If the detection result is no, it is determined that the rider on the electric vehicle is not wearing a helmet.

3. The electric vehicle rider helmet wearing monitoring system based on UAV aerial photography as described in claim 1, characterized in that, The YOLOv7 network outputs bounding boxes for electric bike riders and their helmets in road images, along with the confidence scores for each bounding box.

4. The electric vehicle rider helmet wearing monitoring system based on UAV aerial photography as described in claim 3, characterized in that, The specific methods for identifying electric bike riders who are not wearing helmets are as follows: Treating the electric scooter and its rider as a target object, the system reads the detection box containing the target object and its confidence level in the road image. It then checks whether the confidence level of the target object's detection box reaches a set confidence threshold. If the result is yes, it checks whether a helmet detection box exists within the target object's detection box. If the result is no, it is determined that the rider is not wearing a helmet. If the result is yes, it checks whether the confidence level of the helmet detection box is greater than a confidence threshold. If the result is yes, it is determined that the rider on the electric scooter is wearing a helmet. If the result is no, it is determined that the rider on the electric scooter is not wearing a helmet.

5. The electric vehicle rider helmet wearing monitoring system based on UAV aerial photography as described in claim 1, characterized in that, The head layer uses the KLD loss function to calculate the regression loss of the detection boxes, and outputs ( x,y,w,h,θ Rotate the border, where, ( x,y () represent the coordinates of the center point of the rotated border, w、h These represent the long and short sides of the rotated border, respectively. θ It refers to the angle between the positive x-axis and the longer side w.

Images