Vehicle helmet detection method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a method and device for establishing a vehicle helmet detection model, a vehicle helmet detection method and device, electronic equipment, a computer readable storage medium and a computer program product. The method for establishing the vehicle helmet detection model comprises the following steps: acquiring a vehicle helmet detection data set, wherein the vehicle helmet detection data set comprises a plurality of image samples during the driving or stopping period of a vehicle; for each image sample in the plurality of image samples, extracting edge information and background information of the image sample; and identifying one or more vehicle helmets in the image sample; and training a neural network model based on the edge information, the background information and the identified one or more vehicle helmets to establish a vehicle helmet detection model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a method and device for establishing a vehicle helmet detection model, a vehicle helmet detection method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] As road traffic congestion becomes increasingly serious, vehicles such as two-wheelers and unicycles have become one of the main means of transportation. However, the traffic accident rate of such vehicles has also increased. Among them, the failure of drivers of such vehicles to wear helmets is one of the main causes of serious casualties in traffic accidents.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] According to one aspect of the present disclosure, a method for establishing a vehicle helmet detection model is provided, comprising: acquiring a vehicle helmet detection dataset, the vehicle helmet detection dataset comprising a plurality of image samples of a vehicle while it is moving or stopped; for each image sample in the plurality of image samples: extracting edge information and background information of the image sample; and identifying one or more vehicle helmets in the image sample; and training a neural network model based on the edge information, the background information and the identified one or more vehicle helmets to establish a vehicle helmet detection model.

[0005] According to another aspect of the present disclosure, a vehicle helmet detection method is provided, comprising: capturing one or more images of a vehicle while it is moving or stopped via a sensor; performing target detection on each of the one or more images using a vehicle helmet detection model to determine whether a driver of the vehicle is wearing a vehicle helmet, wherein the vehicle helmet detection model is established according to the above-mentioned method for establishing a vehicle helmet detection model; and issuing an early warning to the driver based on determining that the driver of the vehicle is not wearing a vehicle helmet.

[0006] According to another aspect of the present disclosure, a device for establishing a vehicle helmet detection model is provided, comprising: a data set acquisition unit, configured to acquire a vehicle helmet detection data set, the vehicle helmet detection data set comprising a plurality of image samples of a vehicle during driving or stopping; an extraction unit, configured to extract edge information and background information of each image sample in the plurality of image samples; an identification unit, configured to identify one or more vehicle helmets in the image sample for each image sample in the plurality of image samples; and a training unit, configured to train a neural network model based on the edge information, the background information and the identified one or more vehicle helmets to establish a vehicle helmet detection model.

[0007] According to another aspect of the present disclosure, a vehicle helmet detection device is provided, comprising: a capture unit configured to capture one or more images of a vehicle while it is moving or stopped via a sensor; a target detection unit configured to perform target detection on each of the one or more images using a vehicle helmet detection model to determine whether the driver of the vehicle is wearing a vehicle helmet, wherein the vehicle helmet detection model is established according to the above-mentioned method for establishing a vehicle helmet detection model; and an early warning unit configured to warn the driver based on determining that the driver of the vehicle is not wearing a vehicle helmet.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement the above-mentioned method for establishing a vehicle helmet detection model or the steps of the vehicle helmet detection method.

[0009] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the method for establishing a vehicle helmet detection model or the steps of the vehicle helmet detection method are implemented.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the steps of the method for establishing a vehicle helmet detection model or the vehicle helmet detection method are implemented.

[0011] Further features and advantages of the present disclosure will become apparent from the exemplary embodiments described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 is a flow chart illustrating a method of establishing a vehicle helmet detection model according to an exemplary embodiment;

[0014] Figure 2 is a flow chart illustrating performing a global average pooling operation and a global maximum pooling operation on an image according to an exemplary embodiment;

[0015] Figure 3 is a flow chart illustrating a method for establishing a vehicle helmet detection model according to another exemplary embodiment;

[0016] Figure 4 is a flow chart illustrating a method for establishing a vehicle helmet detection model according to yet another exemplary embodiment;

[0017] Figure 5 is a flow chart illustrating a vehicle helmet detection method according to an exemplary embodiment;

[0018] Figure 6 A structural block diagram of a device for establishing a vehicle helmet detection model according to an embodiment of the present disclosure is shown;

[0019] Figure 7 A structural block diagram of a vehicle helmet detection device according to an embodiment of the present disclosure is shown;

[0020] Figure 8 is a block diagram showing a structure of a computing device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0022] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0023] As road traffic congestion becomes increasingly serious, vehicles such as two-wheelers and unicycles have become one of the main means of transportation. However, the traffic accident rate of such vehicles has also increased. Among them, the failure of drivers of such vehicles to wear helmets is one of the main causes of serious casualties in traffic accidents. Therefore, increasing the helmet wearing rate of drivers of such vehicles is beneficial to reducing casualties in traffic accidents.

[0024] Although it has been clearly stipulated that drivers of such vehicles must wear helmets while driving, regulations and supervision alone are not efficient. At the same time, the inventors found that the existing algorithms for target detection using computer vision technology have a slow processing speed and cannot achieve real-time detection of the driver's helmet wearing status, especially in cases of heavy traffic, which may lead to missed detections and false detections. Furthermore, the inventors also found that the existing target detection algorithms are limited in their applicable scenarios. On the one hand, they may not be able to adapt to different hardware devices and are difficult to apply on a large scale. On the other hand, they may not be able to accurately detect the driver's helmet wearing status in different regions and environments, and their generalization ability is poor.

[0025] In order to solve the above technical problems, the present disclosure provides a new method and device for establishing a vehicle helmet detection model, a vehicle helmet detection method and device, an electronic device, a computer-readable storage medium and a computer program product. It will be clearly seen in the following detailed description that the method for establishing a vehicle helmet detection model according to an embodiment of the present disclosure can obtain a large number of vehicle helmet detection data sets, extract edge information and background information for each image sample in the data set and identify the target "helmet", and train the neural network model based on the edge information, background information and recognition results, so as to establish a vehicle helmet detection model. Since the edge information can locate the contour boundary of the target object, and the background information can determine the environmental range of the target object, therefore, by extracting both edge information and background information to train the neural network model, the position of the helmet can be accurately determined in more complex scenes, and the situation of false detection and missed detection can be reduced. When the target object is partially blocked, the complete shape and position of the blocked part can be better inferred, so as to improve the accuracy of the established vehicle helmet detection model. Further, the vehicle helmet detection method according to an embodiment of the present disclosure can capture one or more images of the vehicle during driving or stopping, and use the established vehicle helmet detection model to determine whether the driver is wearing a vehicle helmet, so as to issue an early warning according to the determination result. Since the established vehicle helmet detection model has high accuracy and fast processing speed, it can realize real-time detection of the driver's helmet wearing status and at the same time improve the accuracy of driver warning.

[0026] An exemplary embodiment of the vehicle warning method disclosed herein will be further described below with reference to the accompanying drawings.

[0027] Figure 1 FIG. 1 is a flow chart of a method 100 for establishing a vehicle helmet detection model according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the method 100 for establishing a vehicle helmet detection model may include: step S110, obtaining a vehicle helmet detection data set, the vehicle helmet detection data set including a plurality of image samples of a vehicle during driving or stopping; step S120, for each of the plurality of image samples, extracting edge information and background information of the image sample; step S130, for each of the plurality of image samples, identifying one or more vehicle helmets in the image sample; and step S140, training a neural network model based on the edge information, background information and the identified one or more vehicle helmets to establish a vehicle helmet detection model.

[0028] Since edge information can locate the contour boundary of the target object, and background information can determine the environmental range of the target object, therefore, extracting both edge information and background information to train the neural network model can accurately determine the position of the helmet in more complex scenes, reduce false detections and missed detections, and better infer the complete shape and position of the obscured part when the target object is partially obscured, thereby improving the accuracy of the established vehicle helmet detection model.

[0029] According to some embodiments of the present disclosure, the vehicle may be a two-wheeled vehicle or a unicycle. For example, the vehicle in the present disclosure may be a two-wheeled vehicle such as a motorcycle, an electric bicycle, a two-wheeled balancing vehicle, an electric scooter, or a unicycle such as a single-wheeled balancing vehicle.

[0030] In step S110, obtaining a vehicle helmet detection data set may include obtaining an image sample set in any suitable manner. In some examples, the vehicle helmet detection data set may be obtained through a web crawler. That is, a script is used to retrieve the keywords "wearing a helmet", "detecting a two-wheeled vehicle helmet", and "riding a two-wheeled vehicle" to obtain a batch of image sample sets, and then images that meet the detection scene are selected as the vehicle helmet detection data set. In other examples, the vehicle helmet detection data set may also be obtained by shooting on a real road. For example, multiple vehicles on real roads may be photographed at different times, in different regions, and in different weather conditions, and clearer images that meet the detection scene may be selected as the vehicle helmet detection data set. Furthermore, when selecting images, the number of targets of different categories in the vehicle helmet detection data set may be relatively balanced.

[0031] Since the acquired vehicle helmet detection dataset contains a large number of detection images from different angles, different times, different regions, and different weather conditions, and preferably the number of targets of different categories is relatively balanced, the diversity and richness of the detection dataset can be ensured, which is conducive to improving the accuracy of the established vehicle helmet detection model and making it adaptable to diverse application scenarios.

[0032] According to some embodiments of the present disclosure, in step S120, for each image sample in a plurality of image samples, obtaining edge information and background information of the image sample may include: performing a global average pooling operation and a global maximum pooling operation on each image sample in the plurality of image samples to obtain the edge information and background information of the image sample.

[0033] In the global average pooling operation, the entire feature map is used as input, and a new feature vector is generated by averaging it in the spatial dimensions (height and width). Specifically, it is assumed that the size of the feature map is H×W×C, where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels. For each channel c (c=1,2,…,C), the average value of all elements on the channel is calculated. After the global average pooling operation, a feature vector of size 1×1×C can be obtained, which compresses the original feature map in height and width, retaining only the information of the channel dimension, and the value of each channel is the average value of all elements of the corresponding channel of the original feature map.

[0034] By applying a global average pooling operation, a large number of connection weights in the fully connected layer of the neural network are reduced. The reduction of parameters helps to reduce the complexity of the neural network and reduce the risk of overfitting, thereby enabling the neural model to have better generalization capabilities on unseen data.

[0035] Similar to the global average pooling operation, in the global maximum pooling operation, the entire feature map is used as input, and a new feature vector is generated by maximizing it in the spatial dimensions (height and width). Specifically, it is assumed that the size of the feature map is H×W×C, where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels. For each channel c (c=1,2,…,C), the maximum value of all elements on the channel is determined. After the global maximum pooling operation, a feature vector of size 1×1×C can also be obtained. This feature vector compresses the original feature map in height and width, retaining only the information of the channel dimension, and the value of each channel is the maximum value of all elements of the corresponding channel of the original feature map.

[0036] By applying the global maximum pooling operation, the most significant features in each channel can be extracted. Since these maximum values ​​may represent key information in the image (for example, in the target detection task, the most representative features of the object (edges, textures, etc.) may exist in the feature map in the form of maximum values), the global maximum pooling operation can help the model quickly find these key information and improve the accuracy and efficiency of target detection.

[0037] Figure 2 FIG. 4 is a flow chart showing a method of performing a global average pooling operation and a global maximum pooling operation on an image according to an exemplary embodiment. Figure 2 As shown, by adding operation 210 (MaxPool2d) and operation 220 (AvgPool2d), the edge information and background information of the image sample can be extracted more efficiently, thereby improving the expression ability of the neural network model.

[0038] In step S130, for each image sample in the plurality of image samples, identifying one or more vehicle helmets in the image sample may include: obtaining bounding boxes of the one or more vehicle helmets in the image sample and position information thereof.

[0039] In some examples, the position information of the bounding box of each vehicle helmet may be the center point of the lower border of the bounding box. In other examples, the position information of the bounding box of each vehicle helmet may be the geometric center point of the bounding box. In still other examples, the position information of the bounding box of each vehicle helmet may be the center point of the upper border of the bounding box. The position information may be, for example, the two-dimensional coordinates or three-dimensional coordinates of the center point.

[0040] It will be understood that the position information of the bounding box may refer to coordinate information of any suitable points of the bounding box, and the scope of the subject matter claimed in the present disclosure is not limited in the foregoing respects.

[0041] Figure 3 FIG. 4 is a flow chart showing a method for establishing a vehicle helmet detection model according to another exemplary embodiment. Figure 3 As shown, the method 300 for establishing a vehicle helmet detection model may include the following steps: Figure 1 Steps S310-S340 similar to steps S110-S140 in the method 100 for establishing a vehicle helmet detection model described above; and step S350, determining a loss function between a predicted bounding box and an annotated bounding box of one or more identified vehicle helmets, wherein step S340, training a neural network model based on edge information, background information and one or more identified vehicle helmets to establish a vehicle helmet detection model may include: training a neural network model based on edge information, background information and a loss function to establish a vehicle helmet detection model.

[0042] The loss function can characterize the difference between the predicted bounding box and the labeled bounding box of each vehicle helmet. By calculating the difference and further training the neural network model based on it, the accuracy of the vehicle helmet detection model can be further improved.

[0043] According to some embodiments of the present disclosure, the loss function may be an InnerCIoU loss function, which is mainly used to measure the difference between the predicted bounding box and the annotated (true) bounding box of the detected object. Specifically, IoU is the ratio of the intersection and union of the predicted bounding box and the annotated (true) bounding box of the object, as shown in the following formula (1):

[0044]

[0045] Among them, Intersection represents the intersection area of ​​the predicted bounding box and the annotated (true) bounding box of the object, and Union represents the union area of ​​the predicted bounding box and the annotated (true) bounding box of the object. InnerCIoU further considers the distance between the predicted bounding box and the annotated (true) bounding box and the aspect ratio on the basis of CIoU. For example, it can be calculated by the following formula (2):

[0046]

[0047] Among them, L represents the distance between the predicted bounding box and the annotated (true) bounding box (for example, when the position information of the bounding box of each vehicle helmet is the geometric center point of the bounding box, L represents the distance between the geometric center point of the predicted bounding box and the geometric center point of the annotated bounding box), area represents the area of ​​the minimum rectangle that can contain both the predicted bounding box and the annotated (true) bounding box, α represents a weight factor (for example, it can take a value of 0.5) for balancing different loss terms (such as the distance between the center points and the aspect ratio difference), and v represents the aspect ratio value of the annotated (true) bounding box.

[0048] Compared with the common IoU loss function, the InnerCIoU loss function considers more geometric factors and can make the neural network model converge more quickly. Because the InnerCIoU loss function not only focuses on the overlapping area between the predicted bounding box and the annotated (real) bounding box, but also considers the distance between the two bounding boxes and the aspect ratio of the annotated (real) bounding box, it can more effectively guide the predicted bounding box to adjust in the direction of the annotated (real) bounding box during the training of the neural network model. In addition, by balancing the weight factors of the distance between the center points and the aspect ratio difference, the InnerCIoU loss function can also improve the accuracy of target detection and positioning. For example, in the case of small target detection or irregular target shape, the InnerCIoU loss function can better adjust the predicted bounding box to match the real object more accurately.

[0049] It will be understood that for illustration purposes, the loss function may also be other suitable loss functions.

[0050] It will also be understood that the above-mentioned neural network model can be a YOLOv8 model or other models based on deep neural network algorithms to achieve target detection for each image.

[0051] According to some embodiments of the present disclosure, the above-mentioned methods 100 and 300 for establishing a vehicle helmet detection model may also include: determining the ReLU function as the activation function of the neural network model; and wherein steps S140 and S340, based on edge information, background information and one or more identified vehicle helmets, training the neural network model to establish a vehicle helmet detection model may include: based on edge information, background information and the activation function, training the neural network model to establish a vehicle helmet detection model.

[0052] Since the ReLU function only needs to perform one comparison operation (see the following formula (3)), that is, to determine whether the input is greater than 0, it is very fast in the forward propagation process of the neural network and is particularly suitable for processing large-scale data.

[0053] In addition, since the SiLU function includes exponential operations (see formula (4) below), and the neural network operation unit on the end side cannot perform exponential operations, it is necessary to transmit the data to the Arm end or the DSP end, and then retransmit the data to the neural network unit after the exponential operation is completed, which will greatly reduce the operation speed of the neural network model. Compared with the traditional SiLU function, the ReLU function includes the operation of judging whether the input is greater than 0, so it can be operated in the neural network operation unit on the end side, thereby avoiding frequent data transmission and further improving the operation speed of the neural network model.

[0054]

[0055] According to some embodiments of the present disclosure, steps S140 and S340, training a neural network model based on edge information, background information and one or more identified vehicle helmets to establish a vehicle helmet detection model may include: performing a de-redundancy operation on the neural network model.

[0056] As mentioned above, when the ReLU activation function is used to train the neural network model, there may be a situation where the output of some neural network operation units is 0, thereby generating sparsity to a certain extent. In addition, for the convolutional neural network model, the number of channels of the convolution kernel is usually redundant. Since the ordinary convolution layer is usually followed by a batch normalization layer, the redundant channels can be removed through this layer, thereby reducing the number of operations and the complexity of the neural network model, speeding up the operation, and making the neural network model pay more attention to the key features in the data, thereby reducing the fitting of irrelevant information.

[0057] Specifically, the batch normalization (BN) layer is used to normalize the input of each layer of the neural network model to normalize the input to a fixed interval, for example, by the following formula (5):

[0058]

[0059] Among them, x is the input of each layer of the neural network model, E(x) and are the mean and standard deviation of the mini-batch data, and γ and β are learnable parameters.

[0060] In the process of training the neural network model, the desparse training of the parameter γ of the BN layer can be achieved by applying the L1 regularization constraint to the parameter γ of the BN layer and adjusting the L1 regularization strength coefficient λ (i.e., changing the value of part of the parameter γ to 0). Furthermore, when the data of a certain channel in the BN layer is 0, it can be proved that the channel corresponding to the convolution layer above the BN layer is also 0, which is a redundant channel. Accordingly, the de-redundancy operation can be achieved by the following formula (6):

[0061]

[0062] Among them, total_loss represents the total loss, represents the difference between the predicted result and the labeled (true) result of the object, and Indicates that an L1 regularization constraint is applied to the parameter γ of the BN layer.

[0063] It will be understood that after the above-mentioned vehicle helmet detection model is established, the trained model can be fitted and finally converted into an onnx offline model to be decoupled from the original deep learning framework, so that it can run on a variety of operating systems and hardware architectures without relying on the training framework. This greatly broadens the scope of application of the model and facilitates the migration of the model from a training environment such as a data center to an actual production environment or terminal device.

[0064] According to some embodiments of the present disclosure, after obtaining the corresponding onnx offline model, the DFL module can be trimmed to avoid operations such as Reshape, Transpose, Softmax, etc., thereby simplifying the operation and further improving the model operation speed.

[0065] Figure 4 FIG. 4 is a flow chart of a method 400 for establishing a vehicle helmet detection model according to another exemplary embodiment. Figure 4 As shown, the method 400 for establishing a vehicle helmet detection model may include the following steps: Figure 1 Steps S110-S140 in the method 100 for establishing a vehicle helmet detection model, or reference Figure 3Steps S410-S440 similar to steps S310-S340 in the method 300 for establishing a vehicle helmet detection model described above; and step S450, after establishing the vehicle helmet detection model, performing graph optimization operations on the convolutional layer, batch normalization layer and activation function of the vehicle helmet detection model.

[0066] Each time the vehicle helmet detection model is used to perform vehicle helmet detection, the data in the memory will be accessed three times in sequence in the convolution layer, batch normalization layer, and activation function for calculation. In some examples, the convolution layer, batch normalization layer, and activation function structure in the vehicle helmet detection model may occupy more than 80% of the entire structure. This means that the number of accesses to the data in the memory will increase significantly. By performing graph optimization operations on the convolution layer, batch normalization layer, and activation function, the operators corresponding to the three can be coupled into one operator, so that the same operation can be completed by accessing the data in the memory only once, which can greatly improve the calculation speed and improve the model reasoning performance.

[0067] According to some embodiments of the present disclosure, step S550, performing graph optimization operations on the convolutional layers, batch normalization layers, and activation functions of the vehicle helmet detection model may include: reparameterizing the vehicle helmet detection model to couple multiple operators of the corresponding convolutional layers, batch normalization layers, and activation functions of the vehicle helmet detection model.

[0068] Typically, a neural network model has a multi-branch network structure in both the training phase and the application reasoning phase, and each branch network structure is associated with a set of parameters. In the application reasoning phase, it is expected that the reasoning result can be obtained quickly. Therefore, as described above, a graph optimization operation can be performed by coupling multiple sets of parameters of a multi-branch structure of a neural network model such as a convolutional layer, a batch normalization layer, and an activation function to realize the operation of a single operator.

[0069] For example, the operation of the convolutional layer can be implemented by the following formula (7):

[0070] y=w*x (7)

[0071] Among them, y represents the convolution calculation result, w represents the weight factor, and x represents the input data (e.g., an image).

[0072] The batch normalization layer and activation function can be implemented by equations (5) and (4) above, respectively.

[0073] In this case, the graph optimization operations performed on the convolutional layers, batch normalization layers, and activation functions of the vehicle helmet detection model can be implemented as follows (8):

[0074]

[0075] By coupling the convolutional layer, batch normalization layer, and multiple sets of parameters of the activation function of the vehicle helmet detection model, a single operator neural network model can be obtained and equivalent operation results can be achieved. Since the data in the memory only needs to be accessed once when executing a single operator operation, the operation speed is faster and the inference results of the neural network model can be obtained more quickly.

[0076] Figure 5 FIG. 5 shows a flow chart of a vehicle helmet detection method 500 according to an exemplary embodiment. Figure 5 As shown, the vehicle helmet detection method 500 may include: step S510, capturing one or more images of the vehicle during driving or stopping via a sensor; step S520, performing target detection on each of the one or more images using a vehicle helmet detection model to determine whether the driver of the vehicle is wearing a vehicle helmet, wherein the vehicle helmet detection model is established according to the above-mentioned method for establishing a vehicle helmet detection model; and step S530, warning the driver based on determining that the driver of the vehicle is not wearing a vehicle helmet.

[0077] Since the established vehicle helmet detection model is based on a large number of image samples obtained at different times, in different regions, and in different weather conditions, it can monitor the wearing of the driver's vehicle helmet under various conditions. At the same time, since the established vehicle helmet detection model can realize real-time and rapid monitoring of the driver's vehicle helmet wearing, it can timely warn the driver, which helps to improve the safety awareness of the rider and reduce the occurrence of traffic accidents.

[0078] In step S510, the sensor may be any suitable sensor capable of capturing an image, such as a camera, a video camera, a camera, etc. Such a sensor may be, for example, a fisheye lens for capturing the driver's upper body. The scope of the subject matter claimed in the present disclosure is not limited in the above aspects.

[0079] In step S510, the sensor may be located at any position in the front of the vehicle as long as it can capture the vehicle driver in real time, such as on the dashboard of the vehicle, or on the front storage box of the vehicle, etc., preferably in the direction of the central axis of the vehicle.

[0080] In step S520, the method 100, 300 or 400 for establishing a vehicle helmet detection model as described above may be used to perform target detection on each captured image, and determine whether the driver of the vehicle is wearing a vehicle helmet based on the detection result.

[0081] In some examples, it can be determined whether the driver of the vehicle is wearing a vehicle helmet based on the position of the detected bounding box of the vehicle helmet and the position of the detected bounding box of the driver. For example, it can be determined that the driver of the vehicle is wearing a helmet based on the distance between the center point of the bounding box of the vehicle helmet and the center point of the detected bounding box of the driver is less than a preset threshold. For another example, it can be determined that the driver of the vehicle is wearing a helmet based on the distance between the center point of the lower boundary of the frame of the vehicle helmet and the center point of the lower boundary of the detected bounding box of the driver is less than a preset threshold.

[0082] In step S530, according to the determination that the driver of the vehicle is not wearing a vehicle helmet, the warning to the driver may include any one or more of the following: according to the determination that the driver of the vehicle is not wearing a vehicle helmet, highlighting on the display screen equipped with the vehicle; according to the determination that the driver of the vehicle is not wearing a vehicle helmet, warning the driver of the vehicle through visual signals such as flashing lights; or according to the determination that the driver of the vehicle is not wearing a vehicle helmet, warning the driver of the vehicle through auditory signals such as alarms. Furthermore, as the length of time that the driver of the vehicle is not wearing a vehicle helmet exceeds a preset threshold, the driver of the vehicle may be warned by increasing the volume and frequency of the warning buzzer, increasing the flashing frequency of the warning indicator light, and highlighting the prompt more significantly.

[0083] It will be understood that how to warn the driver is not limited to the above-mentioned manner, and the scope of the subject matter claimed in the present disclosure is not limited in the above-mentioned aspects.

[0084] According to some embodiments of the present disclosure, the vehicle helmet detection method 500 may further include: preprocessing the one or more images to convert the one or more images into one or more images in a preset format.

[0085] The one or more images captured in step S510 may not be directly applied as input to the trained vehicle helmet detection model. Therefore, these images can be converted into a preset format (for example, converted into a yuv format through a Video Input module, or the images are converted into a format suitable for the neural network model), so that they can be applied to the trained vehicle helmet detection model to achieve fast and accurate detection of vehicle helmets.

[0086] Figure 6 FIG. 6 shows a structural block diagram of an apparatus 600 for establishing a vehicle helmet detection model according to an embodiment of the present disclosure. Figure 6As shown, the device 600 may include: a data set acquisition unit 610, configured to acquire a vehicle helmet detection data set, the vehicle helmet detection data set including a plurality of image samples of a vehicle during driving or stopping; an extraction unit 620, configured to extract edge information and background information of each image sample in the plurality of image samples; an identification unit 630, configured to identify one or more vehicle helmets in the image sample for each image sample in the plurality of image samples; and a training unit 640, configured to train a neural network model based on the edge information, the background information and the identified one or more vehicle helmets to establish a vehicle helmet detection model.

[0087] Since edge information can locate the contour boundary of the target object, and background information can determine the environmental range of the target object, therefore, extracting both edge information and background information to train the neural network model can accurately determine the position of the helmet in more complex scenes, reduce false detections and missed detections, and better infer the complete shape and position of the obscured part when the target object is partially obscured, thereby improving the accuracy of the established vehicle helmet detection model.

[0088] According to some embodiments of the present disclosure, the extraction unit 620 may include: a unit configured to perform a global average pooling operation and a global maximum pooling operation on each image sample of the multiple image samples to obtain edge information and background information of the image sample.

[0089] According to some embodiments of the present disclosure, the device 600 may also include a loss function determination unit configured to determine a loss function between a predicted bounding box and a labeled bounding box of one or more identified vehicle helmets; and the training unit 640 may include a unit configured to train a neural network model based on edge information, background information and a loss function to establish a vehicle helmet detection model.

[0090] According to some embodiments of the present disclosure, the loss function is an InnerCIoU loss function.

[0091] According to some embodiments of the present disclosure, the device 600 may also include an activation function determination unit configured to determine the ReLU function as the activation function of the neural network model; and the training unit 640 may include a unit configured to train the neural network model based on edge information, background information and the activation function to establish a vehicle helmet detection model.

[0092] According to some embodiments of the present disclosure, the training unit 640 may include a de-redundancy unit configured to perform a de-redundancy operation on the neural network model.

[0093] According to some embodiments of the present disclosure, the device 600 may further include a graph optimization unit configured to perform a graph optimization operation on a convolutional layer, a batch normalization layer, and an activation function of the vehicle helmet detection model after establishing the vehicle helmet detection model.

[0094] According to some embodiments of the present disclosure, the graph optimization unit may include a unit configured to perform model reparameterization on the vehicle helmet detection model to couple corresponding convolutional layers, batch normalization layers, and multiple operators of activation functions of the vehicle helmet detection model.

[0095] According to some embodiments of the present disclosure, the vehicle is a two-wheeled vehicle or a unicycle.

[0096] It should be understood that Figure 6 The various units of the device 600 for establishing a vehicle helmet detection model shown in FIG. 6 can be compared with the reference Figure 1 , Figure 3 and Figure 4 The steps in the methods 100, 300 and 400 for establishing a vehicle helmet detection model correspond to each other. Therefore, the operations, features and advantages described above for the methods 100, 300 and 400 are also applicable to the apparatus 600 for establishing a vehicle helmet detection model and the units included therein. For the sake of brevity, some operations, features and advantages are not described in detail here.

[0097] Figure 7 FIG. 7 shows a structural block diagram of a vehicle helmet detection device 700 according to an embodiment of the present disclosure. Figure 7 As shown, the vehicle helmet detection device 700 may include: a capture unit 710, configured to capture one or more images of the vehicle during driving or stopping via a sensor; a target detection unit 720, configured to perform target detection on each of the one or more images using a vehicle helmet detection model to determine whether the driver of the vehicle is wearing a vehicle helmet, wherein the vehicle helmet detection model is established according to the above-mentioned method for establishing a vehicle helmet detection model; and an early warning unit 730, configured to warn the driver according to determining that the driver of the vehicle is not wearing a vehicle helmet.

[0098] Since the established vehicle helmet detection model is based on a large number of image samples obtained at different times, in different regions, and in different weather conditions, it can monitor the wearing of the driver's vehicle helmet under various conditions. At the same time, since the established vehicle helmet detection model can realize real-time and rapid monitoring of the driver's vehicle helmet wearing, it can timely warn the driver, which helps to improve the safety awareness of the rider and reduce the occurrence of traffic accidents.

[0099] According to some embodiments of the present disclosure, the vehicle helmet detection device 700 may further include a preprocessing unit configured to preprocess the one or more images to convert the one or more images into one or more images in a preset format.

[0100] It should be understood that Figure 7 The various units of the vehicle helmet detection device 700 shown in FIG. Figure 5 The above-described vehicle helmet detection method 500 corresponds to each step in the vehicle helmet detection method 500. Therefore, the operations, features and advantages described above for the vehicle helmet detection method 500 are also applicable to the vehicle helmet detection device 700 and the units included therein. For the sake of brevity, some operations, features and advantages are not described in detail here.

[0101] Yet another aspect of the present disclosure may include an electronic device, which may include a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement the above-mentioned method for establishing a vehicle helmet detection model or the steps of the vehicle helmet detection method.

[0102] Yet another aspect of the present disclosure may include a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for establishing a vehicle helmet detection model or the steps of the vehicle helmet detection method are implemented.

[0103] Yet another aspect of the present disclosure may include a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the method for establishing a vehicle helmet detection model or the vehicle helmet detection method are implemented.

[0104] Reference Figure 8 , a computing device 800 will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The computing device 800 can be any machine configured to perform processing and / or computing, and can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal digital assistant, a smart phone, an in-vehicle computer, an access control system, an attendance device, or any combination thereof. The above-mentioned device for establishing a vehicle helmet detection model and the vehicle helmet detection device can be implemented in whole or in part by the computing device 800 or a similar device or system. Although the computing device 800 represents an example of several types of computing platforms, the computing device 800 can include more than one computer. Figure 8 Greater or fewer elements and / or different arrangements of elements may be shown, and the scope of the claimed subject matter is not limited in these respects.

[0105] In some embodiments, the computing device 800 may include an element connected to or in communication with the bus 802 (possibly via one or more interfaces). For example, the computing device 800 may include a bus 802, one or more processors 804, one or more input devices 806, and one or more output devices 808. The one or more processors 804 may be any type of processor, and may include, but are not limited to, one or more general-purpose processors and / or one or more special-purpose processors (e.g., special processing chips). The input device 806 may be any type of device that can input information to the computing device 800, and may include, but are not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote control. The output device 808 may be any type of device that can present information, and may include, but are not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The computing device 800 may also include or be connected to a non-transitory storage device 810. The non-transitory storage device may be any storage device that is non-transitory and can implement data storage, and may include, but is not limited to, a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a magnetic tape, or any other magnetic medium, an optical disk or any other optical medium, a ROM (read-only memory), a RAM (random access memory), a cache memory, and / or any other memory chip or cartridge, and / or any other medium from which a computer can read data, instructions, and / or code. The non-transitory storage device 810 may be detachable from the interface. The non-transitory storage device 810 embodies one or more non-transitory computer-readable media on which a program including instructions is stored, which, when executed by one or more processors of the computing device 800, causes the computing device 800 to perform the above-mentioned method 100, 300, 400 for establishing a vehicle helmet detection model and its variants or the above-mentioned vehicle helmet detection method 500 and its variants. The computing device 800 may also include a communication device 812. The communication device 812 can be any type of device or system that enables communication with an external device and / or with a network, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication device and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0106] In some embodiments, computing device 800 may also include working memory 814, which may be any type of memory that can store programs (including instructions) and / or data useful for operation of processor 804, and may include, but is not limited to, random access memory and / or read-only memory devices.

[0107] The software elements (programs) may be located in the working memory 814, including but not limited to an operating system 816, one or more application programs 818, drivers and / or other data and codes. Instructions for executing the above methods and steps may be included in one or more application programs 818, and the above-mentioned device for establishing a vehicle helmet detection model and the electronic circuit of the vehicle helmet detection device may be implemented by reading and executing the instructions of one or more application programs 818 by the processor 804. The executable code or source code of the instructions of the software elements (programs) may be stored in a non-transitory computer-readable storage medium (such as the above-mentioned storage device 810), and may be stored in the working memory 814 (possibly compiled and / or installed) when executed. The executable code or source code of the instructions of the software elements (programs) may also be downloaded from a remote location.

[0108] It should also be understood that various modifications may be made according to specific requirements. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. For example, some or all of the disclosed methods and apparatus may be implemented by programming hardware (e.g., programmable logic circuits including field programmable gate arrays (FPGAs) and / or programmable logic arrays (PLAs)) in assembly language or hardware programming languages ​​(such as VERILOG, VHDL, C++) using logic and algorithms according to the present disclosure.

[0109] It should also be understood that the aforementioned method can be implemented in a server-client mode. For example, the client can use a camera to collect image data and send the image data to the server for subsequent processing. The client can also perform a portion of the processing in the aforementioned method and send the processed data to the server. The server can receive data from the client and execute the aforementioned method or another portion of the aforementioned method, and return the execution result to the client. The client can receive the execution result of the method from the server and present it to the user, for example, through an output device.

[0110] It should also be understood that the components of computing device 800 can be distributed over a network. For example, some processing can be performed using one processor, while other processing can be performed by another processor remote from the one processor. Other components of computing device 800 can also be similarly distributed. In this way, computing device 800 can be understood as a distributed computing system that performs processing on multiple processors at multiple locations.

[0111] Some exemplary aspects of the disclosure are described below.

[0112] Aspect 1. A method for establishing a vehicle helmet detection model, comprising:

[0113] Acquire a vehicle helmet detection dataset, where the vehicle helmet detection dataset includes a plurality of image samples of the vehicle when the vehicle is moving or stopped;

[0114] For each image sample in the plurality of image samples:

[0115] Extracting edge information and background information of the image sample; and

[0116] identifying one or more vehicle helmets in the image sample; and

[0117] Based on the edge information, the background information and the one or more identified vehicle helmets, a neural network model is trained to establish a vehicle helmet detection model.

[0118] Aspect 2. The method according to aspect 1, wherein, for each image sample in the plurality of image samples, obtaining the edge information and the background information of the image sample comprises:

[0119] A global average pooling operation and a global maximum pooling operation are performed on each image sample of the multiple image samples to obtain the edge information and the background information of the image sample.

[0120] Aspect 3. The method according to aspect 1 further comprises determining a loss function between a predicted bounding box and an annotated bounding box of the one or more identified vehicle helmets; and

[0121] Wherein, based on the edge information, the background information and the one or more identified vehicle helmets, training the neural network model to establish the vehicle helmet detection model includes:

[0122] Based on the edge information, the background information and the loss function, the neural network model is trained to establish the vehicle helmet detection model.

[0123] Aspect 4. The method according to Aspect 3, wherein the loss function is an InnerCIoU loss function.

[0124] Aspect 5. The method according to any one of Aspects 1-4 further comprises determining a ReLU function as an activation function of the neural network model; and

[0125] Wherein, based on the edge information, the background information and the one or more identified vehicle helmets, training the neural network model to establish the vehicle helmet detection model includes:

[0126] Based on the edge information, the background information and the activation function, the neural network model is trained to establish the vehicle helmet detection model.

[0127] Aspect 6. The method according to any one of aspects 1 to 5, wherein the step of training the neural network model based on the edge information, the background information and the one or more identified vehicle helmets to establish the vehicle helmet detection model comprises:

[0128] A redundancy removal operation is performed on the neural network model.

[0129] Aspect 7. The method according to any one of aspects 1 to 6, further comprising:

[0130] After the vehicle helmet detection model is established, a graph optimization operation is performed on the convolution layer, batch normalization layer and activation function of the vehicle helmet detection model.

[0131] Aspect 8. The method according to aspect 7, wherein performing the graph optimization operation on the convolutional layer, the batch normalization layer, and the activation function of the vehicle helmet detection model comprises:

[0132] The vehicle helmet detection model is reparameterized to couple a plurality of operators corresponding to the convolution layer, the batch normalization layer, and the activation function of the vehicle helmet detection model.

[0133] Aspect 9. The method according to any one of aspects 1 to 8, wherein the vehicle is a two-wheeled vehicle or a unicycle.

[0134] Aspect 10. A vehicle helmet detection method, comprising:

[0135] capturing, via a sensor, one or more images of the vehicle while the vehicle is moving or stopped;

[0136] performing object detection on each of the one or more images using a vehicle helmet detection model to determine whether the driver of the vehicle is wearing a vehicle helmet, wherein the vehicle helmet detection model is established according to the method of any one of aspects 1 to 7; and

[0137] Based on determining that the driver of the vehicle is not wearing a vehicle helmet, a warning is given to the driver.

[0138] Aspect 11. The method according to aspect 10 further includes:

[0139] The one or more images are preprocessed to convert the one or more images into one or more images in a preset format.

[0140] Aspect 12. A device for establishing a vehicle helmet detection model, comprising:

[0141] A data set acquisition unit is configured to acquire a vehicle helmet detection data set, wherein the vehicle helmet detection data set includes a plurality of image samples of the vehicle when the vehicle is moving or stopped;

[0142] an extraction unit, configured to extract edge information and background information of each image sample from the plurality of image samples;

[0143] an identification unit configured to identify, for each image sample in the plurality of image samples, one or more vehicle helmets in the image sample; and

[0144] The training unit is configured to train a neural network model based on the edge information, the background information and the one or more identified vehicle helmets to establish a vehicle helmet detection model.

[0145] Aspect 13. A vehicle helmet detection device, comprising:

[0146] a capturing unit configured to capture one or more images of the vehicle during driving or stopping via a sensor;

[0147] an object detection unit configured to perform object detection on each of the one or more images using a vehicle helmet detection model to determine whether the driver of the vehicle is wearing a vehicle helmet, wherein the vehicle helmet detection model is established according to the method of any one of aspects 1 to 7; and

[0148] The warning unit is configured to warn the driver based on determining that the driver of the vehicle is not wearing a vehicle helmet.

[0149] Aspect 14. An electronic device comprising:

[0150] a memory, a processor and a computer program stored on the memory,

[0151] The processor is configured to execute the computer program to implement the steps of the method according to any one of aspects 1 to 9 or 10 to 11.

[0152] Aspect 15. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of aspects 1 to 9 or 10 to 11 are implemented.

[0153] Aspect 16. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of aspects 1 to 9 or 10 to 11 are implemented.

[0154] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for establishing a vehicle helmet detection model, comprising: Acquire a vehicle helmet detection dataset, where the vehicle helmet detection dataset includes a plurality of image samples of the vehicle when the vehicle is moving or stopped; For each image sample in the plurality of image samples: Extract edge information and background information of the image sample; and identifying one or more vehicle helmets in the image sample; as well as Based on the edge information, the background information and the one or more identified vehicle helmets, a neural network model is trained to establish a vehicle helmet detection model.

2. The method according to claim 1, wherein: For each image sample among the multiple image samples, acquiring the edge information and the background information of the image sample includes: A global average pooling operation and a global maximum pooling operation are performed on each image sample of the multiple image samples to obtain the edge information and the background information of the image sample.

3. The method according to claim 1, further comprising determining a loss function between a predicted bounding box and an annotated bounding box of the one or more identified vehicle helmets; and in, Training the neural network model based on the edge information, the background information and the one or more identified vehicle helmets to establish the vehicle helmet detection model includes: Based on the edge information, the background information and the loss function, the neural network model is trained to establish the vehicle helmet detection model.

4. The method according to claim 3, wherein: The loss function is the InnerCIoU loss function.

5. A vehicle helmet detection method, comprising: capturing, via a sensor, one or more images of the vehicle while the vehicle is moving or stopped; performing object detection on each of the one or more images using a vehicle helmet detection model to determine whether the driver of the vehicle is wearing a vehicle helmet, wherein the vehicle helmet detection model is established according to the method of any one of claims 1 to 4; as well as Based on determining that the driver of the vehicle is not wearing a vehicle helmet, a warning is given to the driver.

6. A device for establishing a vehicle helmet detection model, comprising: A data set acquisition unit is configured to acquire a vehicle helmet detection data set, wherein the vehicle helmet detection data set includes a plurality of image samples of the vehicle when the vehicle is moving or stopped; an extraction unit, configured to extract edge information and background information of each image sample from the plurality of image samples; an identification unit configured to identify, for each image sample in the plurality of image samples, one or more vehicle helmets in the image sample; as well as The training unit is configured to train a neural network model based on the edge information, the background information and the one or more identified vehicle helmets to establish a vehicle helmet detection model.

7. A vehicle helmet detection device, comprising: a capturing unit configured to capture one or more images of the vehicle during driving or stopping via a sensor; an object detection unit, configured to perform object detection on each of the one or more images using a vehicle helmet detection model to determine whether the driver of the vehicle is wearing a vehicle helmet, wherein the vehicle helmet detection model is established according to the method of any one of claims 1 to 4; as well as The warning unit is configured to warn the driver based on determining that the driver of the vehicle is not wearing a vehicle helmet.

8. An electronic device comprising: a memory, a processor and a computer program stored on the memory, The processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1 to 4 or 5.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 or 5 are implemented.

10. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 or 5 are implemented.

Images