Non-motor vehicle illegal driving behavior detection method, device and embedded device

Abstract

This application relates to a method, apparatus, embedded device, storage medium, and computer program product for detecting non-motorized vehicle driving violations. The method includes: defining target types for non-motorized vehicle violation detection; creating a violation detection dataset based on the target types; constructing a network structure; training the network using a knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model; acquiring video data; defining regions of interest and detection regions within the video data; detecting targets within the detection regions using the lightweight model to obtain driving behavior detection results; performing driving behavior discrimination based on the driving behavior detection results; and if the discrimination result indicates the existence of a violation, saving the violation type and extracting the violation image. This method effectively balances accuracy and speed, reduces the consumption of device computing power, decreases false positives and false negatives of violations, and is convenient for deployment in embedded device applications.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, device, embedded device, storage medium, and computer program product for detecting non-motorized vehicle driving violations. Background Technology

[0002] With rapid urbanization and increasing urban population, the number of motor vehicles and non-motor vehicles has grown rapidly, leading to greater attention and research into traffic safety. Compared to motor vehicles, non-motor vehicles are characterized by their smaller size, insufficient protective equipment, and fewer restrictions, making them more prone to accidents and causing significant losses. In scenarios involving the absence of helmets and unauthorized passenger transport, non-motor vehicles are more likely to cause injuries or fatalities.

[0003] In recent years, smart cities and the artificial intelligence industry have developed rapidly, leading to significant improvements in corresponding hardware and algorithms. This has driven research into the detection of both motor vehicles and non-motor vehicles, such as the YOLO series of detection algorithms and the FGD algorithm in knowledge distillation. Detection of behaviors by motor vehicles, such as running red lights, speeding, and overloading, has effectively reduced violations and improved driver safety. Similarly, detecting behaviors by non-motor vehicle drivers, such as not wearing helmets, illegally carrying passengers, and driving against traffic, will effectively improve the flow of traffic in non-motor vehicle lanes and enhance the safety of non-motor vehicle drivers.

[0004] Numerous methods have been applied to detect non-motorized vehicle violations, primarily falling into two categories: one-stage and two-stage. One-stage methods focus on detection, judging violations based on logical rules once a target is detected, offering advantages in speed and efficiency. Two-stage methods primarily employ a "detection + tracking" approach, first detecting the target, then tracking its movement trajectory to determine if its behavior violates regulations, offering the advantage of preventing target confusion. However, due to limited datasets, the difficulty of detecting small targets, and the limited computing power of embedded devices, existing methods struggle to achieve good detection results, easily leading to false positives and false negatives. Furthermore, these methods require high-performance embedded devices, resulting in high costs and hindering widespread adoption and practical application. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, device, embedded device, computer-readable storage medium, and computer program product for detecting non-motorized vehicle driving violations that can improve the accuracy and speed of violation detection while reducing computing power, in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a method for detecting non-motorized vehicle driving violations. The method includes:

[0007] Step A: Define the target types for non-motorized vehicle violation detection, and create a violation detection dataset based on the target types;

[0008] Step B: Build the network structure and train the network using the knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model;

[0009] Step C: Acquire video data and delineate the region of interest and detection region within the video data;

[0010] Step D: The targets in the detection area are detected using a lightweight model to obtain the driving behavior detection results;

[0011] Step E: Based on the driving behavior detection results, determine the driving behavior;

[0012] Step F: If the judgment result indicates that there is a violation, save the violation type and capture the violation image; if the judgment result indicates that there is no violation, return to step D to continue processing.

[0013] In one embodiment, step B: building the network structure, training the network using a knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model, including:

[0014] Step B1: Preprocess the image channels and dimensions of the violation detection dataset;

[0015] Step B2: Build the feature extraction module, feature fusion module, and detection information output module used for training the model;

[0016] Step B3: Train the teacher model based on the violation detection dataset;

[0017] Step B4: Train a lightweight model based on the teacher model and the violation detection dataset.

[0018] In one embodiment, step B4: training a lightweight model based on the teacher model and the violation detection dataset includes:

[0019] Step B41: Input the violation detection dataset for training, and obtain the first student feature, second student feature, third student feature, first teacher feature, second teacher feature, third teacher feature, and loss value of the lightweight model before and after feature fusion in each round of the teacher model and the lightweight model;

[0020] Step B42: Input the first student features, second student features, third student features, first teacher features, second teacher features, and third teacher features of the teacher model and the lightweight model into the knowledge distillation optimization network to obtain the first loss, second loss, and third loss;

[0021] Step B43: Obtain the training loss and pass it through iterative steps; the training loss is the sum of the first loss, the second loss, the third loss, and the model's own loss.

[0022] Step B44: Determine whether the training has converged or reached the maximum number of training iterations. If it has converged or reached the maximum number of training iterations, save the lightweight model; otherwise, return to step B41 to continue training.

[0023] In one embodiment, step C: acquiring video data and defining regions of interest and detection regions within the video data, including:

[0024] Step C1: Convert and quantize the lightweight model, and deploy it to an embedded device;

[0025] Step C2: Start the embedded device and acquire video data;

[0026] Step C3: Delineate the region of interest based on the characteristics of the non-motorized vehicle lane, and perform target detection within the region of interest;

[0027] Step C4: Within the region of interest, define the detection area; the detection area is used to determine driving behavior.

[0028] In one embodiment, step E: Based on the driving behavior detection results, driving behavior is judged, including: according to the characteristics of non-motorized vehicles, simultaneously judging not wearing a helmet and illegally carrying passengers:

[0029] When a target passes through the detection area, the target is identified.

[0030] When a non-motorized vehicle is detected among the targets, the video frame at that moment is captured to obtain the captured image, which is then used as the input to the secondary detection model.

[0031] The captured image is preprocessed and then subjected to secondary recognition to obtain the processed image;

[0032] In the secondary detection, the processed image is detected three times consecutively. If a person and a helmet are found to be present in two out of the three consecutive detections, the number of people and helmets is compared.

[0033] The driving behavior is judged based on the comparison results.

[0034] In one embodiment, step E: based on the driving behavior detection results, driving behavior discrimination further includes: determining non-motorized vehicle driving in the wrong direction.

[0035] When a non-motorized vehicle is detected among the targets in the detection area, the non-motorized vehicle is tracked.

[0036] The direction of movement of non-motorized vehicles is determined based on the tracking results. If the direction is consistent with the direction of the detection area, it is considered to be moving in the same direction as the detection area; if the direction is opposite to the direction of the detection area, it is considered to be moving against the flow of traffic.

[0037] Secondly, this application also provides a device for detecting non-motorized vehicle driving violations. The device includes:

[0038] The data acquisition module is used to define the target types for non-motorized vehicle violation detection and to create violation detection datasets based on the target types.

[0039] The model building module is used to build the network structure and train the network using the knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model.

[0040] The video processing module is used to acquire video data and delineate the region of interest and detection region within the video data;

[0041] The detection module is used to detect targets in the detection area using a lightweight model to obtain driving behavior detection results;

[0042] The judgment module is used to judge driving behavior based on the driving behavior detection results;

[0043] If the processing module determines that a violation exists, it saves the violation type and extracts the violation image; if the determination result is that no violation exists, it returns to the detection module for further processing.

[0044] Thirdly, this application also provides an embedded device. The embedded device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0045] Step A: Define the target types for non-motorized vehicle violation detection, and create a violation detection dataset based on the target types;

[0046] Step B: Build the network structure and train the network using the knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model;

[0047] Step C: Acquire video data and delineate the region of interest and detection region within the video data;

[0048] Step D: The targets in the detection area are detected using a lightweight model to obtain the driving behavior detection results;

[0049] Step E: Based on the driving behavior detection results, determine the driving behavior;

[0050] Step F: If the judgment result indicates that there is a violation, save the violation type and capture the violation image; if the judgment result indicates that there is no violation, return to step D to continue processing.

[0051] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0052] Step A: Define the target types for non-motorized vehicle violation detection, and create a violation detection dataset based on the target types;

[0053] Step B: Build the network structure and train the network using the knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model;

[0054] Step C: Acquire video data and delineate the region of interest and detection region within the video data;

[0055] Step D: The targets in the detection area are detected using a lightweight model to obtain the driving behavior detection results;

[0056] Step E: Based on the driving behavior detection results, determine the driving behavior;

[0057] Step F: If the judgment result indicates that there is a violation, save the violation type and capture the violation image; if the judgment result indicates that there is no violation, return to step D to continue processing.

[0058] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0059] Step A: Define the target types for non-motorized vehicle violation detection, and create a violation detection dataset based on the target types;

[0060] Step B: Build the network structure and train the network using the knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model;

[0061] Step C: Acquire video data and delineate the region of interest and detection region within the video data;

[0062] Step D: The targets in the detection area are detected using a lightweight model to obtain the driving behavior detection results;

[0063] Step E: Based on the driving behavior detection results, determine the driving behavior;

[0064] Step F: If the judgment result indicates that there is a violation, save the violation type and capture the violation image; if the judgment result indicates that there is no violation, return to step D to continue processing.

[0065] The aforementioned non-motorized vehicle violation detection method, device, embedded equipment, storage medium, and computer program product, through the creation of a violation detection dataset and the construction of a network structure, trains the network using a knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model. Based on the use of this lightweight model, this application effectively balances accuracy and speed by defining regions of interest and detection regions, and using the detected targets within these regions as valid targets. Multiple simple, reasonable, and effective rules are formulated to judge violations such as not wearing helmets, illegal passenger transport, and driving against traffic. If a violation occurs, photographic evidence is taken. That is, by defining regions to filter out valid targets and formulating simple and reasonable judgment rules, the consumption of equipment computing power is reduced, effectively maintaining road traffic flow and driver safety.

[0066] The non-motorized vehicle violation driving behavior detection method proposed in this application uses a lightweight detection algorithm, which is further optimized using a knowledge distillation algorithm to meet the requirements of high accuracy and high speed, providing more accurate detection results for behavior judgment. Secondly, the non-motorized vehicle violation driving behavior detection method of this application has formulated reasonable and effective judgment rules for behaviors such as not wearing a helmet, illegally carrying passengers, and driving against the flow of traffic. For example, when judging driving against the flow of traffic, the timer tracking algorithm proposed in this invention can ensure judgment accuracy while reducing the amount of computation. Thirdly, the non-motorized vehicle violation driving behavior detection method of this application effectively avoids repeated detection of the same target by delineating the region of interest and the detection region, reducing the amount of computation and helping to reduce the requirements for computing power, thereby reducing costs and promoting the widespread application of the equipment. Finally, the non-motorized vehicle violation driving behavior detection method of this application is simple to operate after deployment, making it convenient for construction personnel to install and apply.

[0067] Therefore, the non-motorized vehicle violation driving behavior detection method, device, embedded device, storage medium and computer program product proposed in this application have high precision and high speed, effectively reducing the missed detection and false detection rate of violations, and are convenient to deploy in embedded devices and install at intersections. Attached Figure Description

[0068] Figure 1 This is an application environment diagram of a non-motorized vehicle violation driving behavior detection method in one embodiment;

[0069] Figure 2 This is a flowchart illustrating a method for detecting non-motorized vehicle driving violations in one embodiment;

[0070] Figure 3 This is a schematic diagram of dataset annotation in one embodiment;

[0071] Figure 4 This is a flowchart of the steps for building a network structure and training a detection model in one embodiment;

[0072] Figure 5 This is a flowchart of the steps for training a lightweight model based on a teacher model and a violation detection dataset in one embodiment.

[0073] Figure 6 This is a schematic diagram illustrating knowledge distillation by combining a teacher model and a student model in one embodiment.

[0074] Figure 7 This is a flowchart illustrating the steps of acquiring video data and defining the region of interest and detection region in one embodiment.

[0075] Figure 8 This is a schematic diagram of the region of interest and detection region defined in one embodiment;

[0076] Figure 9 This is a schematic diagram of image preprocessing in one embodiment;

[0077] Figure 10 This is a schematic diagram of the process for identifying not wearing a helmet and illegal transportation in one embodiment;

[0078] Figure 11 This is a schematic diagram illustrating the determination of non-motorized vehicles traveling in the wrong direction in one embodiment;

[0079] Figure 12 This is a flowchart illustrating a non-motorized vehicle violation driving behavior detection method in another embodiment;

[0080] Figure 13 This is a structural block diagram of a non-motorized vehicle violation driving behavior detection device in one embodiment;

[0081] Figure 14 This is an internal structure diagram of an embedded device in one embodiment. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0083] The non-motorized vehicle violation driving behavior detection method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0084] In one embodiment, such as Figure 2 As shown, a method for detecting non-motorized vehicle driving violations is provided, including the following steps:

[0085] Step A: Define the target types for non-motorized vehicle violation detection, and create a violation detection dataset based on the target types.

[0086] Based on violations by non-motorized vehicles, such as not wearing helmets, illegally carrying passengers, and traveling against traffic, the detection targets are determined to be non-motorized vehicles, people, helmets, and heads. Videos were recorded at multiple intersections and in multiple scenarios targeting these four categories, and compiled into a dataset for algorithm model training and testing.

[0087] Figure 3 This is a schematic diagram of dataset annotation in one embodiment. From... Figure 3 As can be seen, by labeling and defining different target types in the left-hand image, such as non-motorized vehicles, people, helmets, and heads, corresponding data can be obtained, and this data can be compiled into a dataset. Figure 3 The image shown is provided on the right.

[0088] Step B: Build the network structure and train the network using the knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model.

[0089] In one embodiment, building the network structure includes constructing a student network, a teacher network, and a distillation network. A knowledge distillation strategy is then used to optimize the training of a lightweight model.

[0090] Considering the difficulty of detecting small targets such as helmets and heads in the scene and the limited computing power of terminals such as embedded devices, this application uses the knowledge distillation algorithm to train the network, which can effectively balance the relationship between accuracy and speed.

[0091] Knowledge distillation is a classic model compression method. Its core idea is to improve the performance of a lightweight student model by guiding it to "imitate" a teacher model that has better performance and a more complex structure, without changing the structure of the student model.

[0092] Step C: Acquire video data and delineate the region of interest and detection region within the video data.

[0093] Step D: The detection targets in the detection area are detected using a lightweight model to obtain the driving behavior detection results.

[0094] Step E: Based on the driving behavior detection results, determine the driving behavior.

[0095] Step F: If the judgment result indicates that there is a violation, save the violation type and capture the violation image; if the judgment result indicates that there is no violation, return to step D to continue processing.

[0096] When it is determined that the driving behavior constitutes a violation, the violation type is saved and a violation image is captured; when it is determined that the driving behavior does not constitute a violation, the detection of the detection area continues. That is, the detection targets in the real-time acquired detection area are detected using a lightweight model to obtain the detection result of the driving behavior.

[0097] In the aforementioned method for detecting non-motorized vehicle violations, a lightweight model is obtained by creating a violation detection dataset, building a network structure, and training the network using a knowledge distillation algorithm based on the violation detection dataset. This application, based on the use of the lightweight model, effectively balances accuracy and speed by defining regions of interest and detection regions, and using the detected targets within these regions as valid targets. Multiple simple, reasonable, and effective rules are formulated to judge violations such as not wearing helmets, illegal passenger transport, and driving against traffic. If a violation occurs, photographic evidence is taken. That is, by defining regions to filter out valid targets and formulating simple and reasonable judgment rules, the consumption of computing power on equipment is reduced, effectively maintaining road traffic flow and driver safety.

[0098] In one embodiment, such as Figure 4 The diagram shows a flowchart of the steps for building the network structure and training the detection model. Step B involves building the network structure and training the network using a knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model, including:

[0099] Step B1: Preprocess the image channels and dimensions of the violation detection dataset.

[0100] Step B2: Build the feature extraction module, feature fusion module, and detection information output module used for training the model.

[0101] In this application, the lightweight model can be described as the student lightweight network structure yolov5s, the student model, or the student lightweight model; the teacher model can be described as the teacher large model network structure yolov5x, the teacher model, or the teacher large model; based on the width and depth of the modules, the student lightweight network structure yolov5s and the teacher large model network structure yolov5x are constructed.

[0102] Step B3: Train the teacher model based on the violation detection dataset.

[0103] In one embodiment, a large teacher model, yolov5x.pt, is trained using a created violation detection dataset.

[0104] Step B4: Train a lightweight model based on the teacher model and the violation detection dataset.

[0105] In one embodiment, the lightweight model yolov5s.pt is trained using the large teacher model yolov5x.pt and a created violation detection dataset.

[0106] In one embodiment, such as Figure 5 The diagram shows the flowchart of the steps for training a lightweight model based on the teacher model and the violation detection dataset. Step B4: Training a lightweight model based on the teacher model and the violation detection dataset includes:

[0107] Step B41: Input the violation detection dataset for training, and obtain the first student feature, second student feature, third student feature, first teacher feature, second teacher feature, third teacher feature, and loss value of the lightweight model before and after feature fusion in each round of the teacher model and the lightweight model.

[0108] For example, by inputting a violation detection dataset for training, we obtain three output features (first output feature, second output feature, and third output feature) before and after feature fusion for each round of the teacher's large model yolov5x.pt and the student model, as well as the loss value Ls of the student model.

[0109] Step B42: Input the first student features, second student features, third student features, first teacher features, second teacher features, and third teacher features of the teacher model and the lightweight model into the knowledge distillation optimization network to obtain the first loss, second loss, and third loss.

[0110] like Figure 6As shown, the three output features of the large teacher model (first teacher feature, second teacher feature, and third teacher feature) are input into the knowledge distillation optimization network FGD, and the three output features of the lightweight student model (first student feature, second student feature, and third student feature) are input into the knowledge distillation optimization network FGD. The difference between the two models is calculated, resulting in three losses, namely the first loss F... loss1 Second loss F loss2 and the third loss F loss3 .

[0111] Step B43: Obtain the training loss and pass it through iterative steps; the training loss is the sum of the first loss, the second loss, the third loss, and the model's own loss.

[0112] The loss is obtained by combining the model's own loss and the knowledge distillation optimization loss. This loss is then used as the training loss to iteratively optimize the model, thus serving as a teacher guiding students and improving the accuracy of the small model. The loss combination formula is as follows:

[0113] Loss = Ls + F loss1 +F loss2 +F loss3

[0114] Step B44: Determine whether the training has converged or reached the maximum number of training iterations. If it has converged or reached the maximum number of training iterations, save the lightweight model; otherwise, return to step B41 to continue training.

[0115] For example, determine whether the training has converged or reached the maximum number of training iterations. If it has, save the model yolov5s.pt; otherwise, return to step B41 to continue training.

[0116] In one embodiment, such as Figure 7 The flowchart shown illustrates the steps involved in defining the region of interest (ROI) and detection region (DRG) of the video data to acquire the video data. Step C: Acquiring video data and defining the RRO and DRG of the video data includes:

[0117] Step C1: Convert and quantize the lightweight model, and deploy it into an embedded device.

[0118] For example, the obtained yolov5s.pt model is formatted and quantized before being deployed to an embedded device.

[0119] Step C2: Start the embedded device and acquire video data.

[0120] After the device is started, it reads the video to delineate the region of interest and the detection region, as detailed in steps C3 and C4:

[0121] Step C3: Delineate the region of interest based on the characteristics of the non-motorized vehicle lane, and perform target detection within the region of interest.

[0122] Step C4: Within the region of interest, define the detection area; the detection area is used to determine driving behavior.

[0123] like Figure 8 As shown, Figure 8 This is a schematic diagram of the region of interest and detection region defined in one embodiment; Figure 8 The region of interest is represented by a dashed line. Within this region of interest, a detection zone is defined by a solid line. This detection zone is used to determine violations such as not wearing a helmet, driving against traffic, and illegally carrying passengers. The arrows indicate the direction of the detection zone.

[0124] In this implementation, a human-computer interaction method is used to delineate a region of interest (ROI) and a detection area with directional attributes on the non-motorized vehicle lane. Targets detected within these areas are considered valid targets. By delineating the ROI and detection area, duplicate detection of the same target is effectively avoided, reducing computational load and lowering the computational power requirements, thereby reducing costs and promoting the widespread application of the equipment.

[0125] In one embodiment, step E: Based on the driving behavior detection results, driving behavior is judged, including: according to the characteristics of non-motorized vehicles, simultaneously judging not wearing a helmet and illegally carrying passengers:

[0126] When a target passes through the detection area, the target is identified.

[0127] When a non-motorized vehicle is detected among the targets, the video frame at that moment is captured to obtain a cropped image, which is then used as the input to the secondary detection model.

[0128] The captured image is preprocessed and then subjected to secondary recognition to obtain the processed image.

[0129] In the secondary detection, the processed image is detected three times consecutively. If a person and a helmet are found to be present in two out of the three consecutive detections, the number of people and helmets is compared.

[0130] The driving behavior is judged based on the comparison results.

[0131] In one embodiment, such as Figure 10 The diagram illustrates the process of identifying not wearing a helmet and illegally carrying passengers. Based on the characteristics of non-motorized vehicles, not wearing a helmet and illegally carrying passengers are judged simultaneously:

[0132] (1) When a detection target passes through the set detection area, the detection target is identified. If a non-motorized vehicle detection target is found among the detection targets, the video frame at this moment is captured and used as the model input for secondary detection.

[0133] (2) Preprocessing and secondary recognition are performed on the captured images containing non-motorized vehicle detection targets. Specifically, such as... Figure 9 As shown, Figure 9 This is a schematic diagram of image preprocessing in one embodiment.

[0134] (01) Expand the non-motorized vehicle detection target frame to twice its original size as the selected area;

[0135] (02) Blacken the unselected area to filter out the interference of non-motorized vehicles and other targets traveling side by side.

[0136] (03) Use the preprocessed image as the model input for inference.

[0137] (3) In the secondary detection, the processed image is detected three times in a row. If a person or a helmet is found in two of the three consecutive detection targets, the number is recorded as r and h respectively.

[0138] (4) Determine based on the number of people and the number of helmets, such as:

[0139]

[0140] In one embodiment, step E: based on the driving behavior detection results, driving behavior discrimination further includes: determining non-motorized vehicle driving in the wrong direction.

[0141] When a non-motorized vehicle is detected among the targets in the detection area, the non-motorized vehicle is tracked.

[0142] The direction of movement of non-motorized vehicles is determined based on the tracking results. If the direction is consistent with the direction of the detection area, it is considered to be moving in the same direction as the detection area; if the direction is opposite to the direction of the detection area, it is considered to be moving against the flow of traffic.

[0143] For example, such as Figure 11 As shown, this is a schematic diagram for determining non-motorized vehicles traveling in the wrong direction in one embodiment. Figure 11 The dashed box represents the region of interest, the solid box represents the detection area, and the double-dotted box represents the detected non-motorized vehicle target box. The arrows point in the normal driving direction, i.e., the direction of the detection area. The upper left corner schematically indicates the camera's position. (See reference) Figure 11 Judging non-motorized vehicle traffic against the flow of traffic includes:

[0144] (1) When a non-motorized vehicle target is detected in the detection area, it is tracked. The tracking algorithm is as follows:

[0145] (01) Record the center point C of the target frame at this moment. jr and target bounding box;

[0146] (02) Using the target detection speed as a timer, when the timer expires, the center point C of the target (non-motorized vehicle) is determined again. lk If a target exists in the detection frame of the previous time step, the process continues to the next time step; otherwise, the target is cleared. This process continues until the target leaves the set detection area, and the center point coordinates C of the target when it enters and leaves the detection area are saved. jr (x jr ,y jr ) and C lk (x lk ,y lk )

[0147] (2) The direction of movement of the non-motorized vehicle is determined based on the results of the tracking algorithm. If the direction is consistent with the direction of the set detection area, it is considered to be moving in the same direction as the set detection area; if the direction is opposite to the direction of the set detection area, it is considered to be moving in the opposite direction.

[0148]

[0149] In this embodiment, reasonable and effective judgment rules are formulated for behaviors such as not wearing a helmet, illegally carrying passengers, and driving against the flow of traffic. For example, when judging driving against the flow of traffic, the timer tracking algorithm proposed in this invention is used, which can ensure the judgment accuracy while reducing the amount of computation.

[0150] In one embodiment, such as Figure 12 As shown, a method for detecting non-motorized vehicle driving violations is provided, which includes the following steps:

[0151] Step 1202, Dataset Creation: Define the target types in the non-motorized vehicle violation detection dataset, and then create a dataset for training non-motorized vehicle violation detection.

[0152] Step 1204: Establish the student network, teacher network, and distillation network.

[0153] Step 1206: Use the knowledge distillation strategy to optimize the training of the lightweight model.

[0154] Step 1208, Model Format Conversion, Quantization, and Deployment on Device: The obtained lightweight model is converted and quantized, and then deployed to an embedded device.

[0155] Step 1210: The device starts up and reads the video.

[0156] Step 1212: Delineate the region of interest and the detection region.

[0157] Step 1214, Target Detection: Delineate the region of interest based on the characteristics of the non-motorized vehicle lane, and use this region for target detection.

[0158] Step 1216: Determine if there are any non-motorized vehicles. If there are no non-motorized vehicles, return to step 1214 to continue target detection.

[0159] Step 1218: Determine whether there is a violation according to the established rules: If a non-motorized vehicle is detected in step 1216, determine whether there is a violation according to the established rules; if the non-motorized vehicle does not violate the rules, return to step 1214 to continue target detection.

[0160] Step 1220: Output the violation type and save the violation image: When a violation is determined in step 1218, output the violation type and save the violation image.

[0161] The non-motorized vehicle violation detection method provided in this application improves the accuracy and speed of violation detection, while reducing computational requirements and facilitating widespread application, effectively maintaining road traffic flow and driver safety. To address the issue of limited datasets for non-motorized vehicle violation detection, cameras were initially installed at multiple intersections to record non-motorized vehicle lane footage, ensuring multi-scene and multi-category data to create a dataset for target detection. To address the problems of low detection algorithm accuracy and difficulty in embedded applications, the lightweight algorithm YOLOv5S is used, and a knowledge distillation strategy is employed to optimize it, ensuring both speed and improved accuracy. Simultaneously, a detection area is defined to screen effective targets, and several simple and reasonable rules are established to distinguish violations, reducing the computational requirements of embedded devices and ensuring that the equipment can be installed and applied at various intersections with limited economic costs.

[0162] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0163] Based on the same inventive concept, this application also provides a non-motorized vehicle violation driving behavior detection device for implementing the aforementioned non-motorized vehicle violation driving behavior detection method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the non-motorized vehicle violation driving behavior detection device provided below can be found in the limitations of the non-motorized vehicle violation driving behavior detection method described above, and will not be repeated here.

[0164] In one embodiment, such as Figure 13 As shown, a non-motorized vehicle violation driving behavior detection device is provided, including: a data acquisition module 1302, a model construction module 1304, a video processing module 1306, a detection module 1308, a judgment module 1310, and a processing module 1312, wherein:

[0165] Data acquisition module 1302 is used to define the target types of non-motorized vehicle violation detection and to create a violation detection dataset based on the target types;

[0166] The model building module 1304 is used to build the network structure and train the network using the knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model.

[0167] The video processing module 1306 is used to acquire video data and delineate the region of interest and detection region in the video data;

[0168] The detection module 1308 is used to detect the target in the detection area using a lightweight model to obtain driving behavior detection results;

[0169] The judgment module 1310 is used to judge driving behavior based on the driving behavior detection results;

[0170] If the processing module 1312 determines that there is a violation, it saves the violation type and captures the violation image; if the determination result is that there is no violation, it returns to the detection module 1308 for further processing.

[0171] In one embodiment, the model building module 1304 is further used to preprocess the image channels and sizes of the violation detection dataset; build a feature extraction module, feature fusion and detection information output module for training the model; train a teacher model based on the violation detection dataset; and train a lightweight model based on the teacher model and the violation detection dataset.

[0172] In one embodiment, the model building module 1304 is further configured to input a violation detection dataset for training, obtaining the first student features, second student features, third student features, first teacher features, second teacher features, third teacher features, and loss values ​​of the lightweight model before and after feature fusion in each round of the teacher model and the lightweight model; inputting the first student features, second student features, third student features, first teacher features, second teacher features, and third teacher features of the teacher model and the lightweight model into a knowledge distillation optimization network to obtain a first loss, a second loss, and a third loss; obtaining the training loss and iteratively passing the training loss; the training loss is the sum of the first loss, the second loss, the third loss, and the model's own loss; determining whether convergence has been achieved or the maximum number of training iterations has been reached; if convergence has been achieved or the maximum number of training iterations has been reached, saving the lightweight model; otherwise, returning to the input violation detection dataset to continue training.

[0173] In one embodiment, the video processing module 1306 is further configured to perform format conversion and quantization of the lightweight model and deploy it into an embedded device; start the embedded device and acquire video data; delineate the region of interest based on the characteristics of the non-motorized vehicle lane and perform target detection in the region of interest; delineate a detection area within the region of interest; and use the detection area for the determination of driving behavior.

[0174] In one embodiment, the determination module 1310 is further configured to simultaneously determine whether a non-motorized vehicle is not wearing a helmet or illegally carrying passengers based on the characteristics of the non-motorized vehicle: when a detection target passes through the detection area, the detection target is identified; when a non-motorized vehicle is found among the identified detection targets, a video frame at this time is captured to obtain a captured image, which is used as the model input for secondary detection; the captured image is preprocessed and re-identified to obtain a processed image; in the secondary detection, the processed image is detected three times consecutively, and if a person and a helmet are found in two of the three consecutive detection targets, the number of people and helmets is compared; the driving behavior is determined based on the comparison result.

[0175] In one embodiment, the determination module 1310 is also used to determine the non-motorized vehicle's reverse driving behavior: when a non-motorized vehicle is detected among the detection targets in the detection area, the non-motorized vehicle is tracked; the direction of movement of the non-motorized vehicle is determined based on the tracking result, if it is consistent with the direction of the detection area, it is driving in the same direction, and if it is opposite to the direction of the detection area, it is driving in the wrong direction.

[0176] Each module in the aforementioned non-motorized vehicle violation detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the embedded device in hardware form or independently of it, or stored in the memory of the embedded device in software form, so that the processor can call and execute the corresponding operations of each module.

[0177] In one embodiment, an embedded device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 14 As shown, the embedded device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media. The database stores non-motorized vehicle violation detection data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for detecting non-motorized vehicle violation driving behavior.

[0178] Those skilled in the art will understand that Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application and does not constitute a limitation on the embedded device to which the solution of this application is applied. A specific embedded device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0179] In one embodiment, an embedded device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0180] The device in this application is installed about 40 meters away from the non-motorized vehicle lane of the intersection, with a pole height of about 8 meters. After testing and verification under different lighting and weather conditions, the speed and accuracy of the method used in this invention meet the application requirements of various scenarios. Moreover, the lightweight model optimized by the knowledge distillation algorithm in this application has higher accuracy than the original model.

[0181] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0182] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0184] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0185] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0186] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting non-motorized vehicle driving violations, characterized in that, The method includes: Step A: Define the target types for non-motorized vehicle violation detection, and create a violation detection dataset based on the target types; Step B: Build the network structure and train the network using the knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model; Step C: Acquire video data, define the region of interest (ROI) based on the characteristics of the non-motorized vehicle lane, and define the detection region within the ROI; Step D: Detect targets entering the detection area using a lightweight model to obtain driving behavior detection results; Step E: Based on the driving behavior detection results, determine the driving behavior; Step F: If the judgment result indicates that there is a violation, save the violation type and capture the violation image; if the judgment result indicates that there is no violation, return to step D to continue processing. Step E, based on the driving behavior detection results, involves judging driving behavior, including simultaneously judging not wearing a helmet and illegally carrying passengers according to the characteristics of non-motorized vehicles. When a target passes through the detection area, the target is identified; When a non-motorized vehicle is detected among the targets, the video frame at that moment is captured to obtain a captured image, which is then used as the input to the secondary detection model. The captured image is preprocessed and then subjected to secondary recognition to obtain the processed image; In the secondary detection, the processed image is detected three times consecutively. If a person and a helmet are found to be present in two of the three consecutive detections, the number of people and helmets is compared. The driving behavior is judged based on the comparison results.

2. The method according to claim 1, characterized in that, Step B: Constructing the network structure, training the network using the knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model, including: Step B1: Preprocess the image channels and dimensions of the violation detection dataset; Step B2: Build the feature extraction module, feature fusion module, and detection information output module used for training the model; Step B3: Train the teacher model based on the violation detection dataset; Step B4: Train a lightweight model based on the teacher model and the violation detection dataset.

3. The method according to claim 2, characterized in that, Step B4: Training a lightweight model based on the teacher model and the violation detection dataset, including: Step B41: Input the violation detection dataset for training, and obtain the first student feature, second student feature, third student feature, first teacher feature, second teacher feature, third teacher feature, and loss value of the lightweight model before and after feature fusion in each round of the teacher model and the lightweight model; Step B42: Input the first student features, second student features, third student features, first teacher features, second teacher features, and third teacher features of the teacher model and the lightweight model into the knowledge distillation optimization network to obtain the first loss, second loss, and third loss; Step B43: Obtain the training loss and pass it through iterative steps; the training loss is the sum of the first loss, the second loss, the third loss, and the model's own loss. Step B44: Determine whether the training has converged or reached the maximum number of training iterations. If it has converged or reached the maximum number of training iterations, save the lightweight model; otherwise, return to step B41 to continue training.

4. The method according to claim 1, characterized in that, Step C: Acquiring video data, including: Step C1: Convert and quantize the lightweight model, and deploy it into an embedded device; Step C2: Start the embedded device and acquire video data.

5. The method according to claim 1, characterized in that, Step E: Based on the driving behavior detection results, driving behavior discrimination is performed, which further includes: judging non-motorized vehicle driving against traffic. When a non-motorized vehicle is detected among the targets detected in the detection area, the non-motorized vehicle is tracked. The direction of movement of non-motorized vehicles is determined based on the tracking results. If the direction is consistent with the direction of the detection area, it is considered to be moving in the same direction as the detection area; if the direction is opposite to the direction of the detection area, it is considered to be moving against the flow of traffic.

6. A device for detecting non-motorized vehicle driving violations, characterized in that, The device includes: The data acquisition module is used to define the target types for non-motorized vehicle violation detection and to create a violation detection dataset based on the target types. The model building module is used to build the network structure and train the network using the knowledge distillation algorithm based on the violation detection dataset to obtain a lightweight model. The video processing module is used to acquire video data, define a region of interest in the video data according to the characteristics of the non-motorized vehicle lane, and define a detection region within the region of interest; The detection module is used to detect targets entering the detection area using a lightweight model to obtain driving behavior detection results; The determination module is used to determine driving behavior based on the driving behavior detection results; If the processing module determines that a violation exists, it saves the violation type and extracts the violation image; if the determination result is that no violation exists, it returns to the detection module for further processing. The judgment module is also used to simultaneously judge not wearing a helmet and illegally carrying passengers based on the characteristics of non-motorized vehicles: when a detection target passes through the detection area, the detection target is identified; when a non-motorized vehicle is found among the identified detection targets, a video frame at this time is captured to obtain a captured image, which is used as the input to the model for secondary detection; the captured image is preprocessed and re-identified to obtain a processed image; in the secondary detection, the processed image is detected three times consecutively, and if a person and a helmet are found in two out of the three consecutive detection targets, the number of people and helmets is compared; the driving behavior is judged based on the comparison result.

7. An embedded device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images