Motorcycle examination evaluation method based on machine vision and electronic device
By using machine vision technology, combined with YOLOv5, YOLOv8 and target region segmentation networks, the problems of low efficiency of manual evaluation and high false alarm rate of sensors in motorcycle driving tests have been solved, realizing intelligent and accurate evaluation of motorcycle tests.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DUOLUN TECH CO LTD
- Filing Date
- 2024-09-18
- Publication Date
- 2026-07-31
AI Technical Summary
In motorcycle driving tests, manual evaluation is inefficient and inaccurate, while sensor evaluation has a high false alarm rate and is costly, making it difficult to achieve accurate automated evaluation of helmet wearing and pole maneuvering.
Using a machine vision-based approach, leveraging YOLOv5 and YOLOv8 neural network models and a target region segmentation network, the system identifies the positions of helmets, motorcycles, and test cones through real-time video streams. Combined with OpenCV processing, this enables intelligent evaluation of the motorcycle testing process.
It enables fully intelligent evaluation of the motorcycle examination process, making the evaluation more objective and accurate, reducing hardware costs, and improving evaluation efficiency and accuracy.
Smart Images
Figure CN119131693B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of motorcycle driving training technology, specifically relating to a motorcycle examination and evaluation method and electronic equipment based on machine vision. Background Technology
[0002] In motorcycle driving tests, ensuring that candidates wear helmets and operate motorcycles correctly is a crucial aspect of safety. Therefore, the motorcycle driving test process requires simultaneous evaluation of both helmet use and the motorcycle's maneuvering around obstacles. Currently, these evaluations mainly rely on manual monitoring or sensor detection. Manual evaluation suffers from low efficiency and insufficient accuracy. Sensor-based obstacle detection is costly, and sensors are susceptible to environmental factors such as wind and rain, which can damage them and lead to a high false alarm rate. With the development of computer vision technology, the application of deep learning models in image recognition and object detection is becoming increasingly widespread, providing new solutions for automated detection. Summary of the Invention
[0003] In view of the problems of low efficiency and insufficient accuracy of manual evaluation in the existing motorcycle test evaluation process, and high false alarm rate and high cost of sensor evaluation, the present invention provides a motorcycle test evaluation method, electronic device and computer-readable storage medium based on machine vision.
[0004] The first aspect of this invention discloses a machine vision-based method for evaluating motorcycle tests, comprising the following steps:
[0005] The real-time video stream of the motorcycle test site is acquired and converted into an image array feature. The real-time video stream contains relevant information about the examinee, motorcycle, helmet and test stakes in the motorcycle test site.
[0006] The image array features are input into the helmet detection model to obtain the helmet's location information;
[0007] The features of the image array are input into the vehicle-ground contact point detection model to obtain the motorcycle's location information and the location information of multiple motorcycle tire contact points with the ground; the motorcycle tire contact points with the ground are connected sequentially to form a polygonal region and processed by OpenCV to obtain the contact area mask.
[0008] Based on the helmet location information and motorcycle location information, it is determined whether the examinee is wearing a helmet;
[0009] If it is determined that the examinee is wearing a helmet, the features of the image array are input into the target region segmentation network model to obtain the examinee mask;
[0010] Based on the contact area mask and the test cone mask, it is determined whether the motorcycle hits the test cone during the cone-driving process. If it hits the test cone, it is considered a failure in the test.
[0011] The second aspect of the present invention discloses an electronic device comprising a processor and a memory, wherein the processor is configured to invoke a computer program stored in the memory to execute the machine vision-based motorcycle testing and evaluation method described in the first aspect and any optional embodiment of the present invention.
[0012] A third aspect of the present invention discloses a computer-readable storage medium for storing program code for executing the machine vision-based motorcycle testing and evaluation method described in the first aspect and any alternative embodiment of the present invention.
[0013] Beneficial effects:
[0014] The present invention provides a machine vision-based motorcycle examination evaluation method and electronic device, which combines machine vision and deep learning algorithms to identify the examination process, determine whether the examinee is wearing a helmet correctly and whether there is any collision with the pole, so as to achieve intelligent evaluation of the motorcycle examination process.
[0015] Compared to traditional manual monitoring or sensor detection, this invention can achieve intelligent evaluation throughout the entire process, and the evaluation is more objective and accurate, while also saving hardware costs. Attached Figure Description
[0016] Figure 1 A diagram showing the position of the examinee's helmet detected by the helmet detection model;
[0017] Figure 2 In the image: (a) is a schematic diagram of the coordinate regression network, and (b) is a schematic diagram of the heatmap regression network;
[0018] Figure 3 This is a schematic diagram of the contact points between a motorcycle tire and the ground detected by the vehicle-to-ground contact point detection model.
[0019] Figure 4 A schematic diagram showing the location of the test stakes segmented by the target region segmentation network model;
[0020] Figure 5 A schematic diagram illustrating the correct procedure for completing the motorcycle obstacle course test. Detailed Implementation
[0021] To further understand the present invention, the technical solution of the present invention will be further explained and described below with reference to specific embodiments and accompanying drawings.
[0022] This invention discloses a machine vision-based motorcycle testing and evaluation method, which mainly includes the following steps:
[0023] Step 1: Acquire real-time video streams of the motorcycle test site using cameras within the test site. The video streams contain information about the test takers, motorcycles, helmets, and test cones. Use OpenCV to convert the real-time video streams into image array features, which will then be used as input for the subsequent model.
[0024] Step 2: Use the helmet detection model to detect the features of the image array generated in Step 1, detect whether there are helmets in the examination room, and obtain the helmet location information.
[0025] The first step in motorcycle driving test evaluation is to determine if the examinee is wearing a helmet correctly. Only those wearing helmets correctly can proceed to the subsequent motorcycle operation test and evaluation. The image array features output from step 1 are input into the trained helmet detection model to detect whether a helmet is present in the testing area. If a helmet is present, its location information is output; otherwise, the examinee is deemed to have failed the test.
[0026] The helmet detection model can be specifically trained using a YOLOv5 neural network model. The YOLOv5 neural network model adopts a CSP+Focus+SPP structure, where: CSP contains residual structures of convolutional layers, BN layers, and activation layers; Focus contains four slice layers, which mainly fuse multi-scale input images, where multi-scale means that the network will cluster anchors of different scales as sliding boxes during training; SPP contains three max pooling layers, which output features of different scales, mainly including feature maps of scales of 80*80, 40*40, and 20*20.
[0027] The loss function of the YOLOv5 neural network model mainly consists of three parts: classification loss (L... cls The classification loss for positive samples is calculated using BCE (Binary Cross-Entropy Loss), where positive samples represent bounding boxes with an IOU (Intersection over Union, i.e., confidence level) greater than a first preset threshold (e.g., 0.5). The classification loss is mainly divided into two categories: presence or absence of a helmet; the target loss (L... obj The BCE loss is used to calculate the IOU between the predicted detection boxes and the actual detection boxes in the neural network model; the localization loss (L) is used. loc The localization loss for positive samples is calculated using IOU loss. The total loss function is defined as:
[0028] Loss=λ1L cls +λ2L obj +λ3L loc
[0029] In the formula: Loss represents the total loss; λ1, λ 2,λ3 represents the balance coefficient, with values of 0.4, 0.4, and 0.2, which can be adjusted according to the actual situation; L cls L represents classification loss; obj Indicates the target loss; L loc This indicates positioning loss.
[0030] The training process of the YOLOv5 neural network model mainly includes: constructing an initial YOLOv5 neural network model, training the YOLOv5 neural network model with training data, and obtaining the trained YOLOv5 neural network model after the total loss function value of the neural network tends to converge and stabilize.
[0031] The specific training process of the YOLOv5 neural network model is as follows:
[0032] Step a1: Obtain the features of the image array obtained in Step 1;
[0033] Step a2: Use a labeling tool (e.g., labelme) to mark and outline the rectangular detection boxes used to define the candidates' locations, and record the location information, confidence level, and classification of the rectangular detection boxes to form the labels corresponding to the detection boxes, which together with the features of the image array constitute the training data;
[0034] Specifically, a rectangular detection box can be used to define the object (i.e., the helmet) within the image array features. The position of the rectangular detection box can generally be defined in two ways: either by representing the coordinates of the top-left and bottom-right corners of the rectangle, or by representing the coordinates of the top-left corner and the length and width of the rectangle. In addition to face location information, each detection box also has corresponding confidence score and classification data. The classification data includes whether a helmet is present or not. Invalid detection boxes with confidence scores below a first preset threshold are filtered out, and the remaining detection boxes are considered valid.
[0035] Step a3: Input the training data into the initially constructed YOLOv5 neural network model for training. Through continuous backpropagation and data iteration, the total loss of the model tends to converge, and the trained YOLOv5 neural network model, i.e., the helmet detection model, is obtained.
[0036] The backpropagation process involves using matrix multiplication to calculate the gradient derivative of the parameters of each sample in each training batch (i.e., the number of samples randomly extracted from each batch during training and their corresponding labels) in each convolutional layer. The derivatives are then summed, and the results from the previous layer are weighted and summed with the results from the next layer to obtain the final feature value.
[0037] Step 3: Use the vehicle-to-ground contact point detection model to detect the features of the image array generated in Step 1, obtain the position information of the motorcycle and the position information of multiple motorcycle tire contact points with the ground, connect the motorcycle tires and ground contact points into a polygonal region, and then process it through OpenCV to convert it into a binary mask.
[0038] Based on the results obtained in step 2, if multiple helmets are detected in the examination room, the position of each helmet needs to be determined. This invention uses a YOLOv8 neural network model to detect the position of the motorcycle and the position information of the contact point between the motorcycle tires and the ground (referred to as the "vehicle-ground contact point").
[0039] The YOLOv8 neural network model used in this invention can detect not only the position of the motorcycle but also the contact point between the motorcycle tires and the ground; therefore, it is also called the "vehicle-ground contact point detection model." Combined with... Figure 3 As shown, the model detects the motorcycle's position and the contact area between the motorcycle tire and the ground (formed by connecting multiple contact points between the motorcycle tire and the ground). The motorcycle's position can also be defined using the rectangular detection box described in step 2. There can be six vehicle-to-ground contact points, typically six points mapped from the tire towards the ground, meaning each contact point is equidistant from the tire's center. These six contact points are sequentially connected to generate a polygonal region. This polygon is then processed using OpenCV to convert it into a binary mask, resulting in the binary mask of the motorcycle tire-to-ground contact area (referred to as the "contact area mask").
[0040] The vehicle-to-ground contact point detection model was trained using a YOLOv8 neural network. The YOLOv8 neural network model uses a similar network structure to YOLOv5, but with improvements by adding a keypoint detection branch to detect the contact points between the motorcycle tire and the ground. The main network structure of the keypoint detection branch is a heatmap regression branch, which consists of a 5-layer convolutional network. Figure 4 The image shows a comparison between coordinate point regression and heatmap regression. Compared to coordinate point regression, heatmap regression has lower computational complexity and higher accuracy. Furthermore, in addition to adding heatmap detection for key points, the loss function of the YOLOv8 neural network model was modified.
[0041] Wherein: the loss function of the keypoint detection branch:
[0042]
[0043] In the formula, L KThe loss function is defined by keypoints, where α and β are 2 and 4 respectively, and N represents the number of keypoints in the image (number of positive samples), used for standardization. Y is the predicted value, and Y is the labeled true value.
[0044] The overall training loss is:
[0045] Loss=λ1L cls +λ2L obj +λ3L loc +λ4L K
[0046] In the formula, Loss represents the total loss; λ1,λ 2, λ 3, λ4 represents the balance coefficient, with values of 0.3, 0.3, 0.2, and 0.2, which can be adjusted according to the actual situation; L cls L represents classification loss; obj Indicates the target loss; L loc L represents the localization loss. K
[0047] This represents a loss at a critical point.
[0048] The training process of the vehicle-to-ground contact point detection model mainly includes: constructing an initial YOLOV8 neural network model, training the YOLOV8 neural network model with training data, and obtaining the trained YOLOV8 neural network model after the total loss function value Loss of the neural network tends to converge and stabilize.
[0049] The training process of the YOLOv8 neural network model is as follows:
[0050] Step b1: Obtain the features of the image array obtained in Step 1;
[0051] Step b2: Use a labeling tool (e.g., labelme) to mark and outline the detection box for the motorcycle's position and the contact point between the motorcycle tire and the ground, and record the position information of the detection box and the contact point to form the corresponding label, which together with the image array features constitutes the training data;
[0052] Step b3: Construct the initial YOLOv8 neural network model. Input the training data into the YOLOv8 neural network model and continuously backpropagate and iterate the data until the total loss of the model tends to converge, thus obtaining the trained YOLOv8 neural network model, which is the vehicle-to-ground contact point detection model.
[0053] Step 4: Based on the helmet location information detected in Step 2 and the motorcycle location information detected in Step 3, determine whether the examinee is wearing a helmet.
[0054] Based on the helmet position information obtained in step 2 and the motorcycle position information obtained in step 3, especially when multiple helmets are present, the center point position of each helmet is calculated. If the center point of the helmet is within the motorcycle detection frame, the examinee is deemed to be wearing a helmet and can proceed with the test; otherwise, they are deemed to have failed. The position of the helmet center point can be calculated based on the coordinates of the upper left and lower right corners of the helmet detection frame, or the coordinates of the upper left corner of the helmet detection frame along with the length and width of the rectangle.
[0055] Step 5: If it is determined that the examinee is wearing a helmet, use the target region segmentation network model to segment the image array features into a binary mask of the examination stakes.
[0056] Step 5 specifically includes the following sub-steps:
[0057] Step 5.1: Obtain the image array features obtained in Step 1, filter the image array features containing the test stake, and resize the image array features to a size of (512, 512).
[0058] Step 5.2: Input the features of the resized image array into the target region segmentation network model to obtain the segmented binarized mask (referred to as "mask").
[0059] Step 5.3: Further, the binarized mask of the test stake can be visualized to obtain the visualization result of the binarized mask of the test stake, such as... Figure 4 The red test stake shown on the right is the visualization result of the test stake binarization mask.
[0060] The target region segmentation network model mainly consists of a backbone network model and an attention mechanism network layer. The backbone network model comprises 18 convolutional layers, 18 activation layers, 1 pooling layer, and 1 fully connected layer. The attention mechanism network layer encodes and reweights contextual information by encoding rich spatial and detailed information, thus fusing feature information.
[0061] The training process of the target region segmentation network model is as follows:
[0062] Step c1: Obtain the image array features obtained in Step 1, and select the image array features containing test piles under different lighting and weather scenarios.
[0063] Step c2: Use a labeling tool (e.g., LabelMe) to label the polygonal regions of the test stakes in the image array features, and use OpenCV to convert the polygonal regions into labels for training the test stake mask, which together with the image array features form the training data;
[0064] Step c3: Input the training data into the initially constructed region segmentation network model for training. Through continuous backpropagation and data iteration, the total loss function value of the model tends to converge, and the target region segmentation network model is obtained.
[0065] The total loss includes pixel category loss (also known as "primary loss") and auxiliary loss, both of which are ultimately output by the softmax loss function, showing the loss value for each sample pixel in the training set.
[0066]
[0067] In the formula, loss represents the total loss, N represents the number of samples, and i and j both represent samples, i = 1, 2, ..., N, j = 1, 2, ..., N, L i denoted by , represents the loss of the softmax loss function at each sample pixel, and p represents the network's predicted output.
[0068]
[0069] In the formula, L represents the sample loss, α is used to balance the primary and secondary losses, and takes a value between 0 and 1. p Indicates pixel category loss, l i Let X represent the auxiliary loss, W represent the pixel predicted by the network, i represent the i-th pixel, i = 1, 2, ..., K, and K represent the number of pixels.
[0070] Step 6: Based on the test cone mask obtained in Step 5 and the contact area mask obtained in Step 3, determine whether the motorcycle hits the test cone during the cone maneuver. If it hits the test cone, it is considered a failure in the test.
[0071] Step 6.1: Calculate the mask iou of the contact area mask and the test pile mask obtained in Step 3. The formula for calculating mask iou is as follows:
[0072] mask iou=(mask_point∩maks_gan) / (mask_point∪mask_gan)
[0073] In the formula, mask_poin represents the contact area mask, and mask_gan represents the test pile mask.
[0074] Step 6.2: Compare mask iou with a preset value Q (for example, the preset value is set to 0.1). If mask iou is greater than Q, it is considered that the vehicle tire and the test cone have collided.
[0075] Therefore, it can be seen that the present invention can automatically and clearly determine whether a collision with the test cones has occurred during the motorcycle test.
[0076] Furthermore, this embodiment of the invention also discloses an electronic device, which mainly includes a processor and a memory. The processor is mainly used to call the computer program stored in the memory and execute the machine vision-based motorcycle test evaluation method provided in the above embodiments.
[0077] Furthermore, embodiments of the present invention also provide a computer-readable storage medium for storing program code, which is used to execute the machine vision-based motorcycle examination evaluation method provided by the present invention.
[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the program code in the electronic device and computer-readable storage medium described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0079] Finally, it should be noted that although the embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and guiding, and not restrictive. Those skilled in the art, guided by this specification, can make many other forms without departing from the scope of protection of the claims of the present invention, and these all fall within the scope of protection of the present invention.
Claims
1. A method for judging a motorcycle test based on machine vision, characterized in that, Includes the following steps: The real-time video stream of the motorcycle test site is acquired and converted into an image array feature. The real-time video stream contains relevant information about the examinee, motorcycle, helmet and test stakes in the motorcycle test site. The image array features are input into the helmet detection model to obtain the helmet's location information; The image array features are input into the vehicle-ground contact point detection model to obtain the motorcycle's location information and the location information of multiple motorcycle tire contact points with the ground; the motorcycle tire contact points with the ground are connected sequentially to form a polygonal region and processed to obtain the contact area mask. Based on the helmet location information and motorcycle location information, it is determined whether the examinee is wearing a helmet; If it is determined that the examinee is wearing a helmet, the features of the image array are input into the target region segmentation network model to obtain the examinee mask; Based on the contact area mask and the test cone mask, it is determined whether the motorcycle hits the test cone during the cone-driving process. If it hits the test cone, it is determined that the test is failed. The helmet detection model is trained using a YOLOv5 neural network model. The YOLOv5 neural network model consists of a CSP+Focus+SPP structure, where: CSP includes a residual structure of convolutional layers, BN layers, and activation layers; Focus includes four slicing layers for fusing multiple scales of the input image; and SPP includes three max pooling layers for outputting feature maps at three scales. The vehicle-to-ground contact point detection model uses a YOLOv8 neural network model. The YOLOv8 neural network model adds a key point detection branch to the YOLOv5 neural network model to detect the contact point between the motorcycle tire and the ground. The key point detection branch is a heatmap regression branch, which includes a 5-layer convolutional network. The loss function calculation formula for the keypoint detection branch is as follows: In the formula, L K For keypoint loss, α and β are 2 and 4, respectively. N This represents the number of keypoints in the image used for normalization. Y is the predicted value, and Y is the labeled true value.
2. The motorcycle test judging method of claim 1, wherein The real-time video stream is converted into an image array feature using OpenCV; and / or, the polygonal region is processed using OpenCV to obtain a contact area mask.
3. The motorcycle test judging method of claim 1, wherein The loss function of the YOLOv5 neural network model consists of three parts: classification loss L cls The BCE loss is used to calculate the classification loss for positive samples, where positive samples represent detection boxes with an IOU greater than a first preset threshold. The classification loss is divided into two categories: presence or absence of a helmet; the target loss L... obj The BCE loss is used to calculate the IOU between the detection boxes predicted by the neural network model and the ground truth detection boxes; localization loss. L loc The localization loss for positive samples is calculated using IOU loss. Define the total loss function as: Loss = λ1L cls + λ2L obj + λ3L loc In the formula, Loss Represents the total loss; λ1,λ 2, λ3 represents the balance coefficient; L cls L represents classification loss; obj Indicates target loss; L loc Indicates positioning loss; The detection frame is a rectangular detection frame used to define the detection object. The position of the rectangular detection frame is defined by any of the following methods: 1) the coordinates of the upper left and lower right corners of the rectangular detection frame; 2) the coordinates of the upper left corner of the rectangular detection frame and the length and width of the rectangle.
4. The motorcycle testing and evaluation method as described in claim 1 or 3, characterized in that, The vehicle-ground contact point detection model overall total loss calculation formula is: Loss = λ1L cls + λ2L obj + λ3L loc + λ4L K In the formula, Loss Represents the total loss; λ1,λ 2, λ 3, λ 4 Represented as the balance coefficient; L cls L represents classification loss; obj Indicates target loss; L loc Indicates location loss. L K This represents a loss at a critical point.
5. The motorcycle test judging method of claim 1 wherein, Based on the helmet location information and motorcycle location information, determining whether the examinee is wearing a helmet specifically includes: Calculate the center point position of each helmet based on the helmet position information; Determine whether the center point of the helmet is within the rectangular detection frame used to define the position of the motorcycle: if it is, the candidate is wearing a helmet; otherwise, the candidate is deemed unqualified.
6. The motorcycle test judging method of claim 1 wherein, If it is determined that the examinee is wearing a helmet, the features of the image array are input into the target region segmentation network model to obtain the examination marker mask; specifically including: Obtain the image array features, filter the image array features containing the test stakes, and resize the image array features to the target size; The features of the resized image array are input into the target region segmentation network model to obtain the segmented binarized mask, also known as the peg mask.
7. The motorcycle test judging method according to claim 1 or 6, characterized in that, The target region segmentation network model includes a backbone network model and an attention mechanism network layer. The backbone network model has 18 convolutional layers, 18 activation layers, 1 pooling layer, and 1 fully connected layer. The attention mechanism network layer encodes and reweights the context information by encoding rich spatial and detailed information, and fuses the feature information. The total loss during the training process of the target region segmentation network model includes pixel category loss and auxiliary loss, which are ultimately output by the softmax loss function, showing the loss value of each sample pixel in the training set. In the formula, loss Indicates the total loss. N Indicates the number of samples. i and j All represent samples. i =1,2,……, N , j =1,2,……, N , L i This represents the loss of the softmax loss function at each sample pixel. p This represents the network's predicted output; In the formula, L Indicates the loss of the sample. Used to balance the primary and secondary losses, it takes a value between 0 and 1. l p Indicates pixel category loss, l i Indicates auxiliary loss, X This represents the pixel predicted by the network. W Represents the actual label pixels. i Indicates the first i 1 pixel i =1,2,……, K , K This indicates the number of pixels.
8. The motorcycle test judging method of claim 1 wherein, Based on the contact area mask and the cone mask, it is determined whether the motorcycle touched the cone during the cone maneuver, specifically including: The mask iou of the two masks is calculated based on the contact area mask and the test pile mask. The formula for calculating the mask iou is as follows: In the formula, mask_point represents the contact area mask, and mask_gan represents the contact mask. The mask iou is compared with the preset value Q. If the mask iou is greater than Q, it is considered that the car tire and the test cone have collided.
9. An electronic device, comprising: It includes a processor and a memory, wherein the processor is used to call a computer program stored in the memory to execute the machine vision-based motorcycle examination and evaluation method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the machine vision-based motorcycle examination and evaluation method according to any one of claims 1 to 8.