Motorcycle test judgment method based on machine vision and electronic equipment

Abstract

The invention discloses a motorcycle test judgment method based on machine vision. The method comprises the steps of obtaining a real-time video stream of a motorcycle test room and converting the real-time video stream into picture array features; inputting the picture array features into a helmet detection model to obtain position information of the helmet; inputting the picture array features into a motorcycle-ground contact point detection model to obtain position information of the motorcycle and position information of a plurality of motorcycle tires and ground contact points; sequentially connecting the contact points to form a polygonal area, and processing to obtain a contact area mask; judging whether the examinee wears the helmet or not based on the helmet position information and the motorcycle position information; if it is judged that the image is worn, the image array features are input into a target area segmentation network model, and a test pile mask is obtained; based on the contact area mask and the test pile mask, whether the motorcycle touches the test pile in the pile winding process or not is judged, and if yes, the test is judged to be unqualified. According to the invention, the whole-course intelligent judgment can be realized, the judgment is more objective and accurate, and the hardware cost can be saved.

Description

Technical Field

[0001] The invention belongs to the technical field of motorcycle driving training, and in particular relates to a motorcycle test evaluation method and electronic equipment based on machine vision. Background Art

[0002] In the motorcycle driving test, ensuring that the examinee wears a helmet and operates the motorcycle correctly is an important part of ensuring safety. Therefore, the motorcycle driving test process needs to judge both the wearing of the helmet and the operation of the motorcycle around the pole. At present, these judgments mainly rely on manual monitoring or sensor detection. Among them, manual judgment has the problems of low efficiency and insufficient accuracy. On the one hand, judging the pole collision based on sensors is costly. On the other hand, the sensors are easily affected by environmental factors, such as wind and rain, which may cause damage to the sensors, resulting in a high false alarm rate. With the development of computer vision technology, the application of deep learning models in image recognition and target detection has gradually become popular, which can provide new solutions for automated detection. Summary of the invention

[0003] Based on the problems of low efficiency and insufficient accuracy of manual evaluation and high false alarm rate and high cost of sensor evaluation in the existing motorcycle test evaluation process, the present invention provides a motorcycle test evaluation method, electronic device and computer-readable storage medium based on machine vision.

[0004] The first aspect of the present invention discloses a motorcycle test evaluation method based on machine vision, comprising the following steps:

[0005] Obtain a real-time video stream of a motorcycle test site, and convert the real-time video stream into a picture array feature, wherein the real-time video stream contains relevant information about the examinee, motorcycle, helmet, and test pole in the motorcycle test site;

[0006] Inputting the image array features into a helmet detection model to obtain the position information of the helmet;

[0007] Input the image array features into a vehicle-ground contact point detection model to obtain the position information of the motorcycle and the position information of the contact points between the motorcycle tires and the ground; sequentially connect the contact points between the motorcycle tires and the ground to form a polygonal area and obtain a contact area mask through opencv processing;

[0008] Based on the helmet position information and the motorcycle position information, determining whether the examinee is wearing a helmet;

[0009] If it is determined that the examinee is wearing a helmet, the image array features are input into the target area segmentation network model to obtain a test pile mask;

[0010] Based on the contact area mask and the test pole mask, it is determined whether the motorcycle hits the test pole during the process of circling the pole. If the motorcycle hits the test pole, it is determined that the test is failed.

[0011] A second aspect of the present invention discloses an electronic device comprising a processor and a memory, wherein the processor is used to call a computer program stored in the memory to execute the motorcycle test evaluation method based on machine vision described in the first aspect of the present invention and any optional scheme.

[0012] The third aspect of the present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the machine vision-based motorcycle test evaluation method described in the first aspect of the present invention and any optional scheme.

[0013] Beneficial effects:

[0014] The machine vision-based motorcycle test evaluation method and electronic equipment provided by the present invention combine machine vision and deep learning algorithms to identify the test process, determine whether the examinee wears the helmet correctly and whether the pole collision occurs, so as to realize intelligent evaluation of the motorcycle test process.

[0015] Compared with traditional manual monitoring or sensor detection, the present invention can achieve full-process intelligent evaluation, and the evaluation is more objective and accurate, and can also save hardware costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram showing the position of the examinee's helmet detected by the helmet detection model;

[0017] Figure 2 Middle: (a) is a schematic diagram of the coordinate regression network, (b) is a schematic diagram of the heat map regression network;

[0018] Figure 3 Schematic diagram of the contact point between motorcycle tire and ground detected by the vehicle-ground contact point detection model;

[0019] Figure 4 Schematic diagram of the test pile locations segmented by the target area segmentation network model;

[0020] Figure 5 A diagram of the motorcycle test process for correctly completing the pole wrap; DETAILED DESCRIPTION

[0021] In order to further understand the present invention, the technical solution of the present invention is further explained below in conjunction with specific embodiments and drawings.

[0022] The embodiment of the present invention discloses a motorcycle test evaluation method based on machine vision, which mainly includes the following steps:

[0023] Step 1: Obtain a real-time video stream of the motorcycle test site through the camera in the test site. The video stream contains relevant information such as the test taker, motorcycle, helmet, and test pole in the motorcycle test site. Use OpenCV to convert the real-time video stream into image array features as the input of the subsequent model.

[0024] Step 2: Use the helmet detection model to detect the image array features generated in step 1, detect whether there is a helmet in the examination room, and obtain the helmet location information.

[0025] The motorcycle test evaluation must first determine whether the examinee is wearing a helmet correctly. Only when the examinee carries a helmet and wears it correctly can the subsequent motorcycle operation test and evaluation be carried out. The image array features output from step 1 are input into the trained helmet detection model to detect whether there is a helmet in the examination room. If there is, the location information of the helmet is output. If there is no helmet, the examinee is directly judged as failing the test.

[0026] The helmet detection model can be obtained by training the YOLOv5 neural network model. The YOLOv5 neural network model is composed of a CSP+Focus+SPP structure, where: CSP contains a residual structure of a convolutional layer, a BN layer, and an activation layer; Focus contains four slice layers, which mainly fuse the multi-scale of the input image, where multi-scale means that the network will cluster anchors of different scales as sliding boxes during training; SPP contains three maximum pooling layers, which output features of different scales, mainly including feature maps of scales 80*80, 40*40, and 20*20.

[0027] The loss function of the YOLOv5 neural network model mainly consists of three parts, namely: classification loss (L cls ), BCE loss (binary cross entropy loss) is used to calculate the classification loss of positive samples, where the positive samples represent detection boxes whose IOU (intersection over union, i.e., confidence) is greater than a first preset threshold (e.g., 0.5), and the classification loss is mainly divided into two categories: with or without a helmet; the target loss (L obj ), BCE loss is used to calculate the IOU between the detection box predicted by the neural network model and the actual detection box; positioning loss (L loc ), IOU loss is used to calculate the positioning loss of positive samples. The total loss function is defined as:

[0028] Loss = λ1L cls +λ2L obj +λ3L loc

[0029] Where: Loss represents the total loss; λ1,λ 2,λ3 is the balance coefficient, and its values ​​are 0.4, 0.4, and 0.2 respectively, which can be adjusted according to the actual situation; L cls represents the classification loss; L obj represents the target loss; L loc represents the positioning loss.

[0030] The training process of the YOLOv5 neural network model mainly includes: building an initial YOLOV5 neural network model, training the YOLOV5 neural network model through training data, until the total loss function value Loss of the neural network tends to converge and stabilize, and then obtaining the trained YOLOV5 neural network model.

[0031] The specific training process of the YOLOV5 neural network model is as follows:

[0032] Step a1: Get the image array features obtained in step 1;

[0033] Step a2: Use a labeling tool (e.g., labelme) to mark and outline a rectangular detection box used to define the candidate's position, and record the position information, confidence, and classification of the rectangular detection box to form a label corresponding to the detection box, which together with the image array features constitutes training data;

[0034] Specifically, a rectangular detection frame can be used to frame the object (i.e., a helmet) in the image array feature. The position of the rectangular detection frame can generally be defined in two ways, representing the coordinates of the upper left corner and the lower right corner of the rectangular detection frame, or the coordinates of the upper left corner of the rectangular detection frame and the length and width of the rectangle. In addition to the face position information, each detection frame also has corresponding confidence and classification data. The classification data includes helmets and no helmets. Invalid detection frames with confidence lower than the first preset threshold are filtered out, and the remaining detection frames are valid detection frames.

[0035] Step a3: Input the training data into the initially constructed YOLOv5 neural network model for training. Through continuous back propagation and data iteration, the total loss of the model tends to converge, and the trained YOLOV5 neural network model, that is, the helmet detection model, is obtained.

[0036] Among them, the back-propagation process can perform gradient differentiation of the parameters of the samples in each training batch sampling (that is, the number of samples randomly extracted from each batch during the training process and the corresponding labels) in each convolutional layer through matrix multiplication, and then sum the derivative results, and perform weighted summation of the results of the previous layer and the next layer with the corresponding weights to obtain the final eigenvalue.

[0037] Step 3: Use the vehicle-ground contact point detection model to detect the image array features generated in step 1 to obtain the location information of the motorcycle and the location information of multiple motorcycle tire-ground contact points. Connect the motorcycle tire-ground contact points into a polygonal area and process it through OpenCV to convert it into a binary mask.

[0038] Based on the result obtained in step 2, if multiple helmets are detected in the examination room, the position of each helmet needs to be determined. The present invention uses the YOLOv8 neural network model to detect the position of the motorcycle and the contact point between the motorcycle tire and the ground (referred to as the "vehicle-ground contact point").

[0039] The YOLOv8 neural network model used in the present invention can not only detect the position of the motorcycle, but also detect the contact point between the motorcycle tire and the ground, so it is also called the "vehicle-ground contact point detection model". Figure 3 As shown, the model detects the position of the motorcycle and the contact area between the motorcycle tire and the ground (formed by multiple contact points between the motorcycle tire and the ground). The position of the motorcycle can also be defined by the rectangular detection frame described in step 2. There can be 6 vehicle-ground contact points, and the 6 vehicle-ground contact points are usually 6 points mapped to the ground with the tire as the center, that is, each vehicle-ground contact point is equidistant from the center point of the tire. These 6 vehicle-ground contact points are connected in sequence to generate a polygonal area, and the polygon is processed by opencv and converted into a binary mask, that is, a binary mask of the contact area between the motorcycle tire and the ground (referred to as "contact area mask") is obtained.

[0040] The vehicle-ground contact point detection model is trained using the YOLOv8 neural network model. The network structure used by the YOLOv8 neural network model is similar to that of YOLOv5. It is improved on the basis of YOLOv5 and adds a key point detection branch to detect the contact point between the motorcycle tire and the ground. The main network structure of the key point detection branch is the heat map regression branch, which includes a 5-layer convolutional network. Figure 4 As shown in the figure, it is a comparison chart between coordinate point regression and heat map regression. Compared with coordinate point regression, heat map regression has lower computational complexity and higher accuracy. Furthermore, in addition to adding heat map detection key points, the loss function of the YOLOv8 neural network model is also modified.

[0041] Among them: The loss function of the key point detection branch is:

[0042]

[0043] Where, L Kis the key point loss, α and β are 2 and 4 respectively, and N represents the number of key points in the image (the number of positive samples), which is used for normalization. is the predicted value, and Y is the true value.

[0044] The overall training loss is:

[0045] Loss = λ1L cls +λ2L obj +λ3L loc +λ4L K

[0046] Where Loss represents the total loss; λ1,λ 2, λ 3, λ4 is the balance coefficient, and its values ​​are 0.3, 0.3, 0.2, and 0.2 respectively, which can be adjusted according to the actual situation; L cls represents the classification loss; L obj represents the target loss; L loc represents the positioning loss, L K

[0047] is the key point loss.

[0048] The training process of the vehicle-ground contact point detection model mainly includes: constructing an initial YOLOV8 neural network model, training the YOLOV8 neural network model through training data, until the total loss function value Loss of the neural network tends to converge and stabilize, and then obtaining a trained YOLOV8 neural network model.

[0049] The training process of the YOLOV8 neural network model is as follows:

[0050] Step b1: Get the image array features obtained in step 1;

[0051] Step b2: Use a labeling tool (e.g., labelme) to mark and outline a detection frame for the motorcycle position and the contact points between the motorcycle tire and the ground, and record the position information of the detection frame and the contact points to form corresponding labels, which together with the image array features constitute training data;

[0052] Step b3: construct an initial YOLOv8 neural network model, input the training data into the YOLOv8 neural network model, and make the total loss Loss of the model converge through continuous back propagation and data iteration to obtain the trained YOLOv8 neural network model, that is, the vehicle-ground contact point detection model.

[0053] Step 4: Based on the helmet position information detected in step 2 and the motorcycle position information detected in step 3, determine whether the candidate is wearing a helmet.

[0054] Based on the helmet position information obtained in step 2 and the motorcycle position information obtained in step 3, especially when there are multiple helmets, by calculating the center point position of each helmet, if the center point of the helmet is in the motorcycle detection frame, it is judged that the candidate is wearing a helmet and can take the test, otherwise it is directly judged as unqualified. The position of the center point of the helmet can be calculated based on the coordinate information of the upper left corner and the lower right corner of the helmet detection frame, or the coordinates of the upper left corner of the helmet detection frame and the length and width data of the rectangle.

[0055] Step 5: If it is determined that the examinee is wearing a helmet, use the target region segmentation network model to segment the binary mask of the examinee’s features in the image array.

[0056] Step 5 specifically includes the following sub-steps:

[0057] Step 5.1: Get the image array features obtained in step 1, filter the image array features containing the test piles, and resize the image array features to a size of (512, 512).

[0058] Step 5.2: Input the resized image array features into the target region segmentation network model to obtain the segmented test pile binary mask (referred to as "test pile mask").

[0059] Step 5.3: Further, the test pile binary mask can be visualized to obtain the visualization result of the test pile binary mask, such as Figure 4 The red test pile shown on the right is the visualization result of the test pile binary mask.

[0060] The target region segmentation network model mainly includes: a backbone network model and an attention mechanism network layer. The parameters of the backbone network model include 18 convolutional layers, 18 activation layers, 1 pooling layer, and 1 fully connected layer. The attention mechanism network layer can encode and re-weight context information by encoding rich spatial information and detail information, and fuse feature information.

[0061] The training process of the target area segmentation network model is as follows:

[0062] Step c1: Obtain the image array features obtained in step 1, and select the image array features of the test piles under different lighting and weather scenes.

[0063] Step c2: Use a labeling tool (e.g., labelme) to label the polygonal area of ​​the test pile in the image array feature, and use OpenCV to convert the polygonal area into a label for test pile mask training, and the label and the image array feature form the training data;

[0064] Step c3: Input the training data into the initially constructed region segmentation network model for training. Through continuous back propagation and data iteration, until the total loss function value of the model tends to converge, the target region segmentation network model is obtained.

[0065] Among them, the total loss includes pixel category loss (also called "main loss") and auxiliary loss, which are finally unified by the output of the softmax loss function, outputting the loss of each sample pixel value in the training set:

[0066]

[0067] Where loss represents the total loss, N represents the number of samples, i and j represent samples, i = 1, 2, ..., N, j = 1, 2, ..., N, L i represents the loss of the softmax loss function at each sample pixel, and p represents the predicted output of the network.

[0068]

[0069] In the formula, L represents the loss of the sample, α is used to balance the main loss and auxiliary loss, and takes values ​​between 0 and 1. p represents the pixel category loss, l i represents the auxiliary loss, X represents the pixel predicted by the network, W represents the actual label pixel, i represents the i-th pixel, i = 1, 2, ..., K, K represents the number of pixels.

[0070] Step 6: Based on the test pole mask obtained in step 5 and the contact area mask obtained in step 3, determine whether the motorcycle hits the test pole during the process of circling the pole. If it hits the test pole, it is determined that the test is failed.

[0071] Step 6.1: Calculate the mask iou of the contact area mask and the test pile mask obtained in step 3. The calculation formula of mask iou is as follows:

[0072] mask iou=(mask_point∩maks_gan) / (mask_point∪mask_gan)

[0073] In the formula, mask_poin represents the contact area mask, and mask_gan represents the test pile mask.

[0074] Step 6.2: Compare mask iou with a preset value Q (for example, the preset value is set to 0.1). If mask iou is greater than Q, it is considered that the vehicle tire and the test pole have collided.

[0075] It can be seen that the present invention can automatically and clearly determine whether a collision with the test pole occurs during a motorcycle test.

[0076] Furthermore, an electronic device is disclosed in an embodiment of the present invention, which mainly includes a processor and a memory. The processor is mainly used to call a computer program stored in the memory and execute the motorcycle test evaluation method based on machine vision provided in the above embodiment.

[0077] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the motorcycle test evaluation method based on machine vision provided by the present invention.

[0078] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device and program code in the computer-readable storage medium described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0079] Finally, it should be noted that although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields, and the above-mentioned specific embodiments are only illustrative and instructive, rather than restrictive. Under the guidance of this specification, a person of ordinary skill in the art can also make many forms without departing from the scope of protection of the claims of the present invention, all of which belong to the scope of protection of the present invention.

Claims

1. A motorcycle test evaluation method based on machine vision, characterized in that: The following steps are involved: Obtain a real-time video stream of a motorcycle test site, and convert the real-time video stream into a picture array feature, wherein the real-time video stream contains relevant information about the examinee, motorcycle, helmet, and test pole in the motorcycle test site; Inputting the image array features into a helmet detection model to obtain the position information of the helmet; Inputting the image array features into a vehicle-ground contact point detection model to obtain the position information of the motorcycle and the position information of the contact points between the motorcycle tires and the ground; sequentially connecting the motorcycle tires and the ground contact points to form a polygonal area and obtaining a contact area mask through processing; Based on the helmet position information and the motorcycle position information, determining whether the examinee is wearing a helmet; If it is determined that the examinee is wearing a helmet, the image array features are input into the target area segmentation network model to obtain a test pile mask; Based on the contact area mask and the test pole mask, it is determined whether the motorcycle hits the test pole during the process of circling the pole. If the motorcycle hits the test pole, it is determined that the test is failed.

2. The motorcycle test judging method according to claim 1, characterized in that: The real-time video stream is converted into a picture array feature through OpenCV; and / or, the polygonal area is processed through OpenCV to obtain a contact area mask.

3. The motorcycle test judging method according to claim 1, characterized in that: The helmet detection model is obtained by training the YOLOv5 neural network model; the YOLOv5 neural network model is composed of a CSP+Focus+SPP structure, wherein: CSP includes a residual structure of a convolutional layer, a BN layer, and an activation layer; Focus includes four slice layers for fusing multiple scales of the input image; SPP includes three maximum pooling layers for outputting feature maps of three scales; The loss function of the YOLOv5 neural network model consists of three parts, namely: classification loss L cls , BCE loss is used to calculate the classification loss of positive samples, where positive samples represent detection boxes whose IOU is greater than the first preset threshold, and the classification loss is divided into two categories: with or without a helmet; the target loss L obj , BCE loss is used to calculate the IOU between the detection box predicted by the neural network model and the actual detection box; the positioning loss L loc , IOU loss is used to calculate the positioning loss of positive samples; Define the total loss function as: Loss = λ1L cls +λ2L obj +λ3L loc Where Loss represents the total loss; λ1,λ 2, λ3 is the balance coefficient; L cls represents the classification loss; L obj represents the target loss; L loc represents the positioning loss; The detection frame is a rectangular detection frame used to frame the detection object, and the position of the rectangular detection frame is defined by any of the following methods: 1) the coordinates of the upper left corner and the lower right corner of the rectangular detection frame; 2) the coordinates of the upper left corner of the rectangular detection frame and the length and width of the rectangle.

4. The motorcycle test judging method according to claim 1 or 3, characterized in that: The vehicle-ground contact point detection model adopts a YOLOv8 neural network model; the YOLOv8 neural network model adds a key point detection branch on the basis of the YOLOv5 neural network model, which is used to detect the contact point between the motorcycle tire and the ground; the structure of the key point detection branch is a heat map regression branch, and the heat map branch includes a 5-layer convolutional network; The loss function calculation formula of the key point detection branch is: Where, L K is the key point loss, α and β are 2 and 4 respectively, N represents the number of key points of the image used for normalization, is the predicted value, and Y is the labeled true value; The overall total loss calculation formula of the vehicle-ground contact point detection model is: Loss=λ1L cls +λ2L obj +λ3L loc +λ4L K Where Loss represents the total loss; λ1,λ 2, λ 3, λ4 is the balance coefficient; L cls represents the classification loss; L obj represents the target loss; L loc represents the positioning loss, L K is the key point loss.

5. The motorcycle test judging method according to claim 1, characterized in that: Based on the helmet position information and the motorcycle position information, determining whether the examinee is wearing a helmet includes: Calculating the center point position of each helmet based on the helmet position information; Determine whether the center point of the helmet is within the rectangular detection frame used to define the position of the motorcycle: if so, it means that the candidate is wearing a helmet; otherwise, it is determined to be unqualified.

6. The motorcycle test judging method according to claim 1, characterized in that: If it is determined that the examinee is wearing a helmet, the image array features are input into the target area segmentation network model to obtain the examinee mask; specifically including: Obtain the image array features, filter the image array features containing the test piles, and resize the image array features to the target size; The resized image array features are input into the target area segmentation network model to obtain the segmented test pile binary mask, also known as the test pile mask.

7. The motorcycle test judging method according to claim 1 or 6, characterized in that: The target region segmentation network model includes: a backbone network model and an attention mechanism network layer; wherein the parameters of the backbone network model include 18 convolutional layers, 18 activation layers, 1 pooling layer and 1 fully connected layer; the attention mechanism network layer encodes and re-weights the context information by encoding rich spatial information and detail information, and fuses feature information; The total loss of the target area segmentation network model training process includes pixel category loss and auxiliary loss, which are finally unified by the softmax loss function output, outputting the loss of each sample pixel value in the training set: Where loss represents the total loss, N represents the number of samples, i and j represent samples, i = 1, 2, ..., N, j = 1, 2, ..., N, L i represents the loss of the softmax loss function at each sample pixel, and p represents the predicted output of the network; In the formula, L represents the loss of the sample, α is used to balance the main loss and auxiliary loss, and takes values ​​between 0 and 1. p represents the pixel category loss, l i represents the auxiliary loss, X represents the pixel predicted by the network, W represents the actual label pixel, i represents the i-th pixel, i = 1, 2, ..., K, K represents the number of pixels.

8. The motorcycle test judging method according to claim 1, characterized in that: Based on the contact area mask and the test pole mask, judging whether the motorcycle hits the test pole during the slalom process specifically includes: The mask iou of the two masks is calculated based on the contact area mask and the test pile mask. The calculation formula of the mask iou is: mask iou=(mask_point∩maks_gan) / (mask_point∪mask_gan) In the formula, mask_poin represents the contact area mask, and mask_gan represents the test pile mask; Compare mask iou with the preset value Q. If mask iou is greater than Q, it is considered that the vehicle tire and the test pole have collided.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the processor is used to call a computer program stored in the memory to execute the motorcycle test judging method based on machine vision as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the machine vision-based motorcycle test evaluation method described in any one of claims 1 to 8.

Images