A road guardrail collision detection method based on YOLOv5-GRU

Abstract

The present invention relates to a road guardrail collision detection method based on YOLOv5‑GRU. From a visual perspective, the real-time video images of the road monitoring camera are analyzed, and combined with a video behavior recognition algorithm based on deep learning, it is determined whether a motor vehicle collides with the road guardrail, and an alarm signal is promptly sent to the relevant departments. The videos and photos before and after the vehicle collides with the guardrail are automatically captured to provide a basis for traffic accident handling, and the on-site collision photos, videos, time, and location are sent to the system to achieve 24-hour all-weather unmanned monitoring, alarm, and evidence.

Description

Technical Field

[0001] The present invention relates to the field of road guardrail collision, and in particular to a road guardrail collision detection method based on YOLOv5-GRU. Background Art

[0002] With the improvement of economy and society, the traffic volume of domestic urban highways is increasing day by day. Drivers are very likely to lose control and hit the guardrails due to fatigue, speeding or emergencies. Traffic accidents are inevitable. Highways are also damaged to varying degrees, and road property and road rights are affected. However, the collision point and time are uncertain. Road maintenance personnel alone often cannot understand the situation and repair it in the first time. Therefore, real-time monitoring of guardrails can be carried out. When a collision occurs, it can timely alarm and display the corresponding interval position, which helps traffic police rescue trapped people and vehicles in the first time and prevent the further expansion of accidents. In addition, this technology is conducive to 24-hour unmanned inspections on accident-prone sections. At present, most of the collision detection methods for guardrails are based on a combination of pressure sensors, acceleration sensors, optical fibers, etc., but this method is costly, requires multiple equipment, is complex to install, and is difficult to construct. In addition, it can only realize a single alarm function, and cannot realize alarm and evidence at the same time. In addition, the traditional method of using pressure sensors for detection can only detect that the guardrail has collided, but cannot ensure that it is a collision between the motor vehicle and the guardrail, resulting in a wrong judgment. It is also impossible to quickly extract the video of the vehicle colliding with the guardrail, and it is necessary to check the monitoring one by one, resulting in a waste of police force. Therefore, there is a need for a road guardrail collision detection method based on YOLOv5-GRU that can achieve 24-hour all-weather monitoring, anti-interference, sensitive perception, accurate alarm, save human resources, reduce the frequency of manual inspections, and reduce the loss of road property caused by malicious vehicle escape. Summary of the invention

[0003] The problem to be solved by the present invention is to provide a road guardrail collision detection method based on YOLOv5-GRU, which can realize 24-hour all-weather monitoring, anti-interference, sensitive perception, accurate alarm, save human resources, reduce the frequency of manual inspections, and reduce road property losses caused by malicious vehicle escapes.

[0004] In order to ensure stable operation and high working accuracy during use, the present invention relates to a road guardrail collision detection method based on YOLOv5-GRU, comprising the following steps:

[0005] S1: Set the warning area. Determine the road guardrail boundary in the video image of the smart ball camera and mark it as the warning area. Detect passing motor vehicles. When setting the warning area, it is also necessary to install the smart ball camera and terminal server in the area to be detected, adjust the camera installation angle and height, and set the camera resolution, frame rate, exposure rate, and intrusion detection alarm function;

[0006] S2: Record the coordinates of the center point of the alarm vehicle and the original video image. When a motor vehicle is found to enter the warning area, the camera intrusion detection alarm function is automatically triggered. The camera starts to capture the video image at the trigger time t, records the video image and returns the coordinates P (x, y) of the center point of the alarm vehicle frame at the trigger time, as well as the video recording of 6 seconds before and after the trigger time;

[0007] S3: Extract the alarm position time series image based on the vehicle center point coordinates and original video images recorded by the smart ball camera;

[0008] S4: Input the extracted alarm location time series images frame by frame into the convolutional neural network for feature extraction;

[0009] S5: Input the feature vector sequence X extracted in step S4 into the bidirectional GRU network layer to learn the temporal features of the feature vector sequence. First, the weight coefficients of the Bi-GRU output feature vectors at different times are calculated. The weight coefficients are weighted and summed for the Bi-GRU output feature vectors at different times to obtain the feature y of the video sequence. Finally, the probability Pre of a motor vehicle colliding with a guardrail in the video sequence is predicted by the Softmax classifier.

[0010] S6: Target vehicle feature extraction and collision behavior alarm. When Pre>=0.8, it indicates that the target vehicle has collided with the road guardrail. When 0.8>pre>=0.5, it indicates that the target vehicle is suspected of colliding with the road guardrail and further manual judgment is required. When Pre<0.5, it indicates that the target vehicle has not collided with the road guardrail.

[0011] The beneficial effect of the present invention is that, from a visual perspective, the real-time video images of the road monitoring camera are analyzed, combined with a video behavior recognition algorithm based on deep learning, to determine whether a motor vehicle has collided with a road guardrail, and to send an alarm signal to the relevant departments in a timely manner. The videos and photos of the vehicles before and after the collision with the guardrail are automatically captured to provide a basis for handling traffic accidents, and the photos, videos, time and location of the on-site collision are sent to the system, realizing 24-hour unmanned monitoring, alarm and evidence.

[0012] Furthermore, step S3 also includes the following:

[0013] B1: The terminal server receives the video and the target vehicle center point coordinates P (x, y) sent back by the smart ball camera;

[0014] B2: Using the coordinate P in step B1 as the center point, capture a square image of a fixed size frame by frame.

[0015] Furthermore, the step S4 also includes the following steps:

[0016] C1: Input the alarm position time series image Z extracted in step S3 into the convolutional neural network frame by frame to extract features V;

[0017] C2: After extracting V, V needs to be subjected to dimensionality reduction and full connection operations to convert the feature vector sequence V into a one-dimensional feature vector sequence X. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 The processing flow of a road guardrail collision detection method based on YOLOv5-GRU of the present invention is shown in the figure;

[0020] Figure 2 It is a feature extraction diagram of a road guardrail collision detection method based on YOLOv5-GRU of the present invention;

[0021] Figure 3 It is a collision simulation diagram of a road guardrail collision detection method based on YOLOv5-GRU of the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described in detail below in conjunction with specific embodiments:

[0023] The problem to be solved by the present invention is to provide a road guardrail collision detection method based on YOLOv5-GRU, which can realize 24-hour all-weather monitoring, anti-interference, sensitive perception, accurate alarm, save human resources, reduce the frequency of manual inspections, and reduce road property losses caused by malicious vehicle escapes.

[0024] like Figure 1 As shown, in order to ensure stable operation and high working accuracy during use, the present invention relates to a road guardrail collision detection method based on YOLOv5-GRU, comprising the following steps:

[0025] S1: Set the warning area. Determine the road guardrail boundary in the video image of the smart ball camera and mark it as the warning area. Detect passing motor vehicles. When setting the warning area, it is also necessary to install the smart ball camera and terminal server in the area to be detected, adjust the camera installation angle and height, and set the camera resolution, frame rate, exposure rate, and intrusion detection alarm function;

[0026] S2: Record the coordinates of the center point of the alarm vehicle and the original video image. When a motor vehicle is found to enter the warning area, the camera intrusion detection alarm function is automatically triggered. The camera starts to capture the video image at the trigger time t, records the video image and returns the coordinates P (x, y) of the center point of the alarm vehicle frame at the trigger time, as well as the video recording of 6 seconds before and after the trigger time;

[0027] S3: Extract the alarm position time series image based on the vehicle center point coordinates and original video images recorded by the smart ball camera;

[0028] S4: Input the extracted alarm location time series images frame by frame into the convolutional neural network for feature extraction;

[0029] S5: Input the feature vector sequence X extracted in step S4 into the bidirectional GRU network layer to learn the temporal features of the feature vector sequence. First, the weight coefficients of the Bi-GRU output feature vectors at different times are calculated. The weight coefficients are weighted and summed for the Bi-GRU output feature vectors at different times to obtain the feature y of the video sequence. Finally, the probability Pre of a motor vehicle colliding with a guardrail in the video sequence is predicted by the Softmax classifier.

[0030] S6: Target vehicle feature extraction and collision behavior alarm. When Pre>=0.8, it indicates that the target vehicle has collided with the road guardrail. When 0.8>pre>=0.5, it indicates that the target vehicle is suspected of colliding with the road guardrail and further manual judgment is required. When Pre<0.5, it indicates that the target vehicle has not collided with the road guardrail.

[0031] The beneficial effect of the present invention is that, from a visual perspective, the real-time video images of the road monitoring camera are analyzed, combined with a video behavior recognition algorithm based on deep learning, to determine whether a motor vehicle has collided with a road guardrail, and to send an alarm signal to the relevant departments in a timely manner. The videos and photos of the vehicles before and after the collision with the guardrail are automatically captured to provide a basis for handling traffic accidents, and the photos, videos, time and location of the on-site collision are sent to the system, realizing 24-hour unmanned monitoring, alarm and evidence.

[0032] In actual operation, the road guardrail boundary is determined in the video image of the smart ball camera and marked as a warning area, and passing motor vehicles are detected for crossing the boundary. The purpose is to conduct preliminary screening of suspected motor vehicle collisions with guardrails and reduce the computing power requirements of the terminal server for real-time monitoring. At the same time, the video resolution of the smart ball camera is set to 1920*1080 (default, which can be modified according to the ratio of 16:9).

[0033] In actual operation, the motor vehicle guardrail collision detection area is determined in the video image and marked as a warning area, including two lanes in the same direction (both the forward and reverse lanes can be monitored) to eliminate unnecessary interference.

[0034] Furthermore, step S3 also includes the following:

[0035] B1: The terminal server receives the video and the target vehicle center point coordinates P (x, y) sent back by the smart ball camera;

[0036] B2: Using the coordinate P in step B1 as the center point, capture square images of a fixed size (here 640*640) frame by frame.

[0037] In actual operation, the specific calculation method is as follows:

[0038] xi=Max(x,0),xi=Min(1920-xi,xi);

[0039] yi=Max(y,0),yi=Min(1080-yi,yi);

[0040] 1920 is the image width and 1080 is the image height. So far, the time series image of the alarm position can be constructed as follows:

[0041] Z=[z1,z2,…,zt],zt∈R640*640*3,

[0042] zt represents a video image after being intercepted;

[0043] t represents the sequence length;

[0044] 640*640*3 represents the image width, height and number of channels;

[0045] Z represents a video sequence

[0046] Furthermore, the step S4 also includes the following steps:

[0047] C1: Input the alarm position time series image Z extracted in step S3 into the convolutional neural network frame by frame to extract features V;

[0048] C2: After extracting V, V needs to be subjected to dimensionality reduction and full connection operations to convert the feature vector sequence V into a one-dimensional feature vector sequence X.

[0049] In actual operation, the present invention uses the YOLOv5 backbone network (composed of an improved YOLOv5 backbone network, including 1 Focus layer, 6 convolutional layers and 6 downsampling layers) with stronger feature extraction capability, fewer parameters and faster operation speed for feature extraction, such as Figure 2 As shown. After a series of convolution and downsampling operations, a feature vector sequence with a minimum scale of 5*5 can be obtained:

[0050] V=[v1,v2,…,vt],vt∈R5*5*1024,

[0051] vt=YOLOv5(zt)∈R5*5*1024,

[0052] Where YOLOv5 represents the feature extraction operation.

[0053] After extracting V, we need to perform dimension reduction and full connection operations on V to convert the feature vector sequence V into a one-dimensional feature vector sequence X, that is:

[0054] X=[x1,x2,…,xt],vt∈R1*1*1024,

[0055] xt=FC(flatten(vt)),

[0056] Among them, FC represents the fully connected operation;

[0057] Flatten represents a one-dimensional operation.

[0058] Furthermore, the step S5 also includes the following steps:

[0059] D1: Calculate the weight coefficient of the Bi-GRU output feature vector at different times;

[0060] D2: Use the weight coefficients in D1 to perform weighted summation on the Bi-GRU output feature vectors at different times to obtain the feature y of the video sequence;

[0061] D3: Finally, the Softmax classifier is used to predict the probability Pre of a motor vehicle colliding with a guardrail in the video sequence.

[0062] In actual operation, the specific update formula is as follows:

[0063] Input: x input =concat[h t -1,x t ],

[0064] Reset gate neuron: r t =σ(x input *W r +b r ),

[0065] Input gate neuron: I t =σ(x input *W z +b z ),

[0066] Memory gate neurons:

[0067] Forget gate neuron: f t =1-I t ,

[0068] Output:

[0069] Where σ refers to the sigmoid function, ⊙ represents the dot product of vector elements, and the above formula can also be abbreviated as:

[0070] Output: h t =GRU(h t -1,x t ),

[0071] Compared with LSTM, GRU has a simpler structure, fewer training samples, and is easier to apply in practical engineering. In addition, the present invention uses a bidirectional GRU network to fully combine the information before and after the collision for reasoning in the detection of motor vehicle collision guardrail events. Compared with the unidirectional GRU, the prediction is more accurate, namely:

[0072] Output:

[0073] In order to make the model focus on the fragments in the feature sequence that are key to the motor vehicle collision guardrail and reduce the impact of irrelevant fragments on the classification performance, the present invention adds an attention layer after the Bi-GRU network. The specific calculation formula is as follows:

[0074] s i =tanh (W o *h i +b o )

[0075]

[0076] Where S i As the attention layer, the weight coefficient of the Bi-GRU output feature vector at different times is first calculated, and then the weight coefficient is used to perform weighted summation on the Bi-GRU output feature vector at different times to obtain the feature y of the video sequence. Finally, the Softmax classifier is used to predict the probability Pre of a motor vehicle colliding with a guardrail in the video sequence.

[0077] Furthermore, the step S6 further includes the following steps:

[0078] E1: When Pre>=0.8 (preset threshold), it indicates that the target vehicle has collided with the road guardrail;

[0079] E2: When 0.8>pre>=0.5 (preset threshold), it indicates that the target vehicle is suspected to have collided with the road guardrail, and further manual judgment is required;

[0080] E3: When Pre<0.5, it indicates that the target vehicle does not collide with the road guardrail.

[0081] like Figure 3 As shown in the figure, in actual operation, the terminal server extracts the target vehicle's license plate, model, body color and other information from the captured image at time t transmitted by the smart ball camera based on the prediction results of the model, and uploads the video frames captured at time t-2 and t+2, the captured image at time t and the corresponding video recording as the evidence chain of the target vehicle colliding with the guardrail to the background storage server, and sends a motor vehicle collision guardrail alarm signal to the emergency center.

[0082] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A road guardrail collision detection method based on YOLOv5-GRU, characterized in that: The following steps are involved: S1: Set the warning area. Determine the road guardrail boundary in the video image of the smart ball camera and mark it as the warning area. Detect passing motor vehicles. When setting the warning area, it is also necessary to install the smart ball camera and terminal server in the area to be detected, adjust the camera installation angle and height, and set the camera resolution, frame rate, exposure rate, and intrusion detection alarm function; S2: Record the coordinates of the center point of the alarm vehicle and the original video image. When a motor vehicle is found to enter the warning area, the camera intrusion detection alarm function is automatically triggered. The camera starts to capture the video image at the trigger time t, records the video image and returns the coordinates P (x, y) of the center point of the alarm vehicle frame at the trigger time, as well as the video recording of 6 seconds before and after the trigger time; S3: Extract the alarm position time series image based on the vehicle center point coordinates and original video images recorded by the smart ball camera; S4: Input the extracted alarm location time series images frame by frame into the convolutional neural network for feature extraction; S5: Input the feature vector sequence X extracted in step S4 into the bidirectional GRU network layer to learn the temporal features of the feature vector sequence. First, the weight coefficients of the Bi-GRU output feature vectors at different times are calculated. The weight coefficients are weighted and summed for the Bi-GRU output feature vectors at different times to obtain the feature y of the video sequence. Finally, the probability Pre of a motor vehicle colliding with a guardrail in the video sequence is predicted by the Softmax classifier. S6: Target vehicle feature extraction and collision behavior alarm. When Pre>=0.8, it indicates that the target vehicle has collided with the road guardrail. When 0.8>pre>=0.5, it indicates that the target vehicle is suspected of colliding with the road guardrail and further manual judgment is required. When Pre<0.5, it indicates that the target vehicle has not collided with the road guardrail.

2. The road guardrail collision detection method based on YOLOv5-GRU according to claim 1 is characterized in that The step S3 also includes the following contents: B1: The terminal server receives the video and the target vehicle center point coordinates P (x, y) sent back by the smart ball camera; B2: Using the coordinate P in step B1 as the center point, capture a square image of a fixed size frame by frame.

3. The road guardrail collision detection method based on YOLOv5-GRU according to claim 1 is characterized in that The step S4 also includes the following steps: C1: Input the alarm position time series image Z extracted in step S3 into the convolutional neural network frame by frame to extract features V; C2: After extracting V, V needs to be subjected to dimensionality reduction and full connection operations to convert the feature vector sequence V into a one-dimensional feature vector sequence X.

Images