A road traffic safety hazard detection method based on instance segmentation network

Abstract

The present invention discloses a method for checking road traffic safety hazards based on an instance segmentation network, which belongs to the field of road traffic. The method comprises the following steps: recording driving videos with a camera, constructing an instance segmentation network and establishing a data set for training; importing the video into the instance segmentation network for instance segmentation; calculating relevant parameters based on the segmentation results; making risk judgments based on the segmentation results and the relevant parameters; and summarizing and outputting existing risk points. The present invention performs safety hazard checking on road videos, which can effectively reduce dependence on professionals and reduce work difficulty. In addition, deep learning technology is used to automatically identify safety hazards on the road, thereby improving the work efficiency of road hazard checking.

Description

Technical field:

[0001] The present invention belongs to the field of road traffic, and in particular relates to a method for detecting road traffic safety hazards based on an instance segmentation network. Background technology:

[0002] The rectification of road traffic safety hazards can improve the road driving environment, ensure the smoothness of the road to the greatest extent, and reduce the occurrence of traffic accidents. In recent years, large-scale road safety hazard inspections have been carried out across the country, which has put forward higher requirements for the efficiency and accuracy of hazard inspections.

[0003] At present, the inspection of road hazards mainly relies on manual inspection, and the professional ability of the inspectors is required to be high, which makes it difficult to form an inspection team in reality and also affects the accuracy and efficiency of the inspection results.

[0004] Therefore, it is necessary to design a method that can automatically analyze road safety hazards instead of manual work, reduce dependence on professionals, improve the efficiency of road hazard inspection, reduce costs, and reduce the incidence of traffic accidents. Summary of the invention:

[0005] To address the shortcomings of existing technologies, the present invention provides a road traffic safety hazard detection method based on an instance segmentation network. This method directly uses an instance segmentation network to read, identify, and classify images frame by frame. Based on the instance segmentation results, it calculates certain parameters of the current road and determines the risks associated with each road section in each frame of the video. This method, on the one hand, addresses the current reliance on specialized personnel in road hazard detection, reducing labor costs. On the other hand, it utilizes deep learning technology to automatically identify safety hazards on the road, significantly improving the efficiency and accuracy of road hazard detection.

[0006] The technical solutions of the present invention are as follows:

[0007] A method for detecting road traffic safety hazards based on an instance segmentation network includes the following steps:

[0008] (1) Install a monocular camera on a vehicle, drive the vehicle on a road where safety hazards need to be checked, and record driving video from the driver's perspective;

[0009] (2) Establish an instance segmentation network;

[0010] (3) Establish a comprehensive road scene dataset and loss function, initialize the instance segmentation network parameters and the loss function preset conditions, and train the instance segmentation network; when the loss function reaches the preset conditions, obtain the trained instance segmentation network;

[0011] (4) Perform instance segmentation on the vehicle driving video based on the trained instance segmentation network;

[0012] (5) Based on the instance segmentation results of the driving video, the curvature, slope, and driving sight distance of the current road are calculated, and the risk hazards existing on the current road are judged based on the instance segmentation results and the calculated parameters;

[0013] (6) Summarize and output the risk points on the road where the vehicle is traveling.

[0014] The instance segmentation network includes an FPN network layer, an RPN network layer, a RoIAlign layer, a category prediction branch, a bounding box prediction branch, and a mask segmentation branch;

[0015] The FPN network layer consists of a convolutional layer, an activation function and a pooling layer;

[0016] The RPN network layer consists of a convolutional layer and a Softmax regression function; the Softmax regression function is used to determine whether the generated rectangular box is a foreground point or a background point;

[0017] The RoIAlign layer is used to collect proposals generated by the RPN network and unify the input feature images into the same size;

[0018] The category prediction branch calculates the category to which each proposal belongs through a fully connected layer and softmax based on the proposed feature image generated by the RoIAlign layer;

[0019] The bounding box prediction branch calculates the position offset of each proposal and performs regression based on the proposed feature image generated by the RoIAlign layer to obtain a more accurate target detection frame;

[0020] The mask segmentation branch uses a fully convolutional neural network to segment each RoI, and the output dimension is K*m*m, where m is the size of the feature map of RoIAlign, that is, it outputs m*m masks of K categories and binarizes them with the sigmoid function to generate mask segmentation of background and foreground.

[0021] The comprehensive road scene dataset includes labeling vehicles, roads, lane lines, pedestrians, guardrails, lane lines, obstacles, and the surrounding environment of the road in the driving video to determine the labeling results, wherein the labeling results include the true value of the segmented image and the precise edge map corresponding to each frame of the driving video.

[0022] The loss function is a multi-task loss: L = L cls +L box +L mask , where L clsrepresents the classification loss of the prediction box, L box represents the regression loss of the prediction box, L mask represents the loss of the mask segmentation branch.

[0023] The existing risks and hidden dangers are judged as follows:

[0024] When the road environment is a river or lake, check whether there are guardrails on both sides of the lane. If not, the current road section is judged to be a risky section;

[0025] Determine whether there is a lane line on the current road. If not, the current road is considered a risky section.

[0026] Determine whether the curvature radius of the current road meets the design conditions. If not, the current road is considered a risky section.

[0027] Determine whether the current road slope meets the design conditions. If not, the current road is considered a risky section.

[0028] Determine whether there are any obstructions or obstruction areas on the current road. If so, determine that the current road is a risky section.

[0029] Based on the curvature of the current road, the specific steps to determine whether the current road is a risky section are as follows:

[0030] Mark the lane line pixel coordinate set {(x1, y1), (x2, y2), (x3, y3)...}, and use the least squares method to solve the sum of squares of the error The minimum value of the fitting function is obtained, and the lane curve equation x=Ay is calculated 2 +By+C, converted into the world coordinate equation is Where mx and my are the ratios of 1 meter in the real world to pixels in the horizontal and vertical directions respectively, then the road curvature radius is

[0031] When R<r, where r=125 meters for second-class highway, r=60 meters for third-class highway, and R=30 meters for fourth-class highway, the road curvature radius of this section does not meet the design conditions, and the current road is a risky section.

[0032] Based on the current road slope, the specific steps to determine whether the current road is a risky section are as follows:

[0033] Calculate the slope of the current road When the slope α is greater than I%, where I=6 for a second-class highway, I=7 for a third-class highway, and I=8 for a fourth-class highway, the road section is a steep slope section and a risky section;

[0034] When the road is downhill, the vehicle driving mileage is counted starting from when the vehicle enters the slope section until the road surface becomes a horizontal section and the counting stops. If the length of this section is greater than 3 km, it is determined whether the slope α is greater than I’%. For secondary roads, I’ = 4.5; for tertiary roads, I’ = 5; for quaternary roads, I’ = 5.5. If so, this section is a continuous downhill section and a risk section.

[0035] Where e′ is the projection point of the point at infinity on the slope road surface on the camera imaging plane, f′ is the projection point of the corner point of the slope road surface and the horizontal road surface on the camera imaging plane, y e′ 、y f′ are the vertical coordinates of e′ and f′ in the image coordinate system, and f is the camera focal length.

[0036] The specific steps to determine whether the current road is a risk section according to whether there is an occluder / area in the current frame are as follows:

[0037] Determine whether there is an occluder / area in the current frame. If there is, calculate the distance d of the vehicle from the obstacle / area. When d < L, where for secondary roads, L = 150 m; for tertiary roads, L = 80 m; for quaternary roads, L = 60 m, the current section is a section with poor visibility and a risk section, and the same obstacle / area will not be calculated in subsequent frames.

[0038] Where where h is the height of the camera from the ground and β is the camera pitch angle, where v p is the vertical coordinate of the target point in the pixel coordinate system, v0 is the vertical coordinate of the origin of the pixel coordinate system, and f is the camera focal length.

[0039] The advantages of the present invention are:

[0040] 1. Compared with the traditional manual inspection of road risks, the present invention directly uses a deep learning model based on an instance segmentation network to conduct safety hazard inspections on road videos. On the one hand, fewer staff are required, which can save a large amount of manpower and material resources and the cost is low; on the other hand, during the process of using this inspection method, the requirements for the professional knowledge and skills of the inspection personnel are reduced, and the work difficulty is reduced.

[0041] 2. The present invention imports the video of the road section to be checked into the instance segmentation network, reads it frame by frame, and identifies and classifies the roads, lane markings, pedestrians, guardrails, lane markings, traffic signs, obstacles, and the road surroundings in the video. Based on the instance segmentation results, road parameters such as road curvature and driving sight distance are calculated. Based on the results, the risks and hidden dangers existing on the current road are judged, and the risk points existing on the road section where the vehicle is traveling are summarized and output. Reading in frame form and conducting risk screening can fully detect risks on the road, avoiding the risk neglect caused by inertia during manual screening.

[0042] 3. The present invention uses deep learning technology to automatically identify safety hazards on the road, greatly improving the efficiency of road hazard inspection. Description of the drawings:

[0043] Figure 1 Flow chart of the method of the present invention.

[0044] Figure 2 Schematic diagram of the example segmentation network structure of the present invention. Specific implementation method:

[0045] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0046] This embodiment provides a method for checking road traffic safety hazards based on an instance segmentation network. The method flow is as follows: Figure 1 As shown, the following steps are included:

[0047] (1) Install a monocular camera on a vehicle, drive the vehicle on a road where safety hazards need to be checked, and record driving video from the driver's perspective;

[0048] (2) Establish an instance segmentation network;

[0049] (3) Establish a comprehensive road scene dataset and loss function, initialize the instance segmentation network parameters and the loss function preset conditions, and train the instance segmentation network; when the loss function reaches the preset conditions, obtain the trained instance segmentation network;

[0050] (4) Perform instance segmentation on the vehicle driving video based on the trained instance segmentation network; import the driving video recorded by the monocular camera into the instance segmentation network, read it in frame form, identify and classify the roads, lane lines, pedestrians, guardrails, lane lines, traffic signs, obstacles, and road surroundings in the video, and output the instance segmentation results;

[0051] (5) Based on the instance segmentation results of the driving video, the curvature, slope, and driving sight distance of the current road are calculated, and the risk hazards existing on the current road are judged based on the instance segmentation results and the calculated parameters;

[0052] (6) Summarize the risk points on the road where the vehicle is traveling, list them in video time sequence and output them to a CSV file.

[0053] The instance segmentation network model includes: FPN network layer, RPN network layer, RoIAlign layer, category prediction branch, bounding box prediction branch and mask segmentation branch, such as Figure 2 As shown;

[0054] The FPN network layer consists of convolutional layers, activation functions, and pooling layers;

[0055] The RPN network layer consists of a convolutional layer and a Softmax regression function; the Softmax regression function is used to determine whether the generated rectangular box is a foreground point or a background point;

[0056] The RoIAlign layer is used to collect proposals generated by the RPN network and unify the input feature images into the same size;

[0057] The category prediction branch calculates the category to which each proposal belongs through the fully connected layer and softmax in the category prediction branch based on the proposed feature image generated by the RoIAlign layer;

[0058] The bounding box prediction branch calculates the position offset of each proposal based on the proposed feature image generated by the RoIAlign layer and performs regression through the fully connected layer in the bounding box prediction branch to obtain a more accurate object detection frame;

[0059] The mask segmentation branch uses a fully convolutional neural network to segment each RoI, and the output dimension is K*m*m, where m is the size of the feature map of RoIAlign. That is, it outputs m*m masks of K categories and binarizes them with the sigmoid function to generate mask segmentations of the background and foreground.

[0060] A comprehensive road scene dataset, including annotation of vehicles, roads, lane markings, pedestrians, guardrails, lane markings, obstacles, and the surrounding environment of the road in driving videos, to determine the annotation results. The annotation results include the true value of the segmented image and the precise edge map corresponding to each frame of driving video.

[0061] The loss function is a multi-task loss: L = L cls +L box +L mask , where L cls represents the classification loss of the prediction box, L box represents the regression loss of the prediction box, Lmask represents the loss of the mask segmentation branch.

[0062] Make a judgment on the existing risks and hidden dangers, including:

[0063] When the road environment is a river or lake, check whether there are guardrails on both sides of the lane. If not, the current road section is judged to be a risky section;

[0064] Determine whether there is a lane line on the current road. If not, the current road is considered a risky section.

[0065] Determine whether the curvature radius of the current road meets the design conditions. If not, the current road is considered a risky section.

[0066] Determine whether the current road slope meets the design conditions. If not, the current road is considered a risky section.

[0067] Determine whether there are any obstructions or obstruction areas on the current road. If so, determine that the current road is a risky section.

[0068] Based on the curvature of the current road, the specific steps to determine whether the current road is a risky section are as follows:

[0069] Mark the lane line pixel coordinate set {(x1, y1), (x2, y2), (x3, y3)...}, and use the least squares method to solve the sum of squares of the error The minimum value of the fitting function is obtained, and the lane curve equation x=Ay is calculated 2 +By+C, converted into the world coordinate equation is Where mx and my are the ratios of 1 meter in the real world to pixels in the horizontal and vertical directions respectively, then the road curvature radius is

[0070] When R<r, where r=125 meters for second-class highway, r=60 meters for third-class highway, and R=30 meters for fourth-class highway, the road curvature radius of this section does not meet the design conditions, and the current road is a risky section.

[0071] Based on the current road slope, the specific steps to determine whether the current road is a risky section are as follows:

[0072] Calculate the slope of the current road When the slope α is greater than I%, where I=6 for a second-class highway, I=7 for a third-class highway, and I=8 for a fourth-class highway, the road section is a steep slope section and a risky section;

[0073] When the road is downhill, the vehicle mileage is counted from the moment the vehicle reaches the sloped section until the road surface becomes level, at which point the counting stops. If the length of the section is greater than 3 km, the slope α is determined to be greater than I'%, where I' = 4.5 for a second-class highway, I' = 5 for a third-class highway, and I' = 5.5 for a fourth-class highway. If so, the section is a continuous downhill section and is a risky section.

[0074] Where e′ is the projection point of the infinitely far point on the slope road surface on the camera imaging plane, f′ is the projection point of the corner point of the slope road surface and the horizontal road surface on the camera imaging plane, and y e′ 、y f′ is the vertical coordinate of e′ and f′ in the image coordinate system, and f is the focal length of the camera;

[0075] The specific steps to determine whether the current road is a risky section based on whether there are occlusions / areas in the current frame are as follows:

[0076] Determine whether there is an obstruction / area in the current frame. If so, calculate the distance d between the vehicle and the obstacle / area. When d < L, where L = 150 meters for a secondary highway, L = 80 meters for a tertiary highway, and L = 60 meters for a quaternary highway, the current road section is a section with poor visibility and is considered a risky section. The same obstacle / area will not be calculated in subsequent frames.

[0077] in Where h is the height of the camera from the ground, β is the camera pitch angle, where v p is the ordinate of the target point in the pixel coordinate system, v0 is the ordinate of the origin of the pixel coordinate system, and f is the focal length of the camera.

[0078] In the present invention:

[0079] The FPN network consists of convolutional layers, activation functions, and pooling layers. The convolution kernel size of all convolutional layers is 3*3, the edge size is 1, the convolution step is 1, and the input and output matrix sizes are not changed before and after convolution; the pooling kernel size of all pooling layers is 2*2, no edge expansion is performed, and the pooling step is 2. After pooling, the length and width of the feature image become the input All activation functions use the Relu function f(x)=max(0,x);

[0080] Each time an image passes through the convolution kernel of the FPN network, its size is reduced to 1 / 2 of the original size each time, generating a total of 5 feature layers, which are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size respectively. These feature outputs are then upsampled and merged with the last output of the previous convolution layer to output 5 feature layers of sizes, which are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size respectively. A proposal is generated through the fully connected layer, and the feature map is input into the RPN network layer and the RoIAlign layer.

[0081] The RPN network layer consists of a convolutional layer and a Softmax regression function. The convolutional layer consists of a 3*3 convolutional layer and two 1*1 convolutional layers. The Softmax regression function is used to determine whether the generated rectangular box is a foreground point or a background point.

[0082] The RPN network layer uses a 3*3 convolution kernel to convolve each input feature layer, and performs 1*1 convolution to generate the border offset scaling and foreground and background prediction values. Each pixel generates three borders with different rotation ratios and outputs the results to the RoIAlign layer.

[0083] The RoIAlign layer receives the original feature image and candidate detection boxes, and uses bilinear interpolation to uniformly crop the size of each ROI on the corresponding feature image to a fixed size of 7*7*256 and 14*14*256, and outputs them to the category prediction branch, bounding box prediction branch and mask prediction layer respectively.

[0084] The category prediction branch calculates the category to which each proposal belongs through a fully connected layer and softmax based on the proposed feature image generated by the RoIAlign layer.

[0085] The bounding box prediction branch calculates the position offset of each proposal and performs regression based on the proposed feature image generated by the RoIAlign layer to obtain a more accurate target detection frame.

[0086] The mask segmentation branch convolves and upsamples the feature image again, outputs a 28*28 mask prediction, and enlarges the mask according to the detection box size.

[0087] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting road traffic safety hazards based on an instance segmentation network, characterized in that: It includes the following steps: (1) Install a monocular camera on the vehicle and drive the vehicle on the road where potential safety hazards need to be investigated, and record the driving video from the driver's perspective; (2) Establish an instance segmentation network; (3) Establish a road scene comprehensive dataset and a loss function, initialize the parameters of the instance segmentation network and the preset conditions of the loss function, and train the instance segmentation network; when the loss function reaches the preset conditions, obtain the trained instance segmentation network; (4) Perform instance segmentation on the vehicle driving video according to the trained instance segmentation network, and output the instance segmentation result; (5) Calculate the curvature, slope, and driving sight distance of the current road according to the instance segmentation result of the driving video, and judge the potential risk hazards existing on the current road based on the instance segmentation result and the calculated parameters; (6) Summarize and output the risk points existing on the vehicle driving section; Make a judgment on the existing potential risk hazards, specifically as follows: When the road environment is a river or a lake, detect whether there are guardrails on both sides of the lane. If not, judge that the current section is a risk section; Judge whether there are lane lines on the current road. If not, judge that the current road is a risk section; Judge whether the curvature radius of the current road meets the design conditions. If not, judge that the current road is a risk section; Judge whether the slope of the current road meets the design conditions. If not, judge that the current road is a risk section; Judge whether there are obstacles or occlusion areas on the current road. If so, judge that the current road is a risk section; The specific steps to judge whether the current road is a risk section according to the road curvature of the current road are as follows: Mark the lane line pixel coordinate set , use the least squares method to solve the sum of squares of the errors The minimum value of the fitting function is obtained to calculate the lane curve equation , converted into the world coordinate equation , where mx and my are the ratios of 1 meter in the real world to pixels in the horizontal and vertical directions respectively, then the road curvature radius is ; When R < r, where r = 125 meters for secondary highways, r = 60 meters for tertiary highways, and R = 30 meters for quaternary highways, the curvature radius of this section does not meet the design conditions, and the current road is a risk section.

2. The method for detecting road traffic safety hazards based on instance segmentation network according to claim 1, characterized in that: The instance segmentation network includes an FPN network layer, an RPN network layer, a RoIAlign layer, a class prediction branch, a bounding box prediction branch, and a mask segmentation branch; The FPN network layer consists of a convolutional layer, an activation function, and a pooling layer; The RPN network layer consists of a convolutional layer and a Softmax regression function, and the Softmax regression function is used to judge whether the generated rectangular box is a foreground point or a background point; The RoIAlign layer is used to collect the proposals generated by the RPN network and unify the input feature images into the same size; The class prediction branch calculates the category to which each proposal belongs through a fully connected layer and softmax according to the proposal feature image generated by the RoIAlign layer; The bounding box prediction branch calculates the position offset of each proposal through a fully connected layer according to the proposal feature image generated by the RoIAlign layer and performs regression to obtain the object detection box; The mask segmentation branch uses a fully convolutional neural network to segment each RoI, and outputs a dimension of K * m * m, where m is the size of the feature map of the RoIAlign, that is, m * m masks of K categories are output, and a sigmoid function is used for binaryization to generate the mask segmentation of the background and the foreground.

3. The method for detecting road traffic safety hazards based on instance segmentation network according to claim 1, characterized in that: The comprehensive road scene dataset includes annotating vehicles, roads, lane lines, pedestrians, guardrails, lane lines, obstacles, and the road surrounding environment in the driving video to determine the annotation results. Among them, the annotation results include the segmentation image ground truth and the precise edge map corresponding to each frame of the driving video.

4. The method for detecting road traffic safety hazards based on instance segmentation network according to claim 1, characterized in that: The loss function is a multi-task loss: ,in represents the classification loss of the prediction box, represents the regression loss of the prediction box, Indicates the loss of the Mask branch.

5. The method for detecting road traffic safety hazards based on instance segmentation network according to claim 1, characterized in that: According to the current road slope, the specific steps to determine whether the current road is a risky section are as follows: Calculate the slope of the current road , when the slope When it is greater than 1%, where I=6 for Class II highways, I=7 for Class III highways, and I=8 for Class IV highways, the road section is a steep slope section and a risky section; When the road is downhill, the vehicle mileage is counted from the moment the vehicle reaches the slope section until the road surface becomes level, and then the counting stops. If the length of the section is greater than 3km, the slope is determined to be Is it greater than , including secondary roads , a third-level highway , a fourth-level highway , if yes, then the road section is a continuous downhill section and a risky section; in is the projection point of the infinitely far point on the slope road surface on the camera imaging plane, is the projection point of the corner point of the slope road surface and the horizontal road surface on the camera imaging plane, and for and The vertical coordinate in the image coordinate system, is the camera focal length.

6. The method for detecting road traffic safety hazards based on instance segmentation network according to claim 1, characterized in that: The specific steps to determine whether the current road is a risk section according to whether there is an occluder / area in the current frame are as follows: Judge whether there is an occluder / area in the current frame. If there is, calculate the distance d between the vehicle and the occluder / area at present. When d < L, where L = 150 meters for secondary highways, L = 80 meters for tertiary highways, and L = 60 meters for quaternary highways, the current section is a section with poor visibility and is a risk section, and the same obstacle / area will no longer be calculated in subsequent frames; in Where h is the height of the camera from the ground, β is the camera pitch angle, , in is the ordinate of the target point in the pixel coordinate system, is the ordinate of the origin of the pixel coordinate system, is the camera focal length.

Images