A road construction early warning method and device based on video

Abstract

The present application relates to the field of intelligent transportation technology, and provides a video-based road construction warning method and device. Based on the distance and intersection-over-union ratio between the anchor point boxes of multiple training samples and the pre-labeled real boxes, the weight between the two is determined and a weighted bipartite graph is generated. A road construction detection model is obtained based on the weighted bipartite graph training. By using the model, each feature map representing the road construction features in each video frame can be accurately extracted to obtain a feature map set; and for the feature map set corresponding to each video frame, an inter-frame residual network is used to detect whether a road construction event occurs. Since the inter-frame residual network combines the construction features of two video frames at different times, this helps to improve the accuracy of road construction event detection, so that when a road construction event is detected, a construction warning message can be pushed in time, thereby ensuring the normal passage of the highway and reducing the probability of traffic accidents.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a video-based road construction early warning method and device. Background Art

[0002] With the continuous development of the economy, the number of motor vehicles has continued to grow, and the traffic volume of the road network has increased rapidly and steadily, increasing the bearing capacity of the road surface. In particular, highways with heavy traffic volume, due to the high speed of vehicles, the large number of passing trucks, and defects in the construction process, will cause the highways to be damaged to varying degrees. Therefore, the highways need to be maintained and repaired.

[0003] When maintaining and repairing highways, some construction features are often set up, such as roadblocks, construction vehicles, construction signs, construction personnel, etc., to remind vehicles to slow down and detour. However, due to the large volume of traffic on highways, high speeds, and the greater impact of environmental visibility, the probability of traffic accidents on construction sections is high. Therefore, when constructing highways, it is particularly important to ensure the safety and smooth flow of highways. This requires timely reporting of the sections currently under construction to facilitate early warning, so as to do a good job of traffic control in advance, supervise and standardize construction, and ensure the safe passage of highways during construction.

[0004] At present, when on the highway, people often rely on manual real-time monitoring or offline inspections to report construction events for early warning. The real-time performance is poor, resulting in a higher probability of traffic accidents on the construction section, affecting road traffic safety. Summary of the invention

[0005] The embodiments of the present application provide a video-based road construction early warning method and device for timely reporting construction events and improving road traffic safety.

[0006] In a first aspect, an embodiment of the present application provides a video-based road construction early warning method, comprising:

[0007] Get the video stream of the highway;

[0008] For each video frame, a trained road construction detection model is used to extract each feature map in a preset detection area of ​​the video frame to obtain a feature map set, wherein the road construction detection model is trained based on a weighted bipartite graph generated from an anchor point frame set of multiple training samples and a pre-labeled real frame set, and the weights of the anchor point frame and the real frame in the weighted bipartite graph are determined based on the distance and intersection-over-union ratio between the two;

[0009] For the feature map set corresponding to each video frame, an inter-frame residual network is used to detect whether a road construction event occurs;

[0010] When a road construction incident occurs, a construction warning message is pushed to ease road traffic.

[0011] In a second aspect, an embodiment of the present application provides a detection device, including a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus;

[0012] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0013] Obtaining a video stream of the expressway through the communication interface;

[0014] For each video frame, a trained road construction detection model is used to extract each feature map in a preset detection area of ​​the video frame to obtain a feature map set, wherein the road construction detection model is trained based on a weighted bipartite graph generated from an anchor point frame set of multiple training samples and a pre-labeled real frame set, and the weights of the anchor point frame and the real frame in the weighted bipartite graph are determined based on the distance and intersection-over-union ratio between the two;

[0015] For the feature map set corresponding to each video frame, an inter-frame residual network is used to detect whether a road construction event occurs;

[0016] When a road construction event occurs, a construction warning message is pushed through the communication interface to ease road traffic.

[0017] Optionally, the processor uses a trained road construction detection model to extract each feature map in a preset detection area of ​​the video frame to obtain a feature map set, and the specific operation is:

[0018] Using the trained road construction detection model, determine the confidence of each detection box in the detection box set;

[0019] Sort the confidence levels, remove the target detection frame corresponding to the highest confidence level, obtain the remaining detection frame set, and store the target detection frame into the target detection frame set;

[0020] Calculate the intersection-and-union ratio of the area of ​​each detection frame in the remaining detection frame set and the area of ​​the target detection frame, and remove the detection frames whose intersection-and-union ratio is greater than the area threshold;

[0021] Re-determine the target detection frames in the remaining detection frame set until the remaining detection frame set is an empty set, and generate a feature map set using each target detection frame in the target detection frame set.

[0022] Optionally, the processor uses an inter-frame residual network to detect whether a road construction event occurs for a set of feature maps corresponding to each video frame, and the specific operation is:

[0023] For each feature map in the feature map set corresponding to the Nth video frame, multiple convolution operations are performed to obtain multiple feature vectors, where N is an integer greater than or equal to 2;

[0024] Determine a difference vector between each feature map of the Nth video frame and a corresponding feature map of the N-1th video frame;

[0025] Each differential vector is fully connected to multiple feature vectors of the corresponding feature map, and whether a road construction event occurs is detected based on the fully connected target vector.

[0026] Optionally, the processor trains the road construction detection model in the following manner:

[0027] Get a training sample set;

[0028] Input multiple training samples in the training sample set into the initial road construction detection model, and obtain a road construction detection model with a detection loss value within a preset range through multiple rounds of iterative training; wherein, for each round of training, perform the following operations:

[0029] Through the initial road construction detection model, multiple anchor boxes in each training sample are extracted to obtain an anchor box set;

[0030] For each anchor box in the anchor box set, determining a first weight of the two according to an intersection-over-union ratio and a distance between the anchor box and a real box corresponding to the target contained in the anchor box;

[0031] A weighted bipartite graph is established based on the anchor box set of each training sample and the pre-labeled real box set;

[0032] Using the KM algorithm, the best matching result of the weighted bipartite graph is calculated;

[0033] According to the matched anchor box and the real box, the detection loss value of each anchor box is determined;

[0034] According to the detection loss value of each anchor box in each training sample, the parameters of the initial road construction detection model are adjusted.

[0035] Optionally, one side of the weighted bipartite graph is a set of anchor boxes, and the other side is a set of real boxes;

[0036] The processor uses the KM algorithm to calculate the best matching result of the weighted bipartite graph, and the specific operation is:

[0037] Determine, according to the first weights of the anchor boxes and the intersection-and-union ratios corresponding to the first weights, the second weights of the edges formed by the anchor boxes and the corresponding real boxes;

[0038] According to the second weights of the edges, the KM algorithm is adopted to obtain a matching result with the largest number of matches.

[0039] Optionally, the calculation formula of the first weight is:

[0040] W ij =log(D ij *|1-S i / S j |)

[0041] Among them, W ij represents the first weight between the i-th anchor box and the j-th real box, D ij represents the distance between the center point of the i-th anchor box and the j-th ground-truth box, S i / S j It represents the intersection-over-unit ratio of the area of ​​the i-th anchor box and the area of ​​the j-th ground-truth box. Different anchor boxes have different scales.

[0042] Optionally, after obtaining the anchor box set of each training sample, the processor further performs the following operations:

[0043] A position weight is set for each anchor box according to the distance between each anchor box in the anchor box set and the corresponding real box, so that when calculating the best matching result of the weighted bipartite graph, anchor boxes with position weights greater than a position threshold are preferentially regressed.

[0044] In a third aspect, the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the video-based road construction warning method provided in an embodiment of the present application.

[0045] The beneficial effects of this application are as follows:

[0046] The embodiment of the present application provides a video-based road construction warning method and device. For each video frame of a highway video stream, a trained road construction detection model is used to extract each feature map within a preset detection area of ​​the video frame to obtain a feature map set. Since the road construction detection model is trained based on a weighted bipartite graph generated by anchor boxes of multiple training samples and pre-labeled real boxes, the training process takes into account the weight between the anchor box and the real box, which improves the accuracy of the training, so that the feature map set that is most likely to characterize the construction characteristics can be accurately obtained; and for the feature map set corresponding to each video frame, an inter-frame residual network is used to detect whether a road construction event occurs. Since the inter-frame residual network combines the construction features of two video frames at different times, this helps to improve the accuracy of road construction event detection, so that when a road construction event is detected, a construction warning message can be pushed in time, thereby ensuring the normal passage of the highway and reducing the probability of traffic accidents.

[0047] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0049] Figure 1 The flowchart of the training method of the road construction detection model provided in the embodiment of the present application is exemplified;

[0050] Figure 2 The flowchart of the method for calculating the best matching result of a weighted bipartite graph provided in an embodiment of the present application is exemplarily shown;

[0051] Figure 3 The network structure diagram of the road construction detection model provided by the embodiment of the present application is exemplarily shown;

[0052] Figure 4 The network structure diagram for regressing anchor boxes with different position weights provided in an embodiment of the present application is exemplarily shown;

[0053] Figure 5 The flowchart of the video-based road construction early warning method provided in the embodiment of the present application is exemplified;

[0054] Figure 6 The flowchart of the feature map extraction method provided by the embodiment of the present application is exemplarily shown;

[0055] Figure 7 The following is a schematic diagram showing a process of obtaining a feature graph set provided in an embodiment of the present application;

[0056] Figure 8 The flowchart of the method for detecting road construction events provided by the embodiment of the present application is exemplarily shown;

[0057] Fig. 9 The inter-frame residual network structure diagram provided by the embodiment of the present application is exemplarily shown;

[0058] Fig.10 The road construction detection effect diagram provided by the embodiment of the present application is exemplarily shown;

[0059] Fig.11 Another road construction detection effect diagram provided by an embodiment of the present application is exemplarily shown;

[0060] Fig.12 The structural diagram of the detection device provided in the embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] Data provided by the Traffic Information Center of the Ministry of Transport's Research Institute show that the number of motor vehicles on expressways continues to grow steadily. In 26 provinces and cities across the country, the number of trucks alone is about 1.5 billion, with an average daily truck flow of 4.93 million, a year-on-year increase of 14%. In order to ensure the normal passage of expressways, expressways need to be maintained and repaired.

[0063] However, due to the large volume of traffic on highways, high speeds, and the impact of environmental visibility, the probability of traffic accidents occurring on construction sections is high. At present, the reporting of road construction events mainly relies on manual monitoring, which may not be reported in time for traffic diversion, resulting in traffic congestion and collisions caused by construction occupying the road, which poses a serious traffic hazard. Therefore, it is necessary to report and warn highway construction events in a timely manner to improve highway traffic safety.

[0064] The embodiment of the present application provides a video-based road construction early warning method and device. Based on the acquired video stream on the highway, for each video frame, a trained road construction detection model is used to extract each feature map representing the road construction features (such as cones, construction signs, construction vehicles, construction personnel, etc.) to obtain a feature map set. Based on the feature vectors extracted from the feature map sets of two adjacent video frames and the frame difference vectors of the two feature map sets, it is detected whether a road construction event occurs, and when a road construction event is detected, it is reported in time so that the traffic control center can send construction warning messages to moving vehicles to divert road traffic and avoid abnormal situations such as traffic accidents and traffic congestion. At the same time, the traffic control center can also record the construction events to determine whether the construction events occur within the specified time, supervise the operations of illegal construction, and improve road construction management.

[0065] At present, cameras have been installed and configured at preset locations on the highway, and real-time traffic flow images can be uploaded to the background server. Therefore, a training sample set of the road construction detection model in the embodiment of this application can be generated based on the traffic videos collected by various cameras on the highway.

[0066] Since the installation positions, angles, resolutions, etc. of each camera are different, the scales of the targets (including but not limited to vehicles, signs, people, etc.) in the collected traffic images are different, and the image areas where the targets are located are also different. There are usually lawns, trees, houses, etc. on both sides of the highway. In order to eliminate the interference of these backgrounds and improve the accuracy of construction event detection, the traffic images collected by each camera can be pre-labeled with detection areas. Specifically, a closed driving area in the traffic image is labeled as the detection area, denoted as S, and the size of the detection area is denoted as the area of ​​the closed driving area.

[0067] Considering that the installation position, angle, resolution and other parameters of the same camera remain unchanged, it is only necessary to select a reference traffic image from the traffic video collected by each camera to mark the detection area and record the marked detection area, as shown in Table 1.

[0068] Table 1. Detection areas corresponding to each camera

[0069] Camera ID Detection area 1 (x1, y1, w1, h1) 2 (x2, y2, w2, h2) 3 (x3, y3, w3, h3) … …

[0070] Among them, (xi, yi) represents the origin of the detection area box, wi represents the width of the detection area box, and hi represents the height of the detection area box.

[0071] After marking the detection area for the traffic videos collected by each camera in advance, a training sample set can be generated based on the traffic videos collected by each camera to train the road construction detection model. For the specific training process, see Figure 1, mainly includes the following steps:

[0072] S101: Obtain a training sample set.

[0073] In an optional implementation, traffic videos collected by each camera are obtained, and the traffic images of the corresponding camera are cropped based on the pre-marked detection area to obtain a training sample set. The training sample set includes multiple positive training samples and multiple negative training samples. The positive training samples include road construction features, such as construction vehicles, construction signs, cones, construction personnel, etc., and the negative training samples do not include road construction features.

[0074] Optionally, the ratio of positive training samples to negative training samples in the training sample set is 3:1.

[0075] It should be noted that if the number of positive training samples obtained is small, the positive training samples can be obtained from an existing sample library or network resources.

[0076] After obtaining the training sample set, each training sample is labeled with a sample label. For example, the ground truth boxes of the road construction features are marked from the positive training sample, and the ground truth boxes are not included in the negative training sample. A positive training sample can contain multiple road construction features, that is, it can contain multiple ground truth boxes.

[0077] S102: Input multiple training samples in the training sample set into the initial road construction detection model, and obtain a road construction detection model with a detection loss value within a preset range through multiple rounds of iterative training.

[0078] In S102, for each round of training, the following operations are performed:

[0079] S1021: Extract multiple anchor boxes from each training sample through the initial road construction detection model to obtain an anchor box set.

[0080] In an embodiment of the present application, the initial road construction detection model adopts anchor technology to convert the target detection problem into whether there is a target in the anchor box and the distance of the anchor box from the real box of the target.

[0081] For the initial road construction detection model, a set of anchor boxes of different scales and positions are pre-set. These anchor boxes cover the entire area of ​​the training sample. Each anchor box is used to detect targets (such as cones, construction workers, roadblocks, construction signs, etc.) whose intersection-over-union ratio is greater than a threshold. Therefore, in S1021, multiple anchor boxes can be extracted from each training sample through the initial road construction detection model to obtain an anchor box set.

[0082] At present, the existing target detection network (such as Single Shot Multi-box Detector, SSD) matches the anchor box with the real box according to the intersection and union ratio of the anchor box area and the real box area. In this way, one anchor box may match multiple real boxes. At this time, whether the anchor box is matched with any of the multiple real boxes or the anchor box is used as the background, the loss value of the target detection network will be reduced. In addition, in high-speed scenarios, considering that the targets representing road construction features are often small, feature information is insufficient, and anchor box wrapping is inaccurate, conventional anchor matching strategies may not be able to accurately detect targets representing road construction features.

[0083] Therefore, the embodiment of the present application provides a new anchor matching strategy, denoted as ISD (IOU ScaleDistance) matching. Through ISD matching, a real box can be matched for each anchor box. For details, see the following embodiment. Wherein, IOU represents the intersection-over-union ratio of the anchor box and the matched real box area, Scale represents the scale of the anchor box, and the scales of different anchor boxes are different, and Distance represents the distance between the anchor box and the matched real box.

[0084] S1022: For each anchor box in the anchor box set, determine a first weight of the two according to an intersection-over-union ratio and a distance between the anchor box and a real box corresponding to the target contained in the anchor box.

[0085] Taking the i-th anchor box as an example, the real box corresponding to the target contained in the anchor box is the j-th real box, and the calculation process of the first weight is as follows:

[0086] First, the intersection-over-union ratio of the areas between the i-th anchor box and the j-th ground-truth box is calculated.

[0087] Assume that the area of ​​the i-th anchor box is recorded as S i , the area of ​​the jth ground truth box is recorded as S j , then the intersection-and-union ratio IOU = S i / S j .

[0088] Then, the distance between the i-th anchor box and the j-th real box is determined, and the first weight is calculated according to the determined distance and intersection-over-union ratio.

[0089] Theoretically, the closer the i-th anchor box is to the j-th ground truth box, the smaller the distance between the two center points, and the larger the intersection-over-intersection ratio of the two areas. i / S j The smaller the absolute value of , the log function can be used to convert the first weight into a monotonically decreasing function in the transform domain. The calculation formula is as follows:

[0090] W ij =log(D ij *|1-S i / S j |) Formula 1

[0091] Among them, W ij represents the first weight between the i-th anchor box and the j-th real box, which represents the similarity between the anchor box and the real box. The larger the first weight, the more similar the anchor box is to the real box. ij represents the distance between the center point of the i-th anchor box and the j-th true box. The smaller the distance, the closer the anchor box is to the true box. i / S j Represents the intersection and union ratio of the area of ​​the i-th anchor box and the area of ​​the j-th real box, 1-S i / S j The closer the absolute value of is to 0, the closer the anchor box is to the real box.

[0092] In the embodiment of the present application, since different anchor boxes have different scales, their scales can be obtained by S i It means that, therefore, the first weight contains the scale information.

[0093] S1023: Create a weighted bipartite graph based on the anchor box set and the pre-labeled real box set of each training sample.

[0094] Usually, the vertex set of a bipartite graph can be divided into two non-intersecting subsets, and the two vertices attached to each edge in the bipartite graph belong to these two non-intersecting subsets, which is often used to solve matching problems.

[0095] When executing S1023, after determining the first weight for each anchor box in the anchor box set, each anchor box with a weight is used as a binary Figure 1 The vertices on one side are taken as the vertices on the other side of the bipartite graph, and a weighted bipartite graph is established with each real box in the pre-labeled real box set as the vertex on the other side of the bipartite graph.

[0096] S1024: Using the KM algorithm, calculate the best matching result of the weighted bipartite graph.

[0097] The full name of KM algorithm is Kuhn-Munkres Algorithm. KM algorithm can match a real box to each anchor box and maximize the number of matches, that is, maximize the edges in the bipartite graph, thereby improving the convergence speed of the model and the accuracy of detection. Figure 2 :

[0098] S1024_1: Determine the second weight of each edge formed by each anchor box and the corresponding real box according to the first weight of each anchor box and the intersection-and-union ratio corresponding to each first weight.

[0099] Taking the i-th anchor box as an example, it corresponds to the j-th real box, and the first weight of the i-th anchor box is W ij , the intersection-and-union ratio (IOU) corresponding to the first weight is S i / S j , then the second weight of the edge formed by the i-th anchor box and the j-th real box is W ij *IOU.

[0100] S1024_2: According to the second weight of each edge, the KM algorithm is used to obtain a matching result with the largest number of matches.

[0101] In specific implementation, the anchor boxes and real boxes associated with the edges whose second weights are lower than the preset weight threshold are rematched with the unmatched anchor boxes and real boxes until the preset conditions are met. In this way, the matched anchor boxes and real boxes meet the weight threshold requirements, and the number of successful matches is the largest.

[0102] S1025: Determine the detection loss value of each target anchor box according to the matched anchor box and the real box.

[0103] In S1025, the larger the second weight of the edge formed by the matched anchor box and the real box, the more similar they are, and the smaller the detection loss value. Among them, the embodiment of the application does not impose restrictive requirements on the loss function for calculating the detection loss value, for example, it can be a softmax function.

[0104] S1026: Adjust the parameters of the initial road construction detection model according to the detection loss value of each target anchor point box in each training sample.

[0105] The detection loss value obtained in each round of training is used to adjust the parameters of the initial road construction detection model until the detection loss value is within a preset range, thereby obtaining a trained road construction detection model.

[0106] Optionally, the road construction detection model in the above embodiment can be constructed based on the S3FD (single shotscale-invariant face detector) algorithm, such as Figure 3As shown, it is a structural diagram of the road construction detection model provided in an embodiment of the present application, which mainly includes base convolutional layers, extra convolutional layers, normalization layers, predicted convolutional layers and loss function layers (Multi-task Loss Layer).

[0107] Basic convolutional layer: Based on the network structure of VGG16, the structure of conv1_1 to pool5 of VGG16 is used, that is, the top three fully connected layers are removed.

[0108] Add convolutional layers: Convert fc6 and fc7 of VGG16 into convolutional layers to reduce the number of parameters, and add several convolutional layers after them.

[0109] Normalization layer: The outputs of the base convolutional layers conv3_3, conv4_3, and conv5_3 have features of different sizes and are normalized by L2.

[0110] Prediction convolution layer: Each prediction convolution layer is followed by a 3*3 convolution layer. For each anchor, in addition to predicting the offsets of the 4 anchor box coordinates, it is also necessary to predict the scores of Ns categories. For the detection layer of cov3_3, Ns=Nm+1, where Nm is the max-out background label, and for other detection layers, Ns=2.

[0111] Loss function layer: Cross entropy loss softmax is used for classification loss, and SmoothL1 loss is used for regression loss.

[0112] At present, the existing object detection network treats each preset anchor box equally when extracting anchor boxes from training samples. However, in fact, some anchor boxes do not circle the target, or only circle part of the target. This will make it difficult for some anchors to return to the accurate real box, reducing the training efficiency and accuracy of the model.

[0113] In order to solve the above problem, in some embodiments of the present application, after extracting the anchor box set of each training sample through the initial road construction detection model, a position weight is set for each anchor box according to the distance between each anchor box in the anchor box set and the corresponding real box, so that when calculating the best matching result of the weighted bipartite graph, the anchor boxes with position weights greater than the position threshold are preferentially regressed.

[0114] In an optional implementation, the SAPD (Soft Achor-Point Detection) algorithm is used to regress anchor boxes with different position weights. The network structure of the SAPD algorithm is shown in Figure 4 , which mainly includes two parts: soft-weighted anchor points and soft-selected pyramid levels.

[0115] The soft-weighted anchor points part can assign the first weight to the anchor box according to the distance between the anchor box and the true box. The farther the distance, the lower the first weight. In this way, the road construction detection model can focus on the anchor box that is close to the target in the true box. The soft-selected pyramid levels part can better integrate the features of multiple layers of pyramids, so that a certain area of ​​a certain layer of pyramid is activated, and the same area of ​​adjacent levels may also be activated in a similar way, which means that features of multiple pyramid levels can contribute to the detection of a certain target together, but the contributions of features at different levels are different. Therefore, the soft-selected pyramid levels part trains a meta-selection network to predict the weights of each pyramid level for soft feature selection.

[0116] Based on the trained road construction detection model, a video-based road construction early warning method is implemented. The method is executed by the detection device. Figure 5 , mainly includes the following steps:

[0117] S501: Acquire a video stream of a highway.

[0118] In S501, the video stream may be collected by a camera on the highway, and a preset detection area of ​​the video stream may be determined according to the ID of the camera.

[0119] S502: For each video frame, a trained road construction detection model is used to extract each feature map in a preset detection area of ​​the video frame to obtain a feature map set.

[0120] The road construction detection model is trained based on a weighted bipartite graph generated from a set of anchor boxes of multiple training samples and a set of pre-labeled real boxes. The weights of the anchor boxes and the real boxes in the weighted bipartite graph are determined based on the distance and intersection-over-union ratio between the two. For the specific training process of the road construction detection model, see Figure 1 and Figure 2 , which will not be repeated here.

[0121] When executing S502, based on the trained road construction detection model, a non-maximum suppression method (NMS) is used to extract a feature map within a preset detection area of ​​each video frame. Figure 6 :

[0122] S5021: Using the trained road construction detection model, determine the confidence of each detection box in the detection box set.

[0123] In an embodiment of the present application, a video frame may contain multiple targets representing road construction. Therefore, through the road construction detection model, multiple detection frames can be obtained to form a detection frame set. The scale and position of each detection frame are different. The confidence of each detection frame in the detection frame set represents the probability that the corresponding detection frame contains the target representing the road construction. The higher the confidence, the greater the possibility that the detection frame contains the target representing the road construction.

[0124] S5022: Sort the confidence levels, remove the target detection frame corresponding to the highest confidence level, obtain a set of remaining detection frames, and store the target detection frames into the target detection frame set.

[0125] When executing S5022, the confidences of the detection frames are sorted in descending order, and the target detection frame with the highest confidence is removed from the detection frame set to obtain the remaining detection frame set, and the removed target detection frame is stored in the target detection frame set.

[0126] S5023: Calculate the intersection-and-union ratio of the area of ​​each detection frame in the remaining detection frame set and the area of ​​the target detection frame, and remove the detection frames whose intersection-and-union ratio is greater than the area threshold.

[0127] Take detection frame 1 in the remaining detection frames as an example, calculate the area of ​​detection frame 1, and calculate the intersection and union ratio 1 of the area of ​​detection frame 1 and the area of ​​the target detection frame. When the intersection and union ratio 1 is greater than the preset area threshold, it indicates that the overlap area between detection frame 1 and the target detection frame is large, and detection frame 1 is removed from the remaining detection frames and put into the target detection frame set. Optionally, the area threshold is greater than 0.5 and less than 1.

[0128] S5024: Re-determine the target detection frames in the remaining detection frame set until the remaining detection frame set is an empty set, and generate a feature map set using each target detection frame in the target detection frame set.

[0129] After executing S5023, the target detection frame in the remaining detection frame is re-determined, and the operation in S5033 is repeated until the remaining detection frame set is an empty set. In this way, the detection frames with lower confidence can be filtered out, and the detection frames in the target detection frame set with higher confidence can be used to generate a feature map set of the video frame.

[0130] See also Figure 7 , is a schematic diagram of the process of obtaining a feature graph set by the NMS operation provided in an embodiment of the present application, wherein: Figure 7 Table 1 in is the sorted detection frame set. After removing the detection frame A with the highest confidence, the remaining detection frame set is obtained, as shown in Figure 7 As shown in Table 2, the removed detection box A is stored in Figure 7 Table 3 in ; then, the intersection and union ratio of the area of ​​detection frame BD in Table 2 and the area of ​​detection frame A are calculated respectively. When the intersection and union ratio corresponding to detection frame D is greater than the area threshold, detection frame D is removed from Table 2; further, it is determined that the confidence of detection frame C in Table 2 is the highest, stored in Table 3, and the intersection and union ratio of the area of ​​detection frame B and the area of ​​detection frame C is calculated. When the intersection and union ratio corresponding to detection frame B is greater than the area threshold, detection frame D is removed from Table 2, and Table 2 is empty at this time.

[0131] S503: For the feature map set corresponding to each video frame, an inter-frame residual network is used to detect whether a road construction event occurs. If so, execute S504, otherwise obtain the next video frame.

[0132] After obtaining the feature map set of each video frame, each feature map in the feature map set is input into the inter-frame residual network, and the road construction event detection is completed through the network. For the specific detection process, see Figure 8 :

[0133] S5031: Perform multiple convolution operations on each feature map in the feature map set corresponding to the Nth video frame to obtain multiple feature vectors.

[0134] For the first video frame, it is an initial video frame and there is no previous video frame. Therefore, N is an integer greater than or equal to 2.

[0135] When executing S5031, multiple convolution operations are performed on each feature map in the feature map set corresponding to the Nth video frame, wherein the network and convolution kernel used in each convolution operation may be the same or different.

[0136] S5032: Determine a differential vector between a feature map of the Nth video frame and a corresponding feature map of the N-1th video frame.

[0137] For each feature map, a differential vector between the feature map in the Nth video frame and the N-1th video frame is calculated.

[0138] S5033: Fully connect the differential vector and multiple feature vectors, and detect whether a road construction event occurs based on the fully connected target vector.

[0139] Taking a feature map as an example, after a convolution operation is performed on the feature map, convolution dimensionality reduction is performed again, and the differential vector between the Nth frame video frame and the N-1th frame video frame of the feature map is used as a weight and multiplied with the feature vector after dimensionality reduction. The multiplied vector is fully connected with the feature vector after another convolution operation of the feature map, and whether a road construction event occurs is detected based on the fully connected target vector.

[0140] Optionally, the inter-frame residual network in the embodiment of the present application may be a Resnet network.

[0141] See also Fig. 9 , is a structural schematic diagram of the residual network provided in an embodiment of the present application, the residual network performs two convolution operations on a feature map corresponding to the N-th frame of the video frame, the first convolution network uses the Relu network and the BN network, the convolution kernel size is 3*3, the second convolution network uses the BN network, the convolution kernel size is also 3*3, and then the feature map after the second convolution is reduced in dimension using the 3*3 convolution kernel again, and the differential vector between the N-th video frame and the N-1-th video frame is multiplied with the reduced-dimensional feature vector as a weight; further, the multiplied feature vector is fully connected with the feature vector after the first convolution, and whether a road construction event occurs is detected based on the fully connected target vector.

[0142] S504: When a road construction event occurs, a construction warning message is pushed to ease road traffic.

[0143] In S504, when a road construction event occurs, a construction warning message is sent to the traffic control center and moving vehicles to ease road traffic.

[0144] In some embodiments, the traffic control center records the construction warning message after receiving it, and verifies and confirms the construction warning information in the platform, thereby reducing illegal construction and improving traffic safety on highways.

[0145] The embodiment of the present application provides a video-based road construction warning method. For each video frame of the highway video stream, a trained road construction detection model is used to extract each feature map within a preset detection area of ​​the video frame to obtain a feature map set. Since the road construction detection model is trained based on a weighted bipartite graph generated by anchor boxes of multiple training samples and pre-labeled real boxes, the training process takes into account the weight between the anchor box and the real box, which improves the accuracy of the training, so that the feature map set that is most likely to characterize the construction characteristics can be accurately obtained; and for the feature map set corresponding to each video frame, an inter-frame residual network is used to detect whether a road construction event occurs. Since the inter-frame residual network combines the construction features of two video frames at different times, this helps to improve the accuracy of road construction event detection, so that when a road construction event is detected, a construction warning message can be pushed in time, thereby ensuring the normal passage of the highway and reducing the probability of traffic accidents.

[0146] It should be noted that the video-based road construction early warning method provided in the application embodiment can be applied not only to highways, but also to expressways, national highways, ordinary roads and other scenarios.

[0147] Fig.10 and Fig.11 The detection situation of the video-based road construction early warning method provided by the embodiment of the present application is as follows: Fig.10 and Fig.11 It can be seen that this method can accurately detect targets that characterize road construction features, such as cones, construction vehicles, construction workers, etc., so that road construction warning information can be reported in a timely manner, so that highway supervisors can guide and control traffic conditions, maintain highway traffic order, reduce the occurrence of traffic accidents, and ensure safe passage on highways.

[0148] Based on the same technical concept, an embodiment of the present application provides a detection device, which can execute the video-based road construction warning method provided in the above embodiment and achieve the same technical effect, which will not be repeated here.

[0149] See also Fig.12 The detection device includes a processor 1201, a memory 1202, and a communication interface 1203. The communication interface 1203, the memory 1202 and the processor 1201 are connected via a bus 1204. The memory 1202 stores a computer program. The processor 1201 performs the following operations according to the computer program stored in the memory 1202:

[0150] Obtaining a video stream of the highway through the communication interface 1203;

[0151] For each video frame, the trained road construction detection model is used to extract each feature map in the preset detection area of ​​the video frame to obtain a feature map set, wherein the road construction detection model is trained based on a weighted bipartite graph generated from a set of anchor boxes of multiple training samples and a set of pre-labeled real boxes, and the weights of the anchor boxes and the real boxes in the weighted bipartite graph are determined based on the distance and intersection-over-union ratio between the two;

[0152] For the feature map set corresponding to each video frame, an inter-frame residual network is used to detect whether a road construction event occurs;

[0153] When a road construction event occurs, a construction warning message is pushed through the communication interface 1203 to ease road traffic.

[0154] Optionally, the processor 1201 uses a trained road construction detection model to extract each feature map in a preset detection area of ​​a video frame to obtain a feature map set. The specific operation is:

[0155] Using the trained road construction detection model, determine the confidence of each detection box in the detection box set;

[0156] Sort each confidence level, remove the target detection frame corresponding to the highest confidence level, obtain the remaining detection frame set, and store the target detection frame into the target detection frame set;

[0157] Calculate the intersection-and-union ratio of the area of ​​each detection frame in the remaining detection frame set and the area of ​​the target detection frame, and remove the detection frames whose intersection-and-union ratio is greater than the area threshold;

[0158] Re-determine the target detection frames in the remaining detection frame set until the remaining detection frame set is an empty set, and use each target detection frame in the target detection frame set to generate a feature map set.

[0159] Optionally, the processor 1201 uses an inter-frame residual network to detect whether a road construction event occurs for a feature map set corresponding to each video frame. The specific operation is:

[0160] For each feature map in the feature map set corresponding to the Nth video frame, multiple convolution operations are performed to obtain multiple feature vectors, where N is an integer greater than or equal to 2;

[0161] Determine a difference vector between each feature map of the Nth video frame and a corresponding feature map of the N-1th video frame;

[0162] Each differential vector is fully connected to multiple feature vectors of the corresponding feature map, and whether a road construction event occurs is detected based on the fully connected target vector.

[0163] Optionally, the processor 1201 trains the road construction detection model in the following manner:

[0164] Get a training sample set;

[0165] Input multiple training samples in the training sample set into the initial road construction detection model, and obtain a road construction detection model with a detection loss value within a preset range through multiple rounds of iterative training; wherein, for each round of training, perform the following operations:

[0166] Through the initial road construction detection model, multiple anchor boxes in each training sample are extracted to obtain an anchor box set;

[0167] For each anchor box in the anchor box set, determine a first weight of the two according to the intersection-over-union ratio and the distance between the anchor box and the real box corresponding to the target contained in the anchor box;

[0168] A weighted bipartite graph is established based on the anchor box set of each training sample and the pre-labeled real box set;

[0169] The KM algorithm is used to calculate the best matching result of the weighted bipartite graph;

[0170] According to the matched anchor box and the real box, the detection loss value of each anchor box is determined;

[0171] According to the detection loss value of each anchor box in each training sample, the parameters of the initial road construction detection model are adjusted.

[0172] Optionally, one side of the weighted bipartite graph is a set of anchor boxes, and the other side is a set of ground-truth boxes;

[0173] The processor uses the KM algorithm to calculate the best matching result of the weighted bipartite graph. The specific operations are:

[0174] Determine, according to the first weights of the anchor boxes and the intersection-and-union ratios corresponding to the first weights, the second weights of the edges formed by the anchor boxes and the corresponding real boxes;

[0175] According to the second weight of each edge, the KM algorithm is used to obtain the matching result with the largest number of matches.

[0176] Optionally, the calculation formula of the first weight is:

[0177] W ij =log(D ij *|1-S i / S j |)

[0178] Among them, W ij represents the first weight between the i-th anchor box and the j-th real box, D ijrepresents the distance between the center point of the i-th anchor box and the j-th ground-truth box, S i / S j It represents the intersection-over-unit ratio of the area of ​​the i-th anchor box and the area of ​​the j-th ground-truth box. Different anchor boxes have different scales.

[0179] Optionally, after obtaining the anchor box set for each training sample, the processor further performs the following operations:

[0180] According to the distance between each anchor box in the anchor box set and the corresponding real box, a position weight is set for each anchor box, so that when calculating the best matching result of the weighted bipartite graph, the anchor boxes with position weights greater than the position threshold are preferentially regressed.

[0181] It should be noted that Fig.11 It is only the necessary hardware for the detection device to implement the video-based road construction early warning method provided in the embodiment of the present application. Optionally, the detection device also includes conventional hardware such as a display, a video processor, etc.

[0182] The processor involved in the above-mentioned embodiment of the present application can be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. Among them, the memory can be integrated in the processor or can be set separately from the processor.

[0183] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the method of the aforementioned embodiment.

[0184] The embodiments of the present application also provide a computer program product for storing a computer program, wherein the computer program is used to execute the method of the aforementioned embodiment.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0186] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0187] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0188] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0189] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0190] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0191] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A video-based road construction early warning method, characterized in that: include: Get the video stream of the highway; For each video frame, a trained road construction detection model is used to extract each feature map in a preset detection area of ​​the video frame to obtain a feature map set, wherein the road construction detection model is trained based on a weighted bipartite graph generated from an anchor point frame set of multiple training samples and a pre-labeled real frame set, and the weights of the anchor point frame and the real frame in the weighted bipartite graph are determined based on the distance and intersection-over-union ratio between the two; For the feature map set corresponding to each video frame, an inter-frame residual network is used to detect whether a road construction event occurs; When a road construction incident occurs, a construction warning message is pushed to ease road traffic.

2. The method according to claim 1, characterized in that The trained road construction detection model is used to extract each feature map in the preset detection area of ​​the video frame to obtain a feature map set, including: Using the trained road construction detection model, determine the confidence of each detection box in the detection box set; Sort the confidence levels, remove the target detection frame corresponding to the highest confidence level, obtain the remaining detection frame set, and store the target detection frame into the target detection frame set; Calculate the intersection-and-union ratio of the area of ​​each detection frame in the remaining detection frame set and the area of ​​the target detection frame, and remove the detection frames whose intersection-and-union ratio is greater than the area threshold; Re-determine the target detection frames in the remaining detection frame set until the remaining detection frame set is an empty set, and generate a feature map set using each target detection frame in the target detection frame set.

3. The method according to claim 1, characterized in that The feature map set corresponding to each video frame uses an inter-frame residual network to detect whether a road construction event occurs, including: For each feature map in the feature map set corresponding to the Nth video frame, multiple convolution operations are performed to obtain multiple feature vectors, where N is an integer greater than or equal to 2; Determine a difference vector between each feature map of the Nth video frame and a corresponding feature map of the N-1th video frame; Each differential vector is fully connected to multiple feature vectors of the corresponding feature map, and whether a road construction event occurs is detected based on the fully connected target vector.

4. The method according to any one of claims 1 to 3, characterized in that The road construction detection model is trained in the following way: Get a training sample set; Input multiple training samples in the training sample set into the initial road construction detection model, and obtain a road construction detection model with a detection loss value within a preset range through multiple rounds of iterative training; wherein, for each round of training, perform the following operations: Through the initial road construction detection model, multiple anchor boxes in each training sample are extracted to obtain an anchor box set; For each anchor box in the anchor box set, determining a first weight of the two according to an intersection-over-union ratio and a distance between the anchor box and a real box corresponding to the target contained in the anchor box; A weighted bipartite graph is established based on the anchor box set of each training sample and the pre-labeled real box set; Using the KM algorithm, the best matching result of the weighted bipartite graph is calculated; According to the matched anchor box and the real box, the detection loss value of each anchor box is determined; According to the detection loss value of each anchor box in each training sample, the parameters of the initial road construction detection model are adjusted.

5. The method according to claim 4, characterized in that One side of the weighted bipartite graph is a set of anchor boxes, and the other side is a set of real boxes; The KM algorithm is used to calculate the best matching result of the weighted bipartite graph, including: Determine, according to the first weights of the anchor boxes and the intersection-and-union ratios corresponding to the first weights, the second weights of the edges formed by the anchor boxes and the corresponding real boxes; According to the second weights of the edges, the KM algorithm is adopted to obtain a matching result with the largest number of matches.

6. The method according to claim 4, characterized in that The calculation formula of the first weight is: W ij =log(D ij *|1-S i / S j |) Among them, W ij represents the first weight between the i-th anchor box and the j-th real box, D ij represents the distance between the center point of the i-th anchor box and the j-th ground-truth box, S i / S j It represents the intersection-over-unit ratio of the area of ​​the i-th anchor box and the area of ​​the j-th ground-truth box. Different anchor boxes have different scales.

7. The method according to claim 4, characterized in that After obtaining the anchor point frame set of each training sample, the method further includes: A position weight is set for each anchor box according to the distance between each anchor box in the anchor box set and the corresponding real box, so that when calculating the best matching result of the weighted bipartite graph, anchor boxes with position weights greater than a position threshold are preferentially regressed.

8. A detection device, characterized in that: It includes a processor, a memory and a communication interface, wherein the communication interface, the memory and the processor are connected via a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: Obtaining a video stream of the expressway through the communication interface; For each video frame, a trained road construction detection model is used to extract each feature map in a preset detection area of ​​the video frame to obtain a feature map set, wherein the road construction detection model is trained based on a weighted bipartite graph generated from an anchor point frame set of multiple training samples and a pre-labeled real frame set, and the weights of the anchor point frame and the real frame in the weighted bipartite graph are determined based on the distance and intersection-over-union ratio between the two; For the feature map set corresponding to each video frame, an inter-frame residual network is used to detect whether a road construction event occurs; When a road construction event occurs, a construction warning message is pushed through the communication interface to ease road traffic.

9. The detection device according to claim 8, characterized in that The processor uses a trained road construction detection model to extract each feature map in a preset detection area of ​​the video frame to obtain a feature map set. The specific operation is: Using the trained road construction detection model, determine the confidence of each detection box in the detection box set; Sort the confidence levels, remove the target detection frame corresponding to the highest confidence level, obtain the remaining detection frame set, and store the target detection frame into the target detection frame set; Calculate the intersection-and-union ratio of the area of ​​each detection frame in the remaining detection frame set and the area of ​​the target detection frame, and remove the detection frames whose intersection-and-union ratio is greater than the area threshold; Re-determine the target detection frames in the remaining detection frame set until the remaining detection frame set is an empty set, and generate a feature map set using each target detection frame in the target detection frame set.

10. The detection device according to claim 8, characterized in that The processor uses an inter-frame residual network to detect whether a road construction event occurs for a feature map set corresponding to each video frame. The specific operation is as follows: For each feature map in the feature map set corresponding to the Nth video frame, multiple convolution operations are performed to obtain multiple feature vectors, where N is an integer greater than or equal to 2; Determine a difference vector between each feature map of the Nth video frame and a corresponding feature map of the N-1th video frame; Each differential vector is fully connected to multiple feature vectors of the corresponding feature map, and whether a road construction event occurs is detected based on the fully connected target vector.

Images