Intelligent Sensing-Based Monitoring Methods and Systems for Highway Infrastructure

Abstract

This invention provides a method and system for monitoring highway infrastructure based on intelligent sensing. The method extracts feature maps from images and generates probability heatmaps, as well as generating noise boxes. It calculates the denoising budget for each noise box. In each iteration, it predicts the corrected set of noise boxes and their corresponding confidence levels, and determines the consumption value to be deducted from the denoising budget based on the confidence level. If the denoising budget for a noise box is exhausted or the confidence level is lower than a preset value for N consecutive iterations, the subsequent denoising iterations for the noise box are stopped. When the number of iterations reaches a preset number, noise boxes with confidence levels lower than a pruning threshold are removed. If the confidence level of a noise box is higher than a trigger threshold and its size is greater than a size threshold, an auxiliary noise box with the same denoising budget as the current noise box is generated in the neighborhood of the noise box and added to the noise box set for the next iteration. After the denoising steps are completed, the monitoring results of the highway infrastructure are obtained based on the output set.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of monitoring, and particularly relates to a highway infrastructure monitoring method and system based on intelligent perception. BACKGROUND

[0002] Good operation of highways is directly related to the safety and efficiency of transportation. Traditional highway infrastructure monitoring mainly relies on manual inspection, i.e., a dedicated staff member regularly inspects and records the facilities such as the road surface, guardrails, road signs, etc. This method not only consumes a large amount of manpower and resources, but also has low efficiency, and the detection results are easily affected by the responsibility and fatigue level of the inspection personnel. In addition, highway inspection operations themselves have high safety risks, especially on road sections with heavy traffic flow. In order to overcome the drawbacks of manual inspection, many companies and institutions have begun to use computer vision-based automated monitoring, such as the Faster R-CNN series and the YOLO series. Such methods can extract high-level semantic features through end-to-end learning on a large amount of labeled data, and have made a qualitative leap in detection accuracy and speed compared with traditional algorithms. However, these methods usually rely on some artificially designed components, such as anchor boxes and non-maximum suppression, and the parameter settings of these components are complex, and they are not well adapted to defects in highway infrastructure that have irregular shapes and large scale variations. SUMMARY

[0003] In order to improve the accuracy of monitoring, the present application proposes a highway infrastructure monitoring method based on intelligent perception, comprising:

[0004] obtaining an image of a to-be-monitored highway, and extracting a feature map of the image using a backbone network; generating a probability heat map of a highway region using the feature map; generating a noise box on the image of the to-be-monitored highway according to the probability heat map; calculating a gradient variance of a corresponding region of the noise box on the feature map, and obtaining a denoising budget of the noise box according to the gradient variance;

[0005] iterative denoising is performed within the denoising steps, and in each iteration, an image decoder predicts a corrected noise box set and a corresponding confidence based on the feature map and the noise box set of the current iteration step, and determines a consumption value to be deducted from the denoising budget according to the confidence, and if the denoising budget of the noise box is consumed or the confidence is lower than a preset value for consecutive N times, the subsequent denoising iteration of the noise box is stopped;

[0006] when the number of iterations reaches a preset number, noise boxes with a confidence lower than a pruning threshold are removed, and if the confidence of a noise box is higher than a trigger threshold and the size of the noise box is greater than a size threshold, an auxiliary noise box with the same current denoising budget as the noise box is generated in the neighborhood of the noise box and added to the noise box set of the next iteration;

[0007] When the iteration is completed for the denoising number of steps, a monitoring result of the road infrastructure is obtained based on the output bounding box set.

[0008] Preferably, the generating a probability heat map of the road region by using the feature map comprises:

[0009] The feature map extracted by the backbone network is input into a segmentation subnetwork composed of three 3x3 convolution layers;

[0010] The output of the segmentation subnetwork is reduced in channel dimension by a 1x1 convolution layer to obtain a single-channel prediction map;

[0011] The single-channel prediction map is up-sampled to the same resolution as the image of the road to be monitored, and is processed by a Sigmoid activation function, so that the value of each pixel point in the image is normalized to between 0 and 1, to obtain a probability heat map of the road region.

[0012] Preferably, the generating a noise box on the image of the road to be monitored according to the probability heat map comprises:

[0013] The values of all pixel points of the probability heat map are normalized so that the total sum of all pixel values is 1;

[0014] N times of weighted random sampling are performed from the normalized probability heat map to obtain N pixel coordinate points, where N is a preset number of noise boxes;

[0015] A noise box is generated with each pixel coordinate point obtained by sampling as the center and randomly selecting a width and a height from the size range [S min ,S max ].

[0016] Preferably, the obtaining a denoising budget of the noise box according to the gradient variance comprises:

[0017] A rectangular region covered by the noise box on the feature map is determined, a 3x3 Sobel operator is used to calculate the horizontal gradient value and the vertical gradient value of all feature pixel points in the rectangular region, and the gradient amplitude of each pixel point is calculated;

[0018] The variance V grad of the gradient amplitudes of all pixel points in the rectangular region is calculated.

[0019] The initial denoising budget B of the noise box is calculated by the following formula:

[0020] B=B min +(B max -B min )*(V grad / (V grad+V norm )); wherein, B min is a minimum de-noising budget, B max is a maximum de-noising budget, V norm is a preset variance normalization constant.

[0021] Preferably, the consumption value deducted from the de-noising budget according to the confidence level comprises:

[0022] The consumption value C deducted in this iteration is calculated by the following formula: C = p + a * (1 - s conf ); wherein, a is a coefficient, p is a fixed consumption value, s conf is the confidence level.

[0023] Preferably, the generating of the auxiliary noise frame with the same de-noising budget as the current de-noising budget of the noise frame in the neighborhood of the noise frame comprises:

[0024] Obtaining the center coordinates and size of the noise frame, and generating K new center coordinates by Gaussian distribution sampling with the center coordinates as the center and one fourth of the size as the standard deviation;

[0025] For each newly generated center coordinate point, the width and height are randomly selected from uniform distribution U(0.1*w p , 0.5*w p ) and U(0.1*h p , 0.5*h p ) respectively, K auxiliary noise frames are generated, and the K auxiliary noise frames have the same de-noising budget as the current de-noising budget of the noise frame; wherein, w p is the width of the noise frame, and h p is the height of the noise frame.

[0026] Preferably, the generating of the auxiliary noise frame with the same de-noising budget as the current de-noising budget of the noise frame in the neighborhood of the noise frame comprises:

[0027] Calculating the ratio R b of the current de-noising budget of the noise frame to the initial de-noising budget, and the ratio R s of the current size of the noise frame to the size threshold value;

[0028] The number K of auxiliary noise frames is calculated according to the formula K = ceil(C * R b * (R s - 1)), wherein C is a coefficient, and ceil() is the ceiling function;

[0029] Obtaining the center coordinates and size of the noise frame, and generating K new center coordinates by Gaussian distribution sampling with the center coordinates as the center and one fourth of the size as the standard deviation;

[0030] For each newly generated center coordinate point, the width and height are randomly selected from uniform distribution U(0.1*w p ,0.5*w p ) and U(0.1*h p ,0.5*h p ) respectively, K auxiliary noise boxes are generated, and the K auxiliary noise boxes and the current denoising budget of the noise box are the same; wherein w p is the width of the noise box, and h p is the height of the noise box.

[0031] The application also proposes a highway infrastructure monitoring system based on intelligent perception, comprising:

[0032] A feature extraction unit is configured to obtain an image of a to-be-monitored highway and extract a feature map of the image using a backbone network; generate a probability heat map of a highway region using the feature map; generate a noise box on the image of the to-be-monitored highway according to the probability heat map; calculate the gradient variance of the corresponding region of the feature map of the noise box, and obtain the denoising budget of the noise box according to the gradient variance;

[0033] An iteration unit is configured to perform iterative denoising within a denoising step number. In each iteration, an image decoder is used to predict a corrected noise box set and a corresponding confidence based on the feature map and the noise box set of the current iteration step, and a consumption value deducted from the denoising budget is determined according to the confidence. If the denoising budget of the noise box is consumed or the confidence is lower than a preset value for consecutive N times, the subsequent denoising iteration of the noise box is stopped;

[0034] An optimization unit is configured to remove a noise box with a confidence lower than a pruning threshold when the number of iterations reaches a preset number. If the confidence of the noise box is higher than a trigger threshold and the size of the noise box is greater than a size threshold, an auxiliary noise box with the same current denoising budget as the noise box is generated in the neighborhood of the noise box and added to the noise box set of the next iteration;

[0035] A result output unit is configured to obtain a monitoring result of a highway infrastructure based on the output bounding box set when the iteration is completed for the denoising step number.

[0036] Preferably, the generation of the probability heat map of the highway region using the feature map comprises:

[0037] The feature map extracted by the backbone network is input into a segmentation subnetwork composed of three 3x3 convolution layers;

[0038] The output of the segmentation subnetwork is reduced in channel dimension by a 1x1 convolution layer to obtain a single-channel prediction map;

[0039] The single-channel prediction map is upsampled to the same resolution as the image of the highway to be monitored, and then processed by the Sigmoid activation function to normalize the value of each pixel in the map to between 0 and 1, thus obtaining a probability heatmap of the highway area.

[0040] Preferably, generating a noise box on the image of the highway to be monitored based on the probability heatmap includes:

[0041] The values ​​of all pixels in the probability heatmap are normalized so that the sum of all pixel values ​​is 1.

[0042] N weighted random samplings are performed from the normalized probability heatmap to obtain N pixel coordinate points, where N is the preset number of noise boxes;

[0043] Centered on each sampled pixel coordinate point, and from the size range [S min ,S max The width and height of the frame are randomly selected from the frame to generate a noise box.

[0044] Preferably, obtaining the denoising budget of the noise box based on the gradient variance includes:

[0045] The rectangular region covered by the noise box on the feature map is determined. The horizontal and vertical gradient values ​​of all feature pixels within the rectangular region are calculated using the 3x3 Sobel operator, and the gradient magnitude of each pixel is calculated.

[0046] Calculate the variance V of the gradient magnitude of all pixels within the rectangular region. grad ;

[0047] The initial denoising budget B for the noise box is calculated using the following formula:

[0048] B=B min +(B max -B min )*(V grad / (V grad +V norm )); where B min For the minimum noise reduction budget, B max For the maximum noise reduction budget, V norm This is the preset variance normalization constant.

[0049] Preferably, determining the consumption value to be deducted from the denoising budget based on the confidence level includes:

[0050] The consumption value C to be deducted in this iteration is calculated using the following formula: C = p + α * (1 - s) conf ); where α is a coefficient, p is a fixed consumption value, and sconf is the confidence.

[0051] Preferably, the generating the auxiliary noise frame with the same current denoising budget as the noise frame in the neighborhood of the noise frame comprises:

[0052] obtaining the center coordinates and the size of the noise frame, generating K new center coordinates by sampling a Gaussian distribution with the center coordinates as the center and one fourth of the size as the standard deviation;

[0053] for each newly generated center coordinate point, randomly extracting the width and the height from the uniform distributions U(0.1*w p , 0.5*w p ) and U(0.1*h p , 0.5*h p ) respectively, generating K auxiliary noise frames, and the K auxiliary noise frames and the current denoising budget of the noise frame are the same; wherein w p is the width of the noise frame, and h p is the height of the noise frame.

[0054] Preferably, the generating the auxiliary noise frame with the same current denoising budget as the noise frame in the neighborhood of the noise frame comprises:

[0055] calculating the ratio R b of the current denoising budget of the noise frame to the initial denoising budget, and the ratio R s of the current size of the noise frame to the size threshold value;

[0056] calculating the number K of auxiliary noise frames according to the formula K=ceil(C*R b *(R s -1)), wherein C is a coefficient, and ceil() is the upward rounding;

[0057] obtaining the center coordinates and the size of the noise frame, generating K new center coordinates by sampling a Gaussian distribution with the center coordinates as the center and one fourth of the size as the standard deviation;

[0058] for each newly generated center coordinate point, randomly extracting the width and the height from the uniform distributions U(0.1*w p , 0.5*w p ) and U(0.1*h p , 0.5*h p ) respectively, generating K auxiliary noise frames, and the K auxiliary noise frames and the current denoising budget of the noise frame are the same; wherein w p is the width of the noise frame, and h p is the height of the noise frame.

[0059] The present application guides the noise frame to generate in the high probability area of the road surface and road sign through the probability heat map, and the attention of the detection is inclined to the key area from the beginning, which improves the sensitivity to the target defect. In the iteration process, the exploration of the invalid background area is abandoned, when a large area and high suspicious defect is identified, a new boundary frame can be generated in the neighborhood to strengthen it, and the detection completeness and accuracy of complex scenes are enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 Flow chart for example one;

[0061] Figure 2 Road and corresponding probability heat map;

[0062] Figure 3 Schematic diagram of initial noise frame distribution;

[0063] Figure 4 Schematic diagram of auxiliary noise frame. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0065] The terms "first", "second" and corresponding term labels in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, which is only a distinguishing way used in the description of the embodiments of the present application to describe the objects with the same attribute. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the process, method, system, product or equipment containing a series of units does not necessarily limit to those units, but can include other units not clearly listed or inherent to these processes, methods, products or equipment.

[0066] In addition, in the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more. The term "and / or" or the character " / " in the present application is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, or A / B, which can represent the three cases of A alone, A and B together, and B alone.

[0067] In specific embodiments, the present application provides a method for monitoring highway infrastructure based on intelligent perception, as shown in Figure 1 The method comprises the following steps:

[0068] Step 1: Obtain an image of a highway to be monitored, and extract a feature map of the image using a backbone network; generate a probability heat map of a highway region using the feature map; generate a noise box on the image of the highway to be monitored according to the probability heat map; calculate the gradient variance of the corresponding region of the noise box on the feature map, and obtain the noise reduction budget of the noise box according to the gradient variance;

[0069] The image of the highway is captured by a drone or a vehicle-mounted camera, etc. In one embodiment, the image not only includes the highway, but also includes other facilities of the highway, such as road signs, guardrails, etc. The feature map of the captured image is extracted using a backbone network, and a probability heat map is generated. In one embodiment, the feature map is input into a global average pooling layer, and then connected to a fully connected layer to obtain the output weight. The channels of the original feature map are weighted and summed, and the part of the feature map related to the highway infrastructure is enhanced. After upsampling and normalization processing, the probability heat map is obtained, which indicates the possibility of different positions in the image belonging to the highway infrastructure, as shown in Figure 2 In another embodiment, a simple target recognition method or image segmentation is used to segment the region containing the highway infrastructure, and then a high heat value is given to these regions to obtain the probability heat map. Then, the noise box is generated based on the heat map. In one embodiment, the image is divided into m×n grid cells of the same size, and the average probability value of all pixels in each grid cell on the probability heat map is calculated. According to the average probability value, one or more noise boxes are allocated to each grid cell, and the higher the average probability value, the more noise boxes are allocated, as shown in Figure 3 In each cell, the geometric center is taken as the reference, a small random position disturbance is combined, and a size is randomly selected from a preset size library to generate a specified number of noise boxes. The original DiffusionDet model usually uniformly or Gaussian randomly distributes on the entire image when generating the initial noise box, which is redundant for highway monitoring. The method of generating noise boxes in the present application focuses more attention on the region where the highway infrastructure is most likely to appear.

[0070] Meanwhile, the gradient variance or the Shannon entropy of the feature values in the region on the feature map corresponding to the noise box is calculated. The higher the gradient variance or the feature entropy value is, the more complex the texture information of the region is, the greater the uncertainty is, and the more likely the defect is difficult to identify. In an embodiment, the calculated feature entropy value is mapped to a preset denoising budget interval by a piecewise linear function, so as to assign an initial denoising budget related to the feature complexity of the region where the noise box is located to each noise box. The denoising budget of a complex region is larger, which can be finely adjusted, while the denoising budget of a simple region is smaller, thereby stopping faster, avoiding redundant calculation and possible adverse consequences.

[0071] Step 2: Iterative denoising is performed within the denoising steps. In each iteration, the image decoder predicts a modified noise box set and the corresponding confidence based on the feature map and the noise box set of the current iteration step, and determines the consumption value deducted from the denoising budget according to the confidence. If the denoising budget of the noise box is consumed or the confidence is lower than a preset value for consecutive N times, the subsequent denoising iteration of the noise box is stopped.

[0072] In the iteration process, when determining the consumption value of the denoising budget for each noise box, the confidence of the noise box is determined according to various ways, for example, through an MLP network, etc. The present application does not make specific limitations thereon. The greater the confidence is, the greater the credibility of the noise box containing the target is, wherein the target includes but is not limited to cracks, guardrail damage, etc. In an embodiment, the budget consumption value is composed of a basic consumption value and a variable adjustment value. The variable adjustment value is related to the confidence. In an embodiment, the greater the confidence is, the greater the possibility of the noise box containing the target is, in order to accurately locate the target position, less budget is deducted, on the contrary, if the confidence is smaller, the lower the possibility of the noise box containing the target is, more budget is deducted, avoiding repeated iteration on meaningless noise boxes.

[0073] The remaining budget of the noise box and the history of the modification amount of the last N iterations are recorded in the life cycle controller. After each iteration, the controller checks two conditions: one is whether the remaining budget is lower than a minimum value, for example, 1; the other is whether the modification amount of the last several times in the history record is less than a modification threshold. As long as any of the two conditions is met, the noise box is marked as terminated, and is removed from the active iteration set and no longer participates in subsequent denoising calculation. In an embodiment, stopping the subsequent denoising iteration of the noise box does not mean that the noise boxes are directly deleted, but the noise boxes stopped from iteration are added to the result set, and step 4 will further screen according to the confidence, class probability, IOU, etc.

[0074] Step 3, when the iteration number reaches the preset number, the noise frame with a confidence lower than the pruning threshold is removed, and if the confidence of the noise frame is higher than the trigger threshold and the size is greater than the size threshold, an auxiliary noise frame with the same current denoising budget as the noise frame is generated in the neighborhood of the noise frame and added to the noise frame set for the next iteration;

[0075] To eliminate invalid noise frames in advance. In an embodiment, when the iteration proceeds to two checkpoints, for example, 30% and 60% of the total number of iterations, the denoising is paused, and all noise frames still in active iteration are traversed. The confidence of each frame is checked. If the confidence of the frame is identified as background exceeds a high threshold, for example, 0.95, or the confidence of the frame is identified as any valid defect category is lower than a low threshold, for example, 0.05, the noise frame is determined as an invalid frame and permanently removed from the active set. The noise frame removed at this time will not enter step 4.

[0076] To find small defects or defects overlapping with each other, in an embodiment, the auxiliary noise frame is generated by quadtree splitting. Specifically, when a noise frame is detected to have a high confidence and a large size, the parent frame is divided into four equal-sized sub-quadrants along the horizontal and vertical center lines of the parent frame, and an auxiliary noise frame with a size of half of the parent frame is generated in each of the four sub-quadrants. The auxiliary noise frame inherits a denoising budget of the remaining budget of the parent frame or a certain percentage, for example, 70%, of the initial budget of the parent frame. Figure 4 An auxiliary noise frame generation diagram is shown. In an embodiment, after the auxiliary noise frame is generated, the parent noise frame is removed.

[0077] Step 4, when the iteration completes the denoising steps, the monitoring result of the road infrastructure is obtained based on the output bounding box set.

[0078] The entire iteration process is carried out under the limitation of a global maximum step number to prevent infinite loop, and ends when all the boxes are terminated or reach the global maximum step number, obtaining a preliminary detection result set. The bounding boxes in the detection result set may have overlaps, for example, multiple initial noise boxes may all converge to the same road crack. In order to obtain accurate results, further post-processing steps are performed. In an embodiment, a weighted box fusion (WBF) algorithm is used, specifically, all overlapping boxes pointing to the same object are weighted and averaged according to their confidence to calculate a new bounding box after fusion. The position, size and final confidence of the new bounding box are the contribution results of all the boxes participating in the fusion. The present application can not only be used in the detection of road surfaces, but also in the detection of other infrastructure such as guardrails, signal lights, etc. In an embodiment, before running the present application, the captured image is subjected to image segmentation or semantic segmentation to retain the part related to the road infrastructure, which can further improve the accuracy of the present application. Those skilled in the art know that other processing processes can also be included.

[0079] In an optional embodiment, the generating a probability heat map of the road region using the feature map comprises:

[0080] inputting the feature map extracted by the backbone network into a segmentation subnetwork composed of three 3x3 convolution layers;

[0081] performing channel dimension reduction on the output of the segmentation subnetwork through a 1x1 convolution layer to obtain a single-channel prediction map;

[0082] upsampling the single-channel prediction map to the same resolution as the image of the road to be monitored, and processing it through a Sigmoid activation function, so that the value of each pixel in the image is normalized to between 0 and 1, to obtain a probability heat map of the road region.

[0083] Specifically, assuming that the road image to be monitored has a resolution of 1920x1080, and the backbone network such as ResNet extracts a feature map of the road image, for example, the feature map is 60x34 and the number of channels is 512. The feature map is input into the segmentation subnetwork, which is preferably composed of three convolution layers, and its task is to identify the features related to road infrastructure from the 512 channels. A 1x1 convolution layer is then used to fuse the multi-channel features into a single-channel prediction map, obtaining a 60x34 condensed prediction map, where each point represents the possibility that the corresponding original region is road infrastructure. In order to make the small size prediction map correspond to the original image, the resolution is enlarged to 1920x1080 through upsampling operation. The entire image is sent to a Sigmoid activation function for processing, and the sigmoid function converts the value of each pixel to the interval of 0 to 1, thereby generating a probability heat map, which has the same size as the original image.Figure 1 The value of each pixel point in the figure, such as 0.95, represents the probability that the point belongs to the road infrastructure. The closer the value is to 1, the higher the possibility of being road infrastructure, and vice versa.

[0084] In an optional embodiment, the generating a noise frame on the image of the road to be monitored according to the probability heat map comprises:

[0085] The values of all pixel points in the probability heat map are normalized so that the total sum of all pixel values is 1.

[0086] N times of weighted random sampling is performed from the normalized probability heat map to obtain N pixel coordinate points, wherein N is the number of preset noise frames.

[0087] Each pixel coordinate point obtained by sampling is taken as the center, and a width and a height are randomly selected from the size range [S min ,S max ] to generate a noise frame.

[0088] In order to make the noise frame as possible as possible on the road infrastructure, the values of all pixel points in the probability heat map are normalized so that the total sum of all pixel values is 1, so that the probability of a pixel point with a larger value in the heat map being drawn is greater. For example, the value of a pixel point located at the center of a sign may be 0.2, and the value of a pixel point of a roadside green belt is only 0.01. If N is 300, 300 times of sampling is performed, and each time of sampling draws a pixel coordinate point from the figure. Since the probability value of the pixel of the road infrastructure is much higher than that of the background area, most of the 300 coordinate points drawn fall on the road infrastructure rather than the sky. After obtaining 300 high-correlation center coordinate points, a random size is assigned to each point to generate an initial noise frame. Assuming that the preset size range is 32 pixels minimum and 128 pixels maximum. For the first drawn coordinate point, a width and a height are randomly selected from the above range, for example, a width of 60 pixels and a height of 80 pixels, to obtain a noise frame of 60 by 80.

[0089] In an optional embodiment, the obtaining the denoising budget of the noise frame according to the gradient variance comprises:

[0090] A rectangular region covered by the noise frame on the feature map is determined, a 3x3 Sobel operator is used to calculate the horizontal gradient value and the vertical gradient value of all feature pixel points in the rectangular region, and the gradient amplitude value of each pixel point is calculated;

[0091] The variance V grad of the gradient amplitude values of all pixel points in the rectangular region is calculated.

[0092] The initial de-noising budget B of the noise frame is calculated by the following formula:

[0093] B = B min + (B max - B min ) * (V grad / (V grad + V norm ) ) ; wherein B min is the minimum de-noising budget, B max is the maximum de-noising budget, and V norm is a preset variance normalization constant.

[0094] A de-noising budget is assigned according to the image complexity of the area covered by the noise frame. Specifically, the rectangular area corresponding to the noise frame on the feature map is located, and the Sobel operator is used to calculate the variation intensity of each feature point in the area and obtain the gradient amplitude. The higher the gradient amplitude, the more intense the feature variation, such as the edge of an object or complex texture. The variance V grad of all gradient amplitudes in the entire area is calculated. The higher the variance value, the more intense the variation of the area, which is the manifestation of defects such as cracks; on the contrary, a very low variance value represents a single texture of the area, such as the sky in the distance. Assuming that the minimum de-noising budget B min is 10, the maximum de-noising budget B max is 100, and the normalization constant V norm is set to 100. A noise frame is located on a flat and intact road surface, and the gradient variation of the corresponding feature area is small. The calculated variance V grad may be only 0.5, and the initial de-noising budget B obtained is about 10.45. Another noise frame covers a net-like crack, and the gradient variation of the feature area is large. The calculated variance V grad is 400, and the initial de-noising budget B is 82.

[0095] In an optional embodiment, the consumption value deducted from the de-noising budget according to the confidence includes:

[0096] The consumption value C to be deducted in this iteration is calculated by the following formula: C = p + a * (1 - s conf ) ; wherein a is a coefficient, p is a fixed consumption value, and s conf is the confidence.

[0097] Specifically, assume that in an iteration, there are two noise boxes. Box A is a crack, and the calculated confidence sconf is 0.95. Box B is located on a complex background but defect-free road surface, and it is difficult to accurately determine whether it is a defect or not, and sconf is 0.2. If the fixed consumption value p is set to 1, and the coefficient a is 100, then the consumption value C of box A is 6. And the consumption value of box B is 81. Box A with high confidence only pays a very low iteration cost, while box B with low confidence pays a high cost. If the initial denoising budget of box A and box B is 200. After the current iteration, the remaining budget of box A becomes 194, and the remaining budget of box B decreases to 119. Like box B, low-quality and uncertain boxes will quickly run out of all budgets and be eliminated early if their confidence does not improve quickly in the next two or three iterations. And like box A, high-quality boxes can maintain iteration at a very low cost, which gives them enough budget and opportunity to continue fine-tuning in subsequent iterations.

[0098] In an optional embodiment, the generating of the K auxiliary noise boxes in the neighborhood of the noise box includes:

[0099] Obtaining the center coordinates and size of the noise box, and generating K new center coordinates by sampling a Gaussian distribution with the center coordinates as the center and one quarter of the size as the standard deviation;

[0100] For each newly generated center coordinate point, randomly extract the width and height from the uniform distributions U(0.1*w p ,0.5*w p ) and U(0.1*h p ,0.5*h p ) respectively, and generate K auxiliary noise boxes, and the K auxiliary noise boxes and the current denoising budget of the noise box are the same; wherein w p is the width of the noise box, and h p is the height of the noise box.

[0101] Some large defect regions can include multiple defects, for example, a large noise frame with a size of 200x80 pixels includes multiple defects such as road cracks, sign damage, etc. A K number of new center points are generated based on the center of the frame. The positions of the new center points follow a Gaussian distribution, that is, most of them are close to the original center, and a few of them are far away, and the degree of dispersion is determined by one fourth of the size of the original frame. K number of auxiliary frames with smaller sizes are generated using the new center points, the width of which is randomly selected between 20 and 100 pixels, and the height is randomly selected between 8 and 40 pixels, thereby forming a set of sub-region detection frames. All newly generated auxiliary frames inherit the current denoising budget of the parent noise frame. For example, if the 200x80 pixel noise frame has 75 denoising budgets remaining when triggering the generation of auxiliary noise frames, all K newly generated small-size auxiliary frames will start their respective denoising iterations with a budget of 75.

[0102] In an optional embodiment, the generating K number of auxiliary noise frames in the neighborhood of the noise frame with the same current denoising budget as the noise frame comprises:

[0103] calculating the ratio R of the current denoising budget of the noise frame to the initial denoising budget b , and the ratio R of the current size of the noise frame to the size threshold s ;

[0104] calculating the number K of auxiliary noise frames according to the formula K = ceil(C*R b *(R s -1)), wherein C is a coefficient, and ceil() is a rounding up function;

[0105] obtaining the center coordinates and size of the noise frame, and generating K number of new center coordinates by Gaussian distribution sampling with the center coordinates as the center and one fourth of the size as the standard deviation;

[0106] for each newly generated center coordinate point, randomly selecting a width and a height from uniform distributions U(0.1*w p , 0.5*w p ) and U(0.1*h p , 0.5*h p ) respectively, generating K number of auxiliary noise frames, and the K number of auxiliary noise frames have the same current denoising budget as the noise frame; wherein w p is the width of the noise frame, and h p is the height of the noise frame.

[0107] In this embodiment, the number of auxiliary noise blocks is related to the de-noising budget and the size of the noise block. If the de-noising budget is small, it is not meaningful to generate too many auxiliary noise blocks. Similarly, if the size of the noise block is small, generating too many auxiliary noise blocks can even introduce errors. Assuming that when the iteration reaches a certain stage, the size of the noise block is 3 times the preset size threshold, and the remaining de-noising budget is 80. The budget ratio is 0.8 and the size ratio is 3. Assuming that the coefficient is 2, 4 new auxiliary noise blocks need to be generated according to the above formula and the upward rounding.

[0108] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, another division mode can be used. For example, a plurality of units or components can be combined or integrated into another system, or some characteristic data can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0109] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0110] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0111] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A method for monitoring highway infrastructure based on intelligent perception, characterized in that, The method comprises the following steps: obtaining an image of a to-be-monitored road, and extracting a feature map of the image by using a backbone network; generating a probability heat map of a road region by using the feature map; generating a noise box on the image of the to-be-monitored road according to the probability heat map; calculating a gradient variance of a corresponding region of the noise box on the feature map, and obtaining a denoising budget of the noise box according to the gradient variance; iterative denoising is performed within a denoising step number, in each iteration, a modified noise box set and a corresponding confidence are predicted by using an image decoder based on the feature map and the noise box set of the current iteration step, a consumption value deducted from the denoising budget is determined according to the confidence, and if the denoising budget of the noise box is consumed or the confidence is lower than a preset value for consecutive N times, the subsequent denoising iteration of the noise box is stopped; when the iteration number reaches a preset number, noise boxes with a confidence lower than a pruning threshold are removed, if the confidence of a noise box is higher than a trigger threshold and the size of the noise box is greater than a size threshold, an auxiliary noise box with the same current denoising budget as the noise box is generated in the neighborhood of the noise box, and is added to the noise box set of the next iteration; when the iteration is completed for the denoising step number, a monitoring result of a road infrastructure is obtained based on an output bounding box set; the method further comprises the following steps: normalizing the values of all pixel points of the probability heat map, so that the total sum of all pixel values is 1; N times of weighted random sampling are performed on the normalized probability heat map to obtain N pixel coordinate points, wherein N is a preset number of noise boxes; With each sample pixel coordinate point as the center, and randomly extracting the width and height from the size range [S min ,S max ] to generate a noise frame; the method further comprises the following steps: determining a rectangular region covered by the noise box on the feature map, calculating the horizontal gradient value and the vertical gradient value of all feature pixel points in the rectangular region by using a 3x3 Sobel operator, and calculating the gradient amplitude value of each pixel point; calculating the variance V of the gradient amplitudes of all the pixels in the rectangular region grad ; the initial denoising budget B of the noise box is calculated by the following formula: B = B min + (B max - B min ) * (V grad / V grad + V norm )) ; wherein B min is a minimum de-noising budget, B max is a maximum de-noising budget, and V norm is a preset variance normalization constant. the method further comprises the following steps: The consumption value C to be deducted in this iteration is calculated by the following formula: C = p + a * (1 - s conf ); wherein a is a coefficient, p is a fixed consumption value, and s conf is the confidence level. when the iteration is completed for the denoising step number, a monitoring result of a road infrastructure is obtained based on an output bounding box set; the entire iteration process is performed under the limitation of a global maximum step number, and the process is ended when all the boxes are terminated or the global maximum step number is reached, and a preliminary detection result set is obtained, all overlapping boxes pointing to the same object are weighted and averaged according to their confidences, and a fused new bounding box is calculated.

2. The method of claim 1, wherein, the method further comprises the following steps: the feature map extracted by the backbone network is input into a segmentation subnetwork composed of three 3x3 convolution layers; a 1x1 convolution layer is used to reduce the channel dimension of the output of the segmentation subnetwork to obtain a single-channel prediction map; the single-channel prediction map is upsampled to the same resolution as the image of the to-be-monitored road, and is processed by a Sigmoid activation function, so that the value of each pixel point in the image is normalized to between 0 and 1, and a probability heat map of the road region is obtained.

3. The method of claim 1, wherein, The auxiliary noise frame generated in the neighborhood of the noise frame and having the same current denoising budget as the noise frame comprises: obtaining the center coordinates and size of the noise frame, and generating K new center coordinates by sampling a Gaussian distribution with the center coordinates as the center and one quarter of the size as the standard deviation; For each newly generated center coordinate point, width and height are randomly selected from uniform distribution U(0.1*w p ,0.5*w p ) and U(0.1*h p ,0.5*h p ) respectively, K auxiliary noise boxes are generated, and the K auxiliary noise boxes and the current denoising budget of the noise box are the same; wherein w p is the width of the noise box, h p is the height of the noise box.

4. The method of claim 1, wherein, The auxiliary noise frame generated in the neighborhood of the noise frame and having the same current denoising budget as the noise frame comprises: a ratio R of a current de-noising budget of the noise frame to an initial de-noising budget b and a ratio R of a current size of the noise frame to a size threshold s ; The number K of auxiliary noise blocks is calculated according to the formula K = ceil(C*R b *(R s -1)), wherein C is a coefficient, and ceil() is a rounding up function. obtaining the center coordinates and size of the noise frame, and generating K new center coordinates by sampling a Gaussian distribution with the center coordinates as the center and one quarter of the size as the standard deviation; For each newly generated center coordinate point, width and height are randomly selected from uniform distribution U(0.1*w p ,0.5*w p ) and U(0.1*h p ,0.5*h p ) respectively, K auxiliary noise boxes are generated, and the K auxiliary noise boxes and the current denoising budget of the noise box are the same; wherein w p is the width of the noise box, h p is the height of the noise box.

5. A smart sensing based highway infrastructure monitoring system characterized in that, comprise: a feature extraction unit configured to obtain an image of a to-be-monitored road and extract a feature map of the image by using a backbone network; generating a probability heat map of a road region by using the feature map; generating a noise frame on the image of the to-be-monitored road according to the probability heat map; calculating a gradient variance of a corresponding region of the noise frame on the feature map, and obtaining a denoising budget of the noise frame according to the gradient variance; an iteration unit configured to perform iterative denoising within a denoising step number, in each iteration, predict a corrected noise frame set and a corresponding confidence by using an image decoder based on the feature map and the noise frame set of the current iteration step, determine a consumption value deducted from the denoising budget according to the confidence, and stop subsequent denoising iterations of the noise frame if the denoising budget of the noise frame is consumed or the confidence is lower than a preset value for consecutive N times; an optimization unit configured to, when the number of iterations reaches a preset number, remove a noise frame whose confidence is lower than a pruning threshold, generate an auxiliary noise frame in the neighborhood of the noise frame and having the same current denoising budget as the noise frame if the confidence of the noise frame is higher than a trigger threshold and the size of the noise frame is greater than a size threshold, and add the auxiliary noise frame to the noise frame set of the next iteration; a result output unit configured to, when the iteration is completed for the denoising step number, obtain a monitoring result of road infrastructure based on the output bounding box set; The noise frame generated on the image of the to-be-monitored road according to the probability heat map comprises: normalizing the values of all pixel points of the probability heat map so that the total sum of all pixel values is 1; performing N times of weighted random sampling from the normalized probability heat map to obtain N pixel coordinate points, wherein N is a preset number of noise frames; With each sample pixel coordinate point as the center, and randomly extracting width and height from the size range [S min ,S max ] to generate a noise frame; The denoising budget of the noise frame obtained according to the gradient variance comprises: determining a rectangular region covered by the noise frame on the feature map, calculating the horizontal gradient value and the vertical gradient value of all feature pixel points in the rectangular region by using a 3x3 Sobel operator, and calculating the gradient amplitude value of each pixel point; calculating the variance V of the gradient amplitudes of all the pixels in the rectangular region grad ; the initial denoising budget B of the noise frame is calculated by the following formula: B = B min + (B max - B min ) * (V grad / V grad + V norm )) ; wherein B min is a minimum de-noising budget, B max is a maximum de-noising budget, V norm is a preset variance normalization constant; The consumption value deducted from the denoising budget according to the confidence comprises: The consumption value C to be deducted in this iteration is calculated by the following formula: C = p + a * (1 - s conf ); wherein a is a coefficient, p is a fixed consumption value, and s conf is the confidence level. When the iteration is completed for the denoising step number, the monitoring result of road infrastructure is obtained based on the output bounding box set, which comprises: The whole iteration process is carried out under the limitation of a global maximum step number, and ends when all the boxes are terminated or reach the global maximum step number, thereby obtaining a preliminary detection result set, and all the overlapping boxes pointing to the same object are weighted and averaged according to the confidence thereof, so as to calculate a new fused boundary box.

6. The system of claim 5, wherein, The generating of the probability heat map of the road region by using the feature map comprises: inputting the feature map extracted by the backbone network into a segmentation subnetwork composed of three 3x3 convolution layers; performing channel dimension reduction on the output of the segmentation subnetwork through a 1x1 convolution layer to obtain a single-channel prediction map; up-sampling the single-channel prediction map to the same resolution as the image of the road to be monitored, and processing through a Sigmoid activation function, so that the value of each pixel point in the image is normalized to between 0 and 1, thereby obtaining the probability heat map of the road region.

7. The system of claim 5, wherein, The generating of the noise box on the image of the road to be monitored according to the probability heat map comprises: performing normalization processing on the values of all pixel points of the probability heat map, so that the total sum of all pixel values is 1; performing N times of weighted random sampling from the normalized probability heat map to obtain N pixel coordinate points, wherein N is the number of preset noise boxes. With each sample pixel coordinate point as the center, and randomly extracting width and height from the size range [S min ,S max ] to generate a noise frame.

Images