Night road pit identification method and system based on low-illumination enhancement and parallax calculation
By combining sensing systems and image enhancement technology with stereo matching and parallax calculation, the problem of image quality degradation in low-light nighttime pit and pond identification has been solved, achieving efficient and accurate pit and pond detection and three-dimensional parameter measurement, thus improving detection efficiency and security.
Patent Information
- Application Number
- CN202511643162.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-11
- Publication Date
- 2026-07-21
- Estimated Expiration
- 2045-11-11
Smart Images

Figure CN121305415B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent traffic monitoring and road maintenance technology, specifically to a method and system for identifying potholes on highways at night using low-light enhancement and parallax calculation. Background Technology
[0002] Existing drone-based pit and pond identification technologies exhibit significant performance degradation at night or in low-light environments. Under insufficient illumination, acquired images suffer from high noise, low contrast, and blurred details, making it difficult for traditional visual algorithms to accurately extract pit and pond features. Furthermore, due to the lack of effective depth information acquisition methods, existing methods have limited accuracy in measuring the three-dimensional dimensions of pits and ponds, especially their depth.
[0003] Currently, nighttime road inspections mainly rely on two methods: one is manual inspection, but this is limited by human vision, resulting in low inspection efficiency and safety risks; the other is specialized inspections on closed roads, which require traffic interruption, are costly to implement, and disrupt normal road traffic. Furthermore, existing technologies lack adaptability to environmental conditions, struggling to maintain stable inspection results under different weather conditions and flight altitudes, and their robustness in complex nighttime environments such as fog and humidity urgently needs improvement. Summary of the Invention
[0004] Based on the shortcomings of the prior art described above, the purpose of this invention is to provide a method and system for identifying potholes on highways at night using low-light enhancement and parallax calculation, so as to solve the above-mentioned technical problems.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for identifying potholes on highways at night using low-light enhancement and parallax calculation, comprising:
[0006] S1: Collect nighttime highway environmental data through a sensor system, calculate meteorological indices to determine laser compensation power, and acquire stereo images to generate environmental metadata.
[0007] S2: Based on environmental metadata, an enhancement strategy is determined according to preset rules, and the improved Retinex algorithm is used to enhance the stereo image pair to obtain the enhanced image pair;
[0008] S3: Perform stereo matching based on enhanced image pairs, calculate disparity by fusing the pose data of the UAV, and generate a scene depth map;
[0009] S4: Based on the conditional bi-branch recognition model, feature fusion is performed on the enhanced image pair and the depth map to detect and verify the depth of the pit area and obtain the pit area mask.
[0010] S5: Based on the pit area mask, depth map and UAV pose data, calculate the three-dimensional geometric parameters and center position coordinates of the pit, and generate a visual inspection report.
[0011] The present invention is further configured such that S1 includes:
[0012] The drone uses multi-dimensional sensors to collect nighttime highway environmental data, which includes ambient light intensity, flight altitude, visibility, and humidity data.
[0013] A meteorological index characterizing weather conditions is generated by weighted fusion of visibility and humidity data. The meteorological index is positively correlated with visibility and negatively correlated with humidity.
[0014] The present invention is further configured to calculate the basic lighting power based on the ambient light intensity and the preset maximum value of basic lighting, according to the relationship that the weaker the ambient light intensity, the greater the basic lighting power.
[0015] Based on the meteorological index and the preset weather compensation coefficient, the weather compensation power adjustment is calculated according to the relationship that the lower the meteorological index, the greater the weather compensation power adjustment.
[0016] Based on the flight altitude and the preset altitude compensation benchmark, the altitude compensation power adjustment is calculated according to the principle that the higher the flight altitude, the greater the altitude compensation power adjustment, and the adjustment amount increases non-linearly with the increase of altitude.
[0017] The laser compensation power is determined by superimposing the basic lighting power, weather compensation power adjustment, and altitude compensation power adjustment.
[0018] The laser-assisted illumination device is controlled by laser compensation power to drive the binocular optical system and simultaneously acquire stereo image pairs.
[0019] Based on ambient light intensity, flight altitude, weather index, and UAV pose parameters, environmental metadata corresponding to the stereo image is generated.
[0020] The present invention is further configured such that S2 includes:
[0021] The input stereo image is decomposed into reflection and illumination components using a convolutional neural network.
[0022] The illumination components are logarithmically transformed and then optimized through a mapping network, and finally the optimized illumination components are obtained through an inverse exponential transform.
[0023] Edge features of the reflection component are extracted using an edge detection operator, and an edge attention map is generated using a convolutional neural network and activation function.
[0024] Based on the edge attention map, the reflection component is enhanced by convolution to obtain the enhanced reflection component.
[0025] The enhanced image is obtained by multiplying the optimized illumination component with the enhanced reflection component.
[0026] The present invention is further configured to determine an enhancement strategy based on meteorological indices and flight altitude in environmental metadata, and to fine-tune the image enhancement process according to the enhancement strategy;
[0027] The rules for determining the enhancement strategy are as follows: when the meteorological index is below the first threshold, the dense fog enhancement strategy is activated, and the reflection component and illumination component are adjusted by introducing atmospheric transmittance estimation during the image decomposition process; when the flight altitude is above the second threshold, the high-altitude enhancement strategy is activated, and the intensity of the edge enhancement convolution kernel is reduced; otherwise, the standard enhancement strategy is activated.
[0028] When both conditions are met simultaneously—the meteorological index being below the first threshold and the flight altitude being above the second threshold—enhancement strategies are selected according to a preset priority order, with the dense fog enhancement strategy having higher priority than the high-altitude enhancement strategy.
[0029] The present invention is further configured such that S3 includes:
[0030] The SIFT feature detection algorithm is used to extract scale-invariant feature points of the enhanced image pairs and compute descriptors;
[0031] The feature point correspondence between the left and right images in the enhanced image pair is established by a feature matcher, and the RANSAC algorithm is used to remove mismatched points.
[0032] A semi-global stereo matching algorithm is used to calculate the initial disparity map within the disparity search range;
[0033] By performing left-right consistency checks on the initial disparity map, identifying and removing occluded areas and mismatched pixels, and then using weighted least squares method to optimize disparity filling, an optimized stereo disparity map is obtained.
[0034] The present invention is further configured such that, after obtaining the optimized stereo disparity map, motion disparity fusion is performed:
[0035] The inter-frame rotation matrix and translation vector of the UAV are obtained by the inertial measurement unit, and the motion parallax is calculated by combining it with the scene depth map generated in the previous moment.
[0036] Texture richness index is obtained by calculating the gray-level variance and gradient magnitude of the image region, and the fusion weight of stereo parallax and motion parallax is dynamically adjusted based on the texture richness index.
[0037] Stereo parallax and motion parallax are weighted and fused according to fusion weights to generate a fused parallax map;
[0038] Based on preset camera focal length and baseline distance parameters, the fused disparity map is converted into the final scene depth map using triangulation.
[0039] The present invention is further configured such that S4 includes:
[0040] The brightness features of the enhanced image and the depth features of the depth map are extracted separately using a dual-branch convolutional neural network.
[0041] The brightness features, depth features, and conditional vectors are fused to generate fused features;
[0042] Based on the fusion features, candidate regions for pits and ponds are generated and the probability of each candidate region for pits and ponds is output. The depth extreme difference of each candidate region for pits and ponds is calculated.
[0043] The depth extreme difference is compared with the preset depth mutation threshold to verify the candidate pit and pond regions and generate pit and pond region masks.
[0044] Based on the probability of pits and ponds, the difference in depth extreme values, and the reliability coefficient corresponding to the enhancement strategy, the confidence score of the mask for each pit and pond region is calculated.
[0045] The present invention is further configured such that S5 includes:
[0046] Based on the UAV pose data and preset camera parameters, the pixel coordinates of the pit area mask are converted into the world coordinate system.
[0047] Calculate the average of all world coordinates corresponding to the pit area mask to obtain the center coordinates of the pit.
[0048] The minimum bounding rectangle of the pothole area mask in the road surface projection is calculated based on the principal component analysis method to obtain the pothole length and pothole width.
[0049] Calculate the depth of the pit based on the statistical distribution characteristics of the depth values in the depth map;
[0050] By combining the center location coordinates, pit length, pit width, pit depth, and confidence score, a three-dimensional geometric parameter of the pit is generated.
[0051] A visual inspection report is generated by annotating the bounding boxes and depth information of pits and ponds on the enhanced image.
[0052] This invention also provides a nighttime highway pothole identification system based on low-light enhancement and parallax calculation, the system comprising:
[0053] Environmental perception and acquisition module: Collects nighttime highway environmental data through a sensing system, calculates meteorological indices to determine laser compensation power, and acquires stereo images to generate environmental metadata;
[0054] Image enhancement module: Based on environmental metadata, the enhancement strategy is determined according to preset rules, and the improved Retinex algorithm is used to enhance the stereo image pairs to obtain enhanced image pairs;
[0055] The disparity calculation module performs stereo matching based on enhanced image pairs, calculates disparity by fusing the pose data of the UAV, and generates a scene depth map.
[0056] The pit and pond identification module: Based on the conditional bi-branch identification model, it performs feature fusion on the enhanced image pair and the depth map to detect and verify the depth of the pit and pond area, and obtain the pit and pond area mask;
[0057] Results generation module: Based on the pit area mask, depth map and UAV pose data, calculate the three-dimensional geometric parameters and center position coordinates of the pit, and generate a visual inspection report.
[0058] This invention provides a method and system for identifying potholes on highways at night using low-light enhancement and disparity calculation. The method comprises the following steps: S1: Collecting nighttime highway environmental data through a sensing system, calculating meteorological indices to determine laser compensation power, and obtaining stereo image pairs to generate environmental metadata; S2: Determining an enhancement strategy based on the environmental metadata according to preset rules, and enhancing the stereo image pairs using an improved Retinex algorithm to obtain enhanced image pairs; S3: Performing stereo matching based on the enhanced image pairs, fusing UAV pose data to calculate disparity, and generating a scene depth map; S4: Based on a conditional bi-branch recognition model, performing feature fusion on the enhanced image pairs and depth map to detect and verify the depth of pothole areas, obtaining a pothole area mask; S5: Based on the pothole area mask, depth map, and UAV pose data, calculating the three-dimensional geometric parameters and center coordinates of the potholes, and generating a visual detection report. The beneficial effects include:
[0059] Strong environmental adaptability: Through adaptive lighting and image enhancement technology, it can maintain stable pit and pond recognition capabilities in various low-light environments and complex weather conditions, effectively overcoming the problem of image quality degradation caused by insufficient nighttime lighting.
[0060] High measurement accuracy: Based on a depth calculation scheme that integrates stereo vision and motion estimation, it can accurately obtain the three-dimensional geometric parameters of potholes, providing reliable dimensional data support for road maintenance.
[0061] High inspection efficiency: It enables automated nighttime inspections under normal traffic conditions, avoiding road closure operations, significantly improving inspection efficiency and reducing the need for manual intervention.
[0062] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description
[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:
[0064] Figure 1 A flowchart illustrating a nighttime road pothole identification method based on low-light enhancement and parallax calculation, as shown in an exemplary embodiment of the present invention;
[0065] Figure 2 This is a schematic diagram illustrating the structure of a nighttime highway pothole identification system with low-light enhancement and parallax calculation, as an exemplary embodiment of the present invention. Detailed Implementation
[0066] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.
[0067] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0068] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.
[0069] Example 1:
[0070] A method for identifying potholes on highways at night using low-light enhancement and parallax calculation, such as... Figure 1 As shown, it includes:
[0071] S1: Collect nighttime highway environmental data through a sensor system, calculate meteorological indices to determine laser compensation power, and acquire stereo images to generate environmental metadata.
[0072] S2: Based on environmental metadata, an enhancement strategy is determined according to preset rules, and the improved Retinex algorithm is used to enhance the stereo image pair to obtain the enhanced image pair;
[0073] S3: Perform stereo matching based on enhanced image pairs, calculate disparity by fusing the pose data of the UAV, and generate a scene depth map;
[0074] S4: Based on the conditional bi-branch recognition model, feature fusion is performed on the enhanced image pair and the depth map to detect and verify the depth of the pit area and obtain the pit area mask.
[0075] S5: Based on the pit area mask, depth map and UAV pose data, calculate the three-dimensional geometric parameters and center position coordinates of the pit, and generate a visual inspection report.
[0076] The present invention is further configured such that S1 includes:
[0077] The drone uses multi-dimensional sensors to collect nighttime highway environmental data, which includes ambient light intensity, flight altitude, visibility, and humidity data.
[0078] A meteorological index characterizing weather conditions is generated through weighted fusion of visibility and humidity data. The meteorological index is positively correlated with visibility and negatively correlated with humidity. Specifically, the UAV collects road environment data using onboard multi-source sensors. An ambient light sensor monitors ambient light intensity in real time, providing a basis for laser power compensation; flight altitude data is provided by an onboard altitude sensor and used for power compensation and image enhancement strategy selection; visibility and humidity data are acquired through a meteorological sensing unit, jointly characterizing the impact of weather conditions on image quality. In the meteorological index calculation, visibility and humidity data are normalized to ensure values are within the range of 0 to 1. Then, a comprehensive meteorological index is generated through weighted fusion. The comprehensive meteorological index is positively correlated with visibility and negatively correlated with humidity, with visibility having a higher weight (default setting of 0.7). Specific parameter settings are adaptively adjusted according to the actual application scenario. The calculated sum is the meteorological index, used to quantify the degree of influence of environmental conditions on image quality and provide a decision-making basis for subsequent laser power compensation and image enhancement strategy selection.
[0079] The present invention is further configured to calculate the basic lighting power based on the ambient light intensity and the preset maximum value of basic lighting, according to the relationship that the weaker the ambient light intensity, the greater the basic lighting power.
[0080] Based on the meteorological index and the preset weather compensation coefficient, the weather compensation power adjustment is calculated according to the relationship that the lower the meteorological index, the greater the weather compensation power adjustment.
[0081] Based on the flight altitude and the preset altitude compensation benchmark, the altitude compensation power adjustment is calculated according to the principle that the higher the flight altitude, the greater the altitude compensation power adjustment, and the adjustment amount increases non-linearly with the increase of altitude.
[0082] The laser compensation power is determined by superimposing the basic lighting power, weather compensation power adjustment, and altitude compensation power adjustment.
[0083] The laser-assisted illumination device is controlled by laser compensation power to drive the binocular optical system and simultaneously acquire stereo image pairs.
[0084] Based on ambient light intensity, flight altitude, weather index, and UAV pose parameters, environmental metadata corresponding to the stereo image is generated. Specifically, in the laser power compensation calculation process, the basic lighting power is first calculated based on the ambient light intensity value. Basic lighting power is negatively correlated with ambient light intensity; the lower the ambient light intensity, the higher the basic lighting power. In the specific calculation, the ambient light intensity is compared with the preset maximum basic lighting value, and the current basic lighting power value is determined through linear interpolation. The default maximum basic lighting value is set to 1000 milliwatts to ensure sufficient illumination in completely dark environments. Next, the weather compensation power adjustment is calculated based on the weather index. The weather index is negatively correlated with the weather compensation power adjustment; the lower the weather index, the greater the weather compensation power adjustment. A preset weather compensation coefficient is used in the calculation, with a default setting of 500 milliwatts to ensure sufficient power compensation under adverse weather conditions. Simultaneously, the altitude compensation power adjustment is calculated. The altitude compensation power adjustment is positively correlated with flight altitude and exhibits a non-linear growth characteristic with increasing altitude. The default calculation uses a squared growth relationship with a height compensation baseline of 30 meters. The nonlinear relationship is used to accurately compensate for the attenuation of light as the propagation distance increases. The three power components are added together to obtain the final laser compensation power. This final laser compensation power value is used to control the operation of the laser-assisted illumination device and simultaneously drive the binocular optical system to complete the synchronous acquisition of stereo image pairs. Finally, ambient light intensity, flight altitude, weather index, and UAV pose parameters are integrated to generate an environmental metadata data set that strictly corresponds to each stereo image pair.
[0085] The present invention is further configured such that S2 includes:
[0086] The input stereo image is decomposed into reflection and illumination components using a convolutional neural network.
[0087] The illumination components are logarithmically transformed and then optimized through a mapping network, and finally the optimized illumination components are obtained through an inverse exponential transform.
[0088] Edge features of the reflection component are extracted using an edge detection operator, and an edge attention map is generated using a convolutional neural network and activation function.
[0089] Based on the edge attention map, the reflection component is enhanced by convolution to obtain the enhanced reflection component.
[0090] The optimized illumination component is multiplied by the enhanced reflection component to obtain the enhanced image. Specifically, in the image enhancement process, the input stereo image is first decomposed into reflection and illumination components using a convolutional neural network (CNN). The CNN is trained by minimizing the weighted sum of reconstruction loss, illumination smoothing loss, and reflection sparsity loss, where the weight of the reflection sparsity loss is set to 10 times that of the illumination smoothing loss by default to better preserve the details of pit edges. A logarithmic transformation is then applied to the decomposed illumination component, using a very small constant to avoid zero-value problems; the default value is 1 / 10,000. After the transformation, it is optimized using a mapping network with an encoder-decoder structure, trained to make the optimized illumination component approximate a uniform distribution. Subsequently, an inverse exponential transformation is used to obtain the optimized illumination component, with the ideal illumination distribution set to a uniform distribution with a mean of 0.5 by default; the default value is determined based on visual perception research. The Sobel edge detection operator is used to extract edge features of the reflection component, and an edge attention map is generated using a CNN and a Sigmoid activation function. In the attention map, the values in edge regions are close to 1, and the values in non-edge regions are close to 0. Based on the edge attention map, a Laplacian convolution kernel is used to enhance the edge of the reflection component. The enhancement strength is set to 0.3 by default, striking a balance between preserving details and controlling noise. The enhanced reflection component is multiplied by the optimized illumination component to obtain the final enhanced image. A similar approach can be implemented as follows: To adapt to the identification of potholes on highways at night, considering the characteristics of highway nighttime images such as headlight glare, road shadows, and blurred pothole edges, an attention mechanism and a multi-scale fusion module are added to the traditional algorithm. Multi-scale Retinex transform is used to separate the illumination and reflection components of the image; the Retinex-Net algorithm then performs stereo image processing... Decomposed into reflection components (The inherent color / texture characteristics of highways, unaffected by lighting) and lighting components (Ambient light intensity): The core issue in low-light nighttime images is the illumination component. Too dark and unevenly distributed, resulting in reflective components It is masked. The goal of the Retinex-Net algorithm is to... Separate from and By adjusting And retain To obtain an enhanced image ,in, To enhance the image; This is the optimized illumination component.
[0091] The present invention is further configured to determine an enhancement strategy based on meteorological indices and flight altitude in environmental metadata, and to fine-tune the image enhancement process according to the enhancement strategy;
[0092] The rules for determining the enhancement strategy are as follows: when the meteorological index is below the first threshold, the dense fog enhancement strategy is activated, and the reflection component and illumination component are adjusted by introducing atmospheric transmittance estimation during the image decomposition process; when the flight altitude is above the second threshold, the high-altitude enhancement strategy is activated, and the intensity of the edge enhancement convolution kernel is reduced; otherwise, the standard enhancement strategy is activated.
[0093] When both conditions are met simultaneously—meteorological index below the first threshold and flight altitude above the second threshold—enhancement strategies are selected according to a preset priority order, with dense fog enhancement strategy having higher priority than high-altitude enhancement strategy. Specifically, during the enhancement strategy selection and adaptive adjustment process, the system determines the applicable enhancement strategy based on meteorological index and flight altitude parameters in the environmental metadata. When the meteorological index is below the first threshold, the dense fog enhancement strategy is activated. The first threshold is set to 0.3 by default, a default value based on the critical point of fog impact derived from meteorological data analysis. When the flight altitude is above the second threshold, the high-altitude enhancement strategy is activated. The second threshold is set to 30 meters by default, due to the fact that image details begin to be significantly lost above this altitude. If none of the above conditions are met, the standard enhancement strategy is activated. In the dense fog enhancement strategy, atmospheric transmittance estimation is introduced during image decomposition to simultaneously adjust the reflection and illumination components. Atmospheric transmittance is calculated using a dark channel prior method, estimating the fog concentration by finding the minimum brightness value in a local area of the image, thereby simultaneously optimizing the reflection and illumination components. In the high-altitude enhancement strategy, the intensity of the edge enhancement convolution kernel is reduced to balance the needs of detail enhancement and noise suppression. By default, the convolution kernel intensity is adjusted to 50% of the standard value to suppress noise amplification in high-altitude images. In the standard enhancement strategy, the complete edge enhancement and detail restoration process is maintained, using the default execution procedure. When both the meteorological index and flight altitude meet the trigger conditions for both dense fog and high-altitude enhancement strategies, the system selects the strategy according to a preset priority order, with the dense fog enhancement strategy having higher priority than the high-altitude enhancement strategy. This priority setting is based on the fact that fog and haze are more detrimental to the overall image quality than flight altitude.
[0094] The present invention is further configured such that S3 includes:
[0095] The SIFT feature detection algorithm is used to extract scale-invariant feature points of the enhanced image pairs and compute descriptors;
[0096] The feature point correspondence between the left and right images in the enhanced image pair is established by a feature matcher, and the RANSAC algorithm is used to remove mismatched points.
[0097] A semi-global stereo matching algorithm is used to calculate the initial disparity map within the disparity search range;
[0098] By performing left-right consistency checks on the initial disparity map, occluded regions and mismatched pixels are identified and removed. Weighted least squares is then used for disparity filling optimization to obtain an optimized stereo disparity map. Specifically, in the stereo matching and disparity optimization process, the SIFT feature detection algorithm is first used to extract scale-invariant feature points from the enhanced stereo image pairs and calculate 128-dimensional descriptors. The FLANN fast approximation matcher is used to establish the correspondence between feature points of the left and right images, and the RANSAC algorithm is used to remove mismatched points. The default inlier threshold for the RANSAC algorithm is set to 3.0 pixels, which is used to balance matching accuracy and mismatch removal intensity. A semi-global stereo matching algorithm is used to calculate the initial disparity map within a preset disparity search range. The default disparity search range is set to 0 to 256 pixels, determined based on the maximum possible depth difference of potholes in common highway scenes. By performing left-right consistency checks on the initial disparity map, occluded regions and mismatched pixels are identified and removed. The tolerance threshold for consistency checks is set to 1 pixel by default to detect most erroneous matching points. Finally, weighted least squares method is used to optimize the disparity map, with the smoothing term weight set to 10.0 by default to achieve a good balance between maintaining the smoothness of the disparity map and preserving edge details. After the above processing, the optimized stereo disparity map is obtained for subsequent depth calculation.
[0099] The present invention is further configured such that, after obtaining the optimized stereo disparity map, motion disparity fusion is performed:
[0100] The inter-frame rotation matrix and translation vector of the UAV are obtained by the inertial measurement unit, and the motion parallax is calculated by combining it with the scene depth map generated in the previous moment.
[0101] Texture richness index is obtained by calculating the gray-level variance and gradient magnitude of the image region, and the fusion weight of stereo parallax and motion parallax is dynamically adjusted based on the texture richness index.
[0102] Stereo parallax and motion parallax are weighted and fused according to fusion weights to generate a fused parallax map;
[0103] Based on preset camera focal length and baseline distance parameters, the fused disparity map is converted into the final scene depth map using triangulation. Specifically, in the motion disparity fusion and depth calculation process, after obtaining the optimized stereo disparity map, the system begins to perform motion information fusion. First, the inertial measurement unit acquires the inter-frame rotation matrix and translation vector of the UAV at a frequency of 100Hz. This is then combined with the depth values of the corresponding pixels in the scene depth map generated in the previous moment to calculate the motion disparity. For the first frame in the sequence, the system directly calculates the initial depth map using the stereo disparity map of the current frame through triangulation, and uses this as the benchmark for motion estimation. From the second frame onwards, the system combines the depth values of the corresponding pixels in the scene depth map generated in the previous moment to calculate the motion disparity. The motion disparity calculation uses a projection method based on epipolar geometry to convert the camera motion between adjacent frames into disparity changes. Texture richness indices are obtained by calculating the variance and gradient magnitude of grayscale values in local image regions. The grayscale variance threshold is set to 10 by default, and the gradient magnitude threshold is set to 5 by default. These two thresholds effectively distinguish between texture-rich regions and smooth regions. The fusion weights of stereo parallax and motion parallax are dynamically adjusted based on texture richness metrics, specifically using a linear weighting method. Stereo parallax is given higher weight in texture-rich regions (default weight coefficient 0.8), while motion parallax is given higher weight in texture-deficient regions (default weight coefficient 0.7). This weighting scheme leverages the advantages of both parallax estimation methods. Stereo parallax and motion parallax are then weighted and fused according to the dynamically adjusted weights to generate a fused parallax map. Finally, based on preset camera focal length and baseline distance parameters, triangulation is used to convert the fused parallax map into a final scene depth map. The default camera focal length is 2000 pixels, and the default baseline distance is 0.2 meters. These default values were obtained through camera calibration to ensure the accuracy of depth measurements.
[0104] The present invention is further configured such that S4 includes:
[0105] The brightness features of the enhanced image and the depth features of the depth map are extracted separately using a dual-branch convolutional neural network.
[0106] The brightness features, depth features, and conditional vectors are fused to generate fused features;
[0107] Based on the fusion features, candidate regions for pits and ponds are generated and the probability of each candidate region for pits and ponds is output. The depth extreme difference of each candidate region for pits and ponds is calculated.
[0108] The depth extreme difference is compared with the preset depth mutation threshold to verify the candidate pit and pond regions and generate pit and pond region masks.
[0109] Based on the pit probability, depth extreme difference, and reliability coefficient corresponding to the enhancement strategy, the confidence score of the mask for each pit region is calculated. Specifically, in the pit detection and verification process, a dual-branch convolutional neural network is first used to extract the brightness features of the enhanced image and the depth features of the depth map. Both branches contain five convolutional layers, using the ReLU activation function, and finally output a 512-dimensional feature vector. The extracted brightness features, depth features, and the conditional vector composed of the enhancement strategy identifier are concatenated and fused to generate a 1024-dimensional fused feature vector. The dimension of the conditional vector is set to 64 by default, which can effectively convey strategy information without overwhelming the original features. Based on the fused features, pit candidate regions are generated and the pit probability of each candidate region is output. The candidate region generation adopts an anchor point mechanism, with 9 anchor point boxes of different scales set by default, covering the pit size range from 0.2 meters to 2.0 meters. At the same time, the depth extreme difference of each candidate region is calculated, that is, the difference between the maximum and minimum depth values within the region. The depth extreme difference is compared with a preset depth mutation threshold, which is set to 0.1 meters by default. The specific data is determined based on the depth requirements for significant potholes in the highway pothole maintenance standards. The candidate regions are validated through the comparison results, retaining only regions where the depth extreme difference exceeds the threshold, generating the final pothole region mask. Based on the pothole probability, depth extreme difference, and the reliability coefficient corresponding to the enhancement strategy, a confidence score is calculated for each pothole region mask. The confidence score is calculated using a weighted summation method, where the weight of the pothole probability is set to 0.6 by default, the weight of the depth extreme difference is 0.3, and the weight of the strategy reliability coefficient is 0.1. The weight allocation is determined based on the validation principle of prioritizing model prediction and supplementing it with geometric features and strategy reliability. The strategy reliability coefficient in dense fog mode is set to 0.7 by default, and the coefficient in standard mode is 1.0, reflecting the differences in the reliability of the output results from different enhancement strategies.
[0110] The present invention is further configured such that S5 includes:
[0111] Based on the UAV pose data and preset camera parameters, the pixel coordinates of the pit area mask are converted into the world coordinate system.
[0112] Calculate the average of all world coordinates corresponding to the pit area mask to obtain the center coordinates of the pit.
[0113] The minimum bounding rectangle of the pothole area mask in the road surface projection is calculated based on the principal component analysis method to obtain the pothole length and pothole width.
[0114] Calculate the depth of the pit based on the statistical distribution characteristics of the depth values in the depth map;
[0115] By combining the center location coordinates, pit length, pit width, pit depth, and confidence score, a three-dimensional geometric parameter of the pit is generated.
[0116] A visual inspection report is generated by annotating the bounding boxes and depth information of potholes on the enhanced image. Specifically, in the process of calculating the 3D parameters of the potholes and generating the report, the pixel coordinates of the pothole area mask are first transformed to the world coordinate system based on the UAV pose data and the camera intrinsic parameter matrix. The camera intrinsic parameter matrix includes focal length and principal point coordinate parameters, which are pre-obtained using the checkerboard calibration method. The focal length is set to 2000 pixels by default, and the principal point coordinates are set to the image center by default, with specific values determined based on typical parameters of commonly used camera models. The average value of all world coordinate points corresponding to the pothole area mask is calculated to obtain the center position coordinates of the pothole. The arithmetic mean method is used to calculate the center point to accurately reflect the geometric center position of the pothole area. The minimum bounding rectangle of the pothole area mask in the road surface projection is calculated based on the principal component analysis method to obtain the pothole length and width. The reason for choosing the principal component analysis method is that it can effectively handle irregularly shaped pothole areas. The main direction of the pothole is determined by calculating the eigenvectors, with the length corresponding to the span of the first principal component and the width corresponding to the span of the second principal component. The depth of the pit is calculated based on the statistical distribution characteristics of depth values within the pit area in the depth map. The 95th quantile is used as the pit depth value, and the median of the depth values in the boundary area is subtracted to obtain the reference surface height. The 95th quantile is chosen to eliminate noise points in the depth map while accurately reflecting the actual maximum depth of the pit. A structured report containing the pit's 3D geometric parameters is generated by combining the center coordinates, pit length, pit width, pit depth, and confidence score. The report is in JSON format and includes fields such as location, size, depth, and confidence score. The JSON format facilitates subsequent data processing and analysis. A visual inspection report is generated by annotating the pit boundary boxes and depth information on the enhanced image. The boundary boxes are drawn as red rectangles with a default line width of 3 pixels. Depth information is displayed as an overlay of a color spectrum, with warm colors representing deeper areas and cool colors representing shallower areas. The confidence score and key parameters are also annotated, generating a PNG format visual report.
[0117] Example 2:
[0118] Please see Figure 2 This exemplary low-light enhancement and parallax calculation nighttime highway pothole identification system includes:
[0119] Environmental perception and acquisition module: Collects nighttime highway environmental data through a sensing system, calculates meteorological indices to determine laser compensation power, and acquires stereo images to generate environmental metadata;
[0120] Image enhancement module: Based on environmental metadata, the enhancement strategy is determined according to preset rules, and the improved Retinex algorithm is used to enhance the stereo image pairs to obtain enhanced image pairs;
[0121] The disparity calculation module performs stereo matching based on enhanced image pairs, calculates disparity by fusing the pose data of the UAV, and generates a scene depth map.
[0122] The pit and pond identification module: Based on the conditional bi-branch identification model, it performs feature fusion on the enhanced image pair and the depth map to detect and verify the depth of the pit and pond area, and obtain the pit and pond area mask;
[0123] Results generation module: Based on the pit area mask, depth map and UAV pose data, calculate the three-dimensional geometric parameters and center position coordinates of the pit, and generate a visual inspection report.
[0124] It should be noted that the low-light enhancement and parallax calculation nighttime highway pothole recognition system provided in the above embodiments and the low-light enhancement and parallax calculation nighttime highway pothole recognition method provided in the above embodiments belong to the same concept. The specific operation methods of each module and unit have been described in detail in the method embodiments and will not be repeated here. In practical applications, the low-light enhancement and parallax calculation nighttime highway pothole recognition system provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.
[0125] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for identifying potholes on highways at night using low-light enhancement and parallax calculation, characterized in that, include: S1: Collect nighttime highway environment data through a sensing system, calculate the meteorological index to determine the laser compensation power, and acquire stereoscopic image pairs to generate environmental metadata, including: collecting nighttime highway environment data using multi-dimensional sensors mounted on a drone, the nighttime highway environment data including: ambient light intensity, flight altitude, visibility, and humidity data; generating a meteorological index characterizing weather conditions based on weighted fusion of visibility and humidity data, wherein the meteorological index is positively correlated with visibility and negatively correlated with humidity; calculating the basic lighting power based on ambient light intensity and a preset maximum value of basic lighting, according to the relationship that the weaker the ambient light intensity, the greater the basic lighting power; and based on the meteorological index and a preset weather compensation coefficient... Based on the relationship that the lower the meteorological index, the greater the adjustment of weather compensation power, the weather compensation power adjustment is calculated. Based on the flight altitude and the preset altitude compensation benchmark, the altitude compensation power adjustment is calculated according to the relationship that the higher the flight altitude, the greater the adjustment of altitude compensation power, and the adjustment amount increases non-linearly with the increase of altitude. The laser compensation power is determined by superimposing the basic lighting power, the weather compensation power adjustment, and the altitude compensation power adjustment. The laser auxiliary lighting device is controlled according to the laser compensation power to drive the binocular optical system and simultaneously acquire stereo image pairs. Based on the ambient light intensity, flight altitude, meteorological index, and UAV attitude parameters, environmental metadata corresponding to the stereo image pairs is generated. S2: Based on environmental metadata, an enhancement strategy is determined according to preset rules, and the improved Retinex algorithm is used to enhance the stereo image pair to obtain the enhanced image pair; S3: Perform stereo matching based on enhanced image pairs, calculate disparity by fusing the pose data of the UAV, and generate a scene depth map; S4: Based on a conditional dual-branch recognition model, feature fusion is performed on the enhanced image pair and the depth map to detect and verify the depth of pit and pond regions, obtaining a pit and pond region mask. This includes: extracting the brightness features of the enhanced image and the depth features of the depth map through a dual-branch convolutional neural network; fusing the brightness features, depth features, and a conditional vector composed of enhancement strategy identifiers to generate fused features; generating pit and pond candidate regions based on the fused features and outputting the pit and pond probability of each candidate region, calculating the depth extreme value difference of each candidate region; comparing the depth extreme value difference with a preset depth mutation threshold to verify the candidate regions and generate a pit and pond region mask. S5: Based on the pit area mask, depth map and UAV pose data, calculate the three-dimensional geometric parameters and center position coordinates of the pit, and generate a visual inspection report.
2. The method for identifying potholes on highways at night using low-light enhancement and parallax calculation according to claim 1, characterized in that, S2 includes: The input stereo image is decomposed into reflection and illumination components using a convolutional neural network. The illumination components are logarithmically transformed and then optimized through a mapping network, and finally the optimized illumination components are obtained through an inverse exponential transform. Edge features of the reflection component are extracted using an edge detection operator, and an edge attention map is generated using a convolutional neural network and activation function. Based on the edge attention map, the reflection component is enhanced by convolution to obtain the enhanced reflection component. The enhanced image is obtained by multiplying the optimized illumination component with the enhanced reflection component.
3. The method for identifying potholes on highways at night using low-light enhancement and parallax calculation according to claim 2, characterized in that, Enhancement strategies are determined based on meteorological indices and flight altitudes in environmental metadata, and the image enhancement process is fine-tuned according to the enhancement strategies. The rules for determining the enhancement strategy are as follows: when the meteorological index is below the first threshold, the dense fog enhancement strategy is activated, and the reflection component and illumination component are adjusted by introducing atmospheric transmittance estimation during the image decomposition process; when the flight altitude is above the second threshold, the high-altitude enhancement strategy is activated, and the intensity of the edge enhancement convolution kernel is reduced; otherwise, the standard enhancement strategy is activated. When both conditions are met simultaneously—the meteorological index being below the first threshold and the flight altitude being above the second threshold—enhancement strategies are selected according to a preset priority order, with the dense fog enhancement strategy having higher priority than the high-altitude enhancement strategy.
4. The method for identifying potholes on highways at night using low-light enhancement and parallax calculation according to claim 1, characterized in that, S3 includes: The SIFT feature detection algorithm is used to extract scale-invariant feature points of the enhanced image pairs and compute descriptors; The feature point correspondence between the left and right images in the enhanced image pair is established by a feature matcher, and the RANSAC algorithm is used to remove mismatched points. A semi-global stereo matching algorithm is used to calculate the initial disparity map within the disparity search range; By performing left-right consistency checks on the initial disparity map, identifying and removing occluded areas and mismatched pixels, and then using weighted least squares method to optimize disparity filling, an optimized stereo disparity map is obtained.
5. The method for identifying potholes on highways at night using low-light enhancement and parallax calculation according to claim 4, characterized in that, After obtaining the optimized stereo disparity map, motion disparity fusion is performed: The inter-frame rotation matrix and translation vector of the UAV are obtained by the inertial measurement unit, and the motion parallax is calculated by combining it with the scene depth map generated in the previous moment. Texture richness index is obtained by calculating the gray-level variance and gradient magnitude of the image region, and the fusion weight of stereo parallax and motion parallax is dynamically adjusted based on the texture richness index. Stereo parallax and motion parallax are weighted and fused according to fusion weights to generate a fused parallax map; Based on preset camera focal length and baseline distance parameters, the fused disparity map is converted into the final scene depth map using triangulation.
6. The method for identifying potholes on highways at night using low-light enhancement and parallax calculation according to claim 1, characterized in that, S4 includes: Based on the probability of pits and ponds, the difference in depth extreme values, and the reliability coefficient corresponding to the enhancement strategy, the confidence score of the mask for each pit and pond region is calculated.
7. The method for identifying potholes on highways at night using low-light enhancement and parallax calculation according to claim 1, characterized in that, S5 includes: Based on the UAV pose data and preset camera parameters, the pixel coordinates of the pit area mask are converted into the world coordinate system. Calculate the average of all world coordinates corresponding to the pit area mask to obtain the center coordinates of the pit. The minimum bounding rectangle of the pothole area mask in the road surface projection is calculated based on the principal component analysis method to obtain the pothole length and pothole width. Calculate the depth of the pit based on the statistical distribution characteristics of the depth values in the depth map; By combining the center location coordinates, pit length, pit width, pit depth, and confidence score, a three-dimensional geometric parameter of the pit is generated. A visual inspection report is generated by annotating the bounding boxes and depth information of pits and ponds on the enhanced image.
8. A nighttime road pothole identification system based on low-light enhancement and parallax calculation, used to implement the nighttime road pothole identification method based on low-light enhancement and parallax calculation as described in any one of claims 1-7, characterized in that, include: Environmental perception and acquisition module: Collects nighttime highway environmental data through a sensing system, calculates meteorological indices to determine laser compensation power, and acquires stereo images to generate environmental metadata; Image enhancement module: Based on environmental metadata, the enhancement strategy is determined according to preset rules, and the improved Retinex algorithm is used to enhance the stereo image pairs to obtain enhanced image pairs; The disparity calculation module performs stereo matching based on enhanced image pairs, calculates disparity by fusing the pose data of the UAV, and generates a scene depth map. The pit and pond identification module: Based on the conditional bi-branch identification model, it performs feature fusion on the enhanced image pair and the depth map to detect and verify the depth of the pit and pond area, and obtain the pit and pond area mask; Results generation module: Based on the pit area mask, depth map and UAV pose data, calculate the three-dimensional geometric parameters and center position coordinates of the pit, and generate a visual inspection report.
Citation Information
Patent Citations
Pavement structure damage detection method and device, storage medium and electronic equipment
CN119643566A
