Road condition monitoring method and system under complex light based on lightweight enhanced network

Abstract

The invention provides a road condition monitoring method and system under complex light based on a lightweight enhanced network, and belongs to the field of traffic road condition intelligent detection. According to the invention, a weak light enhancement network PLED-Net is adopted, and the low contrast and noise problems of an image in a weak light environment are optimized; an astigmatism enhancement network TSD-Net is introduced on the basis of a weak light enhancement network PLED-Net, so that the definition and detail performance of an image in an astigmatism environment are improved; meanwhile, the backlight enhancement network Expanand-Net is introduced on the basis of the weak light enhancement network PLED-Net, so that the problems of overexposure and shadow in a backlight environment are effectively solved. According to the method, the image quality under the condition of multiple complex light rays is remarkably improved by utilizing the lightweight enhancement network; the road condition detection system comprises a camera and an edge computing platform and can work normally under various complex light conditions, dependence on a multi-mode sensor is reduced, efficient real-time processing is guaranteed, and meanwhile the road condition monitoring cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent detection of traffic conditions, and in particular to a road condition monitoring method based on multiple lightweight image enhancement networks. Background Art

[0002] With the acceleration of urbanization, traffic condition monitoring systems play a vital role in ensuring traffic safety and optimizing traffic management. Most existing traffic condition monitoring systems rely on cameras to collect video data and use image processing and target detection algorithms to monitor and analyze traffic elements such as vehicles and pedestrians. However, complex light environments (such as strong light, shadows, night lighting, etc.) often pose challenges to the accuracy of traffic condition monitoring systems.

[0003] Although existing road condition monitoring systems can work normally under general lighting conditions, the accuracy and reliability of target detection are significantly reduced in complex lighting environments such as weak light (reduced image contrast and increased noise), astigmatism (blurred images and loss of details), and backlight (overexposed images and shadow areas). To meet these challenges, existing systems usually rely on multimodal sensor fusion or complex image processing algorithms, but this not only increases system complexity and cost, but also has high requirements for hardware performance, making it difficult to run efficiently on edge computing platforms, resulting in insufficient real-time performance. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention provides a road condition monitoring method and system under complex light conditions based on a lightweight enhancement network, which belongs to the field of intelligent detection of traffic conditions. The present invention adopts a weak light enhancement network PLED-Net to optimize the low contrast and noise problems of images in weak light environments; on the basis of the weak light enhancement network PLED-Net, a scattered light enhancement network TSD-Net is introduced to improve the clarity and detail performance of images in scattered light environments; at the same time, on the basis of the weak light enhancement network PLED-Net, a backlight enhancement network Expand-Net is introduced to effectively solve the overexposure and shadow problems in backlight environments. The present invention uses a lightweight enhancement network to significantly improve the image quality under multiple complex light conditions; the road condition detection system includes a camera and an edge computing platform, and the road condition detection system can work normally under a variety of complex light conditions, reducing the dependence on multimodal sensors, while ensuring efficient real-time processing, while reducing the cost of road condition monitoring.

[0005] A road condition monitoring system under complex light conditions based on a lightweight enhancement network, the road condition monitoring system includes a camera and an edge computing platform; the edge computing platform includes a lightweight enhancement network and a PP-YOLOE-PLUS-Tiny model; the lightweight enhancement network includes a weak light enhancement network PLED-Net, a scattered light enhancement network TSD-Net and a backlight enhancement network Expand-Net; the weak light enhancement network PLED-Net, the scattered light enhancement network TSD-Net and the backlight enhancement network Expand-Net are three independent enhancement networks; the lightweight enhancement networks operate independently of each other according to the real-time road condition environment, and when the road condition environment is under a variety of complex light conditions, the three independent enhancement networks are combined with each other to enhance the image at the same time; the camera collects road condition video and transmits the video stream to the edge computing platform in real time through the USB transmission protocol; the lightweight enhancement network splits and enhances the video stream, inputs the enhanced image into the PP-YOLOE-PLUS-Tiny model for target monitoring, and combines the monitoring results into a video stream for visual display.

[0006] A road condition monitoring method under complex light conditions based on a lightweight enhanced network, the road condition monitoring method comprising the following steps:

[0007] Step 1, using a USB camera to collect road condition videos, and transmitting the collected road condition video streams to the edge computing platform in real time through the USB transmission protocol; the edge computing platform includes a lightweight enhancement network and a PP-YOLOE-PLUS-Tiny model; the lightweight enhancement network includes a weak light enhancement network PLED-Net, a scattered light enhancement network TSD-Net, and a backlight enhancement network Expand-Net;

[0008] Step 2, initializing the parameters of the lightweight enhanced network and the PP-YOLOE-PLUS-Tiny model: when the road condition monitoring system is started, the parameters are automatically initialized and loaded into the road condition monitoring system at one time; the lightweight enhanced network and the PP-YOLOE-PLUS-Tiny model remain activated during the operation of the road condition monitoring system until all weights are released when the road condition monitoring system exits;

[0009] Step 3: The lightweight enhancement network dynamically selects the weak light enhancement network PLED-Net, the scattered light enhancement network TSD-Net and the backlight enhancement network Expand-Net according to the light environment to perform image enhancement processing to obtain a single-frame enhanced image.

[0010] Step 4: Input the single-frame enhanced image into the PP-YOLOE-PLUS-Tiny model for detection to obtain a road condition monitoring image;

[0011] Step 5, batch-combine the road condition monitoring images into a video stream for visual display; batch combination means combining 5 frames of images into a batch.

[0012] Further, in step 3, the single-frame enhanced image is a low-light enhanced image, an astigmatism enhanced image, a backlight enhanced image or a fused enhanced image; the fused enhanced image is a low-light astigmatism enhanced image, a low-light backlight enhanced image, an astigmatism backlight enhanced image or a low-light astigmatism backlight enhanced image.

[0013] Further, in step 3, the weak light enhancement network PLED-Net is based on Retinex theory, and the weak light enhancement image process is five stages of enhancement, denoising, enhancement, denoising and enhancement in sequence; the weak light enhancement network achieves high-quality image enhancement through progressive enhancement and adaptive denoising; the weak light enhancement network PLED-Net includes a progressive multi-scale residual illumination enhancement module (PMSRIEB) and a detail enhancement and adaptive denoising module (DEANRB);

[0014] In the enhancement stage, the PMSRIEB module is used; the PMSRIEB module includes a multi-scale feature extraction layer, a channel attention module, a color interaction module, a global-local illumination estimation layer and a spatial attention module in sequence; the multi-scale feature extraction layer includes four convolutional layers, and the four convolutional layers are connected in parallel, and the convolution kernels of the four convolutional layers are 1x1, 3x3, 5x5, and 7x7 respectively; the color interaction module is a 3x3 grouped convolutional layer; the global-local illumination estimation layer includes two convolutional layers, and the two convolutional layers are connected in parallel, and the convolution kernels of the two convolutional layers are 3x3 and 7x7 respectively; the low-light image first extracts the illumination features through the multi-scale feature extraction layer, and then the channel attention module pays attention to the illumination features; then the color interaction module improves the correlation of the three RGB channels; then the global-local illumination estimation layer estimates the illumination from the global scale and the local scale, and the spatial attention module applies different enhancement degrees according to different regions to obtain a low-light output image; finally, the low-light output image is added to the low-light image residual to obtain a preliminary enhanced image;

[0015] In the denoising stage, a DEANRB module is used; the DEANRB module includes a differential convolution, a noise estimation module and a spatial attention module; the low-light image and the preliminary low-light enhanced image are simultaneously used as inputs of the differential convolution, and the differential convolution extracts detail features; the preliminary low-light enhanced image is passed through the noise estimation module to obtain a noise estimated image; the illumination component feature map obtained by dividing the low-light image and the preliminary low-light enhanced image is used to calculate the SNR map using the illumination component feature map, the SNR map is passed through the spatial attention module to obtain a noise spatial attention feature map, and the noise attention feature map is multiplied with the estimated noise image to realize adaptive denoising; finally, the noise estimated image is subtracted from the preliminary low-light enhanced image to obtain a preliminary denoised image, and the detail features are added to the preliminary denoised image to obtain a denoised image;

[0016] The calculation process of the SNR map is as follows: first, the illuminance component feature map is subjected to median filtering to obtain a filtered illumination component feature map; then, the illumination component feature map and the filtered illumination component feature map are subtracted and the absolute value is taken to obtain a noise map; then, the filtered illumination component feature map is divided by the noise map to obtain the SNR map.

[0017] Furthermore, the differential convolution includes center differential, horizontal differential, vertical differential and alignment differential.

[0018] Furthermore, the noise estimation module includes 12 convolution blocks in sequence; each convolution block includes a 3x3 convolution layer, a normalization layer and a LeakyReLU layer; the convolution layer, the normalization layer and the LeakyReLU layer are connected in series in sequence.

[0019] Further, in step 3, the astigmatism enhancement network TSD-Net includes a multi-scale attention extraction module, a dual-branch feature extraction module and a multi-feature fusion module;

[0020] The multi-scale attention extraction module includes six convolution units; among the six convolution units, the first convolution unit includes maximum pooling, a 3x3 convolution layer and a LeakyReLU layer; the second convolution unit includes a 3x3 convolution layer and a LeakyReLU layer; the third convolution unit includes three parallel 3x3 dilated convolution layers, and the expansion degrees of the three dilated convolution layers are 5, 7 and 9 respectively; the fourth convolution unit includes a 3x3 convolution layer and a LeakyReLU layer; the fifth convolution unit includes a 3x3 convolution layer and a LeakyReLU layer; the sixth convolution unit includes a 3x3 convolution layer and a Sigmoid layer;

[0021] The dual-branch feature extraction module includes two branches; the first branch adopts the UNet architecture, and the second branch adopts the DeseNet network; the first branch includes a basic feature extraction block, a downsampling block, a basic feature extraction block, a basic feature extraction block, a downsampling block, a basic feature extraction block, a basic feature extraction block, a basic feature extraction block, an upsampling block, a basic feature extraction block, a basic feature extraction block, a basic feature extraction block, an upsampling block, a basic feature extraction block, a basic feature extraction block, a basic feature extraction block, and a basic feature extraction block in sequence; the basic feature extraction block includes a 3x3 convolution layer and a LeakyReLU layer in sequence, the downsampling block includes a 3x3 convolution layer and a LeakyReLU layer in sequence, and the upsampling block is a bilinear interpolation.

[0022] The multi-feature fusion module includes an input block, 5 feature extraction blocks and an output block in sequence; the input block includes a 3x3 convolution layer and a LeakyReLU layer in sequence; each feature extraction block includes two convolution blocks, and each convolution block includes a 3x3 convolution layer, a LeakyReLU layer and a residual block; the 3x3 convolution layer and the LeakyReLU layer are connected in series and then connected to the residual block; the output block includes a 3x3 convolution layer and a LeakyReLU layer in sequence.

[0023] Furthermore, the image enhancement process of the astigmatism enhancement network TSD-Net is as follows:

[0024] First, the astigmatism image is feature extracted through a multi-scale attention extraction module to obtain an astigmatism feature map;

[0025] Then, the astigmatism image and the astigmatism feature image are multiplied pixel by pixel and then added to obtain the input feature extraction map;

[0026] Next, the input feature extraction image is passed through a dual-branch feature extraction module to obtain two preliminary astigmatism-enhanced images;

[0027] Finally, the input feature extraction map is combined with the two preliminary astigmatism enhanced images and input into the multi-feature fusion module to obtain the astigmatism enhanced image.

[0028] Further, in step 3, the backlight enhancement network Expand-Net adopts a convolutional network with two branches; the convolutional network with two branches includes two branches; the first branch includes four 3x3 convolutional layers; the second branch includes seven 3x3 convolutional layers; both branches have 64 channels;

[0029] Step 3.1, the backlit image is processed by the first branch to obtain a detail feature image, and the image size of the detail image remains unchanged;

[0030] Step 3.2, the backlit image is gradually downsampled by the second branch to obtain a global feature image, and the size of the global feature image is reduced to 1x1;

[0031] In step 3.3, the detail feature image is fused with the global feature image and input into a 3x3 convolutional layer. The number of channels of the convolutional layer is reduced from 128 to 3 to obtain a backlight enhanced image.

[0032] Furthermore, in step 4, the detection process of the PP-YOLOE-PLUS-Tiny model is as follows:

[0033] The PP-YOLOE-PLUS-Tiny model includes a target detection module, a target tracking module, and an extended function module;

[0034] First, the target detection module accurately identifies and classifies the single-frame enhanced image and classifies it as a pedestrian or a vehicle.

[0035] Then, the target tracking module saves the classified image of the target detection module and associates the information between the upper and lower frames of the classified image of the target detection module, thereby realizing continuous tracking of the same target;

[0036] Next, the extended function module detects the attributes of pedestrians or vehicles; pedestrian attributes include pedestrian gender, clothing and age characteristics, vehicle attributes include vehicle characteristic information and vehicle violation detection, and vehicle characteristic information includes vehicle model, color and brand; the extended function module detects vehicle characteristic information by segmenting the license plate image and extracting text information, and the extended function module performs violation detection on the vehicle image by combining the vehicle characteristic information and road traffic markings to determine whether the vehicle has violated the law; finally, a road condition monitoring image is obtained.

[0037] Further, in step 4, the scheduling mode of the PP-YOLOE-PLUS-Tiny model is:

[0038] The calling mode of PaddleDetection in the PP-YOLOE-PLUS-Tiny model is reconstructed from command line driven to modular; the modular scheduling mode enables the road condition monitoring system to dynamically select and call the corresponding functional modules according to the input type and target category during operation, thereby enhancing the flexibility and adaptability of the road condition monitoring system; the input type is image or video, and the target category is pedestrian or vehicle.

[0039] The present invention has the following beneficial effects:

[0040] The image enhancement networks of the present invention are all lightweight networks, which can respectively process weak light, scattered light and backlight for the same original image; they can be operated independently to optimize processing for specific light conditions (such as low illumination, foggy days, backlight), or they can be used in combination to cope with more complex mixed light environments; in the case of weak light and backlight at the same time, the road condition monitoring system can first perform weak light enhancement through PLED-Net, and then perform backlight repair through Expand-Net. The flexible use method not only improves the visible quality of the image in harsh light environments, but also significantly improves the accuracy of the detection algorithm, and enhances the robustness and accuracy of the road condition monitoring system working under different light conditions; through lightweight design, the road condition monitoring system reduces the computational complexity of the system while maintaining efficient processing capabilities, so that it can better meet the needs of real-time road condition monitoring;

[0041] Aiming at the requirements of real-time processing of road condition monitoring system, the present invention adopts a fully lightweight model framework, which maintains the high performance of the road condition monitoring system while reducing the consumption of computing resources, realizing low-latency real-time processing, and can respond quickly in a dynamically changing environment, monitor and analyze road conditions in time, and ensure traffic safety. The road condition monitoring system is lightweight, easy to deploy, has good robustness, and strong real-time performance. It can improve image quality while improving the accuracy of traffic detection under a variety of complex lighting conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The overall architecture diagram of the road condition monitoring method in a complex light environment for lightweight enhanced network;

[0043] Figure 2 This is a schematic diagram of the weak light enhancement network structure;

[0044] Figure 3 This is a schematic diagram of the structure of the astigmatism enhancement module;

[0045] Figure 4 This is a schematic diagram of the backlight enhancement module structure;

[0046] Figure 5 Workflow diagram of the PP-YOLOE-PLUS-Tiny model improved for scheduling architecture. DETAILED DESCRIPTION

[0047] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0049] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0050] The present invention provides a method and system for monitoring road conditions under complex light conditions based on a lightweight enhancement network, and belongs to the field of intelligent detection of traffic conditions. The present invention adopts a weak light enhancement network PLED-Net, and optimizes the low contrast and noise problems of images in weak light environments; on the basis of the weak light enhancement network PLED-Net, a scattered light enhancement network TSD-Net is introduced to improve the clarity and detail performance of images in scattered light environments; at the same time, on the basis of the weak light enhancement network PLED-Net, a backlight enhancement network Expand-Net is introduced to effectively solve the overexposure and shadow problems in backlight environments. The present invention uses a lightweight enhancement network to significantly improve the image quality under multiple complex light conditions; the road condition detection system includes a camera and an edge computing platform, and the road condition detection system can work normally under a variety of complex light conditions, reducing the dependence on multimodal sensors, while ensuring efficient real-time processing, reducing the cost of road condition monitoring.

[0051] A road condition monitoring system under complex light conditions based on a lightweight enhancement network, the road condition monitoring system includes a camera and an edge computing platform; the edge computing platform adopts Jetson Orin Nano; the edge computing platform includes a lightweight enhancement network and a PP-YOLOE-PLUS-Tiny model; the lightweight enhancement network includes a weak light enhancement network PLED-Net, a scattered light enhancement network TSD-Net and a backlight enhancement network Expand-Net; the weak light enhancement network PLED-Net, the scattered light enhancement network TSD-Net and the backlight enhancement network Expand-Net are three independent enhancement networks; the lightweight enhancement networks operate independently of each other according to the real-time road condition environment, and when the road condition environment is under a variety of complex light conditions, the three independent enhancement networks are combined with each other to enhance the image at the same time; the camera collects road condition video and transmits the video stream to the edge computing platform in real time through the USB transmission protocol; the lightweight enhancement network splits and enhances the video stream, inputs the enhanced image into the PP-YOLOE-PLUS-Tiny model for target monitoring, and combines the monitoring results into a video stream for visual display.

[0052] A road condition monitoring method under complex light conditions based on a lightweight enhanced network, the road condition monitoring method comprising the following steps:

[0053] like Figure 1 As shown:

[0054] Step 1, using a USB camera to collect road condition videos, and transmitting the collected road condition video streams to the edge computing platform in real time through the USB transmission protocol; the edge computing platform includes a lightweight enhancement network and a PP-YOLOE-PLUS-Tiny model; the lightweight enhancement network includes a weak light enhancement network PLED-Net, a scattered light enhancement network TSD-Net, and a backlight enhancement network Expand-Net;

[0055] Step 2, initialize the parameters of the lightweight enhancement network and the PP-YOLOE-PLUS-Tiny model: when the road condition monitoring system is started, the parameters are uniformly and automatically initialized and loaded into the road condition monitoring system at one time; the lightweight enhancement network and the PP-YOLOE-PLUS-Tiny model remain activated during the operation of the road condition monitoring system until all weights are released when the road condition monitoring system exits; each module can always maintain a low latency state to ensure that the road condition monitoring system can operate efficiently and stably when processing image enhancement and target detection tasks in complex lighting environments in real time.

[0056] Step 3: The lightweight enhancement network dynamically selects the weak light enhancement network PLED-Net, the scattered light enhancement network TSD-Net and the backlight enhancement network Expand-Net according to the light environment for image enhancement processing to obtain a single frame enhanced image; in a single complex light environment, the road condition monitoring system independently calls the corresponding enhancement network: PLED-Net is used in a weak light environment to improve the image contrast and detail performance; TSD-Net is used in a scattered light environment to improve the image clarity and detail retention; Expand-Net is used in a backlight environment to reduce overexposure and shadow problems; for more complex mixed light environments, the mixed light environment includes the coexistence of weak light and backlight, the coexistence of scattered light and backlight, the coexistence of weak light and scattered light, and the coexistence of scattered light, weak light and backlight; the road condition monitoring system can combine and use multiple enhancement networks to perform targeted enhancement processing on the image in turn to ensure that high-quality image output can be obtained under a variety of complex light conditions, providing better input for subsequent target detection; the input image is sent to different enhancement networks in different scenes in turn, and the process of "weak light enhancement → scattered light enhancement → backlight enhancement" is adopted. According to actual needs, the input image can be selected to be processed through one or more enhancement networks.

[0057] Step 4: Input the single-frame enhanced image into the PP-YOLOE-PLUS-Tiny model for detection to obtain a road condition monitoring image;

[0058] Step 5, batch-combine the road condition monitoring images into a video stream for visual display; batch combination means combining 5 frames of images into a batch.

[0059] In step 3, the single-frame enhanced image is a low-light enhanced image, an astigmatism enhanced image, a backlight enhanced image, or a fused enhanced image; the fused enhanced image is a low-light astigmatism enhanced image, a low-light backlight enhanced image, an astigmatism backlight enhanced image, or a low-light astigmatism backlight enhanced image;

[0060] like Figure 2 As shown:

[0061] In step 3, the weak light enhancement network PLED-Net is based on the Retinex theory, and the weak light enhancement image process is five stages: enhancement, denoising, enhancement, denoising and enhancement; the weak light enhancement network achieves high-quality image enhancement through progressive enhancement and adaptive denoising; the weak light enhancement network PLED-Net includes a progressive multi-scale residual illumination enhancement module (PMSRIEB) and a detail enhancement and adaptive denoising module (DEANRB);

[0062] In the enhancement stage, the PMSRIEB module is used; the PMSRIEB module includes a multi-scale feature extraction layer, a channel attention module, a color interaction module, a global-local illumination estimation layer and a spatial attention module in sequence; the multi-scale feature extraction layer includes four convolutional layers, and the four convolutional layers are connected in parallel, and the convolution kernels of the four convolutional layers are 1x1, 3x3, 5x5, and 7x7 respectively; the color interaction module is a 3x3 grouped convolutional layer; the global-local illumination estimation layer includes two convolutional layers, and the two convolutional layers are connected in parallel, and the convolution kernels of the two convolutional layers are 3x3 and 7x7 respectively; the low-light image first extracts the illumination features through the multi-scale feature extraction layer, and then the channel attention module pays attention to the illumination features; then the color interaction module improves the correlation of the three RGB channels; then the global-local illumination estimation layer estimates the illumination from the global scale and the local scale, and the spatial attention module applies different enhancement degrees according to different regions to obtain a low-light output image; finally, the low-light output image is added to the low-light image residual to obtain a preliminary enhanced image;

[0063] In the denoising stage, a DEANRB module is used; the DEANRB module includes a differential convolution, a noise estimation module and a spatial attention module; the low-light image and the preliminary low-light enhanced image are simultaneously used as inputs of the differential convolution, and the differential convolution extracts detail features; the preliminary low-light enhanced image is passed through the noise estimation module to obtain a noise estimated image; the illumination component feature map obtained by dividing the low-light image and the preliminary low-light enhanced image is used to calculate the SNR map using the illumination component feature map, the SNR map is passed through the spatial attention module to obtain a noise spatial attention feature map, and the noise attention feature map is multiplied with the estimated noise image to realize adaptive denoising; finally, the noise estimated image is subtracted from the preliminary low-light enhanced image to obtain a preliminary denoised image, and the detail features are added to the preliminary denoised image to obtain a denoised image;

[0064] The calculation process of the SNR map is as follows: first, the illuminance component feature map is subjected to median filtering to obtain a filtered illumination component feature map; then, the illumination component feature map and the filtered illumination component feature map are subtracted and the absolute value is taken to obtain a noise image; then, the filtered illumination component feature map is divided by the noise image to obtain the SNR map.

[0065] The low-light enhancement network PLED-Net adopts a phased enhancement and noise reduction strategy, which significantly improves the contrast and detail performance of low-light images, while effectively suppressing noise and maintaining low computational complexity, making it suitable for efficient operation on edge computing platforms.

[0066] The differential convolution includes center difference, horizontal difference, vertical difference and alignment difference.

[0067] The noise estimation module includes 12 convolution blocks in sequence; each convolution block includes a 3x3 convolution layer, a normalization layer and a LeakyReLU layer; the convolution layer, the normalization layer and the LeakyReLU layer are connected in series in sequence.

[0068] like Figure 3 As shown:

[0069] In step 3, the astigmatism enhancement network TSD-Net includes a multi-scale attention extraction module, a dual-branch feature extraction module and a multi-feature fusion module;

[0070] The multi-scale attention extraction module includes six convolution units; among the six convolution units, the first convolution unit includes maximum pooling, a 3x3 convolution layer and a LeakyReLU layer; the second convolution unit includes a 3x3 convolution layer and a LeakyReLU layer; the third convolution unit includes three parallel 3x3 dilated convolution layers, and the expansion degrees of the three dilated convolution layers are 5, 7 and 9 respectively; the fourth convolution unit includes a 3x3 convolution layer and a LeakyReLU layer; the fifth convolution unit includes a 3x3 convolution layer and a LeakyReLU layer; the sixth convolution unit includes a 3x3 convolution layer and a Sigmoid layer;

[0071] The dual-branch feature extraction module includes two branches; the first branch adopts the UNet architecture, and the second branch adopts the DeseNet network; the first branch includes a basic feature extraction block, a downsampling block, a basic feature extraction block, a basic feature extraction block, a downsampling block, a basic feature extraction block, a basic feature extraction block, a basic feature extraction block, an upsampling block, a basic feature extraction block, a basic feature extraction block, a basic feature extraction block, an upsampling block, a basic feature extraction block, a basic feature extraction block, a basic feature extraction block, and a basic feature extraction block in sequence; the basic feature extraction block includes a 3x3 convolution layer and a LeakyReLU layer in sequence, the downsampling block includes a 3x3 convolution layer and a LeakyReLU layer in sequence, and the upsampling block is a bilinear interpolation.

[0072] The multi-feature fusion module includes an input block, 5 feature extraction blocks and an output block in sequence; the input block includes a 3x3 convolution layer and a LeakyReLU layer in sequence; each feature extraction block includes two convolution blocks, and each convolution block includes a 3x3 convolution layer, a LeakyReLU layer and a residual block; the 3x3 convolution layer and the LeakyReLU layer are connected in series and then connected to the residual block; the output block includes a 3x3 convolution layer and a LeakyReLU layer in sequence.

[0073] The image enhancement process of the astigmatism enhancement network TSD-Net is as follows:

[0074] First, the astigmatism image is feature extracted through a multi-scale attention extraction module to obtain an astigmatism feature map;

[0075] Then, the astigmatism image and the astigmatism feature image are multiplied pixel by pixel and then added to obtain the input feature extraction map;

[0076] Next, the input feature extraction image is passed through a dual-branch feature extraction module to obtain two preliminary astigmatism-enhanced images;

[0077] Finally, the input feature extraction map is combined with the two preliminary astigmatism enhanced images and input into the multi-feature fusion module to obtain the astigmatism enhanced image.

[0078] like Figure 4 As shown:

[0079] In step 3, the backlight enhancement network Expand-Net adopts a convolutional network with two branches; the convolutional network with two branches includes two branches; the first branch includes four 3x3 convolutional layers; the second branch includes seven 3x3 convolutional layers; both branches have 64 channels;

[0080] Step 3.1, the backlit image is processed by the first branch to obtain a detail feature image, and the image size of the detail image remains unchanged;

[0081] Step 3.2, the backlit image is gradually downsampled by the second branch to obtain a global feature image, and the size of the global feature image is reduced to 1x1;

[0082] Step 3.3, the detail feature image is fused with the global feature image and input into a 3x3 convolutional layer. The number of channels of the convolutional layer is reduced from 128 to 3 to obtain a backlight enhanced image.

[0083] The convolutional network design of dual-branch convolution effectively reduces the ghosting phenomenon that may appear in the repaired image by avoiding the use of upsampling operations, and improves the quality of backlight enhanced images.

[0084] like Figure 5 As shown:

[0085] In step 4, the detection process of the PP-YOLOE-PLUS-Tiny model is as follows:

[0086] The PP-YOLOE-PLUS-Tiny model includes a target detection module, a target tracking module, and an extended function module;

[0087] First, the target detection module accurately identifies and classifies the single-frame enhanced image and classifies it as a pedestrian or a vehicle.

[0088] Then, the target tracking module saves the classified image of the target detection module and associates the information between the upper and lower frames of the classified image of the target detection module, thereby realizing continuous tracking of the same target;

[0089] Next, the extended function module detects the attributes of pedestrians or vehicles; pedestrian attributes include pedestrian gender, clothing and age characteristics, vehicle attributes include vehicle characteristic information and vehicle violation detection, and vehicle characteristic information includes vehicle model, color and brand; the extended function module detects vehicle characteristic information by segmenting the license plate image and extracting text information, and the extended function module performs violation detection on the vehicle image by combining the vehicle characteristic information and road traffic markings to determine whether the vehicle has violated the law; finally, a road condition monitoring image is obtained.

[0090] In step 4, the scheduling mode of the PP-YOLOE-PLUS-Tiny model is:

[0091] The calling mode of PaddleDetection in the PP-YOLOE-PLUS-Tiny model is reconstructed from command line driven to modular; the modular scheduling mode enables the road condition monitoring system to dynamically select and call the corresponding functional modules according to the input type and target category during operation, thereby enhancing the flexibility and adaptability of the road condition monitoring system; the input type is image or video, and the target category is pedestrian or vehicle.

[0092] The PP-YOLOE-PLUS-Tiny model is a lightweight target detection model optimized based on the PP-YOLOE architecture. It is suitable for edge computing platforms and has multi-task adaptation capabilities. It can flexibly switch on and off functional modules such as pedestrian attribute recognition, vehicle attribute recognition, and license plate recognition according to different needs. In the traffic flow monitoring scenario, the pedestrian attribute recognition function can be turned off to focus on vehicle detection and license plate recognition; in the urban road monitoring scenario, pedestrian and vehicle detection functions can be turned on at the same time to achieve comprehensive traffic monitoring. The functional configuration of the PP-YOLOE-PLUS-Tiny model enables the road condition monitoring system to dynamically adjust the detection tasks according to different application scenarios, output corresponding traffic detection images, and improve the practicality and adaptability of the system; the PP-YOLOE-PLUS-Tiny model with improved scheduling mode has a target segmentation function that uses built-in pedestrian attributes, vehicle attributes, license plate recognition and other multi-functional models for recognition.

[0093] The above specific implementation modes are used to explain the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A road condition monitoring method under complex light conditions based on a lightweight enhanced network, characterized in that: The road condition monitoring method comprises the following steps: Step 1, using a USB camera to collect road condition videos, and transmitting the collected road condition video streams to the edge computing platform in real time through the USB transmission protocol; the edge computing platform includes a lightweight enhancement network and a PP-YOLOE-PLUS-Tiny model; the lightweight enhancement network includes a weak light enhancement network PLED-Net, a scattered light enhancement network TSD-Net, and a backlight enhancement network ExpandNet; Step 2, initializing the parameters of the lightweight enhanced network and the PP-YOLOE-PLUS-Tiny model: when the road condition monitoring system is started, the parameters are automatically initialized and loaded into the road condition monitoring system at one time; the lightweight enhanced network and the PP-YOLOE-PLUS-Tiny model remain activated during the operation of the road condition monitoring system until all weights are released when the road condition monitoring system exits; Step 3: The lightweight enhancement network dynamically selects the weak light enhancement network PLED-Net, the scattered light enhancement network TSD-Net and the backlight enhancement network ExpandNet to perform image enhancement processing according to the light environment to obtain a single-frame enhanced image. Step 4: Input the single-frame enhanced image into the PP-YOLOE-PLUS-Tiny model for detection to obtain a road condition monitoring image; Step 5, batch-combine the road condition monitoring images into a video stream for visual display; batch combination means combining 5 frames of images into a batch.

2. The road condition monitoring method according to claim 1, characterized in that: In step 3, the single-frame enhanced image includes a low-light enhanced image, an astigmatism enhanced image, a backlight enhanced image or a fusion enhanced image; the fusion enhanced image includes a low-light astigmatism enhanced image, a low-light backlight enhanced image, an astigmatism backlight enhanced image or a low-light astigmatism backlight enhanced image.

3. The road condition monitoring method according to claim 1, characterized in that: In step 3, the low-light enhancement network PLED-Net is based on the Retinex theory, and the low-light enhancement image process is sequentially divided into five stages: enhancement, denoising, enhancement, denoising and enhancement; The low-light enhancement network achieves high-quality image enhancement through progressive enhancement and adaptive denoising; the low-light enhancement network PLED-Net includes PMSRIEB and DEANRB; In the enhancement stage, the PMSRIEB module is used; the PMSRIEB module includes a multi-scale feature extraction layer, a channel attention module, a color interaction module, a global-local illumination estimation layer and a spatial attention module in sequence; the multi-scale feature extraction layer includes four convolutional layers, and the four convolutional layers are connected in parallel, and the convolution kernels of the four convolutional layers are 1x1, 3x3, 5x5, and 7x7 respectively; the color interaction module is a 3x3 grouped convolutional layer; the global-local illumination estimation layer includes two convolutional layers, and the two convolutional layers are connected in parallel, and the convolution kernels of the two convolutional layers are 3x3 and 7x7 respectively; the low-light image first extracts the illumination features through the multi-scale feature extraction layer, and then the channel attention module pays attention to the illumination features; then the color interaction module improves the correlation of the three RGB channels; Then the global-local illumination estimation layer performs illumination estimation from global and local scales, and the spatial attention module applies different enhancement levels to different regions to obtain a low-light output image. Finally, the low-light output image is added to the low-light image residual to obtain a preliminary enhanced image. In the denoising stage, the DEANRB module is used; The DEANRB module includes differential convolution, noise estimation module and spatial attention module; the low-light image and the preliminary low-light enhanced image are simultaneously used as inputs of the differential convolution, and the differential convolution extracts detail features; The preliminary low-light enhanced image is passed through the noise estimation module to obtain a noise estimated image; the illumination component feature map obtained by dividing the low-light image and the preliminary low-light enhanced image is used to calculate the SNR map, the SNR map is passed through the spatial attention module to obtain the noise spatial attention feature map, and the noise attention feature map is multiplied with the estimated noise image to achieve adaptive denoising; finally, the noise estimated image is subtracted from the preliminary low-light enhanced image to obtain a preliminary denoised image, and the detail features are added to the preliminary denoised image to obtain a denoised image; The calculation process of the SNR map is as follows: first, the illuminance component feature map is subjected to median filtering to obtain a filtered illumination component feature map; then, the illumination component feature map and the filtered illumination component feature map are subtracted and the absolute value is taken to obtain a noise image; then, the filtered illumination component feature map is divided by the noise image to obtain the SNR map.

4. The road condition monitoring method according to claim 3, characterized in that: The differential convolution includes center difference, horizontal difference, vertical difference and alignment difference.

5. The road condition monitoring method according to claim 3, characterized in that: The noise estimation module includes 12 convolution blocks in sequence; each convolution block includes a 3x3 convolution layer, a normalization layer and a LeakyReLU layer; the convolution layer, the normalization layer and the LeakyReLU layer are connected in series in sequence.

6. The road condition monitoring method according to claim 1, characterized in that: In step 3, the astigmatism enhancement network TSD-Net includes a multi-scale attention extraction module, a dual-branch feature extraction module and a multi-feature fusion module; The multi-scale attention extraction module includes six convolution units; among the six convolution units, the first convolution unit includes maximum pooling, a 3x3 convolution layer and a LeakyReLU layer; the second convolution unit includes a 3x3 convolution layer and a LeakyReLU layer; the third convolution unit includes three parallel 3x3 dilated convolution layers, and the expansion degrees of the three dilated convolution layers are 5, 7 and 9 respectively; the fourth convolution unit includes a 3x3 convolution layer and a LeakyReLU layer; the fifth convolution unit includes a 3x3 convolution layer and a LeakyReLU layer; the sixth convolution unit includes a 3x3 convolution layer and a Sigmoid layer; The dual-branch feature extraction module includes two branches; the first branch adopts the UNet architecture, and the second branch adopts the DeseNet network; the first branch includes a basic feature extraction block, a downsampling block, a basic feature extraction block, a basic feature extraction block, a downsampling block, a basic feature extraction block, a basic feature extraction block, a basic feature extraction block, an upsampling block, a basic feature extraction block, a basic feature extraction block, a basic feature extraction block, an upsampling block, a basic feature extraction block, a basic feature extraction block, a basic feature extraction block, and a basic feature extraction block in sequence; The basic feature extraction block includes a 3x3 convolutional layer and a LeakyReLU layer in sequence, the downsampling block includes a 3x3 convolutional layer and a LeakyReLU layer in sequence, and the upsampling block is a bilinear interpolation; The multi-feature fusion module includes an input block, 5 feature extraction blocks and an output block in sequence; the input block includes a 3x3 convolution layer and a LeakyReLU layer in sequence; each feature extraction block includes two convolution blocks, and each convolution block includes a 3x3 convolution layer, a LeakyReLU layer and a residual block; the 3x3 convolution layer and the LeakyReLU layer are connected in series and then connected to the residual block; the output block includes a 3x3 convolution layer and a LeakyReLU layer in sequence.

7. The road condition monitoring method according to claim 6, characterized in that: The image enhancement process of the astigmatism enhancement network TSD-Net is as follows: First, the astigmatism image is feature extracted through a multi-scale attention extraction module to obtain an astigmatism feature map; Then, the astigmatism image and the astigmatism feature image are multiplied pixel by pixel and then added to obtain the input feature extraction map; Next, the input feature extraction image is passed through a dual-branch feature extraction module to obtain two preliminary astigmatism-enhanced images; Finally, the input feature extraction map is combined with the two preliminary astigmatism enhanced images and input into the multi-feature fusion module to obtain the astigmatism enhanced image.

8. The road condition monitoring method according to claim 1, characterized in that: In step 3, the backlight enhancement network ExpandNet adopts a convolutional network with two branches; the convolutional network with two branches includes two branches; the first branch includes four 3x3 convolutional layers; the second branch includes seven 3x3 convolutional layers; both branches have 64 channels; Step 3.1, the backlit image is processed by the first branch to obtain a detail feature image, and the image size of the detail image remains unchanged; Step 3.2, the backlit image is gradually downsampled by the second branch to obtain a global feature image, and the size of the global feature image is reduced to 1x1; In step 3.3, the detail feature image is fused with the global feature image and input into a 3x3 convolutional layer. The number of channels of the convolutional layer is reduced from 128 to 3 to obtain a backlight enhanced image.

9. The road condition monitoring method according to claim 1, characterized in that: In step 4, the detection process of the PP-YOLOE-PLUS-Tiny model is as follows: The PP-YOLOE-PLUS-Tiny model includes a target detection module, a target tracking module, and an extended function module; First, the target detection module accurately identifies and classifies the single-frame enhanced image and classifies it as a pedestrian or a vehicle. Then, the target tracking module saves the classified image of the target detection module and associates the information between the upper and lower frames of the classified image of the target detection module, thereby realizing continuous tracking of the same target; Next, the extended function module detects the attributes of pedestrians or vehicles; pedestrian attributes include pedestrian gender, clothing and age characteristics, vehicle attributes include vehicle characteristic information and vehicle violation detection, and vehicle characteristic information includes vehicle model, color and brand; the extended function module detects vehicle characteristic information by segmenting the license plate image and extracting text information, and the extended function module detects violations of vehicle images by combining vehicle characteristic information and road traffic markings to determine whether the vehicle has violated the law; finally, a road condition monitoring image is obtained; The scheduling mode of the PP-YOLOE-PLUS-Tiny model is: Refactored the calling mode of PaddleDetection in the PP-YOLOE-PLUS-Tiny model from command line driven to modular; The modular scheduling mode dynamically selects and calls the corresponding functional modules according to the input type and target category, thereby enhancing the flexibility and adaptability of the road condition monitoring system; The input type is image or video, and the target category is pedestrian or vehicle.

10. A road condition monitoring system for executing the method according to any one of claims 1 to 9, characterized in that: The road condition monitoring system includes a camera and an edge computing platform; the edge computing platform includes a lightweight enhancement network and a PP-YOLOE-PLUS-Tiny model; the lightweight enhancement network includes a weak light enhancement network PLED-Net, a scattered light enhancement network TSD-Net and a backlight enhancement network ExpandNet; the weak light enhancement network PLED-Net, the scattered light enhancement network TSD-Net and the backlight enhancement network ExpandNet are three independent enhancement networks; the lightweight enhancement networks operate independently of each other according to the real-time road condition environment, and when the road condition environment is under a variety of complex light conditions, the three independent enhancement networks are combined with each other to enhance the image at the same time; The camera collects road condition videos and transmits the video stream to the edge computing platform in real time through the USB transmission protocol; the lightweight enhancement network splits and enhances the video stream, inputs the enhanced image into the PP-YOLOE-PLUS-Tiny model for target monitoring, and combines the monitoring results into a video stream for visual display.

Images