Real-time road surface state analysis and early warning method and system based on semantic segmentation

Abstract

The invention relates to the technical field of road surface information state monitoring, and discloses a real-time road surface state analysis and early warning method and system based on semantic segmentation, and the method comprises the steps: collecting and preprocessing road surface video data; inputting the preprocessed data into a semantic segmentation model for segmentation processing to obtain a semantic segmentation result; semantic segmentation results are input into an analysis module, and the analysis module obtains positions, sizes, area statistics, target categories and the number of single-class targets through analysis and operation; the early warning module receives the data output by the analysis module and generates early warning data according to an early warning rule; the system adopted by the invention pays attention to an area with rich information in an image, reduces the influence of background noise, pays attention to a feature channel, improves the characterization capability of features, effectively solves the problem of low segmentation precision of a small target, inhibits background noise, enhances the generalization capability of a model, uses a clustering algorithm and a post-processing algorithm, quickly analyzes a detection result, and improves the detection efficiency. Therefore, traffic accidents are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of road surface information status monitoring, and in particular to a real-time road surface status analysis and early warning method and system based on semantic segmentation. Background Art

[0002] In the field of smart transportation, road surface information status monitoring is a key component. It involves detecting and analyzing the conditions of the road surface, including cracks, potholes, water accumulation, ice and snow coverage, and the degree of wear and tear of the road surface. With the development of autonomous driving technology, real-time road surface information status detection has become particularly important, and the demand for high-precision, real-time image analysis is growing. Existing road surface information status detection mostly relies on fixed cameras, which is difficult to fully cover and detect in real time, and the detection accuracy is low.

[0003] With the rapid development of deep learning and semantic segmentation technology, real-time road condition detection and analysis technology based on semantic segmentation has emerged. By deploying high-definition vehicle cameras and combining advanced semantic segmentation technology, continuous monitoring of road conditions can be achieved. By analyzing image frames in the video stream in real time, the system can quickly identify abnormal conditions on the road, such as sudden potholes or new cracks, lane line deviations, etc., and send alerts to drivers in a timely manner by counting and analyzing road conditions and distribution. This technology can greatly improve driving safety, reduce the occurrence of traffic accidents, and promote the development of smart transportation.

[0004] However, in the field of smart transportation, semantic segmentation technology still faces some challenges:

[0005] (1) Real-time performance: Autonomous driving and traffic monitoring systems require semantic segmentation technology to be able to process video streams in real time. Currently, many high-precision semantic segmentation models have a very large amount of computation and are difficult to achieve real-time performance.

[0006] (2) Complex road scenes contain objects of different scales, large, medium, and small. This requires semantic segmentation to be able to extract multi-scale features in order to accurately classify various targets.

[0007] (3) Deployment environment requirements: High-precision semantic segmentation networks usually have high computing resource requirements and high power consumption, making them difficult to deploy directly in vehicle environments. Therefore, model compression and acceleration are required to adapt to the computing power and energy consumption limitations of vehicle-mounted devices. Currently, many high-precision semantic segmentation networks are difficult to deploy in vehicle environments.

[0008] In short, the high labeling cost, low semantic segmentation accuracy, poor real-time performance and difficult deployment of semantic segmentation technology limit its widespread application in the field of smart transportation. Summary of the invention

[0009] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a real-time road condition analysis and early warning method and system based on semantic segmentation.

[0010] In order to achieve the above object, the present invention provides the following technical solutions:

[0011] A real-time road condition analysis and early warning method based on semantic segmentation includes the following steps:

[0012] Collect road video data and pre-process it;

[0013] The preprocessed data is input into the semantic segmentation model for segmentation processing to obtain the semantic segmentation result;

[0014] The semantic segmentation results are input into the analysis module, which obtains the position, size, area statistics, target category and quantity of single-class targets through analysis and calculation;

[0015] The early warning module receives the data output by the analysis module and generates early warning data according to the early warning rules.

[0016] In the present invention, preferably, the method further includes the steps of optimizing the semantic segmentation model and communicating with the vehicle-mounted system.

[0017] In the present invention, preferably, performing segmentation processing in the semantic segmentation model specifically includes:

[0018] Preliminary processing: Perform two standard convolutions on the input image to obtain feature F c ;

[0019] Feature fusion: Feature F c The input is sent to the pc2f convolution unit, and multi-scale features are obtained through feature extraction and concatenation at different scales.

[0020] Repeated fusion: After multiple pc2f convolution units and standard convolution units, feature F is obtained 4 ;

[0021] Dual-channel weighting: Feature F 4 Input to the dual-branch attention mechanism unit, and obtain features F of different scales after weighting, concatenation and pooling of spatial features and channel features 5 ;

[0022] Post-processing: feature F 5 Upsampling and decoding are performed to obtain the semantic segmentation results.

[0023] In the present invention, preferably, the loss function FL(p t )、TL(α,β)、L DFL , the loss function FL(pt ) focuses on the feature level loss, and the loss function TL(α, β) focuses on the target detection loss, and the loss function L DFL For the semantic segmentation task, the loss function FL(p t ) and the loss function TL(α, β) is the sample class balance loss, which is used to adjust the sample weights, L DFL is the bounding box classification loss, which is used to better regress the target bounding box. The various loss functions are defined as follows:

[0024] L = FL (p t )+TL(α,β)+L DFL (1)

[0025]

[0026] FL(p t )=-α t (1-p t ) γ log(p t ) (4).

[0027] Among them, the formula L DFL In it, M is the number of samples; N is the number of categories; i represents the i-th sample; j represents the j-th category; k represents the four parameters of the bounding box (usually x, y, w, h, representing the center point coordinates and width and height respectively); is the probability distribution of the true label after processing. For the positive sample position, its value is a specific probability distribution, and for the negative sample position, its value is 0; p ijk is the probability distribution predicted by the model; j is the category weight coefficient, which is used to balance the importance of different categories; γ is the focusing parameter, which is used to adjust the weight of difficult and easy samples. When the sample is easy to classify, The value of is small, thereby reducing the loss weight of easy-to-classify samples and making the model pay more attention to difficult-to-classify samples. α is another adjustment parameter used to further adjust the characteristics of the loss function. In the formula TL(α, β), p0 i is the probability that the i-th target is the target, p1 i The probability that the i-th target is not the target, α, β are used to weigh the false positives and true positives. t ) t is the predicted probability value, α t , γ are parameters.

[0028] In the present invention, preferably, the analysis operation specifically includes the steps of:

[0029] Determine target distribution: cluster the semantic segmentation results using a clustering algorithm to obtain the position distribution of the target center point in the feature map, and affine transform the position distribution on the feature map to the target position on the actual road surface;

[0030] Target statistics: For each type of target, a post-processing algorithm is used to obtain its position, size and area.

[0031] In the present invention, preferably, the warning rules include:

[0032] If a pedestrian is detected on the road surface, a warning message of "Pay attention to pedestrians" is generated;

[0033] If a vehicle is detected on the road and the target area ratio is greater than 65% of the current frame, a warning message of "Watch out for the vehicle ahead" is generated;

[0034] When cracks are detected on the road surface, their location distribution is analyzed. If the crack length is greater than 20cm and the width is greater than 5cm, a warning message is generated to pay attention to cracks on a certain side of the road.

[0035] When a pothole is detected on the road, its location and distribution are analyzed. If the area is larger than 0.01 square meters, a warning message "Pay attention to potholes" is generated;

[0036] When water is detected on the road surface, the size of the area is analyzed. If the area is larger than 1 square meter, a warning message of road water accumulation is generated;

[0037] If a foreign object is detected on the road, its location and area are analyzed. If the area is larger than 0.01 square meters, a warning message "Pay attention to foreign objects on the road" is generated;

[0038] Road lane line detection: Based on the lane line detection results, it analyzes whether the current vehicle is separated from the lane line. If separated, a lane departure warning message is generated.

[0039] In the present invention, preferably, the specific steps of determining lane separation are:

[0040] a. Extracting lane line edge points from lane line masks based on the analysis results obtained by the analysis module;

[0041] b. Use a quadratic polynomial to fit the lane lines on the left and right sides:

[0042] y l =A l x 2 +B l x+C l ,

[0043] y r =A r x 2 +B r x+C r ,

[0044] Among them, A, B, and C are polynomial coefficients, l represents the left lane line, and r represents the right lane line;

[0045] c. Calculate the midpoint of the lane line corresponding to the y value closest to the lane line:

[0046]

[0047] d. Calculate the vehicle offset in the current frame. If it is greater than the threshold, it is considered that the lane is offset. The offset calculation formula is as follows:

[0048] where x w is the width of the current frame, α is the offset;

[0049] e. If the offsets of 10 consecutive frames are all greater than the threshold, a warning message is generated to indicate lane departure.

[0050] A real-time road condition analysis and warning system based on semantic segmentation includes an acquisition module, a preprocessing module, a semantic segmentation model, an analysis module and a warning module connected in sequence. The acquisition module is used to acquire road video data, and the preprocessing module is used to preprocess the road video data. The preprocessing steps include video frame extraction, data cleaning, data annotation, data statistical analysis, data enhancement and data splitting. The semantic segmentation model adopts a deep learning model, which performs semantic segmentation on the preprocessed image. The analysis module is provided with a clustering algorithm and a post-processing algorithm for performing road condition analysis on the semantically segmented image. The warning module gives corresponding warning information based on the road condition analysis given by the analysis module.

[0051] In the present invention, preferably, the semantic segmentation model includes a feature extraction module, a two-branch attention mechanism unit, an SPPF pooling unit and a post-processing module connected in sequence; the feature extraction module includes a plurality of standard convolution units and pc2f convolution units that are alternately arranged; the two-branch attention mechanism unit includes a spatial attention layer, a channel attention layer and a splicing layer, and the splicing layers are connected to the spatial attention layer and the channel attention layer; the post-processing module includes a plurality of upsampling units, convolution units and fusion units.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] (1) The present invention uses a variety of data statistical methods in data preprocessing to facilitate the understanding of the distribution of training data. The data annotation method uses an automatic annotation method, first using multi-scene data to train a semantic segmentation model as a pre-trained automatic annotation model, and then using the existing pre-trained model to automatically annotate new data.

[0054] (2) The present invention designs a feature extraction network pc2f convolution unit composed of partial convolutions. The pc2f convolution unit fuses feature maps of different levels, realizes feature fusion of information, is beneficial to the segmentation of targets of different scales in complex scenes, and designs partial convolution branches. The partial convolution reduces the high computational complexity caused by redundant features. In addition, the channels and sizes of the feature maps are kept unchanged in the pc2f convolution unit, which helps to maintain the integrity of spatial information at different scales, which can not only reduce unnecessary feature redundancy, but also improve model performance.

[0055] (3) The present invention adopts a dual-branch attention mechanism unit, which not only focuses on the information-rich areas in the image to reduce the impact of background noise, but also focuses on useful feature channels to improve the representation ability of features, effectively solving the problem of low accuracy in small target segmentation, suppressing background noise, and enhancing the generalization ability of the model.

[0056] (4) The present invention designs an analysis module for semantic segmentation detection results, uses clustering algorithms and post-processing algorithms to quickly analyze detection results, issue instant warnings, and reduce the occurrence of traffic accidents.

[0057] (5) The present invention quantizes and deploys the optimized model on the computing box to improve the model reasoning speed and realize real-time reasoning. It is also physically connected to the vehicle-mounted system, providing GPU computing, data storage and real-time display functions. It is easy to deploy, easy to implement and low-cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 The present invention is a flowchart of a real-time road condition analysis and early warning method based on semantic segmentation.

[0059] Figure 2 Schematic diagram of the structure of the semantic segmentation model described in the present invention.

[0060] Figure 3 It is a structural schematic diagram of the pc2f convolution unit of the present invention.

[0061] Figure 4 It is a structural schematic diagram of the dual-branch attention unit of the present invention.

[0062] Figure 5 It is a structural schematic diagram of the analysis module of the present invention.

[0063] Figure 6 It is a cluster analysis result diagram of the semantic segmentation result of the present invention.

[0064] Figure 7 This is a graph of the location distribution results of a single type of target analyzed by listing the semantic segmentation results of the present invention.

[0065] Figure 8This is a statistical diagram of the number of various types of targets analyzed by listing the semantic segmentation results of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0068] Please also see Figures 1 to 3 A preferred embodiment of the present invention provides a real-time road state analysis and early warning method based on semantic segmentation, which mainly collects road video data, processes, segments and analyzes the video data to obtain early warning information, has a fast data processing speed, can achieve timely early warning, and reduce the occurrence of traffic accidents, and specifically includes the following steps:

[0069] S1, collecting road video data and preprocessing;

[0070] S2, the preprocessed data is input into the semantic segmentation model for segmentation processing to obtain the semantic segmentation result;

[0071] S3, the semantic segmentation results are input into the analysis module, and the analysis module obtains the position, size, area statistics, target category and quantity of a single type of target through analysis and calculation;

[0072] S4. The early warning module receives the data output by the analysis module and generates early warning data according to early warning rules.

[0073] Specifically, in step S1, road surface video data collection includes:

[0074] Install high-resolution and high-frame-rate cameras on the vehicle, adjust the camera installation angle and position to ensure that the complete road surface and subtle changes in the road surface can be captured, and ensure smooth video. Real-time data collection can be carried out after installation.

[0075] Specifically, in step S1, the preprocessing includes:

[0076] S11. Use a video frame extraction algorithm to obtain frame images in the video data. The video frame extraction algorithm can select a basic video frame extraction algorithm: use Python and OpenCV library to implement equal-time frame extraction, or use FFmpeg tools to obtain high-quality video frames. In addition, a motion detection-based frame extraction algorithm can also be used. The appropriate frame extraction algorithm can be selected according to the actual data situation.

[0077] S12: Data cleaning: removing unclear road information images and enhancing image data.

[0078] (1) Image clarity assessment: Use an image clarity scoring model to filter out blurry images and remove low-quality images caused by unclear images. The image clarity scoring model refers to the grayscale variance method. In addition, gradient-based methods such as Brenner gradient and Tenengrad gradient; frequency-domain-based methods such as the fast Fourier transform (FFT) method can also be used.

[0079] (2) Image quality assessment: removing low-quality images through an image quality assessment model, where the image quality assessment model refers to a full-reference image quality assessment method. In addition, semi-reference image quality assessment and no-reference image quality assessment methods can also be used.

[0080] (3) Image denoising: Image denoising is performed using an image processing algorithm. The selectable image processing algorithms include: median filter algorithm, Gaussian filter algorithm, total variation denoising algorithm, etc.

[0081] (4) Image restoration: using generative adversarial networks (GANs) to fill in missing areas of images, repair artifacts, etc.

[0082] (5) Data enhancement: Perform operations such as rotating, translating, scaling, and flipping images to increase data diversity and improve the effectiveness of subsequent model training.

[0083] Specifically, in step S2, the preprocessed image is input into the semantic segmentation model for segmentation processing:

[0084] The input image is first processed by performing two standard convolution operations, reducing the image resolution by two times and increasing the number of channels to 64, to obtain the feature F. c In a specific embodiment, the shape of the input image is (1, 3, 640, 640), and the shape of the output image after two standard convolutions is (1, 64, 160, 160). Through two standard convolution operations, the complexity of the feature map is increased, laying the foundation for subsequent deep processing.

[0085] Then perform feature fusion: feature F cThe input is sent to the pc2f convolution unit, and multi-scale features are obtained through feature extraction and concatenation at different scales. Figure 3 The pc2f convolution unit consists of three branches, each of which processes features of different scales. c After three branch feature operations in the pc2f convolution unit, for the input feature Fc, the first branch is a short connection, and the original feature is directly retained, which is recorded as feature F 1 , the second branch is the output feature F after the input feature undergoes a partial convolution 2 , the third branch is feature F 2 After two partial convolutions, the features and F 2 Fusion gets feature F 3 Finally, F 1 、F 2 、F 3 After splicing together, subsequent convolution operations are performed to obtain multi-scale features.

[0086] Then the output multi-scale features are processed by multiple standard convolution units and pc2f convolution units in turn, and repeated fusion is performed to finally obtain the feature F 4 .

[0087] Then perform dual-channel weighted processing: feature F 4 Input to the dual-branch attention mechanism unit. The dual-branch attention mechanism unit processing process is shown in Figure 4 The dual-branch attention mechanism unit is respectively provided with a spatial attention layer, a channel attention layer and a splicing layer. The spatial attention layer and the channel attention layer weight the spatial features and the channel features respectively. Finally, the features output by the two layers are spliced ​​through the splicing layer. The spliced ​​features are input into the fast spatial pyramid pooling module to obtain features of different scales F 5 .

[0088] Finally, for feature F 5 Post-processing: feature F is processed by upsampling unit, convolution unit and fusion unit 5 Multiple upsampling, convolution and decoding are performed to obtain the semantic segmentation result.

[0089] Specifically, after obtaining the semantic segmentation result, step S3 is performed, and the semantic segmentation result is input into the analysis module, such as Figure 5 As shown, the analysis module obtains the location, size, area statistics, target category and quantity of a single type of target through analysis and calculation. The analysis and calculation specifically includes the following steps:

[0090] Determine the target distribution: The semantic segmentation results are saved in json format, which contains the center point coordinates of each target. The semantic segmentation results are clustered using a clustering algorithm to obtain the position distribution of the target center points in the feature map. The center point coordinates of each target are used as the input data of the clustering algorithm. The K-means clustering algorithm, DBSCAN clustering algorithm or other clustering algorithms are used to set cluster clusters to obtain the distribution of the center point of each target in the image. The target position distribution results in the image are shown in the following example. Figure 6 As shown in the figure, the horizontal and vertical coordinates represent the width and height of the image respectively. The points in the figure represent the center points of each type of target in the semantic segmentation result. The positions of the points in the figure represent the positions of the targets in the image. The positions of these points directly reflect the actual positions of the targets in the image. Then, by considering the angle and height of the camera, an affine transformation is used to map the target position in the image to the position on the actual road surface. This ensures accurate positioning and distance estimation.

[0091] Target statistics: For each type of target, a post-processing algorithm is used to obtain the location, size, and area of ​​the target. The post-processing algorithm includes statistics for a single type of target and a number of targets. For each type of target, its location, size, and area are analyzed. These statistics are essential for understanding the distribution and size of specific targets. The statistical results can be displayed in the form of graphs, such as Figure 7 , the horizontal axis represents the width of the target box of the semantic segmentation result, and the vertical axis represents the height of the target box. The distribution of the points in the figure represents the width and height distribution of the target box, such as Figure 7 On the left, the width and height of the target box of category tq are both within 500 pixels. Figure 7 On the right, most of the target boxes of category bx are within 1500 pixels, and a few target boxes have larger widths and heights. Figure 7 The area distribution of the target can be calculated intuitively, and the size characteristics of the target in the image can be analyzed and understood in a clear and intuitive way; for multi-class target statistics, the number of all categories is counted, which helps to quickly evaluate the diversity and density of targets in the image, such as Figure 8 As shown, the horizontal axis represents the categories in the detection results, and the vertical axis represents the number of targets of each category. It can be seen from the figure that the number of targets of different categories can be clearly displayed in the statistical chart.

[0092] Finally, step S4 is performed, where the early warning module receives the data output by the analysis module and generates early warning data according to early warning rules. The early warning rules include:

[0093] When the target category is a pedestrian, that is, a pedestrian is detected on the road surface, a warning message of paying attention to pedestrians is generated;

[0094] When the target category is a vehicle, that is, a vehicle is detected on the road surface and the target area ratio is greater than 65% of the current frame, a warning message of "watch out for the vehicle ahead" is generated;

[0095] When the target category is cracks, that is, cracks are detected on the road surface, their location distribution is analyzed, and if the crack length is greater than 20cm and the width is greater than 5cm, a warning message of paying attention to cracks on a certain side of the road is generated;

[0096] When the target category is a pothole, that is, a pothole is detected on the road surface, its location distribution is analyzed, and if the area is greater than 0.01 square meters, a warning message of road potholes is generated;

[0097] When the target category is road waterlogging, that is, road waterlogging is detected on the road surface, the area size is analyzed, and if the area is greater than 1 square meter, a warning message of road waterlogging is generated;

[0098] When the target category is a foreign object, that is, a foreign object is detected on the road, the location and area are analyzed, and if the area is greater than 0.01 square meters, a warning message of "Pay attention to foreign objects on the road" is generated;

[0099] When the target category is a lane line, the road lane line detection is performed. According to the lane line detection results, it is analyzed whether the current vehicle is separated from the lane line. If separated, a lane departure warning message is generated.

[0100] The specific steps of determining lane separation are as follows:

[0101] a. Extracting lane line edge points from lane line masks based on the analysis results obtained by the analysis module;

[0102] b. Use a quadratic polynomial to fit the lane lines on the left and right sides:

[0103] y l =A l x 2 +B l x+C l ,

[0104] y r =A r x 2 +B r x+C r ,

[0105] Among them, A, B, and C are polynomial coefficients, l represents the left lane line, and r represents the right lane line;

[0106] c. Calculate the midpoint of the lane line corresponding to the y value closest to the lane line:

[0107]

[0108] d. Calculate the vehicle offset in the current frame. If it is greater than the threshold, it is considered that the lane is offset. The offset calculation formula is as follows:

[0109] where xw is the width of the current frame, α is the offset;

[0110] e. If the offsets of 10 consecutive frames are all greater than the threshold, a warning message is generated to indicate lane departure.

[0111] In this embodiment, the loss function FL(p t )、TL(α,β)、L DFL , the loss function FL(p t ) focuses on the feature level loss, and the loss function TL(α, β) focuses on the target detection loss, and the loss function L DFL For the semantic segmentation task, the three loss functions work together to help optimize the semantic segmentation network parameters so that the model can achieve the best performance in multiple dimensions and obtain the optimal semantic segmentation model. The definitions of each loss function are as follows:

[0112] L = FL (p t )+TL(α,β)+L DFL (1)

[0113]

[0114]

[0115] FL(p t )=-α t (1-p t ) γ log(p t ) (4),

[0116] Wherein, in formula (2), M is the number of samples; N is the number of categories; i represents the i-th sample; j represents the j-th category; k represents the four parameters of the bounding box (usually x, y, w, h, representing the center point coordinates and width and height, respectively); is the probability distribution of the true label after processing. For the positive sample position, its value is a specific probability distribution, and for the negative sample position, its value is 0; p ijk is the probability distribution predicted by the model; j is the category weight coefficient, which is used to balance the importance of different categories; γ is the focusing parameter, which is used to adjust the weight of difficult and easy samples. When the sample is easy to classify, The value of is small, thereby reducing the loss weight of easy-to-classify samples and making the model pay more attention to difficult-to-classify samples. α is another adjustment parameter used to further adjust the characteristics of the loss function. In formula (3), p0 i is the probability that the i-th target is the target, p1 iThe probability that the i-th target is not the target, α, β are used to weigh the false positives and true positives. In formula (4), p t is the predicted probability value, α t , γ are parameters.

[0117] In this embodiment, the steps are also included: optimizing the semantic segmentation model and communicating with the vehicle system. The semantic segmentation model is accelerated using TensorRT software to simplify the model structure, reduce the number of weights and resource utilization. The optimized semantic segmentation model, analysis module and warning module are then deployed on the JetsonTX2 computing box. The performance of the JetsonTX2 computing box is as follows: 256-core NVIDIA Pascal GPU, dual-core Denver 64-bit CPU and quad-core Arm Cortex-A57 MPCore processor, 8GB video memory, and 32GB storage space. Finally, a test is performed. To ensure that the accuracy and response speed of the overall model meet the real-time requirements of the vehicle system, a series of tests are performed, including functional tests: verifying whether the model can correctly complete the semantic segmentation task; performance tests: testing the reasoning speed of the model; stability tests: whether the model reasoning is stable and reliable for a long time. Connection between the vehicle system and the computing box. Connect the Jetson TX2 to the vehicle's display, which may require an appropriate interface converter or adapter to ensure signal compatibility. Set display parameters to ensure that images and warning information can be clearly displayed on the driver's display screen, and implement a voice feedback system to provide auditory warnings. This completes feedback warnings and ensures safe driving.

[0118] Another preferred embodiment of the present invention provides a real-time road surface state analysis and early warning system based on semantic segmentation, including an acquisition module, a preprocessing module, a semantic segmentation model, an analysis module and an early warning module connected in sequence, wherein the acquisition module is used to acquire road surface video data, the preprocessing module is used to preprocess the road surface video data, and the preprocessing steps include video frame extraction, data cleaning, data annotation, data statistical analysis, data enhancement and data splitting, the semantic segmentation model adopts a deep learning model, which performs semantic segmentation on the preprocessed image, and the analysis module is provided with a clustering algorithm and a post-processing algorithm for performing road condition analysis on the semantically segmented image, and the early warning module gives corresponding early warning information according to the road condition analysis given by the analysis module. This system can realize real-time semantic segmentation of road surface information using a vehicle-mounted device, and realize real-time road surface state detection, analysis and early warning by analyzing the semantic segmentation results and preset early warning rules. This can effectively improve the accuracy of scene semantic segmentation, quickly analyze the detection results, and provide immediate feedback and early warning, especially in a traffic environment with poor road conditions, which helps to reduce the occurrence of traffic accidents and escort safe driving.

[0119] The semantic segmentation model includes a feature extraction module, a dual-branch attention mechanism unit, an SPPF pooling unit and a post-processing module connected in sequence; the feature extraction module includes a number of standard convolution units and pc2f convolution units that are alternately arranged; the dual-branch attention mechanism unit includes a spatial attention layer, a channel attention layer and a splicing layer, and the splicing layers are connected to the spatial attention layer and the channel attention layer; the post-processing module includes a number of upsampling units, convolution units and fusion units.

[0120] In this embodiment, the preprocessing module also includes the step of automatically labeling data:

[0121] (1) Take a portion of images from various scenes for manual annotation, including different types of scenes such as urban streets, rural roads, and highways. In addition, scene images with different lighting, angles, and occlusion conditions should also be selected.

[0122] (2) Pre-trained model training.

[0123] Select a pre-trained model. First, select ResNet as the pre-trained model. You can also select other models, such as VGG or Inception, which are deep learning models pre-trained on the ImageNet dataset.

[0124] Model training, initial stage. Freeze some or all layers of the pre-trained model and train the last few layers with the new dataset to adapt to the new task.

[0125] Model training and model fine-tuning. Unfreeze some layers and fine-tune the model to optimize performance. During model fine-tuning, use data augmentation methods such as rotation, scaling, and cropping to improve the generalization ability of the model. In addition, hyperparameters such as learning rate, batch size, and optimizer need to be adjusted to obtain the best performance.

[0126] Model training and rigorous evaluation. Fine-tune the trained model with new data. After completing the initial training, the model needs to be rigorously evaluated, such as through indicators such as accuracy, recall rate, F1 score, confusion matrix, etc., and adjust and optimize the model based on the evaluation results to obtain the optimal pre-trained model.

[0127] (3) Automatically label the remaining data. During the automatic labeling process, a certain confidence threshold is set, generally between 0.8 and 0.95. Only when the confidence of the model prediction is higher than a certain threshold, the labeling result is accepted. Then, the automatic labeling results are manually checked, and samples that do not reach the confidence threshold are manually labeled. The wrong samples are added back to the training set for further model optimization. Through continuous model evaluation and optimization, the performance of the model can be gradually improved, the need for manual labeling can be reduced, and the high quality of data labeling can be guaranteed.

[0128] (4) Split the labeled data into training set, test set and validation set in a ratio of 8:1:1. Use the holdout method to divide the data into three parts: 80% of the training set is used for semantic segmentation model learning, 10% of the validation set is used for semantic segmentation model tuning (such as hyperparameter selection), and the last 10% of the test set is used for final semantic segmentation model evaluation. It is also necessary to ensure that the data distribution of each subset is consistent during the data splitting process to avoid a specific category or feature appearing too much or too little in a certain subset. The semantic segmentation model that has passed the training and testing can perform feature segmentation well.

[0129] In some other preferred embodiments of the present invention, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method described in the above embodiment.

[0130] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0131] The above description is a detailed description of the preferred feasible embodiments of the present invention, but the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modified changes completed under the technical spirit suggested by the present invention should fall within the patent scope covered by the present invention.

Claims

1. A real-time road condition analysis and early warning method based on semantic segmentation, characterized in that: Includes steps: Collect road video data and pre-process it; The preprocessed data is input into the semantic segmentation model for segmentation processing to obtain the semantic segmentation result; The semantic segmentation results are input into the analysis module, which obtains the position, size, area statistics, target category and quantity of single-class targets through analysis and calculation; The early warning module receives the data output by the analysis module and generates early warning data according to the early warning rules.

2. The real-time road condition analysis and early warning method based on semantic segmentation according to claim 1 is characterized in that: The method also includes the steps of optimizing the semantic segmentation model and communicating with the vehicle-mounted system.

3. The real-time road condition analysis and early warning method based on semantic segmentation according to claim 2 is characterized in that: The segmentation process in the semantic segmentation model specifically includes: Preliminary processing: Perform two standard convolutions on the input image to obtain feature F c ; Feature fusion: Feature F c The input is sent to the pc2f convolution unit, and multi-scale features are obtained through feature extraction and concatenation at different scales. Repeated fusion: After multiple pc2f convolution units and standard convolution units, feature F4 is obtained; Dual-channel weighting: Feature F4 is input into the dual-branch attention mechanism unit, and features F5 of different scales are obtained through weighting, concatenation, and pooling of spatial features and channel features; Post-processing: Feature F5 is upsampled and decoded to obtain the semantic segmentation result. Through this process, the spatial resolution is gradually restored, so that the final output feature map is consistent in size with the original input image, achieving pixel-level classification.

4. The real-time road condition analysis and early warning method based on semantic segmentation according to claim 1 is characterized in that: The loss function FL(p t )、TL(α,β)、L DFL , the loss function FL(p t ) focuses on the feature level loss, the loss function TL(α,β) focuses on the target detection loss, the loss function L DEL For semantic segmentation tasks.

5. The real-time road condition analysis and early warning method based on semantic segmentation according to claim 1 is characterized in that: The analysis operation specifically includes the steps of: Determine target distribution: cluster the semantic segmentation results using a clustering algorithm to obtain the position distribution of the target center point in the feature map, and affine transform the position distribution on the feature map to the target position on the actual road surface; Target statistics: For each type of target, a post-processing algorithm is used to obtain its position, size and area.

6. The real-time road condition analysis and early warning method based on semantic segmentation according to claim 5 is characterized in that: The warning rules include: If a pedestrian is detected on the road surface, a warning message of "Pay attention to pedestrians" is generated; If a vehicle is detected on the road and the target area ratio is greater than 65% of the current frame, a warning message of "Watch out for the vehicle ahead" is generated; When cracks are detected on the road surface, their location distribution is analyzed. If the crack length is greater than 20cm and the width is greater than 5cm, a warning message of road cracks on a certain side is generated; When a pothole is detected on the road, its location and distribution are analyzed. If the area is larger than 0.01 square meters, a warning message "Pay attention to potholes" is generated; When water is detected on the road surface, the size of the area is analyzed. If the area is larger than 1 square meter, a warning message of road water accumulation is generated; If a foreign object is detected on the road, its location and area are analyzed. If the area is larger than 0.01 square meters, a warning message "Pay attention to foreign objects on the road" is generated; Road lane line detection: Based on the lane line detection results, it analyzes whether the current vehicle is separated from the lane line. If separated, a lane departure warning message is generated.

7. The real-time road condition analysis and early warning method based on semantic segmentation according to claim 6 is characterized in that: The specific steps of determining lane separation are as follows: a. Extracting lane line edge points from lane line masks based on the analysis results obtained by the analysis module; b. Use a quadratic polynomial to fit the lane lines on the left and right sides: y l =A l x 2 +B l x+C l , y r =A r x 2 +B r x+C r , Among them, A, B, and C are polynomial coefficients, l represents the left lane line, and r represents the right lane line; c. Calculate the midpoint of the lane line corresponding to the y value closest to the lane line: d. Calculate the vehicle offset in the current frame. If it is greater than the threshold, it is considered that the lane is offset. The offset calculation formula is as follows: where x w is the width of the current frame, α is the offset; e. If the offsets of 10 consecutive frames are all greater than the threshold, a warning message is generated to indicate lane departure.

8. A real-time road surface condition analysis and early warning system based on semantic segmentation, used to implement the real-time road surface condition analysis and early warning method based on semantic segmentation according to any one of claims 1 to 7, characterized in that: It includes an acquisition module, a preprocessing module, a semantic segmentation model, an analysis module and an early warning module which are connected in sequence. The acquisition module is used to acquire road video data. The preprocessing module is used to preprocess the road video data. The preprocessing steps include video frame extraction, data cleaning, data labeling, data statistical analysis, data enhancement and data splitting. The semantic segmentation model adopts a deep learning model, which performs semantic segmentation on the preprocessed image. The analysis module is provided with a clustering algorithm and a post-processing algorithm for performing road condition analysis on the semantically segmented image. The early warning module gives corresponding early warning information according to the road condition analysis given by the analysis module.

9. The real-time road condition analysis and early warning system based on semantic segmentation according to claim 8, characterized in that: The semantic segmentation model includes a feature extraction module, a dual-branch attention mechanism unit, an SPPF pooling unit and a post-processing module connected in sequence; the feature extraction module includes a number of standard convolution units and pc2f convolution units that are alternately arranged; the dual-branch attention mechanism unit includes a spatial attention layer, a channel attention layer and a splicing layer, and the splicing layers are connected to the spatial attention layer and the channel attention layer; the post-processing module includes a number of upsampling units, convolution units and fusion units.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images