Road traffic signs and markings rapid detection and troubleshooting methods
Through the technology combined with CCD camera and laser scanner, road traffic signs and markings are quickly identified and positioned, and the problem of low detection efficiency in the existing technology is solved, intelligent traffic signs and markings are realized and problem detection is improved, and work is improved. Efficiency and normativeness.
Patent Information
- Application Number
- CN202111597696.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-08-31
- Filing Date
- 2021-12-24
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-12-24
AI Technical Summary
In the existing technology, the inspection of road traffic signs and markings is inefficient and relies on manual investigation, resulting in incomplete detection of hidden dangers and the inaccurate identification and correction of problems of irregular or missing settings.
The CCD camera is used to collect videos, combine laser scanner to obtain point cloud data, and use GNSS, IMU and DMI positioning devices to obtain vehicle position information. The traffic signs and markings are identified through image processing and feature extraction, and spatial data is generated, and compared with the standard specification database. Yes, generate a problem investigation report intelligently.
It has achieved rapid and accurate detection and identification of traffic signs and markings, improved work efficiency, generated intelligent inspection reports, improved the existing work conditions, and improved the standardization and safety of traffic signs and markings.
Smart Images

Figure CN114299247B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to road traffic technology, and in particular to a method for rapid detection and troubleshooting of road traffic signs and markings. Background Art
[0002] Road traffic signs and markings are the basic infrastructure for road traffic management. They can regulate drivers' driving behavior and quickly provide them with information about driving bans, warnings, and tips. In recent years, with the development of road traffic construction, the use rate of road traffic signs and markings has gradually increased, and higher requirements have been put forward for their standard setting.
[0003] With the rapid development of social economy, the number of cars has grown excessively, causing serious traffic congestion and traffic accidents. Among them, the lack of traffic signs and markings, irregular and unreasonable settings, and drivers' confusion or failure to pay attention to traffic signs are one of the important reasons for these traffic accidents. There are still many problems in the use of traffic signs and markings in traffic management practice, which urgently need to be systematically sorted out and studied in depth in order to formulate corresponding improvement measures in a targeted manner.
[0004] In order to further standardize the setting and management of road traffic signs and signboards, and prevent and reduce the occurrence of road traffic accidents, the traffic management department often organizes the inspection and rectification of hidden dangers of road traffic signs and signboards. The centralized inspection and rectification focuses on the inspection of whether the road signs and signboards are set irregularly, irrationally, missing, damaged, etc., and timely maintenance and replacement of problematic signs and signboards to ensure that the traffic signs and signboards are clear, eye-catching, accurate and intact.
[0005] The traditional way of conducting road traffic sign and marking problem investigation and rectification through manual field inspection has problems such as low investigation efficiency, high reliance on investigators' work experience, and incomplete hidden danger investigation. The rapid detection and problem investigation method of road traffic sign and marking can improve the existing working conditions and improve the work efficiency of staff. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide a method for rapid detection and troubleshooting of road traffic signs and markings in view of the defects in the prior art.
[0007] The technical solution adopted by the present invention to solve the technical problem is: a method for rapid detection and troubleshooting of road traffic signs and markings, comprising the following steps:
[0008] 1) Collect the target road video through the CCD camera, and extract the frame-by-frame image containing the target traffic sign and marking from the target video;
[0009] 2) Obtain target road point cloud data through laser scanner;
[0010] 3) Obtaining vehicle positioning data through a vehicle positioning device;
[0011] The vehicle positioning device composed of the GNSS (Global Navigation Satellite System), IMU (Inertial Measurement Unit) and DMI (Distance Measurement Indicator) can quickly and accurately collect vehicle location information;
[0012] 4) Identify the image containing the target traffic sign and marking, output the type of traffic sign and marking, and number the traffic sign and marking and the road to which it belongs;
[0013] Performing text recognition on traffic sign characters and outputting layout information on the traffic sign; the layout information includes the characters on the traffic sign;
[0014] 5) According to the positioning data of the target road video, the location information of the output traffic signs and markings is obtained, including the latitude and longitude data of the traffic signs and markings;
[0015] 6) Calculate the traffic sign and marking spatial data including the traffic sign and marking interval, the relative position of the traffic sign and marking to the intersection / crossing, and the distance between traffic signs based on the traffic sign and traffic marking edge point coordinates and the longitude and latitude of the target road stop line;
[0016] 7) Develop a sign and marking criteria database based on standards and specifications related to urban road and highway traffic signs and markings;
[0017] 8) Compare the acquired traffic sign and marking spatial data and attribute data of the target road with the sign and marking criteria database based on the standards and specifications of urban road and highway traffic signs and markings, detect the non-standard or unreasonable problems in the presence, location, sign layout information, etc. of the traffic signs and markings at specific locations, formulate corresponding improvement measures in a targeted manner, and intelligently generate problem investigation reports;
[0018] The attribute data includes the number, type, direction and layout information of traffic sign markings.
[0019] According to the above scheme, the image containing the target traffic sign and marking is identified in step 4), specifically as follows:
[0020] 4.1) Image preprocessing: For the image containing the target traffic sign and marking, the frequency domain-based filtering is used to smooth and reduce the noise of the image, the histogram equalization method is used to enhance the regional features of the traffic sign and marking of the denoised image, and the Canny edge detection method is used to extract the edge of the traffic marking to obtain the preprocessed image containing the target traffic sign and marking;
[0021] 4.2) For the pre-processed image containing the target traffic sign and marking, roughly locate the traffic sign based on the YCbCr color space to obtain an image retaining the target traffic sign area;
[0022] 4.3) The image of the retained target traffic sign area is subjected to shape analysis to locate the circular area, rectangular area, and triangular area (the area with a rectangularity / circularity greater than 0.8 is determined to be a rectangular / circular area, and the area with a triangle interval threshold between 0.85-1.15 is determined to be a triangular area); the area is cropped out and input into the designed traffic sign classifier as the traffic sign to be identified for identification; the traffic sign classifier is a recognition model based on a BP neural network.
[0023] 4.4) using an improved random sampling consensus algorithm to identify the traffic markings in the image containing the target traffic sign markings;
[0024] According to the above scheme, the position information of the traffic sign and marking in step 5) is determined based on the image containing the target traffic sign and the target road point cloud data, as follows:
[0025] 5.1) Obtain a stereoscopic image through the CCD camera; the stereoscopic image includes the image information of the target traffic sign, the internal orientation elements of the image and the external orientation element information of the camera at the moment of shooting; wherein, in the stereoscopic image, a certain ground point (X A ,Y A ,Z A ) in the left and right images and The internal orientation elements are obtained by calibration, and the external orientation elements are calculated by obtaining the position information and posture information of the vehicle at the moment of shooting through the vehicle positioning device;
[0026] 5.2) performing image matching; using an image matching method based on grayscale matching to identify points of the same name between the two images, wherein the grayscale matching determines the relationship between the template and the matching area by calculating the matching measure of the grayscale between the template and the image;
[0027] 5.3) Setting matching constraints; the matching constraints are the conditions that must be satisfied first during the matching process of the stereo image pair. In the matching of homonymous points of sequential images, the constraints are epipolar constraints; wherein the epipolar constraints are: epipolar lines of the same name must pass through the same name points;
[0028] 5.4) Grayscale matching: firstly, a matching template is selected, and then a traffic sign is searched in the image containing the traffic sign, a search subwindow of the template size is slid across the entire image, and a matching measure R between the template and the image containing the traffic sign in the subwindow is calculated using a specified method f(x, y) and saved in an array. After the calculation is completed, the maximum value in the array is the best template matching position;
[0029] 5.5) Feature extraction: Extract features from traffic signs and use the extracted feature points as target points for automatically acquiring image points;
[0030] 5.6) Matching of points with the same name;
[0031] Based on the image containing the traffic sign, edge extraction is performed on the area to be detected; based on the edge image, edge corner points in the traffic sign are selected as matching feature points, and the corner point coordinates are obtained by using the corner point extraction operator and the straight line corner points extracted by Hough transform, and further corner points are obtained; based on the edge image, Hough straight line extraction is performed on it, and the extracted straight lines are merged, and the points where the straight line intersections are in the detected area are formed into an intersection point set; the union of the corner point set and the intersection point set is used as the feature point set P to be matched F ; Taking the points in the feature point set as known image point coordinates, generating the same-name epipolar lines in the stereo image pair according to the properties of the epipolar lines;
[0032] The feature point set P extracted from the left image F (L) Pick any point P1 (L) , calculated through P1 (L) The epipolar line K with the same name is located in the right image. R , according to the kernel line constraint, calculate the right image point set P F (R) Midpoint to epipolar line K R Distance d, select points with d<∈ to form the point set P of the check core line constraint condition K (R) ; Select P1 (L) The grayscale matrix with the size of n×n as the center is used as the matching template, along the kernel line K R Perform grayscale matching on the right image and find the point set P that satisfies the maximum matching coefficient and is located on the same-name epipolar line. m (R) ; Point set P K (R) With point set P m (R) It is judged that if there is a corresponding relationship between the points in the two point sets, they are a pair of image points with the same name.
[0033] 5.7) Traffic sign positioning;
[0034] Based on feature matching, we can get the same-name image points on the stereo pair, and then we can get two spatial rays with the same name. Then, according to the collinear equation, we know that the two rays must intersect at the spatial position of the ground point of the same-name image point. There are three unknowns in the simultaneous equations of the unknown ground points, namely the ground coordinates (X, Y, Z). Using the coordinates of the same-name image points on the stereo pair (x0, y0) (x1, y1), four equations can be listed to solve the three unknowns, namely the positions of the ground points corresponding to the same-name image points. The calculated ground point coordinates are counted, and the points with large deviations are regarded as gross errors. The gross errors are eliminated using the robust estimation method, and finally the average value of the remaining coordinate points is taken as the three-dimensional coordinates of the traffic sign. Among them, the three-dimensional coordinates of the traffic sign are the coordinates in the image space coordinates. Based on the internal orientation elements and the conversion relationship between the image space coordinate system and the vehicle body coordinate system, the coordinates of the traffic sign in the vehicle body coordinate system are obtained. Since the vehicle coordinate system is a right-handed system and the Gaussian coordinate system is a left-handed system, a transition coordinate system is assumed and then the conversion relationship between the coordinates in the vehicle coordinate system and the Gaussian coordinate system can be obtained according to the exterior orientation elements. According to the conversion relationship between the image space coordinate system and the vehicle coordinate system and the conversion relationship between the vehicle coordinate system and the Gaussian coordinate system, the coordinates of the traffic sign in the Gaussian coordinate system can be obtained, and finally, according to the conversion relationship between the Gaussian coordinate system and the earth coordinate system, the longitude and latitude of the traffic sign can be obtained.
[0035] According to the above scheme, the sign and marking criteria database in step 7) is used to store traffic sign setting specifications and traffic marking setting specifications. The traffic sign setting specifications include traffic sign number, type, name, meaning, layout information, relative setting position of the sign, and the relative position of the sign and the intersection; the traffic marking setting specifications include traffic marking number, type, name, meaning, layout information, marking interval, and the relative position of the marking and the intersection.
[0036] According to the above scheme, generating the troubleshooting report in step 8) includes the following process:
[0037] 8.1) According to the sign and marking criteria database, determine whether there are corresponding traffic signs and markings at the corresponding position of the intersection / crossing; if there are, it meets the specifications and proceed to the next step; if it does not meet the specifications, record the problems of traffic signs and markings in the database, and propose corresponding improvement measures according to the sign and marking criteria database;
[0038] 8.2) According to the detected traffic markings, determine whether there are corresponding traffic signs at the corresponding position of the traffic markings; if there are, it meets the specifications and proceed to the next step; if it does not meet the specifications, record the problems of traffic signs and markings in the database and solve them according to the sign and marking standard data
[0039] 8.3) According to the direction of the traffic markings, determine whether the direction of the traffic markings at the intersection is consistent with the direction of the corresponding traffic signs. If they are consistent, they meet the specifications and proceed to the next step; if they do not meet the specifications, record the problems of the traffic signs and markings in the database, and propose corresponding improvement measures according to the sign and marking criteria database;
[0040] 8.4) Based on the distance between the traffic sign and the intersection, determine whether the relative position of the traffic sign and the intersection meets the specifications. If so, proceed to the next step. If not, record the problems of the traffic sign in the database and propose corresponding improvement measures based on the sign and marking criteria database.
[0041] 8.5) Determine the distance between traffic signs and markings based on the distance between traffic signs and markings
[0042] Whether it meets the specifications. If it does, proceed to the next step. If it does not meet the specifications, record the problems of traffic signs and markings in the database and propose corresponding improvement measures according to the sign and marking standard database;
[0043] The beneficial effects produced by the present invention are:
[0044] The method of the present invention can quickly and accurately detect and identify the type of traffic signs and markings on the target road, obtain the position information of traffic signs and markings, realize the detection and storage of traffic sign and marking information on the target road, and intelligently generate a road inspection and rectification list, thereby improving the existing working conditions and improving the work efficiency of the staff. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:
[0046] Figure 1 is a method flow chart of an embodiment of the present invention;
[0047] Figure 2 It is a flowchart of generating a troubleshooting report according to an embodiment of the present invention. DETAILED DESCRIPTION
[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0049] like Figure 1 As shown, a method for rapid detection and troubleshooting of road traffic signs and markings, the method comprising:
[0050] 1) Collect the target road video through the CCD camera, and extract the frame-by-frame image containing the target traffic sign and marking from the target video;
[0051] The traffic signs in this embodiment are safety facilities that convey legal information about traffic management and control to drivers and pedestrians, such as warning signs, prohibition signs, instruction signs, guide signs, tourist area signs, and road construction safety signs; traffic markings are traffic facilities consisting of various lines, arrows, words, patterns, facade marks, physical marks, raised road signs, and contour marks drawn or installed on urban roads, such as instruction markings, prohibition markings, and warning markings. The extracted frame-by-frame images contain traffic signs and markings.
[0052] 2) Obtain target road point cloud data through laser scanner;
[0053] 3) Obtaining vehicle positioning data through a vehicle positioning device;
[0054] In this embodiment, the vehicle location information is quickly and accurately collected through a vehicle positioning device composed of a GNSS (Global Navigation Satellite System), an IMU (Inertial Measurement Unit) and a DMI (Distance Measurement Indicator);
[0055] in,
[0056] GNSS (Global Navigation Satellite System); A technology integration based on RTK to obtain the geodetic coordinates of the vehicle at the moment of movement in real time;
[0057] IMU (Inertial Measurement Unit): obtains the attitude data of the device during operation, including pitch angle, roll angle and yaw angle;
[0058] DMI (Distance Measurement Indicator): Installed on one wheel of the vehicle, it measures the rotation of the tire to calculate the distance traveled by the vehicle. It provides auxiliary information for GNSS (Global Navigation Satellite System) and IMU (Inertial Measurement Unit);
[0059] 4) Identify the image containing the target traffic sign and marking, output the type of traffic sign and marking, and number the traffic sign and marking and the road to which it belongs;
[0060] Identify images containing target traffic signs and markings as follows:
[0061] 4.1) Image preprocessing: For the image containing the target traffic sign and marking, the frequency domain-based filtering is used to smooth and reduce the noise of the image, the histogram equalization method is used to enhance the regional features of the traffic sign and marking of the denoised image, and the Canny edge detection method is used to extract the edge of the traffic marking to obtain the preprocessed image containing the target traffic sign and marking;
[0062] 4.2) For the pre-processed image containing the target traffic sign and marking, roughly locate the traffic sign based on the YCbCr color space to obtain an image retaining the target traffic sign area;
[0063] 4.3) The image of the retained target traffic sign area is subjected to shape analysis to locate the circular area, rectangular area, and triangular area (the area with a rectangularity / circularity greater than 0.8 is determined to be a rectangular / circular area, and the area with a triangle interval threshold between 0.85-1.15 is determined to be a triangular area); the area is cropped out and input into the designed traffic sign classifier as the traffic sign to be identified for identification; the traffic sign classifier is a recognition model based on a BP neural network.
[0064] 4.4) using an improved random sampling consensus algorithm to identify the traffic markings in the image containing the target traffic sign markings;
[0065] Performing text recognition on traffic sign characters and outputting layout information on the traffic sign; the layout information includes the characters on the traffic sign;
[0066] In this embodiment, the traffic sign layout information recognition adopts the following method, including:
[0067] Using a K-means color clustering method based on RGB color space, the image containing traffic signs and markings is operated to obtain a binary image, and the background is filtered out to obtain a candidate area of the text;
[0068] Performing a connected domain analysis on the candidate area, segmenting and extracting the text using text features, and obtaining a segmented text image;
[0069] Inputting the segmented text image into an OCR recognition system to recognize the text information on the traffic sign;
[0070] 5) According to the positioning data of the target road video, the location information of the output traffic signs and markings is obtained, including the latitude and longitude data of the traffic signs and markings;
[0071] 5.1) Obtain a stereoscopic image through the CCD camera; the stereoscopic image includes the image information of the target traffic sign, the internal orientation elements of the image and the external orientation element information of the camera at the moment of shooting; wherein, in the stereoscopic image, a certain ground point (X A ,Y A ,Z A ) in the left and right images and The internal orientation elements are obtained by calibration, and the external orientation elements are calculated by obtaining the position information and posture information of the vehicle at the moment of shooting through the vehicle positioning device;
[0072] 5.2) performing image matching; using an image matching method based on grayscale matching to identify points of the same name between the two images, wherein the grayscale matching determines the relationship between the template and the matching area by calculating the matching measure of the grayscale between the template and the image;
[0073] 5.3) Setting matching constraints; the matching constraints are the conditions that must be satisfied first during the matching process of the stereo image pair. In the matching of homonymous points of sequential images, the constraints are epipolar constraints; wherein the epipolar constraints are: epipolar lines of the same name must pass through the same name points;
[0074] 5.4) Grayscale matching: firstly, a matching template is selected, and then a traffic sign is searched in the image containing the traffic sign, a search subwindow of the template size is slid across the entire image, and a matching measure R between the template and the image containing the traffic sign in the subwindow is calculated using a specified method f(x, y) and saved in an array. After the calculation is completed, the maximum value in the array is the best template matching position;
[0075] 5.5) Feature extraction: Extract features from traffic signs and use the extracted feature points as target points for automatically acquiring image points;
[0076] 5.6) Matching of points with the same name;
[0077] Based on the image containing the traffic sign, edge extraction is performed on the area to be detected; based on the edge image, edge corner points in the traffic sign are selected as matching feature points, and the corner point coordinates are obtained by using the corner point extraction operator and the straight line corner points extracted by Hough transform, and further corner points are obtained; based on the edge image, Hough straight line extraction is performed on it, and the extracted straight lines are merged, and the points where the straight line intersections are in the detected area are formed into an intersection point set; the union of the corner point set and the intersection point set is used as the feature point set P to be matched F ; Taking the points in the feature point set as known image point coordinates, generating the same-name epipolar lines in the stereo image pair according to the properties of the epipolar lines;
[0078] The feature point set P extracted from the left image F (L) Pick any point P1 (L) , calculated through P1 (L) The epipolar line K with the same name is located in the right image. R , according to the kernel line constraint, calculate the right image point set P F (R) Midpoint to epipolar line K R Distance d, select points with d<∈ to form the point set P of the check core line constraint condition K (R) ; Select P1 (L)The grayscale matrix with the size of n×n as the center is used as the matching template, along the kernel line K R Perform grayscale matching on the right image and find the point set P that satisfies the maximum matching coefficient and is located on the same-name epipolar line. m (R) ; Point set P K (R) With point set P m (R) It is judged that if there is a corresponding relationship between the points in the two point sets, they are a pair of image points with the same name.
[0079] 5.7) Traffic sign positioning;
[0080] Based on feature matching, we can get the same-name image points on the stereo pair, and then we can get two spatial rays with the same name. Then, according to the collinear equation, we know that the two rays must intersect at the spatial position of the ground point of the same-name image point. There are three unknowns in the simultaneous equations of the unknown ground points, namely the ground coordinates (X, Y, Z). Using the coordinates of the same-name image points on the stereo pair (x0, y0) (x1, y1), four equations can be listed to solve the three unknowns, namely the positions of the ground points corresponding to the same-name image points. The calculated ground point coordinates are counted, and the points with large deviations are regarded as gross errors. The gross errors are eliminated using the robust estimation method, and finally the average value of the remaining coordinate points is taken as the three-dimensional coordinates of the traffic sign. Among them, the three-dimensional coordinates of the traffic sign are the coordinates in the image space coordinates. Based on the internal orientation elements and the conversion relationship between the image space coordinate system and the vehicle body coordinate system, the coordinates of the traffic sign in the vehicle body coordinate system are obtained. Since the vehicle coordinate system is a right-handed system and the Gaussian coordinate system is a left-handed system, a transition coordinate system is assumed and then the conversion relationship between the coordinates in the vehicle coordinate system and the Gaussian coordinate system can be obtained according to the exterior orientation elements. According to the conversion relationship between the image space coordinate system and the vehicle coordinate system and the conversion relationship between the vehicle coordinate system and the Gaussian coordinate system, the coordinates of the traffic sign in the Gaussian coordinate system can be obtained, and finally, according to the conversion relationship between the Gaussian coordinate system and the earth coordinate system, the longitude and latitude of the traffic sign can be obtained.
[0081] This embodiment also provides a traffic marking positioning method, including:
[0082] Traffic marking positioning method. Including:
[0083] 5.8) Image-based edge detection of road traffic markings. First, image preprocessing is performed, including ROI (region of interest) selection, image grayscale, filtering and denoising, etc., and then the Canny edge detection method is used to detect the edges of road traffic markings. The specific steps are: based on the collected image containing traffic markings, ROI (region of interest) selection is performed, and interference information such as trees and buildings that are useless for traffic marking detection in the image containing traffic markings is deleted, and the part of the image containing traffic markings is highlighted. Based on the ROI area of the image, grayscale processing is performed using the weighted average method. Based on the grayscale image, filtering and denoising is performed using frequency domain-based filtering to smooth the noise points in the grayscale image, and then the preprocessed image is obtained. Based on the preprocessed image, the Canny edge detection method is used to detect the edges of road traffic markings to obtain the edges of traffic markings in the preprocessed image.
[0084] 5.9) Extraction of internal points of road traffic markings based on point cloud. Based on the target road point cloud data, ROI selection is performed to remove most of the outliers and noise points. The point cloud is projected onto the image, the point cloud is registered with the image, the ROI range of the point cloud is determined according to the ROI range on the image, and then the point cloud data ROI is obtained. Based on the point cloud data ROI, a voxel filter is used to further remove the noise points and outliers of the point cloud data, and then the preprocessed point cloud data is obtained. Based on the preprocessed point cloud data, the point cloud is divided into two parts, the background and the traffic marking, according to the point cloud echo intensity, the inter-class variance of the two parts of the point cloud is calculated, and the optimal segmentation threshold t is obtained when the inter-class variance is the largest. The point clouds that exceed the threshold t are determined to be internal points of the road traffic markings.
[0085] 5.10) Geometric correction of images and point clouds and positioning of traffic markings. Based on the edges of the road traffic markings and the internal points of the road traffic markings in the image, a statistical filter is used to use the edge points to filter the internal points, and the internal points to filter the edge points, and the false detection data are filtered out mutually. After the false detection data is eliminated, if the ROI area is too large and the ground is uneven, the K-means clustering algorithm is used to cluster the edge points of the road traffic markings, classify the road traffic markings, and then redetermine the geometric correction area based on the clustering results and the maximum and minimum values of the point cloud between the X and Y axes. The geometric correction area is the road traffic marking area. The point cloud of the geometric correction area is fitted into a plane by the least squares method, and the rotation matrix R and translation matrix T between the plane coordinate system and the two-dimensional coordinate system with only x and y coordinates are calculated and converted to the two-dimensional coordinate system. The two-dimensional plane coordinate system is defined as the intermediate coordinate system. The polynomial correction method in the digital image geometric correction method is used to calculate the parameters of the polynomial correction through the coordinates of the point cloud in the geometric correction area in the radar coordinate system and the corresponding pixel coordinates in the image coordinate system. Then, the pixel coordinates of the edge points of the road traffic markings are substituted into the polynomial correction equation to calculate their two-dimensional coordinates in the intermediate coordinate system. Finally, according to the previously calculated rotation matrix R and translation matrix T, the two-dimensional coordinates of the road traffic markings are converted to three-dimensional coordinates in the radar coordinate system. Only x and y are taken, and the longitude and latitude of the edge points of the traffic markings are obtained according to the conversion relationship between the radar coordinate system and the earth coordinate system.
[0086] 6) Calculate the traffic sign and marking space data of the traffic sign and marking interval and the relative position of the traffic sign and marking to the intersection / crossing according to the traffic sign and marking edge point coordinates and the longitude and latitude of the stop line of the target road;
[0087] Among them, the following methods are used to extract the stop line and lane line of the target road:
[0088] Use a laser scanner to obtain the target road surface point cloud, and based on the road surface point cloud, set the reflectivity threshold to effectively extract the road marking point cloud. Based on the road marking point cloud, determine the size of the three-dimensional grid according to the point spacing of the scanned points on the road marking, delete the grids with fewer points than the threshold, and obtain the road marking point cloud without outliers. Based on the road marking point cloud without outliers, obtain the position information of the stop line and lane line of the target road in the radar coordinate system, retain only the X and Y values, convert the radar coordinate system to the geodetic coordinate system, and obtain the longitude and latitude of the stop line and lane line of the target road in the geodetic coordinate system.
[0089] 7) Develop a sign and marking criteria database based on standards and specifications related to urban road and highway traffic signs and markings;
[0090] With reference to major standard normative documents such as "Design Specifications for Urban Road Traffic Facilities" GB 50688, "Road Traffic Signs and Markings" GB5786, "Highway Traffic Signs and Markings Setting Specifications" JTG D82, "Highway Traffic Safety Facilities Design Specifications" JTG D81, "Urban Road Traffic Signs and Markings Setting Specifications" GB 51038, etc., a sign and marking criteria database is developed based on the standards and specifications related to urban road and highway traffic signs and markings;
[0091] The sign and marking criteria database is used to store traffic sign setting specifications and traffic marking setting specifications. Traffic sign setting specifications include traffic sign number, type, name, meaning, layout information, relative setting position of signs, and the relative position of signs and intersections; traffic marking setting specifications include traffic marking number, type, name, meaning, layout information, marking interval, and the relative position of markings and intersections.
[0092] 8) Compare the acquired traffic sign and marking spatial data and attribute data of the target road with the sign and marking criteria database based on the standards and specifications of urban road and highway traffic signs and markings, detect the non-standard or unreasonable problems in the presence, location, sign layout information, etc. of the traffic signs and markings at specific locations, formulate corresponding improvement measures in a targeted manner, and intelligently generate problem investigation reports;
[0093] The attribute data includes the number, type, direction and layout information of traffic sign markings.
[0094] Generating a troubleshooting report includes the following steps:
[0095] 8.1) According to the sign and marking criteria database, determine whether there are corresponding traffic signs and markings at the corresponding position of the intersection / crossing; if there are, it meets the specifications and proceed to the next step; if it does not meet the specifications, record the problems of traffic signs and markings in the database, and propose corresponding improvement measures according to the sign and marking criteria database;
[0096] 8.2) According to the detected traffic markings, determine whether there are corresponding traffic signs at the corresponding position of the traffic markings; if there are, it meets the specifications and proceed to the next step; if it does not meet the specifications, record the problems of traffic signs and markings in the database and solve them according to the sign and marking standard data
[0097] 8.3) According to the direction of the traffic markings, determine whether the direction of the traffic markings at the intersection is consistent with the direction of the corresponding traffic signs. If they are consistent, they meet the specifications and proceed to the next step; if they do not meet the specifications, record the problems of the traffic signs and markings in the database, and propose corresponding improvement measures according to the sign and marking criteria database;
[0098] 8.4) Based on the distance between the traffic sign and the intersection, determine whether the relative position of the traffic sign and the intersection meets the specifications. If so, proceed to the next step. If not, record the problems of the traffic sign in the database and propose corresponding improvement measures based on the sign and marking criteria database.
[0099] 8.5) Determine the distance between traffic signs and markings based on the distance between traffic signs and markings
[0100] Whether it meets the specifications. If it does, proceed to the next step. If it does not meet the specifications, record the problems of traffic signs and markings in the database and propose corresponding improvement measures according to the sign and marking standard database;
[0101] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.
Claims
1. A method for rapid detection and troubleshooting of road traffic signs and markings, characterized in that: The following steps are involved: 1) Obtain the target road video through a CCD camera, and extract frame-by-frame images containing target traffic signs and markings from the target video; 2) Obtain target road point cloud data through laser scanner; 3) Obtaining vehicle positioning data through a vehicle positioning device; 4) Identify the image containing the target traffic sign and marking, obtain the type of the traffic sign and marking, and number the traffic sign and marking and the road to which it belongs; Performing text recognition on traffic sign characters and outputting layout information on the traffic sign; the layout information includes the characters on the traffic sign; 5) Obtaining the location information of traffic signs and markings, including the latitude and longitude data of traffic signs and markings; 6) Calculate the traffic sign and marking space data of the traffic sign and marking interval and the relative position of the traffic sign and marking to the intersection / crossing according to the traffic sign and marking edge point coordinates and the longitude and latitude of the stop line of the target road; 7) Develop a sign and marking criteria database based on standards and specifications related to urban road and highway traffic signs and markings; 8) Compare the acquired traffic sign and marking spatial data and attribute data of the target road with the sign and marking criteria database based on the standards and specifications of urban road and highway traffic signs and markings, detect the non-standard or unreasonable problems in the presence, location, sign layout information, etc. of the traffic signs and markings at specific locations, formulate corresponding improvement measures in a targeted manner, and intelligently generate problem investigation reports; The attribute data includes the number, type, direction and layout information of traffic sign markings.
2. The method for rapid detection and troubleshooting of road traffic signs and markings according to claim 1 is characterized in that: The step 4) identifies an image containing target traffic signs and markings, specifically as follows: 4.1) Image preprocessing: For the image containing the target traffic sign and marking, the frequency domain-based filtering is used to smooth and reduce the noise of the image, the histogram equalization method is used to enhance the regional features of the traffic sign and marking of the denoised image, and the Canny edge detection method is used to extract the edge of the traffic marking to obtain the preprocessed image containing the target traffic sign and marking; 4.2) For the pre-processed image containing the target traffic sign and marking, roughly locate the traffic sign based on the YCbCr color space to obtain an image retaining the target traffic sign area; 4.3) The image of the target traffic sign area is subjected to shape analysis to locate a circular area, a rectangular area, and a triangular area; wherein an area with a rectangularity / circularity greater than 0.8 is a rectangular / circular area, and an area with a triangle interval threshold between 0.85 and 1.15 is a triangular area; the area is cut out and input into the designed traffic sign classifier as a traffic sign to be identified for identification; the traffic sign classifier is a recognition model based on a BP neural network; 4.4) Using an improved random sampling consensus algorithm to identify the traffic markings in the preprocessed image containing the target traffic sign markings.
3. The method for rapid detection and troubleshooting of road traffic signs and markings according to claim 1 is characterized in that: The position information of the traffic sign and marking in step 5) is determined based on the image containing the target traffic sign and the target road point cloud data, as follows: 5.1) Obtain a stereoscopic image through the CCD camera; the stereoscopic image includes the image information of the target traffic sign, the internal orientation elements of the image and the external orientation element information of the camera at the moment of shooting; wherein, in the stereoscopic image, a certain ground point (X A ,Y A ,Z A ) in the left and right images and are called image points of the same name, the internal orientation elements are obtained by calibration, and the external orientation elements are calculated by the position information and attitude information obtained by the vehicle positioning device; 5.2) performing image matching; using an image matching method based on grayscale matching to identify points of the same name between the two images, wherein the grayscale matching determines the relationship between the template and the matching area by calculating the matching measure of the grayscale between the template and the image; 5.3) Setting matching constraints; the matching constraints are the conditions that must be satisfied first during the matching process of the stereo image pair. In the matching of homonymous points of sequential images, the constraints are epipolar constraints; wherein the epipolar constraints are: epipolar lines of the same name must pass through the same name points; 5.4) Grayscale matching: firstly, a matching template is selected, and then a traffic sign is searched in the image containing the traffic sign, a search subwindow of the template size is slid across the entire image, and a matching measure R between the template and the image containing the traffic sign in the subwindow is calculated using a specified method f(x, y) and saved in an array. After the calculation is completed, the maximum value in the array is the best template matching position; 5.5) Feature extraction: Extract features from traffic signs and use the extracted feature points as target points for automatically acquiring image points; 5.6) Matching of points with the same name; Based on the image containing the traffic sign, edge extraction is performed on the area to be detected; based on the edge image, edge corner points in the traffic sign are selected as matching feature points, and the corner point coordinates are obtained by using the corner point extraction operator and the straight line corner points extracted by Hough transform, and further corner points are obtained; based on the edge image, Hough straight line extraction is performed on it, and the extracted straight lines are merged, and the points where the straight line intersections are in the detected area are formed into an intersection point set; the union of the corner point set and the intersection point set is used as the feature point set P to be matched F ; Taking the points in the feature point set as known image point coordinates, generating the same-name epipolar lines in the stereo image pair according to the properties of the epipolar lines; The feature point set P extracted from the left image F (L) Pick any point P1 (L) , calculated through P1 (L) The epipolar line K with the same name is located in the right image. R , according to the kernel line constraint, calculate the right image point set P F (R) Midpoint to epipolar line K R Distance d, select points with d<∈ to form the point set P of the check core line constraint condition K (R) ; Select P1 (L) The grayscale matrix with the size of n×n as the center is used as the matching template, along the kernel line K R Perform grayscale matching on the right image and find the point set P that satisfies the maximum matching coefficient and is located on the same-name epipolar line. m (R) ; Point set P K (R) With point set P m (R) If there is a corresponding relationship between the points in the two point sets, they are a pair of image points with the same name; 5.7) Traffic sign positioning; Based on feature matching, we can get the same-name image points on the stereo pair, and then get two spatial rays with the same name. Then, according to the collinearity equation, we know that the two rays must intersect at the spatial position of the ground point with the same-name image point. There are three unknowns in the simultaneous equations of the unknown ground points, namely the ground coordinates (X, Y, Z). Using the coordinates of the same-name image points on the stereo pair, we can list four equations to solve the three unknowns, namely the positions of the ground points corresponding to the same-name image points. The calculated ground point coordinates are counted, and the points with large deviations are regarded as gross errors. The robust estimation method is used to eliminate the gross errors, and finally the average value of the remaining coordinate points is taken as the three-dimensional coordinates of the traffic sign, wherein the traffic sign The three-dimensional coordinates of the traffic sign are the coordinates in the image space coordinate system. Based on the internal orientation elements and the conversion relationship between the image space coordinate system and the vehicle body coordinate system, the coordinates of the traffic sign in the vehicle body coordinate system are obtained. Since the vehicle body coordinate system is a right-handed system and the Gaussian coordinate system is a left-handed system, a transition coordinate system is assumed. According to the external orientation elements, the conversion relationship between the coordinates in the vehicle body coordinate system and the Gaussian coordinate system can be obtained. According to the conversion relationship between the image space coordinate system and the vehicle body coordinate system and the conversion relationship between the vehicle body coordinate system and the Gaussian coordinate system, the coordinates of the traffic sign in the Gaussian coordinate system can be obtained. Finally, according to the conversion relationship between the Gaussian coordinate system and the geodetic coordinate system, the longitude and latitude of the traffic sign are obtained.
4. The method for rapid detection and troubleshooting of road traffic signs and markings according to claim 1 is characterized in that: The sign and marking criteria database in step 7) is used to store traffic sign setting specifications and traffic marking setting specifications. The traffic sign setting specifications include traffic sign number, type, name, meaning, layout information, relative setting position of the sign, and the relative position of the sign and the intersection; the traffic marking setting specifications include traffic marking number, type, name, meaning, layout information, marking interval, and the relative position of the marking and the intersection.
5. The method for rapid detection and troubleshooting of road traffic signs and markings according to claim 1 is characterized in that: Generating the troubleshooting report in step 8) includes the following process: 8.1) According to the sign and marking criteria database, determine whether there are corresponding traffic signs and markings at the corresponding position of the intersection / crossing; if there are, it meets the specifications and proceed to the next step; if it does not meet the specifications, record the problems of traffic signs and markings in the database, and propose corresponding improvement measures according to the sign and marking criteria database; 8.2) According to the detected traffic markings, determine whether there are corresponding traffic signs at the corresponding position of the traffic markings; if there are, it meets the specifications and proceed to the next step; if it does not meet the specifications, record the problems of traffic signs and markings in the database and solve them according to the sign and marking standard data 8.3) According to the direction of the traffic markings, determine whether the direction of the traffic markings at the intersection is consistent with the direction of the corresponding traffic signs. If they are consistent, they meet the specifications and proceed to the next step; if they do not meet the specifications, record the problems of the traffic signs and markings in the database, and propose corresponding improvement measures according to the sign and marking criteria database; 8.4) Based on the distance between the traffic sign and the intersection, determine whether the relative position of the traffic sign and the intersection meets the specifications. If so, proceed to the next step. If not, record the problems of the traffic sign in the database and propose corresponding improvement measures based on the sign and marking criteria database. 8.5) Based on the distance of traffic signs and markings, determine whether the distance of traffic signs and markings meets the specifications. If it does, proceed to the next step; if it does not meet the specifications, record the problems of traffic signs and markings in the database and propose corresponding improvement measures based on the sign and marking standard database.
