A method for detecting missing road traffic signs based on vehicle-mounted forward-looking images

Abstract

The present invention discloses a method for detecting missing road traffic signs based on vehicle-mounted forward-view images, and relates to the technical field of road facility detection. The method comprises the following steps: based on the collected images, a vehicle-mounted forward-view image set is established, and camera external parameters are calculated; based on the vehicle-mounted forward-view image set, a sign highway image is screened, a high-resolution traffic sign image data set is established, and a sign segmentation model is constructed and trained based on the annotated high-resolution traffic sign image data set; based on the trained sign segmentation model, a highway scene image is input, and the pixel coordinates of the left and right sign pixel frames are output, and converted into longitude and latitude positioning coordinates, and the sign is judged whether it is missing according to the longitude and latitude positioning coordinates of multiple vehicle-mounted forward-view image sets. The present invention can improve detection efficiency, reduce human subjective influence, and is suitable for large-scale and long-distance road sign missing detection.

Description

Technical Field

[0001] The present invention relates to the technical field of road facility detection, and in particular to a method for detecting missing road traffic signs based on vehicle-mounted front-view images. Background Art

[0002] With the continuous advancement of infrastructure construction and the continuous development of highway construction in my country, my country has the largest highway network in the world. Highway facilities have become strategic facilities to support national economic development, serve people's lives, and ensure national security. Among them, highway signs can help drivers understand the road ahead, the distance to the destination, and other information in advance, while the damage of signs will seriously affect road driving safety.

[0003] At present, although the accuracy of the facility detection algorithms along the line can meet the requirements, they have disadvantages such as large computational complexity and long training time, and cannot meet the real-time requirements; at the same time, these algorithms are also easily affected by external conditions such as weather and light, and cannot cope with complex climatic conditions; in addition, these algorithms also have limitations in the detection of facility categories, cannot meet the requirements of actual production, and are difficult to deploy.

[0004] There are few publicly published technical research materials on highway signboard damage detection at home and abroad, and only a few technologies are applied to signboard damage detection in other fields. For example, in solving the common damage and scratches of phase signboards in power facilities, some domestic scholars use guided filtering, grayscale, fixed threshold segmentation, SURF algorithm and FLANN algorithm to identify phase signboards, and then determine whether there are defects by calculating the intersection coefficient percentage of the B channel histogram; although the above algorithms have good effects and stability in identifying and detecting phase signboard defects, due to the use of template matching methods in the algorithm, the effect is not ideal when dealing with signs with complex shapes and types.

[0005] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] In view of the shortcomings of the prior art, the present invention provides a method for detecting missing road traffic signs based on vehicle-mounted forward-looking images, which has the advantages of automatically identifying missing traffic signs, greatly improving detection efficiency, and reducing human subjective influence, thereby solving the problems in the prior art that the detection algorithm has a large amount of calculation, a long training time, cannot meet real-time requirements, and the effect is not ideal when dealing with signs with complex and diverse shapes and types.

[0008] (II) Technical solution

[0009] In order to realize the above-mentioned advantages of automatic identification of missing traffic signs, greatly improve detection efficiency, and reduce human subjective influence, the specific technical solutions adopted by the present invention are as follows:

[0010] A method for detecting missing road traffic signs based on vehicle-mounted front-view images comprises the following steps:

[0011] S1. Based on the collected images, a vehicle-mounted front-view image set is established, and camera extrinsic parameters are calculated, wherein the vehicle-mounted front-view image set includes a calibration field scene image and a highway scene image;

[0012] S2. According to the vehicle-mounted forward-view image set, the highway images of traffic signs are screened to establish a high-resolution traffic sign image dataset. Based on the labeled high-resolution traffic sign image dataset, a sign segmentation model is constructed and trained.

[0013] S3. Based on the trained sign segmentation model, the highway scene image is input, the pixel coordinates of the left and right sign pixel frames are output, and the pixel coordinates are converted into longitude and latitude positioning coordinates. According to the longitude and latitude positioning coordinates of multiple vehicle-mounted forward-view image sets, it is determined whether the sign is missing.

[0014] Furthermore, based on the collected images, establishing a vehicle-mounted forward image set and calculating the camera extrinsic parameters include the following steps:

[0015] S11. Use the high-resolution camera on the vehicle to collect the scene image of the inspection field, and use the AI ​​uniform light algorithm to correct the image exposure in real time to obtain the pixel coordinates;

[0016] S12. Based on the geographic coordinates and pixel coordinates of the same-name points, a fusion parameter calculation model is constructed, and the translation and rotation of the camera relative to the integrated navigation are solved to obtain the camera external parameters;

[0017] S13, based on the fusion parameter calculation model, the correction amount of the camera extrinsic parameter is solved by adjustment, and the fusion parameter is optimized by using the least squares iteration method;

[0018] S14. Continuously collect highway scene images using a vehicle-mounted high-resolution camera, and establish a vehicle-mounted forward-view image set in combination with the inspection field scene images.

[0019] Furthermore, the high-resolution camera on the vehicle is used to collect the scene image of the inspection field, and the AI ​​light homogenization algorithm is used to correct the image exposure in real time. The pixel coordinates are obtained in the following steps:

[0020] S111, using the high-resolution camera on board to continuously collect images of the inspection and calibration field scene;

[0021] S112, establishing an image brightening curve model, and using a brightening function to process each pixel point in the calibration scene image to obtain an image pixel coordinate set;

[0022] S113. Based on the image brightening curve model, we introduced depthwise separable convolution and dilated convolution to improve the model network structure.

[0023] S114, using the image pixel coordinate set to train an image brightening curve model, and establishing a loss function of the image brightening curve model, wherein the model loss function includes a spatial consistency error and a color constancy error;

[0024] S115, evaluate the peak signal-to-noise ratio and structural similarity index of the trained image brightening curve model, repeat steps S112 to S114 until the evaluation result meets the preset requirements, save the optimal weight of the model, and output the corresponding image pixel coordinate set.

[0025] Furthermore, the loss function expression of the image brightening curve model is:

[0026] ;

[0027] ;

[0028] Where, L spa represents the spatial consistency error; k represents the number of pixels; i represents the traversal of pixels; j represents the average brightness; Ω(i) represents the 4-neighborhood of the i-th pixel; Y and I represent the enhanced image and the input image respectively; L col Indicates color constant error; j p With j q They represent the average brightness of color channel p and color channel q respectively; ε represents the color channel combination traversed by (p, q); (R, G), (R, B), (G, B) represent all pairwise combinations of the three color channels (R, G, B).

[0029] Furthermore, according to the vehicle-mounted forward-view image set, the highway image of the traffic sign is screened to establish a high-resolution traffic sign image dataset. Based on the labeled high-resolution traffic sign image dataset, the traffic sign segmentation model is constructed and trained, including the following steps:

[0030] S21. Based on the highway scene images, the highway images containing traffic signs are screened to construct a high-resolution traffic sign image dataset;

[0031] S22. Use image annotation tools to annotate a high-resolution traffic sign image dataset;

[0032] S23, dividing the labeled high-resolution traffic sign image dataset based on the division ratio to obtain a training set, a validation set, and a test set;

[0033] S24. Construct a sign segmentation model, and train the sign segmentation model based on the training set, the validation set, and the test set.

[0034] Furthermore, based on the vehicle-mounted forward-view image set, the highway images containing traffic signs are screened, and the construction of a high-resolution traffic sign image dataset includes the following steps:

[0035] S211, pre-processing the highway scene image and performing preliminary filtering;

[0036] S212, based on the pre-trained deep learning model, input the initially filtered image, output the preliminary sign detection result, and establish a pseudo label according to the detected sign position;

[0037] S213, based on weakly supervised learning algorithm, using pseudo labels to train the sign detection model;

[0038] S214. Use the trained traffic sign detection model to process highway scene images, filter out highway images containing traffic signs, and establish a high-resolution traffic sign image dataset.

[0039] Furthermore, using image annotation tools, annotating the high-resolution traffic sign image dataset includes the following steps:

[0040] S221, drawing the outer contour of the signboard at the pixel level and marking it, to obtain a marked partial data set;

[0041] S222, using the labeled partial data set to train a preliminary model;

[0042] S223. Use the trained preliminary model to process the high-resolution traffic sign image data and convert it into labeled data.

[0043] Furthermore, a sign segmentation model is constructed, and based on the training set, the validation set and the test set, the training of the sign segmentation model includes the following steps:

[0044] S241. Building a sign segmentation model based on the mask region convolutional neural network model, training the sign segmentation model using the training set, and calculating the loss using a multi-task loss function;

[0045] S242, calculating prediction accuracy based on the trained sign segmentation model;

[0046] S243. Repeat steps S241 and S242 until the loss, prediction accuracy and recall rate of the sign segmentation model reach the experimental required thresholds.

[0047] Furthermore, the expression for calculating the loss using the multi-task loss function is:

[0048] ;

[0049] ;

[0050] In the formula, L represents the multi-task loss function, L cls represents the classification loss, N cls Indicates the number of positive samples of a category, p i * represents the correct category prediction probability, i represents the current number, L box represents the bounding box regression loss, N box represents the number of positive samples at the boundary, t i * represents the correct bounding box prediction value, t i represents the model prediction bounding box, L mask represents the instance segmentation loss, N mask represents the number of positive samples for instance segmentation, represents the prediction mask;

[0051] The expression for calculating prediction accuracy is:

[0052] ;

[0053] Where Ac represents the prediction accuracy, TP represents the true positive example, TN represents the true negative example, FP represents the false positive example, and FN represents the false negative example.

[0054] Furthermore, based on the trained sign segmentation model, the highway scene image is input, the pixel coordinates of the left and right sign pixel frames are output, and converted into longitude and latitude positioning coordinates. According to the longitude and latitude positioning coordinates of multiple vehicle-mounted front view image sets, it is determined whether the sign is missing, including the following steps:

[0055] S31, using the trained sign segmentation model to process the highway scene image to obtain the pixel coordinates of the left and right sign pixel frames;

[0056] S32, extracting features based on the pixel coordinates of the left and right signboard pixel frames, matching the features of the signboards in the left and right images using a brute force matching algorithm, obtaining the parallax between the two, and calculating the depth distance between the signboard and the camera, and obtaining the spatial rectangular coordinates of the signboard image based on the similarity;

[0057] S33, based on the rectangular coordinates of the sign image space, combined with the camera external parameters, converting them into inertial navigation coordinate system coordinates to obtain longitude and latitude positioning coordinates;

[0058] S34. Generate signboard location data based on latitude and longitude location coordinates using geographic information system technology, and combine the signboard location data of multiple vehicle-mounted forward-looking image sets to determine the missing signboards.

[0059] (III) Beneficial effects

[0060] Compared with the prior art, the present invention provides a method for detecting missing road traffic signs based on vehicle-mounted front-view images, which has the following beneficial effects:

[0061] (1) The present invention innovatively improves the detection accuracy and response speed of traffic signs by combining a deep learning model with the principle of image ranging. This combination not only optimizes the detection algorithm, but also enhances the adaptability to various highway scenarios, thereby improving the performance and speed of traffic sign detection.

[0062] (2) The present invention utilizes vehicle-mounted forward-looking imaging technology to realize automated detection of missing traffic signs, significantly improving the efficiency of detection and effectively reducing interference from human factors. It is particularly suitable for performing large-scale and long-distance road sign missing detection tasks, ensuring the accuracy and reliability of the detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0064] Figure 1 It is a flow chart of a method for detecting missing road traffic signs based on a vehicle-mounted front-view image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in the field should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0066] According to an embodiment of the present invention, a method for detecting missing road traffic signs based on a vehicle-mounted front view image is provided.

[0067] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to an embodiment of the present invention, a method for detecting missing road traffic signs based on a vehicle-mounted front view image comprises the following steps:

[0068] S1. Based on the collected images, a vehicle-mounted front-view image set is established, and the camera extrinsic parameters are calculated, wherein the vehicle-mounted front-view image set includes a calibration field scene image and a highway scene image;

[0069] S2. According to the vehicle-mounted forward-view image set, the highway images of traffic signs are screened to establish a high-resolution traffic sign image dataset. Based on the labeled high-resolution traffic sign image dataset, a sign segmentation model is constructed and trained.

[0070] S3. Based on the trained sign segmentation model, the highway scene image is input, the pixel coordinates of the left and right sign pixel frames are output, and the pixel coordinates are converted into longitude and latitude positioning coordinates. According to the longitude and latitude positioning coordinates of multiple vehicle-mounted forward-view image sets, it is determined whether the sign is missing.

[0071] In one embodiment, establishing a vehicle-mounted forward-view image set based on the collected images and calculating the camera extrinsic parameters includes the following steps:

[0072] S11. Use the high-resolution camera on the vehicle to collect the scene image of the inspection field, and use the AI ​​uniform light algorithm to correct the image exposure in real time to obtain the pixel coordinates;

[0073] S12. Based on the geographic coordinates and pixel coordinates of the same-name points, a fusion parameter calculation model is constructed, and the translation and rotation of the camera relative to the integrated navigation are solved to obtain the camera external parameters;

[0074] In addition, it should be noted that the fusion parameter calculation model in this embodiment adopts the Euler angle conversion model. The Euler angle conversion model is a mathematical model used to describe the direction and position of objects in three-dimensional space. It can define the rotation of an object relative to a fixed coordinate system through three angle values; these three angles are called Euler angles, including yaw angle (rotation around the Z axis, usually used to describe the left and right rotation of an object), pitch angle (rotation around the Y axis, used to describe the up and down rotation of an object) and roll angle (rotation around the X axis, used to describe the forward and backward rolling of an object).

[0075] Specifically, the geographic coordinates and pixel coordinates of the same-name points are input into the fusion parameter calculation model to solve the translation and rotation of the camera relative to the inertial measurement system, that is, the camera external parameters (including X s , Y s , Z s , fai, omi, ka; , , are the coordinate values ​​of the origin of the rectangular coordinate system in the image space in the directions of the three coordinate axes in the inertial navigation system; fai, omi, ka represent the rotation angles of the inertial navigation coordinate system around the y-axis, around the x-axis, and around the z-axis respectively). The expression to be solved is:

[0076] ;

[0077] in,

[0078] ; ;

[0079] ;

[0080] ;

[0081] ;

[0082] Based on the camera imaging geometry model, the expression of the collinear equation is obtained:

[0083] ;

[0084]

[0085] In the formula, ( , , ) is the spatial rectangular coordinate; ( , , )Inertial navigation coordinate system coordinates;( , , ) is the coordinate value of the origin of the image space rectangular coordinate system in the three coordinate axis directions in the inertial navigation coordinate system; fai, omi, ka represent the rotation angles of the inertial navigation coordinate system around the y-axis, around the x-axis, and around the z-axis respectively; R is a 3×3 attitude matrix; are the factor matrices of R around the y-axis, x-axis, and z-axis respectively; a1, a2, and a3 are the three column elements of the first row of the rotation matrix R, b1, b2, and b3 are the three column elements of the second row of the attitude matrix R, and c1, c2, and c3 are the three column elements of the third row of the rotation matrix R.

[0086] In addition, it should be noted that the camera imaging geometry model is a basic concept in the field of computer vision and image processing. This model describes how objects in the three-dimensional world are imaged on a two-dimensional image plane. It involves the physical principles and geometric relationships of optical imaging and has a wide range of applications in multiple fields.

[0087] S13. Based on the fusion parameter calculation model, the measurement adjustment method is used to solve the correction of the camera extrinsic parameters.

[0088] In addition, it should be noted that measurement adjustment is a commonly used technique in photogrammetry, remote sensing and computer vision, which is used to accurately determine the camera's extrinsic parameters (i.e., the camera's position and attitude). These extrinsic parameters are usually expressed as rotation matrices and translation vectors. The basic idea of ​​measurement adjustment is to optimize camera parameters by minimizing observation errors.

[0089] Specifically, the fusion parameter calculation model is used to measure the adjustment to solve the correction of the unknown quantity (camera extrinsic parameter), the measured value of the instrument installation position is used as the initial value for constraint, and the fusion parameters are optimized by the least squares iteration method until the function converges.

[0090] S14. Continuously collect highway scene images using a vehicle-mounted high-resolution camera, and establish a vehicle-mounted forward-view image set in combination with the inspection field scene images.

[0091] Specifically, a high-resolution binocular camera is fixed on the vehicle, and the camera is always aimed at the center line of the highway. Images on both sides of the highway are continuously collected at a speed of 80km / h and a frequency of 4 frames / s. The collected road images include road trees, road signs, and road materials to ensure data diversity.

[0092] In one embodiment, a high-resolution vehicle-mounted camera is used to collect a calibration scene image, and an AI light-evening algorithm is used to correct the image exposure in real time. Obtaining pixel coordinates includes the following steps:

[0093] S111, using the high-resolution camera on board to continuously collect images of the inspection and calibration field scene;

[0094] Specifically, in this embodiment, the vehicle-mounted high-resolution camera adopts a binocular camera. First, the binocular camera is used to simultaneously shoot a black and white checkerboard calibration plate, and the checkerboard calibration plate is moved to fill the image field of view as much as possible. The image is input into the binocular calibration toolbox of the MATLAB software to automatically calculate the focal length, pixel value and other internal parameters of the left and right cameras and the baseline length between the two. Then, artificial markers such as the checkerboard and the target box are evenly arranged in the selected calibration field, and the feature points are measured by a high-precision RTK measuring instrument to obtain the precise geographic coordinates of the feature points, which is convenient for subsequent external parameter calculations. Next, the high-resolution binocular camera is fixed on the vehicle, and the camera is always aimed at the center line of the calibration field. The vehicle travels at a speed of 5 km per hour and continuously and uninterruptedly collects images of the calibration field scene at a frequency of 4 frames / s.

[0095] S112, establishing an image brightening curve model, and using a brightening function to process each pixel point in the calibration scene image to obtain an image pixel coordinate set;

[0096] Specifically, an image brightening curve model (Light-Enhancement Curve, LE-curve) is established. This curve model can apply the same brightening function to each pixel in the image. If the brightness of the pixel is high, the pixel will not be processed.

[0097] Specifically, the expression of the brightening function is:

[0098] ;

[0099] ;

[0100] Where x is the pixel coordinate; α is a learnable parameter and α∈[-1,1]; I(x) is the input content and I(x)∈[0,1]; LE(I(x)α) is the enhanced image of the input I(x), and the three color channels of the enhanced image are processed separately.

[0101] S113. Based on the image brightening curve model, we introduced depthwise separable convolution and dilated convolution to improve the model network structure.

[0102] Specifically, depthwise separable convolution and dilated convolution are introduced to improve the network structure. When building the network, depthwise separable convolution and dilated convolution are introduced to replace the traditional standard convolution operations to improve the computational efficiency and receptive field of the algorithm.

[0103] S114, using the image pixel coordinate set to train an image brightening curve model, and establishing a loss function of the image brightening curve model, wherein the model loss function includes a spatial consistency error and a color constancy error;

[0104] Specifically, the established image pixel coordinate set is used for training, spatial consistency error and color constancy error are selected as loss functions, and the stability of model training is ensured by adjusting hyperparameters such as learning rate and batch size.

[0105] S115, evaluate the peak signal-to-noise ratio and structural similarity index of the trained image brightening curve model, repeat steps S112 to S114 until the evaluation result meets the preset requirements, save the optimal weight of the model, and output the corresponding image pixel coordinate set.

[0106] Specifically, the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) are used as evaluation indicators to assess the accuracy of the trained model. When the performance indicators meet the requirements, the optimal weights of the model are saved.

[0107] In one embodiment, the loss function of the image brightening curve model is expressed as:

[0108] ;

[0109] ;

[0110] Where, L spa represents the spatial consistency error; k represents the number of pixels; i represents the traversal of pixels; j represents the average brightness; Ω(i) represents the 4-neighborhood of the i-th pixel; Y and I represent the enhanced image and the input image respectively; L col Indicates color constant error; j p With j q They represent the average brightness of color channel p and color channel q respectively; ε represents the color channel combination traversed by (p, q); (R, G), (R, B), (G, B) represent all pairwise combinations of the three color channels (R, G, B).

[0111] In one embodiment, based on the vehicle-mounted forward-view image set, the highway image of the traffic sign is screened to establish a high-resolution traffic sign image dataset, and based on the annotated high-resolution traffic sign image dataset, a sign segmentation model is constructed and trained, including the following steps:

[0112] S21. Based on the highway scene images, the highway images containing traffic signs are screened to construct a high-resolution traffic sign image dataset;

[0113] S22. Use image annotation tools to annotate a high-resolution traffic sign image dataset;

[0114] S23, dividing the labeled high-resolution traffic sign image dataset based on the division ratio to obtain a training set, a validation set, and a test set;

[0115] Specifically, the labeled high-resolution traffic sign image dataset is divided, wherein the ratio of the training set, the validation set, and the test set is 8:1:1.

[0116] S24. Construct a sign segmentation model, and train the sign segmentation model based on the training set, the validation set, and the test set.

[0117] In one embodiment, based on the vehicle-mounted forward image set, selecting highway images containing traffic signs and constructing a high-resolution traffic sign image dataset includes the following steps:

[0118] S211, pre-processing the highway scene image and performing preliminary filtering;

[0119] Specifically, the highway scene images are preprocessed, and simple image processing techniques are used to preliminarily filter the images to preliminarily filter out images that obviously do not contain signboards.

[0120] S212, based on the pre-trained deep learning model, input the initially filtered image, output the preliminary sign detection result, and establish a pseudo label according to the detected sign position;

[0121] Specifically, we use an open source deep learning object detection model to pre-train the signboards, and then use the pre-trained deep learning model to detect the initially filtered images to generate preliminary signboard detection results, and use the signboard positions detected by the model as pseudo labels. These pseudo labels may not be completely accurate, but they can provide a basis for subsequent weakly supervised learning.

[0122] S213, based on weakly supervised learning algorithm, using pseudo labels to train the sign detection model;

[0123] Specifically, a self-training method is adopted to continue training with the generated pseudo labels based on the pre-trained deep learning model. After training, a sign detection model is obtained, and the pseudo labels are gradually updated with the results predicted by the model. The training process is repeated, and the accuracy of the model is gradually improved through multiple iterations.

[0124] Specifically, a multi-view learning method is adopted, and different detection models or different feature extraction methods are used to detect the same image, and the detection results from multiple perspectives are integrated to improve the quality of pseudo labels.

[0125] S214. Use the trained traffic sign detection model to process highway scene images, filter out highway images containing traffic signs, and establish a high-resolution traffic sign image dataset.

[0126] Specifically, the sign detection model trained with pseudo labels is used to detect all collected images and filter out highway images containing signboards.

[0127] Specifically, the screened images are organized into a high-resolution traffic sign image (Highway Road Traffic Sign, HRTS) dataset without labels.

[0128] In one embodiment, annotating a high-resolution traffic sign image dataset using an image annotation tool includes the following steps:

[0129] S221, drawing the outer contour of the signboard at the pixel level and marking it, to obtain a marked partial data set;

[0130] S222, using the labeled partial data set to train a preliminary model;

[0131] S223. Use the trained preliminary model to process the high-resolution traffic sign image data and convert it into labeled data.

[0132] Specifically, a small portion of the image data in the HRTS data set obtained in step S214 is manually and finely annotated using automatic annotation software (Labelme software is used in this embodiment), and the outer contour of the sign is accurately drawn at the pixel level, and it is labeled as "Signboard"; then, the annotated data set is used to train the initial mask region convolutional neural network model (Mask RCNN) pre-trained based on the COCO data set, so that the trained preliminary model can obtain basic sign recognition capabilities; then, the trained preliminary model is used to automatically identify and annotate the remaining unlabeled data, and convert it into labeled data; finally, through manual review and correction, the accuracy and consistency of the annotation are ensured, thereby constructing a high-quality, complete, high-resolution traffic sign image data set.

[0133] In one embodiment, a sign segmentation model is constructed, and based on a training set, a validation set, and a test set, training the sign segmentation model comprises the following steps:

[0134] S241. Building a sign segmentation model based on the mask region convolutional neural network model, training the sign segmentation model using the training set, and calculating the loss using a multi-task loss function;

[0135] Specifically, the high-resolution traffic sign image dataset is imported into the Masked Region Convolutional Neural Network model (MaskRCNN) for pre-training. Each time, a minimum batch-size of data is input into the Masked Region Convolutional Neural Network model to obtain the model output, and the loss is calculated in combination with the annotated true labels and the multi-task loss function.

[0136] S242, calculating prediction accuracy based on the trained sign segmentation model;

[0137] S243. Repeat steps S241 and S242 until the loss, prediction accuracy and recall rate of the sign segmentation model reach the experimental required thresholds.

[0138] Specifically, step S241 to step S242 are repeatedly performed, and when the loss, precision and recall rate reach the threshold required by the experiment, the training is terminated.

[0139] In one embodiment, the expression for calculating the loss using the multi-task loss function is:

[0140]

[0141] ;

[0142] In the formula, L represents the multi-task loss function, L cls represents the classification loss, N cls Indicates the number of positive samples of a category, p i * represents the correct category prediction probability, i represents the current number, L box represents the bounding box regression loss, N box represents the number of positive samples at the boundary, t i * represents the correct bounding box prediction value, t i represents the model prediction bounding box, L mask represents the instance segmentation loss, N mask represents the number of positive samples for instance segmentation, represents the prediction mask;

[0143] In addition, it should be noted that smooth L1 Represents the smooth L1 loss function, which is a commonly used loss function in machine learning, especially when dealing with regression problems (such as bounding box regression in target detection). The smooth L1 loss function combines the characteristics of L1 loss (absolute value loss) and L2 loss (square loss), which can significantly improve the robustness of the model and the stability of training.

[0144] The expression for calculating prediction accuracy is:

[0145] ;

[0146] Where Ac represents the prediction accuracy, TP represents the true positive example, TN represents the true negative example, FP represents the false positive example, and FN represents the false negative example.

[0147] In one embodiment, based on the trained sign segmentation model, a highway scene image is input, pixel coordinates of left and right sign pixel frames are output, and converted into longitude and latitude positioning coordinates. According to the longitude and latitude positioning coordinates of multiple vehicle-mounted front view image sets, determining whether the sign is missing includes the following steps:

[0148] S31, using the trained sign segmentation model to process the highway scene image to obtain the pixel coordinates of the left and right sign pixel frames;

[0149] Specifically, the images captured by the binocular camera are preprocessed, and the present invention only needs to crop the images to below the maximum value of the model training.

[0150] Specifically, the cropped image is passed into the trained sign segmentation model, and the output image is a pixel classification map of each road element in the image, a road traffic sign map is obtained, and the pixel position of the sign in the image is recorded.

[0151] S32, extracting features based on the pixel coordinates of the left and right signboard pixel frames, matching the features of the signboards in the left and right images using a brute force matching algorithm, obtaining the parallax between the two, and calculating the depth distance between the signboard and the camera, and obtaining the spatial rectangular coordinates of the signboard image based on the similarity;

[0152] Specifically, the pixel coordinates of the left and right signboards obtained by the binocular camera are used to find feature points through SIFT, and then the brute force matching algorithm is used to calculate whether the feature points of the left and right signboards can form a feature point pair, and then the feature point pair of the left and right images is obtained. The horizontal pixel coordinates of the feature point pair of the left and right images are subtracted to obtain the parallax between the two, and then the depth distance Z between the signboard and the camera is calculated, and then the X and Y coordinates are obtained based on the similarity, that is, the rectangular coordinates of the image space of the signboard.

[0153] Specifically, the expression for calculating the depth distance between the sign and the camera is:

[0154] ;

[0155] Where Z represents the depth distance between the sign and the vehicle-mounted high-resolution camera, f represents the focal length of the vehicle-mounted high-resolution camera, b represents the baseline length of the vehicle-mounted high-resolution camera, and d represents the parallax of the target object in the image.

[0156] S33, based on the rectangular coordinates of the sign image space, combined with the camera external parameters, converting them into inertial navigation coordinate system coordinates to obtain longitude and latitude positioning coordinates;

[0157] Specifically, based on the sign image space rectangular coordinates (X, Y, Z) calculated in step S32 and combined with the camera external parameters, the image space rectangular coordinates are translated into the inertial navigation coordinate system centered on the inertial navigation system, and the center position and attitude information recorded by the inertial navigation system are further converted into CGCS2000 coordinate system coordinates, with a positioning accuracy of up to 30 cm.

[0158] S34. Generate signboard location data based on latitude and longitude location coordinates using geographic information system technology, and combine the signboard location data of multiple vehicle-mounted forward-looking image sets to determine the missing signboards.

[0159] Specifically, the geographic information system (GIS) technology is used to generate SHP data with the information such as the signpost number, type, and location collected each time. The data records information such as the photo serial number, geographic coordinates, and type. By comparing multiple periods of SHP data, information such as the location and type of missing signboards can be obtained.

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting missing road traffic signs based on vehicle-mounted front-view images, characterized in that: The method for detecting missing road traffic signs based on vehicle-mounted front-view images comprises the following steps: S1. Based on the collected images, a vehicle-mounted front-view image set is established, and camera extrinsic parameters are calculated, wherein the vehicle-mounted front-view image set includes a calibration field scene image and a highway scene image; S2. According to the vehicle-mounted forward-view image set, the highway images of traffic signs are screened to establish a high-resolution traffic sign image dataset. Based on the labeled high-resolution traffic sign image dataset, a sign segmentation model is constructed and trained. S3, based on the trained sign segmentation model, input the highway scene image, output the pixel coordinates of the left and right sign pixel frames, and convert them into longitude and latitude positioning coordinates, and determine whether the sign is missing according to the longitude and latitude positioning coordinates of multiple vehicle-mounted front view image sets; The process of establishing a vehicle-mounted forward-view image set based on the collected images and calculating the camera extrinsic parameters comprises the following steps: S11. Use the high-resolution camera on the vehicle to collect the scene image of the inspection field, and use the AI ​​uniform light algorithm to correct the image exposure in real time to obtain the pixel coordinates; S12. Based on the geographic coordinates and pixel coordinates of the same-name points, a fusion parameter calculation model is constructed, and the translation and rotation of the camera relative to the integrated navigation are solved to obtain the camera external parameters; S13, based on the fusion parameter calculation model, the correction amount of the camera extrinsic parameter is solved by adjustment, and the fusion parameter is optimized by using the least squares iteration method; S14, using a high-resolution vehicle-mounted camera to continuously collect highway scene images, and combining them with the inspection field scene images to establish a vehicle-mounted forward-view image set; The method of using a high-resolution vehicle-mounted camera to collect the calibration scene image, and using an AI light homogenization algorithm to correct the image exposure in real time, and obtaining pixel coordinates includes the following steps: S111, using the high-resolution camera on board to continuously collect images of the inspection and calibration field scene; S112, establishing an image brightening curve model, and using a brightening function to process each pixel point in the calibration scene image to obtain an image pixel coordinate set; S113. Based on the image brightening curve model, we introduced depthwise separable convolution and dilated convolution to improve the model network structure. S114, using the image pixel coordinate set to train an image brightening curve model, and establishing a loss function of the image brightening curve model, wherein the model loss function includes a spatial consistency error and a color constancy error; S115, evaluate the peak signal-to-noise ratio and structural similarity index of the trained image brightening curve model, repeat steps S112 to S114 until the evaluation result meets the preset requirements, save the optimal weight of the model, and output the corresponding image pixel coordinate set.

2. The method for detecting missing road traffic signs based on vehicle-mounted front-view images according to claim 1, characterized in that: The expression of the loss function for establishing the image brightening curve model is: Where, L spa represents the spatial consistency error; k represents the number of pixels; i represents the traversal of pixels; j represents the average brightness; Ω(i) represents the 4-neighborhood of the i-th pixel; Y and I represent the enhanced image and the input image respectively; L col Indicates color constant error; j p With j q They represent the average brightness of color channel p and color channel q respectively; ε represents the color channel combination traversed by (p, q); (R, G), (R, B), (G, B) represent all pairwise combinations of the three color channels (R, G, B).

3. The method for detecting missing road traffic signs based on vehicle-mounted front-view images according to claim 1, characterized in that: The method comprises the following steps: screening highway images of traffic signs according to the vehicle-mounted forward-view image set, establishing a high-resolution traffic sign image dataset, and constructing and training a traffic sign segmentation model based on the annotated high-resolution traffic sign image dataset: S21. Based on the highway scene images, the highway images containing traffic signs are screened to construct a high-resolution traffic sign image dataset; S22. Use image annotation tools to annotate a high-resolution traffic sign image dataset; S23, dividing the labeled high-resolution traffic sign image dataset based on the division ratio to obtain a training set, a validation set, and a test set; S24. Construct a sign segmentation model, and train the sign segmentation model based on the training set, the validation set, and the test set.

4. The method for detecting missing road traffic signs based on vehicle-mounted front-view images according to claim 3 is characterized in that: The method of screening highway images containing traffic signboards based on the vehicle-mounted forward-view image set and constructing a high-resolution traffic sign image dataset comprises the following steps: S211, pre-processing the highway scene image and performing preliminary filtering; S212, based on the pre-trained deep learning model, input the initially filtered image, output the preliminary sign detection result, and establish a pseudo label according to the detected sign position; S213, based on weakly supervised learning algorithm, using pseudo labels to train the sign detection model; S214. Use the trained traffic sign detection model to process highway scene images, filter out highway images containing traffic signs, and establish a high-resolution traffic sign image dataset.

5. The method for detecting missing road traffic signs based on vehicle-mounted front-view images according to claim 3, characterized in that: The method of using an image annotation tool to annotate a high-resolution traffic sign image dataset comprises the following steps: S221, drawing the outer contour of the signboard at the pixel level and marking it, to obtain a marked partial data set; S222, using the labeled partial data set to train a preliminary model; S223. Use the trained preliminary model to process the high-resolution traffic sign image data and convert it into labeled data.

6. The method for detecting missing road traffic signs based on vehicle-mounted front-view images according to claim 3, characterized in that: The step of constructing a sign segmentation model and training the sign segmentation model based on a training set, a validation set, and a test set includes the following steps: S241. Building a sign segmentation model based on the mask region convolutional neural network model, training the sign segmentation model using the training set, and calculating the loss using a multi-task loss function; S242, calculating prediction accuracy based on the trained sign segmentation model; S243. Repeat steps S241 and S242 until the loss, prediction accuracy and recall rate of the sign segmentation model reach the experimental required thresholds.

7. The method for detecting missing road traffic signs based on vehicle-mounted front-view images according to claim 6, characterized in that: The expression for calculating the loss using the multi-task loss function is: L=L cls +L box +L mask In the formula, L represents the multi-task loss function, L cls represents the classification loss, N cls Indicates the number of positive samples of a category, p i * represents the correct category prediction probability, i represents the current number, L box represents the bounding box regression loss, N box represents the number of positive samples at the boundary, t i * represents the correct bounding box prediction value, t i represents the model prediction bounding box, L mask represents the instance segmentation loss, N mask represents the number of positive samples for instance segmentation, represents the prediction mask; The expression for calculating the prediction accuracy is: Where Ac represents the prediction accuracy, TP represents the true positive example, TN represents the true negative example, FP represents the false positive example, and FN represents the false negative example.

8. The method for detecting missing road traffic signs based on vehicle-mounted front-view images according to claim 1, characterized in that: The method based on the trained sign segmentation model inputs a highway scene image, outputs pixel coordinates of left and right sign pixel frames, and converts them into longitude and latitude positioning coordinates. According to the longitude and latitude positioning coordinates of multiple vehicle-mounted front view image sets, judging whether the sign is missing includes the following steps: S31, using the trained sign segmentation model to process the highway scene image to obtain the pixel coordinates of the left and right sign pixel frames; S32, extracting features based on the pixel coordinates of the left and right signboard pixel frames, matching the features of the signboards in the left and right images using a brute force matching algorithm, obtaining the parallax between the two, and calculating the depth distance between the signboard and the camera, and obtaining the spatial rectangular coordinates of the signboard image based on the similarity; S33, based on the rectangular coordinates of the sign image space, combined with the camera external parameters, converting them into inertial navigation coordinate system coordinates to obtain longitude and latitude positioning coordinates; S34. Generate signboard location data based on latitude and longitude location coordinates using geographic information system technology, and combine the signboard location data of multiple vehicle-mounted forward-looking image sets to determine the missing signboards.

Images