Monocular traffic sign online detection and photogrammetric positioning method and system

Abstract

The application belongs to the technical field of automatic driving high-precision map, and discloses a monocular traffic sign online detection and photogrammetry positioning method and system, which comprises the following steps: in the process of the mobile system carrying a combined navigation module, a monocular camera is used to collect air traffic sign images in real time; according to the camera positioning information of each traffic sign image frame detected by the combined navigation module, a target photogrammetry positioning equation is used to solve the three-dimensional object coordinate of the geometric center of each traffic sign image frame, so as to realize the positioning of the traffic sign image frame online. The application uses a lightweight RT-DETR model to detect traffic signs online, uses a SIFT matching algorithm to obtain a high-precision sign sequence geometric center, and proposes a new monocular photogrammetry method for traffic signs, further completes the three-dimensional coordinate solution of the traffic signs, and realizes the real-time detection and positioning of the traffic signs under low-cost monocular vision.

Description

Technical Field

[0001] This invention belongs to the field of high-precision map technology for autonomous driving, specifically relating to a method and system for online detection and photogrammetric positioning of monocular traffic signs. Background Technology

[0002] In recent years, autonomous driving technology and advanced driver assistance systems (ADAS) have become research hotspots. The increasing intelligence of automobiles has led to the emergence of high-precision maps, and information and location data of aerial features such as traffic signs are crucial components of these maps. Information acquisition primarily involves the detection and localization of traffic signs. Currently, traditional methods for traffic sign detection mainly extract features based on color and shape information and use template matching for recognition. However, these methods are easily affected by changes in lighting conditions, resulting in low detection success rates. Computer vision-based traffic sign detection has shifted from traditional image processing algorithms to popular deep learning algorithms, which exhibit excellent generalization performance. These algorithms are mainly divided into two categories: CNN-based and Transformer-based. CNN-based detection algorithms are represented by the YOLO series. For example, Wang et al. constructed the AF-FPN structure based on YOLOv5, a novel feature pyramid that improves multi-scale object detection capabilities. They also applied a new sample data augmentation method to enhance the robustness of the traffic sign detection model. However, one-stage or two-stage detection algorithms based on CNNs generate multiple non-unique anchor boxes, while object detection requires unique anchor boxes. Therefore, non-maximum suppression (NMS) is needed to remove redundant detection boxes, but the application of NMS results in a very large number of network parameters. To address this shortcoming, the DETR series of algorithms emerged, transforming the object detection problem into a geometric prediction problem, directly detecting the category and location of the object. Xia et al. proposed DSRA-DETR based on Transformer, integrating a pyramid pooling model and a multi-scale feature residual aggregation method to obtain features at different scales, minimizing noise and feature loss in feature extraction to improve the detection rate. However, Transformer-based traffic sign detection algorithms also suffer from complex network parameters and huge resource consumption such as memory. How to use a more lightweight model for online traffic sign detection is an urgent problem to be solved.

[0003] Real-time localization of traffic signs in mobile measurement has always been a research hotspot. Liu et al. established a calibration mapping table, obtained disparity through pyramid template matching, and measured the distance to traffic signs, but the localization accuracy was not high. Maged et al. improved upon the PointNet model by using local features for training, which improved accuracy and efficiency, but it relies on point cloud data, and radar costs are high. Liu et al. used a binocular vision method combined with local weighted regression smoothing to obtain higher-precision data, but the short installation baseline of the binocular vision system in autonomous vehicles cannot guarantee the accuracy of localizing relatively distant traffic signs. Therefore, how to use the lower-cost monocular vision to solve the real-time position of traffic signs in mobile platforms has been a problem that needs to be solved urgently, with limited research both domestically and internationally. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for online detection and photogrammetric positioning of traffic signs using monocular vision, enabling rapid online detection of traffic signs and solving the problem of real-time positioning of traffic signs in low-cost monocular vision.

[0005] The technical solution adopted in this invention is as follows:

[0006] This invention provides a method for online detection and photogrammetric positioning of monocular traffic signs, comprising the following steps:

[0007] Step S1: During the journey, the mobile system equipped with the integrated navigation module uses a monocular camera to collect real-time online images of aerial traffic signs.

[0008] Step S2: Using a traffic sign detection model, traffic sign detection is performed on the traffic sign images in real time online, and the traffic sign detection box corresponding to each traffic sign image is identified, thereby obtaining the traffic sign image frame corresponding to the traffic sign detection box;

[0009] Step S3: Select several traffic sign image frames within a reliable range, arranged from far to near, to form a traffic sign image frame sequence;

[0010] Step S4: Use the SIFT operator to perform feature matching on each traffic sign image frame in the traffic sign image frame sequence to obtain the geometric center of each traffic sign image frame;

[0011] Step S5: Based on the camera positioning information of each traffic sign image frame detected by the integrated navigation module, the three-dimensional object coordinates of the geometric center of each traffic sign image frame are calculated using the target photogrammetry positioning equation, thereby realizing online positioning of the traffic sign image frame.

[0012] Furthermore, step S2 specifically involves:

[0013] Step S21: The traffic sign detection model adopts an improved RT-DETR model, which includes a backbone network, an encoder neck network, and a decoder head network; wherein, the encoder neck network includes a same-scale feature interaction module and a cross-scale feature fusion module.

[0014] Step S22: The traffic sign image is input into the backbone network, and the backbone network performs multi-scale feature extraction on the traffic sign image, extracting deep semantic features, intermediate layer features and low-level features respectively.

[0015] Step S23: The deep semantic features are input to the same-scale feature interaction module, and self-attention behavior is performed on the elevation features to output high-level features;

[0016] Step S24: The high-level features, the intermediate-level features, and the low-level features are input into the cross-scale feature fusion module for cross-scale feature fusion, and the fused features are output.

[0017] In step S25, the decoder head network decodes the fused features and outputs the decoded traffic sign detection box.

[0018] Furthermore, step S4 specifically involves:

[0019] Step S41: According to the sequence of traffic sign image frames from far to near, use the SIFT operator to perform feature matching on every two adjacent traffic sign image frames. Based on the matching point pairs and feature descriptors, calculate the homography matrix between the two traffic sign image frames to represent the affine transformation relationship between the two traffic sign image frames.

[0020] Step S42: Determine the geometric center of the first traffic sign image frame in the traffic sign image frame sequence, and then perform perspective transformation using the corresponding homography matrix to obtain the matching center of the second traffic sign image frame, which serves as the geometric center of the second traffic sign image frame; then, using the geometric center of the second traffic sign image frame as the initial value, perform perspective transformation using the corresponding homography matrix to obtain the matching center of the third traffic sign image frame, which serves as the geometric center of the third traffic sign image frame, and so on, to determine the geometric center of each traffic sign image frame.

[0021] Furthermore, step S41 specifically includes:

[0022] Step S411: For every two adjacent traffic sign image frames, namely the first traffic sign image frame and the second traffic sign image frame; perform feature matching on the first traffic sign image frame and the second traffic sign image frame to obtain at least four matching point pairs;

[0023] The image coordinates of any matching point pair are represented as (x1, y1, z1) and (x2, y2, z2), respectively. Since the image is a two-dimensional plane, let z1 = z2 = 1.

[0024] Step S412, establish the image coordinate transformation equation shown in formula (1):

[0025]

[0026] Where: H is the homography matrix, expressed as:

[0027]

[0028] Step S413, based on formula (1), we obtain formula (2) and formula (3):

[0029]

[0030] in: x'2 and y'2 are intermediate variables;

[0031] Establish formulas (4), (5), and (6):

[0032] h=(H 11 H 12 H 13 H 21 H 22 H 23 H 31 H 32 H 33 ) T (4)

[0033]

[0034] Where: h is an intermediate variable; intermediate variable a x =(-x1,-y1,-1,0,0,0,x2′x1,x2′y1,x2′) T intermediate variable a y =(0,0,0,-x1,-y1,-1,y2′x1,y2′y1,y2′) T ;

[0035] Step S414: Therefore, for any two matching point pairs, a system of equations with formulas (1) to (6) is established; for at least four matching point pairs obtained, a system of equations with formulas (1) to (6) is established; the system of equations is solved using the least squares principle to obtain the homography matrix H.

[0036] Furthermore, step S5 specifically involves:

[0037] For a sequence of traffic sign image frames, a position difference constraint is formed based on the camera positioning information detected by the integrated navigation module, and the photogrammetric collinearity equation is modified to solve the three-dimensional object coordinates of the geometric center of the traffic sign in the direction of travel, vertical direction and vertical direction of the mobile system at the time of the first and last frames, thereby completing the positioning of the traffic sign image frames of the first and last frames; and further, the positioning of the traffic sign image frames of the sequence frames between the first and last frames is realized.

[0038] Furthermore, step S5 specifically involves:

[0039] Step S51, for each traffic sign image frame, perform image-side radial distortion correction at the geometric center using formulas (7) and (8):

[0040] x r =(cx-x0)(1+k1r 2 +k2r 4 +k3r 6 (7)

[0041] y r =(cy-y0)(1+k1r 2 +k2r 4 +k3r 6 (8)

[0042] in:

[0043] (x0, y0) are the original image-side coordinates of the geometric center of the traffic sign image frame; (cx, cy) are the coordinates of the principal point in the camera's interior orientation elements; (k1, k2, k3) are the radial distortion parameters in the interior orientation elements, obtained by Zhang's calibration method; r is the distance from the geometric center of the sign to the principal point; (x r ,y r () are the image coordinates after radial distortion correction is performed on the geometric center of the image.

[0044] Step S52, using formulas (9) and (10), image-side tangential distortion correction is performed:

[0045]

[0046] Where: (p1,p2) are the tangential distortion parameters in the interior orientation elements, obtained by Zhang's calibration method; (x,y) are the image-side coordinates of the traffic sign image frame after geometric center distortion correction;

[0047] Step S53, establish the target photogrammetric positioning equation:

[0048]

[0049] Where: (pX, pY, pZ) are the object coordinates, (a1, a2, ..., c2, c3) are the angular elements in the camera's exterior orientation elements, (X...) s ,Y s Z s ) is the line element of the camera's exterior orientation, obtained by Zhang's calibration method; f is the camera's focal length;

[0050] Step S54, for the traffic sign image frames at the beginning and end, according to formula (11), we can obtain:

[0051]

[0052] pZ1=pZ2 (14)

[0053] Where: (x1, y1) and (x2, y2) are the image-space coordinates of the geometric center of the traffic sign image frames after distortion correction in the first and last frames, respectively; (pX1, pY1, pZ1) and (pX2, pY2, pZ2) are the object-space coordinates of the geometric center of the traffic sign image frames in the first and last frames, respectively; and pZ1 and pZ2 are the elevations of the traffic sign center; f x and f y , which are the components of the camera focal length along the x-axis and y-axis, respectively; where the x-axis is the horizontal direction perpendicular to the direction of travel of the moving system; and the y-axis is the forward direction.

[0054] Furthermore, due to:

[0055] pY2-pY1=D (15)

[0056] Where: D is the distance the camera travels between the first and last frames provided by the integrated navigation module, and pY1 and pY2 are the distances from the traffic sign to the camera in the direction of travel at the first and last frames, respectively.

[0057] Let the intermediate quantity intermediate quantity

[0058] By combining formulas (12) to (15), we can obtain:

[0059]

[0060] Then, the three-dimensional object coordinates (pX1, pY1, pZ1) and (pX2, pY2, pZ2) of the geometric center of the traffic sign image frame of the first and last frames are calculated according to formulas (12) and (13), and the positioning of the traffic sign image frame of the first and last frames is completed.

[0061] Step S55 further realizes the positioning of traffic sign image frames in the sequence frames between the first and last frames.

[0062] Furthermore, step S55 specifically includes:

[0063] By using the distance traveled between the first and last frames of each traffic sign image frame from the time of capture to the position of the first frame, provided by the combined navigation module, and then using the three-dimensional object coordinates of the first frame, the three-dimensional object coordinates of each traffic sign image frame between the first and last frames are determined, thereby achieving the positioning of traffic sign image frames in the sequence between the first and last frames.

[0064] The present invention also provides a system for the monocular traffic sign online detection and photogrammetric positioning method, comprising:

[0065] A monocular camera for mobile systems is used to acquire real-time online images of aerial traffic signs while the mobile system is equipped with a combined navigation module.

[0066] A traffic sign detection model is used to perform traffic sign detection on the traffic sign images in real time online, identify the traffic sign detection box corresponding to each traffic sign image, and thus obtain the traffic sign image frame corresponding to the traffic sign detection box;

[0067] The SIFT operator is used to select several traffic sign image frames from far to near within a reliable range to form a traffic sign image frame sequence; the SIFT operator is used to perform feature matching on each traffic sign image frame in the traffic sign image frame sequence to obtain the geometric center of each traffic sign image frame.

[0068] The traffic sign image frame positioning module is used to calculate the three-dimensional object coordinates of the geometric center of each traffic sign image frame by using the target photogrammetry positioning equation based on the camera positioning information of each traffic sign image frame detected by the integrated navigation module, thereby realizing online positioning of the traffic sign image frame.

[0069] The present invention also provides a monocular traffic sign online detection and photogrammetric positioning device, the device comprising:

[0070] Memory, used to store computer programs;

[0071] A processor is used to execute the computer program to implement the steps of the above-described monocular traffic sign online detection and photogrammetric positioning method.

[0072] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described monocular traffic sign online detection and photogrammetric positioning method.

[0073] The monocular traffic sign online detection and photogrammetric positioning method and system provided by this invention have the following advantages:

[0074] This invention relates to a method and system for online detection and photogrammetric positioning of traffic signs using a lightweight RT-DETR model. It utilizes the SIFT matching algorithm to obtain the geometric center of the sign sequence with high precision and proposes a new monocular photogrammetric ranging method for traffic signs. Furthermore, it completes the three-dimensional coordinate calculation of traffic signs, realizing real-time detection and positioning of traffic signs under low-cost monocular vision. Attached Figure Description

[0075] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 A flowchart of the monocular traffic sign online detection and photogrammetric positioning method provided by the present invention;

[0077] Figure 2 The structural diagram of the RT-DETR traffic sign detection model provided by this invention;

[0078] Figure 3 The image shows the effect of RT-DETR traffic sign detection provided by the present invention; where (a) is the first frame, (b) is the middle frame, and (c) is the last frame.

[0079] Figure 4 The SIFT geometric center matching diagram provided by the present invention; wherein (a) is the first frame and the middle frame, and (b) is the middle frame and the last frame. Detailed Implementation

[0080] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0081] This invention provides a method and system for online detection and photogrammetric localization of traffic signs using monocular vision. This method addresses the issues of complex parameters and memory consumption in existing deep learning detection models, as well as the lack of research on low-cost monocular vision-based sign localization technology. It utilizes a lightweight RT-DETR model for online traffic sign detection, employs the SIFT matching algorithm to obtain the high-precision geometric center of the sign sequence, and proposes a novel monocular photogrammetric ranging method for traffic signs. Furthermore, it completes the 3D coordinate calculation of the traffic signs, achieving real-time detection and localization of traffic signs under low-cost monocular vision.

[0082] like Figure 1 As shown, the monocular traffic sign online detection and photogrammetric positioning method provided by the present invention includes the following steps:

[0083] Step S1: During the journey, the mobile system equipped with the integrated navigation module uses a monocular camera to collect real-time online images of aerial traffic signs.

[0084] Step S2: Using a traffic sign detection model, traffic sign detection is performed on the traffic sign images in real time online, and the traffic sign detection box corresponding to each traffic sign image is identified, thereby obtaining the traffic sign image frame corresponding to the traffic sign detection box;

[0085] Step S2 is as follows:

[0086] Step S21: The traffic sign detection model adopts an improved RT-DETR model, which includes a backbone network, an encoder neck network, and a decoder head network; wherein, the encoder neck network includes a same-scale feature interaction module and a cross-scale feature fusion module.

[0087] Step S22: The traffic sign image is input into the backbone network, and the backbone network performs multi-scale feature extraction on the traffic sign image, extracting deep semantic features, intermediate layer features and low-level features respectively.

[0088] Step S23: The deep semantic features are input to the same-scale feature interaction module, and self-attention behavior is performed on the elevation features to output high-level features;

[0089] Step S24: The high-level features, the intermediate-level features, and the low-level features are input into the cross-scale feature fusion module for cross-scale feature fusion, and the fused features are output.

[0090] In step S25, the decoder head network decodes the fused features and outputs the decoded traffic sign detection box.

[0091] Step S3: Select several traffic sign image frames within a reliable range, arranged from far to near, to form a traffic sign image frame sequence;

[0092] Step S4: Use the SIFT operator to perform feature matching on each traffic sign image frame in the traffic sign image frame sequence to obtain the geometric center of each traffic sign image frame;

[0093] Step S4 is as follows:

[0094] Step S41: According to the sequence of traffic sign image frames from far to near, use the SIFT operator to perform feature matching on every two adjacent traffic sign image frames. Based on the matching point pairs and feature descriptors, calculate the homography matrix between the two traffic sign image frames to represent the affine transformation relationship between the two traffic sign image frames.

[0095] Step S42: Determine the geometric center of the first traffic sign image frame in the traffic sign image frame sequence, and then perform perspective transformation using the corresponding homography matrix to obtain the matching center of the second traffic sign image frame, which serves as the geometric center of the second traffic sign image frame; then, using the geometric center of the second traffic sign image frame as the initial value, perform perspective transformation using the corresponding homography matrix to obtain the matching center of the third traffic sign image frame, which serves as the geometric center of the third traffic sign image frame, and so on, to determine the geometric center of each traffic sign image frame.

[0096] Step S41 is as follows:

[0097] Step S411: For every two adjacent traffic sign image frames, namely the first traffic sign image frame and the second traffic sign image frame; perform feature matching on the first traffic sign image frame and the second traffic sign image frame to obtain at least four matching point pairs;

[0098] The image coordinates of any matching point pair are represented as (x1, y1, z1) and (x2, y2, z2), respectively. Since the image is a two-dimensional plane, let z1 = z2 = 1.

[0099] Step S412, establish the image coordinate transformation equation shown in formula (1):

[0100]

[0101] Where: H is the homography matrix, expressed as:

[0102]

[0103] Step S413, based on formula (1), we obtain formula (2) and formula (3):

[0104]

[0105] in: x'2 and y'2 are intermediate variables;

[0106] Establish formulas (4), (5), and (6):

[0107] h=(H 11 H 12 H 13 H21 H 22 H 23 H 31 H 32 H 33 ) T (4)

[0108]

[0109]

[0110] Where: h is an intermediate variable; intermediate variable a x =(-x1,-y1,-1,0,0,0,x2′x1,x2′y1,x2′) T intermediate variable a y =(0,0,0,-x1,-y1,-1,y2′x1,y2′y1,y2′) T ;

[0111] Step S414: Therefore, for any pair of matching points, a system of equations (1) to (6) is established; for at least four pairs of matching points, a system of equations (1) to (6) is established; the system of equations is solved using the least squares principle to obtain the homography matrix H.

[0112] Step S5: Based on the camera positioning information of each traffic sign image frame detected by the integrated navigation module, the three-dimensional object coordinates of the geometric center of each traffic sign image frame are calculated using the target photogrammetry positioning equation, thereby realizing online positioning of the traffic sign image frame.

[0113] Step S5 is as follows:

[0114] For a sequence of traffic sign image frames, a position difference constraint is formed based on the camera positioning information detected by the integrated navigation module, and the photogrammetric collinearity equation is modified to solve the three-dimensional object coordinates of the geometric center of the traffic sign in the direction of travel, vertical direction and vertical direction of the mobile system at the time of the first and last frames, thereby completing the positioning of the traffic sign image frames of the first and last frames; and further, the positioning of the traffic sign image frames of the sequence frames between the first and last frames is realized.

[0115] Furthermore, step S5 specifically involves:

[0116] Step S51, for each traffic sign image frame, perform image-side radial distortion correction at the geometric center using formulas (7) and (8):

[0117] x r =(cx-x0)(1+k1r 2 +k2r 4+k3r 6 (7)

[0118] y r =(cy-y0)(1+k1r 2 +k2r 4 +k3r 6 (8)

[0119] in:

[0120] (x0, y0) are the original image-side coordinates of the geometric center of the traffic sign image frame; (cx, cy) are the coordinates of the principal point in the camera's interior orientation elements; (k1, k2, k3) are the radial distortion parameters in the interior orientation elements, obtained by Zhang's calibration method; r is the distance from the geometric center of the sign to the principal point; (x r ,y r () are the image coordinates after radial distortion correction is performed on the geometric center of the image.

[0121] Step S52, using formulas (9) and (10), image-side tangential distortion correction is performed:

[0122]

[0123] Where: (p1,p2) are the tangential distortion parameters in the interior orientation elements, obtained by Zhang's calibration method; (x,y) are the image-side coordinates of the traffic sign image frame after geometric center distortion correction;

[0124] Step S53, establish the target photogrammetric positioning equation:

[0125]

[0126] Where: (pX, pY, pZ) are the object coordinates, (a1, a2, ..., c2, c3) are the angular elements in the camera's exterior orientation elements, (X...) s ,Y s Z s ) is the line element of the camera's exterior orientation, obtained by Zhang's calibration method; f is the camera's focal length;

[0127] Step S54, for the traffic sign image frames at the beginning and end, according to formula (11), we can obtain:

[0128]

[0129] pZ1=pZ2 (14)

[0130] Where: (x1, y1) and (x2, y2) are the image-space coordinates of the geometric center of the traffic sign image frames after distortion correction in the first and last frames, respectively; (pX1, pY1, pZ1) and (pX2, pY2, pZ2) are the object-space coordinates of the geometric center of the traffic sign image frames in the first and last frames, respectively; pZ1 and pZ2 are the elevations of the traffic sign center. Since the distance covered by the same sign sequence image is relatively short, the elevation can be considered stable and unchanged in a short time, therefore pZ1 and pZ2 are equal. x and f y , which are the components of the camera focal length along the x-axis and y-axis, respectively; where the x-axis is the horizontal direction perpendicular to the direction of travel of the moving system; and the y-axis is the forward direction.

[0131] Furthermore, due to:

[0132] pY2-pY1=D (15)

[0133] Where: D is the distance the camera travels between the first and last frames provided by the integrated navigation module, and pY1 and pY2 are the distances from the traffic sign to the camera in the direction of travel at the first and last frames, respectively.

[0134] Let the intermediate quantity intermediate quantity

[0135] By combining formulas (12) to (15), we can obtain:

[0136]

[0137] Then, the three-dimensional object coordinates (pX1, pY1, pZ1) and (pX2, pY2, pZ2) of the geometric center of the traffic sign image frame of the first and last frames are calculated according to formulas (12) and (13), and the positioning of the traffic sign image frame of the first and last frames is completed.

[0138] Step S55 further realizes the positioning of traffic sign image frames in the sequence frames between the first and last frames.

[0139] Step S55 is as follows:

[0140] By using the distance traveled between the first and last frames of each traffic sign image frame from the time of capture to the position of the first frame, provided by the combined navigation module, and then using the three-dimensional object coordinates of the first frame, the three-dimensional object coordinates of each traffic sign image frame between the first and last frames are determined, thereby achieving the positioning of traffic sign image frames in the sequence between the first and last frames.

[0141] The following is an example:

[0142] Step 1: Online detection of traffic signs using monocular vision based on RT-DETR

[0143] In this invention, the traffic sign detection model adopts an improved RT-DETR model to achieve online detection of traffic signs using monocular vision.

[0144] In this invention, the improved RT-DETR model is an end-to-end object detection network architecture based on Transformer. It defines the number of objects through object queries and uses Hungarian matching to determine unique detection boxes, thus avoiding the use of non-maximum suppression to uniquely determine anchor boxes.

[0145] The model architecture comprises a backbone network, an encoder neck network, and a decoder head network. The backbone network is a ResNet, and the neck network is a high-efficiency hybrid encoder, including a same-scale feature interaction module and a cross-scale feature fusion module. It performs self-attention on semantically rich elevation features to improve the detection capability of the feature fusion module. Specifically, traffic sign images, after passing through the CNN backbone network, obtain features S3, S4, and S5. S5 has deeper, higher-level semantic features but a smaller spatial scale; therefore, only S5 features are input to the AIFI module of the neck network. This aligns with the attention mechanism's focus on semantic connections between features, avoiding the encoder's interaction between S3 and S5 multi-scale features, which would consume significant computational resources. The CCFM module of the RT-DETR model introduces multiple convolutional blocks during feature fusion to further perform cross-scale feature fusion, enhancing the overall performance of the model. Specifically, the high-level features extracted from traffic signs are input into a cross-scale feature fusion module along with the unextracted low-level features for feature fusion. Finally, a query optimization method considering the Intersection over Union (IoU) is used to generate the sign detection box and confidence score, significantly enhancing the location accuracy of the target detection box. This end-to-end target detection framework breaks through the traditional methods of predefined anchor boxes and non-maximum suppression, reducing redundancy while ensuring the accuracy of the bounding boxes. The inference process is smooth and efficient, significantly improving efficiency and making it highly suitable for online real-time traffic sign detection applications. Image data of traffic signs is acquired using a monocular camera, and the RT-DETR model architecture is used to train the data. Compared to YOLOv8, RT-DETR has a shorter training period (75-80 epochs), does not require mosaic enhancement, but achieves better efficiency and balance, fully realizing the accuracy and real-time performance of traffic sign detection.

[0146] Step 2: Aerial Photogrammetry Positioning Method for Traffic Signs Based on Monocular View

[0147] The SIFT operator is used for sequence frame matching to obtain the high-precision geometric center of the sign. Combined with the combined positioning information of the mobile system, a new target photogrammetric positioning equation is proposed to further complete the three-dimensional coordinate calculation of the traffic sign, realizing the real-time detection and positioning of traffic signs under low cost monocular vision.

[0148] The specific steps for using the SIFT operator to perform sequence frame matching to obtain high-precision marker geometric centers are as follows:

[0149] Subsequent traffic sign photogrammetric localization algorithms require highly stable calculated sign geometric centers; otherwise, the accuracy of photogrammetric localization will be affected. Therefore, feature matching is needed for the traffic sign images detected in the sequence of images to control the relative pixel error of the geometric center on the sequence of images. This invention uses the SIFT operator to perform feature matching on the detected sequence of traffic sign images and uses the RANSAC method to calculate the homography matrix, representing the affine transformation relationship between different sign images, in order to match a more accurate sign geometric center on the sequence of frames. Because the localization algorithm uses the first and last frames for photogrammetric calculation, the geometric center of the sign on the first frame is used as the standard for pairwise matching in the sequence of images. The selection of the first frame is extremely important. The selection principle is: select the sign detection box with the highest score (confidence) obtained by RT-DETR from the first 20 frames of the sequence of sign images. This box most accurately segments the traffic sign from the background, and its geometric center accuracy is the highest. It can be set as the initial geometric center for subsequent pairwise matching.

[0150] Combining the combined positioning information from the mobile system, a new target photogrammetric positioning equation is proposed. The specific steps for further calculating the three-dimensional coordinates of traffic signs are as follows:

[0151] Traffic signs are aerial features located far from the ground. Although they are physically connected to the ground in the real world, in imagery, the connecting rods of most signs are obscured by road features such as flower beds. Therefore, it is impossible to identify the connection point between the traffic sign and the ground, nor can the position coordinates be calculated by combining camera projection with the horizontal geometric constraints of the road surface. For low-cost mobile acquisition systems, a novel monocular aerial photogrammetric positioning method for traffic signs is proposed. This method generates and tracks detection boxes for traffic signs in real time on the sequence of images acquired by the mobile system using an RT-DETR model. Detection boxes from the first and last frames within a reliable range are selected. Based on camera positioning information provided by the mobile system's integrated navigation module, position difference constraints are formed, and the photogrammetric collinearity equation is modified. The three-dimensional coordinates of the target in the first frame (direction of travel, perpendicular direction, and vertical direction) are calculated, thus completing the positioning of all aerial traffic signs in the first and last frames and subsequent frames.

[0152] The advantages and beneficial effects of this invention are as follows:

[0153] (1) The RT-DETR model is used to detect traffic signs online in sequence images with varying distances, effectively resisting image blurring caused by distance changes outside the camera's depth of field. Compared with the YOLOv8m model, the accuracy and recall of traffic sign detection based on RT-DETR are improved, the average detection accuracy is comparable, and the number of model parameters and computational load are comparable or better, and the detection efficiency fully meets the real-time requirements of online detection. (2) In the monocular photogrammetric positioning process, the SIFT operator is used to match sequence frames to obtain high-precision geometric centers of signs, ensuring the stability of the geometric center calculation and reducing the image-side error of photogrammetric positioning. (3) The position difference constraint is combined to modify the photogrammetric collinearity equation, and a new monocular photogrammetric ranging method for traffic signs is proposed. The three-dimensional coordinate calculation of the geometric centers of all aerial traffic signs on the first and last frames and the sequence frames in between is completed, realizing the online accurate positioning of signs.

[0154] This invention also provides a system for realizing online detection and photogrammetric positioning of monocular traffic signs, comprising:

[0155] A monocular camera for mobile systems is used to acquire real-time online images of aerial traffic signs while the mobile system is equipped with a combined navigation module.

[0156] A traffic sign detection model is used to perform traffic sign detection on the traffic sign images in real time online, identify the traffic sign detection box corresponding to each traffic sign image, and thus obtain the traffic sign image frame corresponding to the traffic sign detection box;

[0157] The SIFT operator is used to select several traffic sign image frames from far to near within a reliable range to form a traffic sign image frame sequence; the SIFT operator is used to perform feature matching on each traffic sign image frame in the traffic sign image frame sequence to obtain the geometric center of each traffic sign image frame.

[0158] The traffic sign image frame positioning module is used to calculate the three-dimensional object coordinates of the geometric center of each traffic sign image frame by using the target photogrammetry positioning equation based on the camera positioning information of each traffic sign image frame detected by the integrated navigation module, thereby realizing online positioning of the traffic sign image frame.

[0159] In practice, this invention also provides a monocular traffic sign online detection and photogrammetric positioning device, the device comprising:

[0160] Memory, used to store computer programs;

[0161] A processor is used to execute the computer program to implement the steps of the monocular traffic sign online detection and photogrammetric positioning method.

[0162] In practice, this invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the monocular traffic sign online detection and photogrammetric positioning method.

[0163] Here is another example:

[0164] The dataset used in this embodiment consists of 9000 images of traffic signs along a 15-kilometer stretch of urban Qingdao, collected by a self-developed mobile system. The images are 1280×720 pixels in resolution, RGB three-channel color mode, and JPEG format. The training set contains 7200 images, and the test set contains 1800 images. The model training environment includes an Intel Core i9 CPU, an RTX 4090 GPU, 64GB of RAM, Ubuntu 18.04 operating system, and is written in Python 3.9 and PyTorch 1.7.2. The training cycle is 120, and the batch size is 8. The training results are shown in Table 1. Compared to the YOLOv8m model, the traffic sign detection based on RT-DETR shows improvements in accuracy (P) and recall (R), while the average detection precision (mAP) is comparable (difference of only 0.6, which can be considered equivalent). The number of model parameters and computational cost are comparable or better, and the number of images detected per second (FPS) is 106 frames. Although the performance is slightly lower than that of the YOLOv8m model, the difference is not significant, and the performance fully meets the real-time requirements of online detection.

[0165] The trained RT-DETR traffic sign detection model is used to perform real-time online detection of traffic sign sequences. The detection results are as follows: Figure 3 Because mobile systems acquire traffic sign images from far to near, signs that are far apart become blurred due to the distance being much greater than the depth of field, and the degree of blur varies with distance. The RT-DETR traffic sign detection model effectively combats the varying degrees of blur caused by different distances, maintaining the accuracy and stability of the detection boxes and ensuring the stable calculation of the geometric center on the image sequence, thus providing a foundation for subsequent photogrammetric positioning of the geometric center.

[0166] Table 1 Comparison of traffic sign detection models based on RT-DETR and YOLOv8m

[0167]

[0168] like Figure 4 Following the image sequence from far to near, the SIFT operator is used to perform pairwise feature matching on the marker images within the detection box (e.g., ...). Figure 4As shown by the color matching lines in the middle, based on the matching point pairs and feature descriptors, the homography matrix between the two images is calculated, representing the coordinate transformation relationship between their pixels. Based on the center of the marker in the first frame, a perspective transformation is performed using the homography matrix to calculate the matching center of the marker in the next frame (e.g., ...). Figure 4 (As shown by the bold red matching line in the middle), the pixel coordinates of traffic signs on all image sequences are calculated in the same way.

[0169] As shown in Table 2, a total of 298 traffic signs were detected in the 15-kilometer online image sequence (the same sign has the same ID on different images). The monocular photogrammetric positioning method of this invention was used to locate the aerial signs online, calculate the three-dimensional coordinates, and convert them into coordinates in the global coordinate system based on the positioning information provided by the integrated navigation system. The coordinates were compared with the road element data provided on the existing high-precision map, and the positioning errors in three directions (longitudinal, lateral, and vertical) were calculated and statistically analyzed. The overall average error was less than 0.2m, which is comparable to the accuracy of the high-precision map.

[0170] To demonstrate the advantage of using SIFT matching to calculate a more accurate marker geometric center, the SIFT marker center matching module was disabled in the comparative experiment. The geometric center of the marker detection box was directly used to calculate the three-dimensional coordinates of the marker center. As shown in Table 2, the overall geometric error is greater than 0.2m, indicating that the pixel coordinate accuracy of the marker geometric center has a significant impact on the final positioning error.

[0171] Table 2 Statistics on Monocular Positioning Accuracy of Traffic Signs

[0172]

[0173] This invention addresses the challenges of complex detection models and low-cost monocular vision-based localization by proposing a novel online traffic sign detection and photogrammetric localization method. Utilizing the RT-DETR model, it performs online traffic sign detection on image sequences with varying distances, effectively mitigating image blurring caused by distance changes outside the camera's depth of field. During monocular photogrammetric localization, the SIFT operator is employed for frame matching to obtain high-precision sign geometric centers, ensuring the stability of geometric center calculation and reducing image-side errors in photogrammetric localization. Simultaneously, a new monocular photogrammetric ranging method for traffic signs is proposed, incorporating position difference constraints to improve the photogrammetric collinearity equation. This method completes the 3D coordinate calculation of the geometric centers of all aerial traffic signs in the first and last frames and subsequent frames, achieving precise online sign localization. The method boasts high detection accuracy, fully meeting real-time requirements, and provides accurate sign localization, filling a gap in the field of monocular traffic sign localization technology and demonstrating broad application prospects.

[0174] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for online detection and photogrammetric positioning of monocular traffic signs, characterized in that, Includes the following steps: Step S1: During the journey, the mobile system equipped with the integrated navigation module uses a monocular camera to collect real-time online images of aerial traffic signs. Step S2: Using a traffic sign detection model, traffic sign detection is performed on the traffic sign images in real time online, and the traffic sign detection box corresponding to each traffic sign image is identified, thereby obtaining the traffic sign image frame corresponding to the traffic sign detection box; Step S3: Select several traffic sign image frames within a reliable range, arranged from far to near, to form a traffic sign image frame sequence; Step S4: Use the SIFT operator to perform feature matching on each traffic sign image frame in the traffic sign image frame sequence to obtain the geometric center of each traffic sign image frame; Step S5: Based on the camera positioning information of each traffic sign image frame detected by the integrated navigation module, the three-dimensional object coordinates of the geometric center of each traffic sign image frame are calculated using the target photogrammetry positioning equation, thereby realizing online positioning of the traffic sign image frame; Step S5 is as follows: For a sequence of traffic sign image frames, a position difference constraint is formed based on the camera positioning information detected by the integrated navigation module, and the photogrammetric collinearity equation is modified to solve the three-dimensional object coordinates of the geometric center of the traffic sign in the direction of travel, vertical direction and vertical direction of the mobile system at the time of the first and last frames, thereby completing the positioning of the traffic sign image frames of the first and last frames; and further, the positioning of the traffic sign image frames of the sequence frames between the first and last frames is realized.

2. The method for online detection and photogrammetric positioning of monocular traffic signs according to claim 1, characterized in that, Step S2 is as follows: Step S21: The traffic sign detection model adopts an improved RT-DETR model, which includes a backbone network, an encoder neck network, and a decoder head network; wherein, the encoder neck network includes a same-scale feature interaction module and a cross-scale feature fusion module. Step S22: The traffic sign image is input into the backbone network, and the backbone network performs multi-scale feature extraction on the traffic sign image, extracting deep semantic features, intermediate layer features and low-level features respectively. Step S23: The deep semantic features are input to the same-scale feature interaction module, and self-attention behavior is performed on the elevation features to output high-level features; Step S24: The high-level features, the intermediate-level features, and the low-level features are input into the cross-scale feature fusion module for cross-scale feature fusion, and the fused features are output. In step S25, the decoder head network decodes the fused features and outputs the decoded traffic sign detection box.

3. The method for online detection and photogrammetric positioning of monocular traffic signs according to claim 1, characterized in that, Step S4 is as follows: Step S41: According to the sequence of traffic sign image frames from far to near, use the SIFT operator to perform feature matching on every two adjacent traffic sign image frames. Based on the matching point pairs and feature descriptors, calculate the homography matrix between the two traffic sign image frames to represent the affine transformation relationship between the two traffic sign image frames. Step S42: Determine the geometric center of the first traffic sign image frame in the traffic sign image frame sequence, and then perform perspective transformation using the corresponding homography matrix to obtain the matching center of the second traffic sign image frame, which serves as the geometric center of the second traffic sign image frame; then, using the geometric center of the second traffic sign image frame as the initial value, perform perspective transformation using the corresponding homography matrix to obtain the matching center of the third traffic sign image frame, which serves as the geometric center of the third traffic sign image frame, and so on, to determine the geometric center of each traffic sign image frame.

4. The method for online detection and photogrammetric positioning of monocular traffic signs according to claim 3, characterized in that, Step S41 is as follows: Step S411: For every two adjacent traffic sign image frames, namely the first traffic sign image frame and the second traffic sign image frame; perform feature matching on the first traffic sign image frame and the second traffic sign image frame to obtain at least four matching point pairs; The image coordinates of any pair of matching points are represented as follows: and Since the image is a two-dimensional plane, let ; Step S412, establish the image coordinate transformation equation shown in formula (1): （1） in: The homography matrix is ​​expressed as: ； Step S413, according to formula (1), we obtain formula (2) and formula (3): （2） （3） in: , , and As an intermediate variable; Establish formulas (4), (5), and (6): （4） （5） （6） in: Intermediate variables; intermediate variables intermediate variables ; Step S414: Therefore, for any two matching point pairs, a system of equations (1) to (6) is established; for at least four matching point pairs obtained, a system of equations (1) to (6) is established; the system of equations is solved using the least squares principle to obtain the homography matrix. .

5. The method for online detection and photogrammetric positioning of monocular traffic signs according to claim 1, characterized in that, Step S5 is as follows: Step S51, for each traffic sign image frame, perform image-side radial distortion correction at the geometric center using formulas (7) and (8): （7） （8） in: These are the original image-side coordinates of the geometric center of the traffic sign image frame; These are the coordinates of the principal image point in the camera's orientation elements; It is the radial distortion parameter in the interior orientation element, obtained by Zhang's calibration method; It is the distance from the geometric center of the image to the principal point; These are the image coordinates after radial distortion correction is performed on the geometric center of the image. Step S52, using formulas (9) and (10), image-side tangential distortion correction is performed: （9） （10） in: It is the tangential distortion parameter in the interior orientation element, obtained by Zhang's calibration method; These are the image-side coordinates of the traffic sign image frame after geometric center distortion correction; Step S53, establish the target photogrammetric positioning equation: （11） in: These are object coordinates. It is the corner element in the camera's external orientation elements. These are the line elements of the camera's exterior orientation, obtained using Zhang's calibration method; It is the camera's focal length; Step S54, for the traffic sign image frames at the beginning and end, according to formula (11), we can obtain: （12） （13） （14） in: , These are the geometrically distorted image coordinates of the traffic sign image frames, specifically the first and last frames. , These are the object-space coordinates of the geometric centers of the traffic sign image frames in the first and last frames, respectively. and The elevation of the traffic sign center; and , respectively, along the camera focal length towards and along The component of the direction; among which, The direction is perpendicular to the direction of travel of the mobile system and horizontal; The direction of progress; Furthermore, due to: (15) Where: D is the distance the camera travels between the first and last frames, provided by the integrated navigation module. and These represent the distances from the traffic sign to the camera in the direction of travel at the first and last frame times, respectively. Let the intermediate quantity Intermediate quantity ; By combining formulas (12) to (15), we can obtain: （16） Then, the three-dimensional object coordinates of the geometric center of the traffic sign image frame in the first and last frames are calculated according to formulas (12) and (13). , Complete the positioning of traffic sign image frames at the beginning and end; Step S55 further realizes the positioning of traffic sign image frames in the sequence frames between the first and last frames.

6. The method for online detection and photogrammetric positioning of monocular traffic signs according to claim 5, characterized in that, Step S55 is as follows: By using the distance traveled between the first and last frames of each traffic sign image frame from the time of capture to the position of the first frame, provided by the combined navigation module, and then using the three-dimensional object coordinates of the first frame, the three-dimensional object coordinates of each traffic sign image frame between the first and last frames are determined, thereby achieving the positioning of traffic sign image frames in the sequence between the first and last frames.

7. A system for the online detection and photogrammetric positioning method for monocular traffic signs according to any one of claims 1-6, characterized in that, include: A monocular camera for mobile systems is used to acquire real-time online images of aerial traffic signs while the mobile system is equipped with a combined navigation module. A traffic sign detection model is used to perform traffic sign detection on the traffic sign images in real time online, identify the traffic sign detection box corresponding to each traffic sign image, and thus obtain the traffic sign image frame corresponding to the traffic sign detection box; The SIFT operator is used to select several traffic sign image frames from far to near within a reliable range to form a traffic sign image frame sequence; the SIFT operator is used to perform feature matching on each traffic sign image frame in the traffic sign image frame sequence to obtain the geometric center of each traffic sign image frame. The traffic sign image frame positioning module is used to calculate the three-dimensional object coordinates of the geometric center of each traffic sign image frame by using the target photogrammetry positioning equation based on the camera positioning information of each traffic sign image frame detected by the integrated navigation module, thereby realizing online positioning of the traffic sign image frame.

8. A monocular traffic sign online detection and photogrammetric positioning device, characterized in that, The device includes: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the monocular traffic sign online detection and photogrammetric positioning method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the monocular traffic sign online detection and photogrammetric positioning method as described in any one of claims 1 to 6.

Images