3D annotation of traffic management objects
By calculating the 2D bounding boxes of traffic management objects in 3D space and generating image cropping blocks, combined with neural networks and clustering algorithms, the limitations of 2D annotation and the impracticality of manual 3D annotation are solved, realizing automated and reliable annotation of traffic management objects and generation of training datasets.
Patent Information
- Application Number
- CN202580029241.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-04-29
- Filing Date
- 2025-04-28
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, 2D annotation of traffic management objects cannot effectively capture the object's depth, orientation, and spatial relationships. Manual 3D annotation is complex and inconsistent, which leads to compromised data reliability in autonomous driving systems.
By obtaining 2D bounding boxes and positional orientation data from multiple image frames, the center of the 3D bounding box is calculated, 2D image cropping blocks are generated, and the model is trained for classification. Traffic management objects are automatically labeled using image detector neural networks and clustering algorithms.
It achieves robust, fast and efficient 3D annotation of traffic management objects, generates training datasets with improved variability, and improves the recognition accuracy and robustness of autonomous driving systems.
Smart Images

Figure CN123125622A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to labeling traffic management objects in three-dimensional (3D) space and generating training datasets. In particular, this invention relates to methods for automatically labeling traffic management objects in 3D space, methods for generating training datasets, and corresponding computer-readable media, apparatus, and systems. Background Technology
[0002] Traffic on roads is typically controlled and managed by traffic management objects. These objects play a crucial role in ensuring the safe and efficient operation of vehicles by regulating traffic flow, providing necessary warnings, and guiding road users. Accurate and reliable identification of these traffic management objects is essential for the development of autonomous driving systems, which heavily rely on sensor data and environmental understanding to safely navigate roads.
[0003] Traditionally, traffic management objects are labeled in two dimensions (2D). This process involves tagging images with labels that identify and categorize different traffic signs, signals, and other relevant road markings. While 2D labeling was helpful in the early stages of autonomous driving technology, it presents significant limitations when applied to the increasingly complex, dynamic, and inherently 3D nature of real-world driving environments. 2D labeling cannot fully capture the depth, orientation, and spatial relationships of objects, which are crucial for accurate perception and decision-making processes in autonomous vehicles or traffic management tasks. For example, the distance and angle of traffic signs, which are critical for timely and appropriate responses, cannot be effectively conveyed through 2D labeling.
[0004] Furthermore, manual annotation presents significant challenges in both 2D and 3D environments. It is a labor-intensive, time-consuming, and expensive process requiring substantial human resources and expertise. Manual 3D annotation is particularly cumbersome due to the increased complexity of accurately defining objects in 3D space. This process is not only slow but also prone to inconsistencies and errors, potentially compromising the reliability of data used to train and validate autonomous driving systems.
[0005] US 2023 / 186499 A1 addresses the automatic localization of static objects in urban environments. Noisy image data is used to identify and determine the location of objects in large-scale environments. Static objects are located using a voting-based triangulation technique.
[0006] KR 2023104841 A relates to object labeling in map data used for autonomous driving vehicles. This method is based on LiDAR data of the environment surrounding the autonomous vehicle and is used to register target objects to the map data. Another method relying on LiDAR and camera data is described in the following literature: Wonho Song and Hyun Myung: “3D TrafficSign Detection Using Camera-LiDAR Projection”, Recent Trends in MechatronicsTowards Industry 4.0, Lecture Notes in Electrical Engineering, vol. 730, 2021. This method uses deep learning for 2D object detection, followed by LiDAR camera calibration data and outlier removal algorithms to collect 3D point cloud data of traffic signs.
[0007] EP 4205025 A1 discloses a method for training a classifier to identify traffic signal states in images captured by a vehicle. The identified states are used by the vehicle when making movement decisions in the environment.
[0008] Simone Mentasti et al.: “Traffic lights detection and tracking for HDmap creation”, Frontiers in Robotics and AI, 2023, discloses a pipeline for creating high-definition maps of lights, which is designed to provide geographic reference locations and descriptions of traffic lights as observed by cameras mounted on survey vehicles.
[0009] Nathaniel Fairfield and Chris Urmson: “Traffic Light Mapping and Detection”, IEEE International Conference on Robotics and Automation, 2011, disclose a method for automatically applying the 3D position of traffic lights and detecting the status of onboard traffic lights using a camera. Summary of the Invention
[0010] The limitations of 2D annotation and the impracticality of existing 3D annotation methods underscore the need for automated, reliable, and efficient methods for 3D annotation of traffic management objects. In summary, there is a need for improved methods for automated 3D annotation of traffic management objects that overcome the shortcomings of existing techniques.
[0011] Therefore, the object of this invention is to provide, according to the independent claims, a method for automatically labeling traffic management objects in three-dimensional space and a method for generating training datasets. Furthermore, corresponding (electronic) devices or apparatuses, machine-readable media, and (computer) systems are defined.
[0012] A first aspect of the present invention provides a method for annotating (preferably automatically annotating) traffic management objects in three-dimensional space, comprising: obtaining a plurality of image frames, at least some of the image frames including two-dimensional 2D bounding boxes of the traffic management objects, and each image frame being associated with position and orientation data; locating the center of a 3D bounding box of the traffic management object using the plurality of image frames including the 2D bounding boxes and the corresponding position and orientation data; for the center of the located 3D bounding box of each traffic management object, calculating the size and orientation of the 3D bounding box based on the image frame including the 2D bounding box of the traffic management object; projecting the 3D bounding box of each traffic management object onto the image frame associated with the traffic management object; generating a plurality of 2D image cutouts of the traffic management objects based on the projected 3D bounding boxes; and training at least one model to classify the traffic management objects in the plurality of 2D image cutouts.
[0013] The method for automatically labeling traffic management objects is a computer-implemented method. The method can be implemented in a computing device or on a system having one or more computing devices, the computing devices including one or more processors or processing units (PUs) that can be coupled to or communicate with each other and configured to execute individual method steps. The one or more processors or PUs can implement hardware and / or software modules that execute the individual method steps.
[0014] The first aspect of the method operates on multiple image frames containing 2D bounding boxes of traffic management objects. The image frames may originate from a moving camera or multiple fixed cameras placed at different viewpoints within the environment. Each image frame is associated with its position and orientation data (in 3D space) within the environment. The 2D bounding boxes in the multiple image frames, along with their positions and orientations, are used to determine the center of the traffic management object in 3D space.
[0015] A 2D bounding box is a rectangular box that defines the position of an object within an image frame. It is "2D" because it operates within the two-dimensional space of the image frame. A 2D bounding box can be specified by two points, such as the coordinates of the top-left and bottom-right corners of the rectangle. Alternatively, it can be defined by the coordinates of a corner (typically the top-left) along with the width and height of the bounding box. The purpose of a 2D bounding box is to highlight or isolate the area within an image frame that contains the traffic management object. The coordinates of the 2D bounding box are typically in pixels relative to the size of the image frame, where the origin (0, 0) is typically located at the top-left corner of the image. The x-coordinate can increase to the right, and the y-coordinate can increase downwards. However, it should be understood that other parameters can be used to specify the rectangular area for the 2D bounding box. Using multiple 2D bounding boxes from multiple corresponding image frames and combining the position and orientation data of the respective image frames makes it possible to locate the center of the traffic management object in 3D space, where each center can constitute the center of the corresponding 3D bounding box of the traffic management object. In addition, each center of the corresponding 3D bounding box of a traffic management object can be associated with a set of image frames that include the 2D bounding box that identifies the traffic management object.
[0016] 3D bounding boxes extend the concept of 2D bounding boxes into 3D space. While a 2D bounding box in an image frame is defined by a rectangle on a plane, enclosing the dimensions of a traffic management object in width and height, a 3D bounding box encloses the dimensions of a traffic management object in width, height, and depth, effectively creating a volumetric boundary around the object. A 3D bounding box can be represented by a three-dimensional rectangular prism (or cuboid) and is typically defined by its center, dimensions, and orientation (rotation) relative to the principal axes (usually X, Y, and Z). Additionally or alternatively, a 3D bounding box can be defined by the coordinates of its eight corners, or by one corner and its dimensions of width, height, and depth along the three principal axes.
[0017] In the method according to the first aspect, the (3D) center can be considered as a reference point for constructing a corresponding 3D bounding box around the traffic management object, providing a comprehensive representation of the traffic management object's position, size, and orientation in the environment. As previously described, each located 3D center can correspond to a set of corresponding image frames including the 2D bounding box of the respective traffic management object. This set of image frames can be used to calculate the size and orientation of the 3D bounding box relative to the located center.
[0018] The method also uses specified 3D bounding boxes to generate multiple 2D image crop blocks of traffic management objects. The 2D image crop blocks are generated based on the projected 3D bounding boxes of the respective traffic management objects. Specifically, each 3D bounding box of a traffic management object is projected onto those image frames associated with that traffic management object, such as image frames that include the 2D bounding boxes of the traffic management object. Subsequently, at least one model is trained to classify the traffic management objects in the multiple 2D image crop blocks.
[0019] 2D image crop patches are cropped portions from larger image frames, created by selecting and extracting specific regions of interest using projection-based 3D bounding boxes. By focusing on specific regions containing traffic management objects, these 2D image crop patches provide clearer, more focused data for training, enabling the model to learn relevant features without being affected by background noise or irrelevant parts of the image.
[0020] Each 3D bounding box defining the spatial dimensions of a traffic management object in 3D space is projected onto a 2D plane of the image frame. This projection converts the volumetric information of the 3D bounding box into a 2D viewpoint, preferably taking into account the camera's viewpoint, angle, and the spatial position of the traffic management object. The projection process produces a rectangular region (which can again be represented as a 2D bounding box) that outlines the area of the image frame where the traffic management object appears, taking into account its 3D position and size. The rectangular region is then cropped from the original image frame to obtain a 2D image crop block. The 2D image crop block is essentially a smaller image containing the traffic management object specified by the 3D bounding box, constructed based on the rectangular region. Preferably, the 2D image crop block can be created using image frames located within a predetermined distance of the traffic management object in 3D space. Preferably, the 3D bounding box of the traffic management object can be projected onto an image frame 200 meters or less away from the traffic management object.
[0021] 2D image cropping patches reduce computational requirements by focusing on smaller image regions containing traffic management objects rather than processing the entire image frame. They further focus on the traffic management objects, which helps at least one model learn relevant features more effectively, thus improving recognition accuracy. Furthermore, generating 2D image cropping patches from various angles and distances using 3D bounding boxes introduces desired variability into the training dataset, contributing to the creation of more robust models.
[0022] The method provides a robust, fast, and efficient approach to achieve automatic 3D annotation of traffic management objects based on readily available 2D annotations. The obtained 3D annotations are used to generate a training dataset with improved variability to effectively train at least one model with improved accuracy and robustness.
[0023] In a first embodiment of the method according to the first aspect, an image detector neural network is used to obtain 2D bounding boxes, wherein the method further includes detecting a confidence level of the detected 2D bounding boxes and using image frames with 2D bounding boxes having confidence levels higher than a threshold to locate the center of a 3D bounding box of a traffic management object. Along with the 2D bounding boxes, the image detector neural network can provide a confidence level for each detection. This confidence level is typically expressed as a percentage or probability between 0 and 1, which indicates the network's determinism in detecting and classifying traffic management objects within the bounding box.
[0024] (Image Detector) A neural network scans image frames, searching for patterns, features, or characteristics that match those of traffic management objects, having been trained to recognize them. Depending on the neural network's training data, this can include a wide variety of traffic management objects. Once an object is detected, the neural network determines its location within the image frame and draws a 2D bounding box around it. This 2D bounding box represents the outline of the size of the traffic management object observed in the image frame. The confidence level reflects how certain the neural network is that the bounding box contains a specific type of traffic management object. For example, a 95% confidence level indicates that, based on the categories the model was trained on, there is a 95% probability that the object within the bounding box is correctly identified as a traffic management object. The confidence level can be calculated based on the output of a specific layer of the neural network, such as a softmax layer, which can be combined with the accuracy of the 2D bounding box or other relevant factors. For example, the confidence level can take into account the probability of the traffic management object's category and the accuracy of its spatial location and size.
[0025] Thresholds can be set to filter out detections with low confidence levels. For example, only detections with confidence levels above a certain threshold (such as 50%, 60%, 70%, 80%, or 90% or others) can be considered valid. It should be understood that thresholds can be adjusted to improve the robustness of the method. Using thresholds helps reduce false positives, where the network might incorrectly identify irrelevant objects or noise as valid traffic management objects.
[0026] In another embodiment of the method according to the first aspect, locating the center of the 3D bounding box of the traffic management object includes: for each image frame including the 2D bounding box of the traffic management object, projecting a centerline from the camera position associated with that image frame to the center of the 2D bounding box in that image frame to generate multiple centerlines for multiple image frames. This 3D reconstruction and localization task can be solved based on image frames including sensor data acquired from different angles or positions. Each 2D bounding box can be defined by its angular coordinates or by a corner and its dimensions (width and height). This can be used to calculate the center point of each 2D bounding box. This can be done by averaging the coordinates of diagonally opposite corners or by adding half of the width and height to the coordinates of the upper left corner. From the camera position associated with each image frame, a line can be projected toward the center of the 2D bounding box. This line represents the projection of the center of the traffic management object from the camera's viewpoint. The camera's position and orientation (extrinsic parameters) and its internal characteristics (such as focal length and lens distortion) (intrinsic parameters) can be further considered to accurately project the centerline in 3D space. The centerline can be used to determine the center of the 3D bounding box of the corresponding traffic management object. By utilizing multiple viewpoints and the geometry of projections within image frames, this method effectively bridges the gap between 2D image data and 3D spatial understanding, enabling comprehensive spatial analysis and decision-making processes in automated systems and applications.
[0027] In another embodiment of the method according to the first aspect, the centering of the 3D bounding box of the traffic management object further includes: identifying a set of centerlines among the plurality of centerlines whose distances from each other are less than a predetermined distance; generating a plurality of candidate points by iterating along each pair of centerlines in each set to determine the candidate point closest to the corresponding centerline pair; clustering the plurality of candidate points to form clusters; and using the candidate points of each cluster to compute the center of the 3D bounding box of the traffic management object. To identify different traffic management objects, the centerlines are analyzed to identify a set of lines whose distances from each other are less than a predetermined distance. This step is crucial because it helps to distinguish different traffic management objects and filter out lines that are unlikely to intersect due to errors in detection or camera positioning. For each set of centerlines identified as sufficiently close to each other (according to the predetermined distance), a pair of lines is established, and for each pair, the candidate point closest to these two lines is determined. This is typically achieved through a geometric method or algorithm that finds the shortest distance between the two lines in 3D space and identifies the midpoint of that shortest segment as a candidate point. A clustering algorithm is then performed on the candidate points obtained from all considered centerline pairs. Clustering groups points based on their proximity to each other. Various clustering algorithms can be used, such as K-means, DBSCAN, or hierarchical clustering. The choice of algorithm can depend on the expected distribution of points and the specific requirements of the task, such as the need to specify the number of clusters in advance. Within each cluster of candidate points, the cluster centroid can be calculated. This can be done, for example, by calculating the geometric median, centroid, or another representative point based on the cluster distribution.
[0028] Preferably, the center of a cluster is calculated using the average value of the candidate points within the cluster. The center point of each cluster can be considered as the center of the 3D bounding box corresponding to the traffic management object in that cluster. Therefore, each cluster can represent a different traffic management object.
[0029] In another embodiment of the method according to the first aspect, candidate points are clustered if the number of candidate points exceeds a first threshold and if their distance from each other is less than a second threshold. This groups tightly clustered points together and can help identify points in low-density areas as outliers. Clustering is performed based on the proximity of candidate points to each other. The aim here is to group candidate points that are close to each other together because they may represent the same physical traffic management object in space. To improve the robustness of the method, clusters that are too small (i.e., clusters whose number of center points is less than the second threshold) are discarded as outliers.
[0030] In another embodiment of the method according to the first aspect, calculating the size of the 3D bounding box of the traffic management object includes: calculating a plurality of cross sections by means of the following steps: for each image frame including the 2D bounding box of the traffic management object, calculating the angular projection lines from the camera position associated with the image frame toward the 2D bounding box of the traffic management object, and calculating the intersection of the lines with a plane at the center of the 3D bounding box, the plane being parallel to the vertical axis and perpendicular to the horizontal component of the line-of-sight vector, to calculate the cross section; and using the plurality of cross sections to estimate the size of the 3D bounding box. The plane at the center of the 3D bounding box (also referred to as the center plane) can be used as a reference for projecting the corners of the 2D bounding box (as seen in the image frame) into 3D space to estimate the size of the traffic management object. The center plane can be placed at the center of the 3D bounding box of the traffic management object. This plane can be parallel to the vertical axis, which in most cases is aligned with gravity and points upwards. This alignment ensures that the object can be measured in a manner consistent with how the object is typically oriented and perceived in the real world. The gaze vector can be a direction line from the camera's position to the center of the 3D bounding box of the traffic management object, as it is presented in a 2D image frame. This vector represents the direction of view of the traffic management object from the camera's position. The horizontal component of the gaze vector can refer to the projection of the gaze vector onto a horizontal plane, essentially removing any vertical height or angle from the camera to the traffic management object. This horizontal direction indicates the viewpoint direction at the ground plane. The central plane is perpendicular to the horizontal direction, which corresponds to the horizontal component of the gaze vector. The central plane can be constructed to pass through the center of the 3D bounding box, aligned with the vertical axis (e.g., ensuring it is parallel to the direction of gravity), and perpendicular to the horizontal component of the gaze vector. The intersection of the line projected angularly toward the 2D bounding box and the central plane defines a cross section. The dimensions of each cross section can preferably be defined by its width and height. Additionally or alternatively, each segment can be characterized by its position (such as its position relative to the center of the corresponding 3D bounding box) and orientation (such as the orientation of the central plane).
[0031] In another embodiment of the method according to the first aspect, the method further includes estimating the size of the 3D bounding box using the average or maximum size of a plurality of cross sections. The size of each cross section may be defined by at least one of the width or height of the cross section. The size of the 3D bounding box may be defined by at least one of the width, depth, or height of the 3D bounding box. Estimating the size of the 3D bounding box of a traffic management object based on a plurality of calculated cross sections may involve analyzing the width and / or height of one or more cross sections to determine the overall size of the corresponding 3D bounding box in 3D space. For example, the average width and / or maximum width may be used to estimate one dimension of the 3D bounding box (e.g., width), and / or the average height and / or maximum height may represent another dimension of the 3D bounding box (e.g., height). The depth of the bounding box may be estimated based on the dispersion or distribution of the dimensions of the cross sections along a reference plane.
[0032] Using the average of the cross-sectional dimensions provides a representation of the typical size of a traffic management object as observed from multiple viewpoints. Using the maximum value of the width and / or height observed across all cross-sections ensures that the estimated 3D bounding box will cover the entire traffic management object, regardless of the viewpoint. The choice between using the average or maximum size depends on the uniformity and accuracy requirements of the labeled traffic management object. For objects that maintain consistent dimensions from various angles, the average provides a more accurate representation. For irregularly shaped objects or when ensuring complete coverage is crucial, the maximum size can be used. For example, if the labeled traffic management object represents a traffic light, the width of the 3D bounding box can be set to the average width of multiple cross-sections, and the height of the 3D bounding box can be set to the average height of multiple cross-sections. In yet another example, if the labeled traffic management object represents a traffic sign, the width of the 3D bounding box can be set to the maximum width of multiple cross-sections, and the height of the 3D bounding box can be set to the maximum height of multiple cross-sections.
[0033] In another embodiment of the method according to the first aspect, the method further includes setting a predetermined value for the depth of the 3D bounding box dimensions, or the same value for both width and depth. The depth value can be set based on the spatial characteristics of the traffic management object being labeled. For example, if it is known that the labeled traffic management object has a consistent depth across different instances, or when precise depth is not important for labeling, a predetermined depth can be set for at least some of the 3D bounding boxes. This can be applied, for example, to certain types of traffic signs or obstacles with standardized dimensions. In these cases, the depth value can be selected based on domain knowledge, regulatory standards, or empirical data regarding the traffic management object of interest. The value can be implemented as a constant and applied uniformly across at least some or all of the 3D bounding boxes of the traffic management object. For example, the depth can be set to a value in the range of 5 cm to 25 cm, preferably in the range of 7 cm to 15 cm, and most preferably in a value including 10 cm, 11 cm, or 12 cm. It should be understood that these values are preferred examples, and other values can be selected based on the type characteristics of the traffic management object.
[0034] On the other hand, a 3D bounding box can be defined for traffic management objects that are expected to be approximately symmetrical in a plane perpendicular to the viewing direction, such that the width equals the depth. This can be applied when the traffic management objects of interest tend to have a uniform shape, such as traffic lights. Therefore, after determining the width of the 3D bounding box, the depth value of the 3D bounding box dimensions can be set to the same value, such as by using the maximum or average width derived from cross-sectional analysis. The choice of depth can be guided by the characteristics of the annotated traffic management objects and the requirements of the use case. For example, setting a predetermined depth may be a practical approach if the accuracy of spatial representation is less critical than processing efficiency, or if the traffic management objects are known to have standard dimensions. If the traffic management objects are more uniform in shape and size, using the same value for width and depth can provide a reasonable balance between simplicity and accuracy.
[0035] In another embodiment of the method according to the first aspect, calculating the orientation of the 3D bounding box of the traffic management object includes: identifying a reference frame in an image frame that includes the 2D bounding box of the traffic management object, wherein the reference frame has a position at a predetermined distance from the traffic management object; determining a direction of movement at the position of the reference frame; and calculating the orientation of the 3D bounding box so that the orientation is opposite to the direction of movement. Calculating the orientation of the 3D bounding box of the traffic management object involves using contextual cues from the environment, particularly the relationship between the traffic management object and available image frames, especially the relationship with the reference frame and the (vehicle's) direction of movement associated with the reference frame. The reference frame can be used to establish a basic orientation within an image frame containing the 2D bounding box of the traffic management object. Among multiple image frames capturing the traffic management object, image frames that meet the criterion of being at a predetermined distance from the object are identified and selected as reference frames. The distance can be determined using position and orientation data of the image frames, which can be derived, for example, from camera metadata, for example, which can be compared with the center of an established corresponding 3D bounding box. Other parameters used to determine distance may include object size comparisons across image frames or depth sensors to ensure the appropriate distance of the selected frame for accurate orientation analysis. A reference frame may indicate the direction of movement based on camera or vehicle metadata. If this information is unavailable, the direction of movement can be determined by analyzing a sequence of corresponding image frames, including the reference frame. Additionally or alternatively, environmental cues can be analyzed to determine the direction of movement relative to the traffic management object. This may involve observing traffic flow, the orientation of lane markings, or the direction pedestrians are facing. Additionally or alternatively, visual indicators within the reference frame (such as vehicle orientation, arrows on the road, or the configuration of surrounding infrastructure) can be used to estimate the direction of movement at the location of the reference frame or to confirm an already determined direction of movement. When the direction of movement is determined from the reference frame, the orientation of the traffic management object's 3D bounding box can be set opposite to that direction. This alignment ensures that the front of the 3D bounding box faces the opposite direction of movement, reflecting how the traffic management object typically positions itself to face oncoming vehicles. By treating the reference frame as one of the image frames at a specific distance from the traffic management object, this approach leverages direct observation and analysis of the traffic management object's environment to infer its correct orientation, thereby improving the accuracy of the resulting 3D bounding box.
[0036] In another embodiment of the method according to the first aspect, calculating the orientation of the 3D bounding box of the traffic management object includes: identifying a reference frame of the 2D bounding box of the traffic management object with the maximum width, and setting the orientation according to the reference frame to the opposite direction of the horizontal component of the line-of-sight vector of the traffic management object. Alternatively, or to further refine or improve the determination of the reference frame, the width of the traffic management object in each image frame including the corresponding 2D bounding box may be analyzed. Preferably, for generally flat and / or rectangular traffic management objects, the reference frame is identified based on the maximum width of the depicted traffic management object (or its corresponding 2D bounding box), making the image frame that makes the traffic management object appear the widest likely to be the image frame most directly facing the camera. This image frame may be selected as the reference frame because it may capture the traffic management object completely, thus providing an indication of its expected viewing direction. This applies, for example, to traffic signs and other types of traffic management objects with flat sides.
[0037] Among the collected image frames containing the traffic management object, the frame with the largest width of the 2D bounding box around the traffic management object is selected as the reference frame. The reference frame is considered to best represent the orientation of the traffic management object because the maximum width indicates that the traffic management object is being viewed from a frontal perspective, providing an indication of its facing direction. For the traffic management object in the reference frame, a gaze vector can be calculated. This vector represents the direction from the camera's viewpoint to the center of the traffic management object. In practice, this can be considered as a straight line from the camera to the center of the 2D bounding box in the selected reference frame or the center of the 3D bounding box of the traffic management object. Subsequently, the horizontal component of the vector can be extracted. The horizontal component is the projection of the gaze vector onto a horizontal plane, essentially ignoring any vertical height or tilt. The horizontal component indicates the main direction the camera faces relative to the object. Given the horizontal component of the gaze vector, the orientation of the 3D bounding box can be set directly opposite to this component. Since the traffic management object is designed to face oncoming vehicles or viewers, setting an orientation opposite to the gaze vector ensures that the 3D bounding box accurately reflects the real-world orientation of the traffic management object.
[0038] In another embodiment of the method according to the first aspect, calculating the orientation of the 3D bounding box of a traffic management object includes: using an image-based regression model to determine the orientation of the 3D bounding box. The image-based regression model can be learned directly from image frame data. The model can employ regression to predict continuous variables, such as orientation parameters or angles that define the orientation of the traffic management object in 3D space. Initially, a dataset of image frames characterizing traffic management objects and their known orientations can be compiled. Orientation can be represented in various formats, such as Euler angles (yaw, pitch, roll) or quaternions. The model can learn to identify features within the image that are related to the object's orientation. This may involve the shape of the traffic management object, its appearance at different angles, contextual cues from the surrounding environment, and how these aspects change relative to the camera's viewpoint. Once trained, the regression model can predict the orientation of the traffic management object in the image frame. Using an image-based regression model to calculate the orientation of the 3D bounding box of a traffic management object provides a powerful, data-driven approach that leverages the capabilities of machine learning for accurate and efficient orientation estimation.
[0039] In another embodiment of the method according to the first aspect, the method further includes training a first model to classify the states of traffic management objects in multiple 2D image crop blocks. The training dataset for the first model may include any labeled dataset, such as any proprietary labeled dataset or a publicly available labeled dataset, such as the OpenLane-V2 dataset. The training dataset may include images with labeled states for various lighting conditions, angles, distances, and backgrounds. Diversity in the training dataset is crucial for training a robust model that performs well under real-world conditions.
[0040] The state can depend on the type of traffic-managed object. The state can be static or dynamic. The state can include occlusion level. This can be used for static traffic signs. The state can include color and occlusion level. This can be used for example, traffic lights or dynamic message signs. Color states can be classified into multiple categories, such as red, green, red-yellow, yellow, and one or more unknown, which can cover the basic signals needed to navigate intersections or control traffic, where "unknown" provides a fallback for ambiguous situations. Occlusion can be viewed as a binary classification task. A first model predicts the probability that an image cropped patch is not a traffic-managed object, corresponding to an occluded object. This can be used to identify and ignore occluded views to ensure the system relies on clear, visible signals.
[0041] In one example, the training dataset used to train the classifier model (including the first model) may include the types of traffic management objects, such as the type of traffic sign, color status, and mask type of traffic lights. The training dataset can be manually labeled. It may include proprietary (internal) data as well as publicly available data. Occlusion can be trained in a fully automated manner by selecting 2D bounding boxes identified with high confidence by an image detector neural network and labeling the corresponding image cropped patches in the training dataset as unoccluded. Subsequently, the 2D bounding boxes can be placed at random locations within the image frame, and the corresponding image cropped patches can be labeled as occluded in the training dataset. This is based on observations that the random locations are most likely not to contain any traffic management objects. The classifier model can be trained as an occlusion prediction network on this automatically labeled data in the training dataset.
[0042] In another embodiment of the method according to the first aspect, the first model is a convolutional neural network. Preferably, the first model can be a two-headed convolutional neural network. In this case, the neural network can perform two tasks simultaneously, such as color state classification and occlusion estimation. The dual-task approach allows for simultaneous processing, thereby optimizing the efficiency and effectiveness of the neural network.
[0043] In another embodiment of the method according to the first aspect, the method further includes: inferring the state of a traffic management object using a first model based on one or more image frames associated with the traffic management object, including: for each image frame in the one or more image frames, performing majority voting on the inferred state for a plurality of previous and subsequent image frames to determine the state of the traffic management object. This combines model inference with temporal analysis to improve accuracy and reliability. Utilizing majority voting across a sequence of image frames takes into account temporal consistency and minimizes the impact of potential misclassification or transient occlusion. A first model, trained to classify the state of traffic management objects, is applied to each image frame within a given sequence. This sequence includes a target frame for which the state is determined, as well as a specified number of previous and subsequent frames. For each frame, the model predicts the state of the traffic management object. Assuming the model is designed to handle the state of traffic management objects, each inference provides an estimate of the current state of the traffic management object, such as color and / or visibility. A time window may be defined around the target frame, including a predetermined number of frames before and after the target frame. The size of this window may be selected based on the expected duration of the state (such as traffic light cycles) or other relevant time factors (including expected occlusion). Within this window, inferred states from each image frame can be aggregated, and the most frequently occurring state is selected as the final determination of the object's state for the target frame. Further weighted majority voting can be achieved by associating the inferred state of the target frame with a larger weighting factor and the remaining states in the window with a lower weighting factor. If occlusion detection is part of the output of the first model, frames where the object is determined to be occluded can be excluded from the voting process or processed accordingly to ensure they do not incorrectly influence the final state determination.
[0044] In another embodiment of the method according to the first aspect, the method further includes: obtaining a plurality of 2D image cropped blocks for each traffic management object; and training a second model to classify the mask type of the traffic management object using the plurality of 2D image cropped blocks. The second model can be tailored to subtle variations in the appearance of the traffic management object, which can be represented by a corresponding mask type. Mask type classification essentially refers to recognizing the presence of specific functional symbols or designs displayed by the traffic management object, such as the direction indicated by arrows on dynamic message signs or traffic lights, or the presence of pedestrian crossing symbols or construction site symbols. Each image cropped block can be labeled with the specific mask type it represents. This can include various symbols on traffic lights, dynamic message signs or traffic signs such as directional arrows, pedestrian symbols, construction site symbols, etc., or specific shapes or designs on traffic signs, etc. The second model can be a neural network architecture suitable for image classification tasks. Training the second model on a labeled dataset of 2D image cropped blocks can involve feeding the image cropped blocks into the model, allowing it to learn visual features associated with each mask type through an optimization process, typically employing a loss function that measures the difference between predicted and actual labels. Subsequently, a separate subset of the data (not seen by the second model during training) can be used for validation and testing. The second model can be evaluated using performance metrics such as accuracy, precision, recall, and F1 score. The training dataset for the second model can include a labeled dataset with the obtained 2D image cropped patches, which can be manually labeled to train the classifier model, including the second model. However, it should be understood that the training dataset for the second model can also include any proprietary labeled dataset or publicly available labeled dataset.
[0045] In another embodiment of the method according to the first aspect, the traffic management object is a dynamic or static traffic management object. Preferably, the traffic management object can be one or more of traffic signs, traffic lights, and dynamic message signs. Each type of traffic management object serves a different purpose and presents unique challenges for detection, classification, and interpretation in automated systems, such as those used in autonomous driving. Static traffic management objects can be permanent or semi-permanent fixed installations in the traffic environment. They maintain a constant position and function and do not dynamically change their state. Examples include traffic signs or barriers and bollards. Detection and classification must accurately identify traffic management objects regardless of changes in lighting, weather conditions, and occlusion. Interpretation requires understanding the specific regulations or guidance conveyed by each object (e.g., speed limits, directional information, etc.). Dynamic traffic management objects can be those objects capable of changing their state or position, providing real-time information or control based on current traffic conditions. Examples include dynamic message signs, also known as variable message signs (VMS), traffic lights, automated barriers, or gates, etc. State detection of dynamic traffic management objects requires not only identifying the traffic management object but also accurately determining its current state or message. Time analysis may be necessary for predicting changes in the state of traffic management objects or understanding patterns. Therefore, considering environmental changes, the training data must cover a wide range of appearances for each object type. The focus can be on identifying objects and understanding their fixed meanings. For dynamic traffic management objects, the model should preferably be trained based on data capturing the different states or locations that the traffic management objects may exhibit.
[0046] The training of the first and / or second models can improve the accuracy and utility of traffic object detection and classification through a series of machine learning models and filtering techniques. High-precision 3D bounding boxes of the traffic objects are generated. Triangulation and computer vision methods achieve an accuracy within 0.2 to 0.3 meters of ground reality. From the 3D bounding boxes, 2D image clippings of the traffic objects are generated by projecting the 3D bounding boxes onto the camera image plane (image frame) from a distance preferably starting at 200 meters. This distance can be adjusted according to parameters and conditions. The method captures the traffic objects as they would appear in the camera feed.
[0047] Depending on the type of traffic management object, develop a dedicated model (preferably a CNN) to handle one or two tasks. This may include: classifying the color status of dynamic traffic management objects into multiple categories, such as classifying traffic lights into five categories, including red, green, red-yellow, yellow, and unknown; and / or estimating occlusion as a binary classification, including visible or occluded. The model may use a combination of proprietary annotations and data from publicly available datasets, such as the OpenLane-V2 dataset.
[0048] A reasonable number of image crop blocks, such as at least approximately 100,000; 150,000; 180,000; or 190,000 image crop blocks, including automatically generated occluded versions, can be used to train and evaluate the model to achieve high accuracy in color state classification and / or occlusion detection. To ensure consistent state prediction over time and mitigate misclassification, smoothing techniques that consider frames before and after the current frame can be applied. A majority vote determines the final color state, taking into account the temporal changes in light state within the dynamic traffic management object.
[0049] For individual traffic management objects depicting symbols or having different designs, accurate detection and classification of the traffic management object can serve as the basis for further classification of the specific mask type for each traffic management object. The mask type can represent a variety of symbols or indications that the traffic management object may display, such as directional arrows, pedestrian symbols, or construction site symbols. For example, another model can be trained on proprietary or publicly available data to identify the mask type of each traffic management object from corresponding image cropped patches, thereby covering a comprehensive range of possible traffic indications. Preferably, the model can achieve at least 99% accuracy on a validation dataset.
[0050] Preferably, to reduce false alarms in traffic management object detection, the average intersection-union ratio (IOU) between the 2D projection of the 3D bounding box and the initial 2D detection (based on the initial 2D bounding box) can be calculated. This helps in identification based on spatial overlap and retaining only the most accurate detections. Detection can be further refined by grouping candidates falling on the same line of sight and selecting the candidate with the highest IOU. Predictions with an average occlusion rate higher than, for example, 85%, especially those less than 20 meters from the observer, can be removed to ensure the visibility and relevance of detected traffic management objects.
[0051] According to the multi-stage process described in the embodiments of the first aspect, from high-precision 3D mapping and 2D image cropping block generation to a sophisticated machine learning model for state, occlusion, and / or mask type classification, a robust system capable of accurately detecting and classifying traffic management objects is formed. Preferably, the combination of temporal smoothing and false alarm reduction techniques further enhances the reliability and applicability of the system in real-world scenarios.
[0052] Preferably, the method further includes providing 3D bounding boxes in a map. The 3D bounding boxes provide a volumetric spatial representation of the traffic management object, detailing its location, orientation, and size in 3D space. The generated 3D bounding boxes can be provided in an HD map and used with any dataset including image data with location / orientation information to generate 2D clippings of the corresponding traffic management object. This can be used for classification or for training purposes.
[0053] In another embodiment of the method according to the first aspect, the method further includes: generating a training dataset comprising 3D bounding boxes of traffic management objects. Preferably, the training dataset may include one or more of the state, occlusion, and mask type for each traffic management object. Generating a training dataset comprising 3D bounding boxes of traffic management objects and comprehensive details such as state, occlusion, and mask type for each object provides a solid foundation for training complex machine learning models. These models can then be used in a variety of applications, including autonomous driving, traffic management systems, and urban planning. The dataset can be used to train or validate machine learning models, particularly deep learning models such as convolutional neural networks (CNNs), to identify, classify, and understand the characteristics and states of traffic management objects from various sensor inputs.
[0054] According to the second aspect, a method for generating a training dataset is defined, wherein the training dataset is generated based on 3D bounding boxes of traffic management objects, and wherein the training dataset is obtained by performing a method according to any embodiment of the first aspect. Specifically, the training dataset can be obtained by automatically annotating traffic management objects in three-dimensional space, comprising: obtaining a plurality of image frames, at least some of which include two-dimensional 2D bounding boxes of traffic management objects, and each image frame being associated with position and orientation data; locating the center of a 3D bounding box of a traffic management object using the plurality of image frames including the 2D bounding boxes and the corresponding position and orientation data; for each located center of a 3D bounding box of a traffic management object, calculating the size and orientation of the 3D bounding box based on the image frame including the 2D bounding box of the traffic management object; for each 3D bounding box of a traffic management object, projecting the 3D bounding box onto the image frame associated with the traffic management object; generating a plurality of 2D image cropping blocks of traffic management objects based on the projected 3D bounding boxes; and training at least one model to classify traffic management objects in the plurality of 2D image cropping blocks. It should be understood that the preferred embodiments of the second aspect may include features of the embodiments or implementations of the first aspect in any combination.
[0055] A third aspect of the invention relates to a computer-readable storage medium storing instructions thereon that, when executed by a computing device, configure the computing device to perform the method according to one of the first or second aspects. Specifically, the computing device may be configured to automatically annotate traffic management objects in three-dimensional space, comprising: obtaining a plurality of image frames, at least some of which include two-dimensional 2D bounding boxes of the traffic management objects, and each image frame being associated with position and orientation data; locating the center of a 3D bounding box of the traffic management object using the plurality of image frames including the 2D bounding boxes and the corresponding position and orientation data; for each located center of a 3D bounding box of a traffic management object, calculating the size and orientation of the 3D bounding box based on the image frame including the 2D bounding box of the traffic management object; for each 3D bounding box of a traffic management object, projecting the 3D bounding box onto the image frame associated with the traffic management object; generating a plurality of 2D image cropped blocks of the traffic management object based on the projected 3D bounding boxes; and training at least one model to classify the traffic management objects in the plurality of 2D image cropped blocks. It should be understood that the preferred embodiments of the third aspect may include features of the embodiments or implementations of the first or second aspects in any combination.
[0056] A fourth aspect of the invention defines an apparatus comprising: a memory configured to store image data; and a processor configured to access the memory and perform a method according to one of the first or second aspects. Specifically, the apparatus may preferably be a computing device configured to automatically annotate traffic management objects in three-dimensional space, comprising: acquiring a plurality of image frames, at least some of which include two-dimensional 2D bounding boxes of the traffic management objects, and each image frame being associated with position and orientation data; locating the center of a 3D bounding box of the traffic management object using the plurality of image frames including the 2D bounding boxes and the corresponding position and orientation data; for each located center of a 3D bounding box of a traffic management object, calculating the size and orientation of the 3D bounding box based on the image frames including the 2D bounding boxes of the traffic management object; for each 3D bounding box of a traffic management object, projecting the 3D bounding box onto an image frame associated with the traffic management object; generating a plurality of 2D image cropped blocks of the traffic management object based on the projected 3D bounding boxes; and training at least one model to classify the traffic management objects in the plurality of 2D image cropped blocks.
[0057] In a preferred embodiment of the device according to the fourth aspect, the memory may be configured to store one or more of image frames, 2D bounding boxes, determined 3D bounding boxes, and 2D image cropping blocks in any combination.
[0058] It should be understood that other embodiments of the fourth aspect may include features of the embodiments or implementations of the first or second aspect in any combination.
[0059] According to another aspect of the invention, at least one computer-readable medium may store a training dataset comprising 3D bounding boxes of traffic management objects. Preferably, the training dataset may include one or more of the following: state, occlusion, and mask type for each traffic management object. Preferably, the training dataset may be obtained by performing a method described in any of the embodiments of the first or second aspect.
[0060] It should be understood that embodiments related to a computing device or system may include logic, processors, or functional devices that may be configured in any combination according to embodiments or implementations of the first or second aspect of this disclosure, or any implementation of the first or second aspect. Preferably, the computing device and system may be configured in any combination according to embodiments of the method of the first or second aspect. Similarly, the method according to embodiments of the first or second aspect may include processing steps that, in any combination, reflect the structural features or functionality of components of embodiments of the computing device or the system of the fourth aspect. Attached Figure Description
[0061] To more clearly illustrate the technical features of the embodiments of the present invention, the accompanying drawings provided for describing the embodiments are briefly described below. The drawings in the following description are merely some embodiments of the present invention, and modifications to these embodiments are possible without departing from the scope of the invention as defined in the claims.
[0062] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Figure 2 This is a flowchart of a method according to another embodiment of the present invention. Figure 3 This is a schematic diagram of the method steps for locating the center of a 3D bounding box according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the method steps for calculating the size of a 3D bounding box according to an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the method steps for calculating the orientation of a 3D bounding box according to an embodiment of the present invention; and Figure 6 This is a schematic diagram of the classification process according to an embodiment of the present invention. Detailed Implementation
[0063] The following description refers to the accompanying drawings, which illustrate various embodiments by way of illustration. Furthermore, various embodiments will be described below with reference to several examples. It should be understood that embodiments may include changes in design and structure without departing from the scope of the claimed subject matter.
[0064] The techniques described herein can be implemented in a variety of computing systems, examples of which are described in more detail below. Such systems typically involve the use of appropriately configured computing devices that implement multiple modules, each providing one or more operations necessary to perform such techniques. Each module may be implemented in its own way; none need to be implemented in the same way. As used herein, a module can be a structural component of a system that functions, and can be part of or an entire software element (e.g., a process, discrete process, or the functionality of any other suitable embodiment). Modules may include computer-executable instructions and may be encoded on a computer storage medium. Modules may execute in parallel or serially as needed and may exchange information with each other using shared memory on the computer on which they are executing, using messaging protocols, or in any other suitable manner. Exemplary modules performing one or more tasks are described below, but it should be understood that the division of modules and tasks described may be used only to illustrate the types of modules that can implement the exemplary techniques described herein, and the invention is not limited to implementation with any particular number, division, or type of modules. In some implementations, all functionality may be implemented in a single module. Furthermore, for clarity, modules are discussed below as all executing on a single computing device; however, it should be understood that in some implementations, modules may be implemented on separate computing devices suitable for communicating with each other.
[0065] Various abbreviations may be used in this disclosure. For example, CNN may be used to refer to a convolutional neural network. LiDAR may be used to refer to light detection and ranging. HD may be used to refer to high definition, which can be used in conjunction with maps. Other abbreviations well known to those skilled in the art may be used in this disclosure.
[0066] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.
[0067] Method 100 may be a computer-implemented method for annotating traffic management objects in three-dimensional (3D) space.
[0068] Method 100 can begin in project 102 and proceed to project 104, wherein multiple image frames can be acquired. At least some of the image frames may include two-dimensional (2D) bounding boxes of the traffic management object. Each image frame may be associated with location and orientation data. Image frames can be acquired by reading image data from memory. However, image frames can also be acquired by receiving a data stream, which may be provided by, for example, an image sensor. The 2D bounding boxes can be acquired using an image detector neural network. Confidence levels for the detected 2D bounding boxes can be established.
[0069] Method 100 can continue with item 106 by using multiple image frames including 2D bounding boxes and corresponding position and orientation data to locate the center of the 3D bounding box of the traffic management object. For example, in item 106, only image frames containing 2D bounding boxes with a confidence level above a threshold can be used to locate the center of the 3D bounding box of the traffic management object. For each image frame including a corresponding 2D bounding box, a centerline can be generated by projecting a line from the camera position associated with the image frame onto the center of the 2D bounding box in the image frame. Subsequently, a set of centerlines that are less than a predetermined distance apart can be identified. The set of centerlines can be used to generate multiple candidate points. A candidate point can be the point closest to a corresponding pair of centerlines. Multiple candidate points can be clustered. Clusters and corresponding candidate points can be used to calculate the center of the 3D bounding box of the traffic management object. Preferably, candidate points form a cluster if the number of candidate points exceeds a first threshold and if they are less than a second threshold apart. Other suitable clustering techniques can be used.
[0070] In Project 108, Method 100 can calculate the size and orientation of a 3D bounding box based on a set of image frames including 2D bounding boxes of the corresponding traffic management objects. This can be done at the center of each location of the 3D bounding box. Multiple cross-sections can be used to calculate the size of the 3D bounding box. A cross-section is constructed by projecting angular lines from the camera position associated with the image frame onto the 2D bounding box of the traffic management object in the image frame, and calculating the intersection of these lines with a plane at the center of the 3D bounding box. This is done for each image frame including the 2D bounding box of the traffic management object. The plane at the center of the 3D bounding box can be parallel to the vertical axis and perpendicular to the horizontal component of the line-of-sight vector in the image frame. The cross-sections are then used to estimate the size of the corresponding 3D bounding box.
[0071] In Project 110, for each 3D bounding box of a traffic management object, the 3D bounding box can be projected onto the image frame associated with that traffic management object. In Project 112, the projected 3D bounding boxes are used to generate multiple 2D image clipping blocks of the traffic management object.
[0072] Method 100 also includes, in item 114, training at least one model to classify traffic management objects in multiple 2D image crop blocks.
[0073] Method 100 for labeling traffic management objects in 3D space offers several technical advantages for enhancing accuracy, efficiency, and scalability in traffic management and autonomous driving systems. The method utilizes 3D bounding boxes derived from 2D detection and associated location and orientation data, providing a more accurate representation of objects in space, crucial for precise navigation and interaction in autonomous driving. Leveraging image detector neural networks to identify 2D bounding boxes and establish confidence levels simplifies the detection process, reduces manual labeling work, and improves scalability. Focusing on data with high-confidence detection prioritizes computational resources for the most reliable information, thereby improving the overall efficiency and accuracy of the labeling process. Advanced techniques for locating 3D bounding boxes (including centerline generation and candidate point clustering) and calculating the size and orientation of 3D bounding boxes through cross-sectional analysis provide a deep understanding of object size, location, and spatial orientation within the 3D environment. Generating 2D image cropped patches from projected 3D bounding boxes focuses analysis on relevant image portions, enhancing the effectiveness of subsequent machine learning model training. The structured process generates rich datasets that facilitate efficient training of machine learning models for accurate classification of traffic management objects under various conditions.
[0074] Figure 2 This is a flowchart of a method according to another embodiment of the present invention. Method 200 may include... Figure 1 The processing steps of method 100.
[0075] Method 200 can be applied to traffic lights. Traffic lights represent a dynamic type of traffic management object with different states. However, it should be understood that Method 200 can be applied to any other type of traffic management object.
[0076] Traffic lights and their 3D bounding boxes can be located in a geocentric Earth-fixed coordinate system (ECEF) to generate a 3D map of the traffic lights. Method 200 may begin by detecting traffic lights in a 2D image frame in item 202. In item 204, a 3D line passing through the center of the 2D bounding box of the traffic light in the 2D image frame can be calculated. In item 206, candidate points for the center can be calculated based on the 3D line and intersections to estimate the position, size, and orientation of the corresponding 3D bounding box. Specifically, candidate points defined by narrow or intersecting 3D lines can be established and clustered. In item 208, the average value of the candidate points in the cluster can be set as the center of the corresponding 3D bounding box. Furthermore, in item 210, the size and orientation of the 3D bounding box can be estimated using cross-sections defined by the 3D lines and other line projections and intersections based on the 2D bounding box.
[0077] In Project 212, the 3D bounding boxes of traffic lights are transformed into the instantaneous coordinate system (volume coordinate system) of each image frame by projecting the 3D bounding boxes onto the image plane. In Project 214, the image cropping patches generated by the projection in Project 212 can be classified with respect to color state and occlusion. The output of Project 216 can be a dataset of 3D-annotated traffic lights for each image frame, containing color state, mask, and occlusion information.
[0078] By locating traffic lights in a geocentric Earth-fixed (ECEF) coordinate system and estimating their 3D bounding boxes, Method 200 provides high accuracy in spatial localization, crucial for applications requiring accurate real-world mapping. To address dynamic traffic management objects such as traffic lights, Method 200 appropriately handles traffic management objects with variable states, providing a robust solution for analyzing objects that change over time. Beyond simple detection, Method 200 provides full-spectrum information on detected traffic management objects and their attributes, including location, size, orientation, color state, and occlusion state, enabling detailed analysis and application across various domains. The resulting dataset contains 3D annotations and extensive attribute information for each traffic management object, serving as a vital resource for training machine learning models, advancing traffic research, and improving autonomous navigation systems. In summary, Method 200 introduces sophisticated methods for 3D annotation of traffic management objects, combining 3D spatial analysis with detailed attribute classification to generate richly annotated datasets, contributing to advanced traffic management and autonomous driving technologies.
[0079] Figure 3 This is a schematic diagram of a method for locating the center of a 3D bounding box according to an embodiment of the present invention. Method 300 can be applied to... Figure 1 Method 100 Project 106 or Figure 2 Method 200 in Project 208.
[0080] Based on multiple image frames 302a, 302b, which may include images of traffic management objects 304 such as traffic lights, the traffic management objects 304 can be detected in image frames 302 as 2D bounding boxes 306a, 306b of the traffic management objects 304. This can be accomplished, for example, by using a precise image detector neural network, such as DINO (https: / / github.com / IDEA-Research / DINO). Only those predicted 2D bounding boxes with high confidence are used, thus excluding false positive 2D detections. In the case of method 100 or method 200, this step does not lead to a reduction in the recall of 3D detection because most image frames 302 will include traffic management objects 304 that are close enough and large enough to produce highly confident 2D predictions.
[0081] Using the position and orientation data (such as GPS data) of the observer 308 along the trajectory and using lines 310a, 310b pointing to the center of the 2D bounding box in the corresponding image frames 302a and 302b respectively, the center of the 3D bounding box 312 in the global coordinate system can be determined using triangulation techniques.
[0082] Preferably, for all lines 310, those lines whose distance to each other is less than a predetermined threshold (e.g., 5 cm) can be considered. Other thresholds can be used, such as less than 15 cm, less than 10 cm, preferably 9 cm, 8 cm, 7 cm, 6 cm, or 5 cm. For each line pair, the coordinates of candidate points of the line closest to that pair can be calculated. For example, the DBSCAN clustering method can be used to cluster multiple candidate points. If there are more than a predetermined number of 3D points (e.g., 5) and they are closer to each other than another threshold (e.g., 10 cm), the 3D points can form a cluster. Other numbers can be used, such as at least 5, at least 10, at least 15, preferably 5, 6, 7, 8, 9, or 10. Furthermore, other thresholds can be used, such as less than 5 cm, less than 10 cm, less than 20 cm, or less than 25 cm. After identifying the clusters, the average of the clusters can be used as the final prediction of the center c of the traffic management object 312. The center c can be represented by ECEF coordinates.
[0083] The distance filtering and clustering steps provide method 300 with robustness against different random errors associated with GPS location, orientation, or camera calibration.
[0084] Figure 4 This is a schematic diagram illustrating the steps of a method for calculating the dimensions of a 3D bounding box according to an embodiment of the present invention. Method 400 can be applied to... Figure 1 Method 100 Project 108 or Figure 2 Method 200 in Project 210. Can be based on Figure 3 The result of method 300 is used to execute method 400. Therefore, with Figure 3 The same reference numerals used in the figures are also Figure 4 Used in.
[0085] Method 400 can use a map centered on the traffic light frame, such as using Figure 3 The center is generated by method 300. In order to determine the size of the corresponding 3D bounding box 312, the intersection of the line pointing to the corner of the 2D bounding box 306 in each frame 302 with planes 402a and 402b is determined. Planes 402a and 402b are vertically aligned, contain the center of the 3D bounding box 312, and are perpendicular to the line of sight in the xy plane.
[0086] Using several image frames, multiple cross sections 404a, 404b for each 3D bounding box from several viewpoints can be estimated. In one example, such as for a traffic light, the average width and height of cross section 404 can be used to estimate the width, depth, and height of the 3D bounding box 312. Setting the width and depth to the same value can be a good estimate for more frequently occurring vertically aligned traffic lights. For other types of traffic management objects, other heuristics can be used. For example, for traffic signs, a predetermined depth can be set.
[0087] Figure 5 This is a schematic diagram illustrating the steps of a method for calculating the orientation of a 3D bounding box according to an embodiment of the present invention. Method 500 can be applied to... Figure 1 Method 100 Project 108 or Figure 2 Method 200 in Project 210. Can be based on Figure 4 The result of method 400 is used to execute method 500. Therefore, in Figure 5 Used with Figure 3 and Figure 4 The same reference numerals are used in the figures.
[0088] Figure 4 Method 400 can generate multiple 3D bounding boxes with centers and dimensions. Various methods can be used to determine the orientation of the 3D bounding boxes 312. This may depend on the type of traffic management object 304.
[0089] A preferred heuristic for determining the orientation of a traffic management object 304 represented by a 3D bounding box 312 may consider an image frame just in front of the traffic management object, such as at a distance of 10 meters, and assume that the 3D bounding box orientation is opposite to the direction of movement 502. Other distances, such as 5m, 10m, 20m, 30m, 40m, or 50m, may be used. This heuristic may provide a suitable orientation for multiple traffic management objects, but its prediction may be incorrect for traffic management objects located in specific locations, such as traffic lights associated with cross traffic.
[0090] Other methods (not shown) may include orientation prediction using an image-based regression model. Furthermore, orientation prediction may be based on the type of traffic management object. For example, the orientation may be set to be opposite to the viewing direction of the image frame with the maximum width of the traffic sign.
[0091] Figure 6 This is a schematic diagram of a classification process according to an embodiment of the present invention. The classification process 600 can be used for... Figure 1 Method 100 or Figure 2 Of the 200 methods, one can be based on... Figure 5The result of method 500 is used to execute process 600. Therefore, in Figure 6 Used with Figure 3 , Figure 4 and Figure 5 The same reference numerals are used in the figures.
[0092] Process 600 can operate with high accuracy on an HD map of the 3D bounding box 312 of the traffic management object 304, preferably within 0.2-0.3 meters above the ground. The HD map can be used in other operational design areas (such as rain, night, snow, etc.).
[0093] From the HD map, a 2D image clipping block 602 of the traffic management object is generated by projecting a 3D bounding box 312 onto the camera image plane of the corresponding image frame 302, preferably starting from image frame 302 at a distance of 200 meters from the location of the traffic management object 304. It should be understood that other distances can be used, such as less than 100m, less than 300m, preferably 150m, 200m, or 250m.
[0094] For classifying the states of traffic management objects 304, such as color state classification or occlusion estimation, neural networks 604, such as dual-head convolutional neural networks, can be used. Neural network 604 can be trained on proprietary labeled composite datasets as well as publicly available data (such as the OpenLane-V2 dataset (https: / / github.com / OpenDriveLab / OpenLane-V2 / tree / master / data)), where colors can be classified into five categories: red, green, red-yellow, yellow, and unknown.
[0095] Occlusion calculation can be viewed as a binary classification task, where the output of neural network 604 can be a 1-probability that an image cropped patch is a traffic management object.
[0096] Occluded image crop patches can be automatically generated by moving the coordinates of image crop patches from the combined dataset to random positions within the original image frames. These image crop patches can then be labeled as occluded and of unknown color state. In one example, Neural Network 604 was trained and evaluated on approximately 188,000 (~188,000) image crop patches, half of which were occluded. The model achieved performance of 97% and 96% on the unseen test set, respectively, related to color state and occlusion.
[0097] To provide stable and temporally consistent state predictions during inference, additional smoothing can be applied, where multiple image frames 302 preceding and following the current image frame can be considered. The color state of the image frame can be set based on majority voting. In this way, the past and future can be considered to avoid temporal inconsistencies in the prediction of the state of traffic management objects due to misclassification by the neural network.
[0098] The results of process 600 can be used to obtain traffic management objects and their states in 3D, including the color and occlusion of each timestamp or image frame. For specific types of traffic management objects that can be depicted dynamically as individual signs or designs, such as traffic lights, this information can be used to collect image cropped patches of the traffic management objects, where the mask type can also be classified. For each traffic management object, multiple image cropped patches (e.g., approximately 10, 15, or 20) can be transmitted to an additional classifier network that provides the mask type of the traffic management object.
[0099] For traffic lights, possible mask types may include one or more of the following: forward arrow, left / right arrow, left-front / right-front arrow, slight left / slight right arrow, slight left hard left / slight right hard right arrow, left and right arrow, left and right forward arrow, U-turn, pedestrian, hand, full, bus, tram light, and unknown.
[0100] Such an additional network can be trained on proprietary data and achieve 99% accuracy on unseen validation datasets.
[0101] Despite the application of the above regarding Figures 1 to 6 Despite the aforementioned filtering methods, false positives in traffic management object detection may still occur. This can be further avoided by determining the average IOU between the 2D projection of the 3D bounding box 312 and the initial 2D bounding box 306, then finding groups of candidates falling on the same line of sight, and retaining the candidate with the highest IOU in each group. Furthermore, considering the distance to the observer is less than 20 meters, predictions with an average occlusion higher than 85% can be removed.
[0102] The embodiments of this disclosure do not require point cloud data for 3D localization. Instead, the embodiments of this disclosure rely on 2D image detection and location information. The embodiments of this disclosure offer several advantages. The method is faster than point cloud-based methods. The method does not suffer from calibration errors associated with LiDAR data. Furthermore, the method can detect traffic management objects where LiDAR will not detect any reflections (traffic lights are typically above the LiDAR sensing area).
[0103] In a preferred embodiment, this disclosure may cover one or more processing devices having multiple means configured to perform the functions of embodiments of the invention. Specifically, at least one device may be configured to annotate (preferably automatically annotate) traffic management objects in three-dimensional space and may include means for: obtaining a plurality of image frames, at least some of which include two-dimensional 2D bounding boxes of the traffic management objects, and each image frame being associated with position and orientation data; locating the center of a 3D bounding box of the traffic management object using the plurality of image frames including the 2D bounding boxes and the corresponding position and orientation data; for each located center of a 3D bounding box of a traffic management object, calculating the size and orientation of the 3D bounding box based on the image frame including the 2D bounding box of the traffic management object; for each 3D bounding box of a traffic management object, projecting the 3D bounding box onto the image frame associated with the traffic management object; generating a plurality of 2D image cropped blocks of the traffic management object based on the projected 3D bounding boxes; and training at least one model to classify the traffic management objects in the plurality of 2D image cropped blocks.
[0104] Preferred embodiments of the one or more processing devices may include other means for implementing, in any combination, the aforementioned... Figures 1 to 6 And details of the aspects disclosed in the corresponding description, its implementation methods and other embodiments.
[0105] It should be understood that Figures 1 to 6 The implementation details provided represent preferred examples. Other implementations using different components, modules, blocks, units, circuits, connections, and links may be employed, and the invention is not limited to any particular implementation in silicon.
[0106] While some embodiments have been described in detail, it should be understood that various aspects of this disclosure can take many forms. Specifically, the claimed subject matter may be practiced or implemented differently from the described examples, and the described features and characteristics may be practiced or implemented in any combination. The embodiments shown herein are intended to illustrate, and not limit, the invention as defined by the claims.
Claims
1. A computer-implemented method for annotating traffic management objects in three-dimensional space, comprising: Multiple image frames are obtained, at least some of which include a two-dimensional 2D bounding box of the traffic management object, and each image frame is associated with location and orientation data; Using the plurality of image frames including the 2D bounding box and the corresponding position and orientation data, locate the center of the 3D bounding box of the traffic management object; For the center of the 3D bounding box of each traffic management object located, the size and orientation of the 3D bounding box are calculated based on the image frame including the 2D bounding box of the traffic management object. For each traffic management object's 3D bounding box, project the 3D bounding box onto the image frame associated with that traffic management object; Based on the projection of the 3D bounding box, multiple 2D image clipping blocks of the traffic management object are generated; as well as Train at least one model to classify traffic management objects in the plurality of 2D image crop blocks.
2. The method according to claim 1, wherein, The 2D bounding box is obtained using an image detector neural network. The method further includes: detecting the confidence level of the detected 2D bounding box, and using image frames with 2D bounding boxes having a confidence level higher than a threshold to locate the center of the 3D bounding box of the traffic management object.
3. The method according to claim 2, wherein, The center of the 3D bounding box of the traffic management object is defined as follows: for each image frame including the 2D bounding box of the traffic management object, a center line is projected from the camera position associated with the image frame to the center of the 2D bounding box in the image frame to generate multiple center lines for the multiple image frames.
4. The method according to claim 3, wherein, The center of the 3D bounding box of the traffic management object is further defined by: identifying a set of centerlines among the plurality of centerlines whose distances from each other are less than a predetermined distance; generating a plurality of candidate points by iterating on each pair of centerlines in each set to determine the candidate point closest to the corresponding pair of centerlines; clustering the plurality of candidate points to form a cluster; and using the candidate points of each cluster to calculate the center of the 3D bounding box of the traffic management object.
5. The method according to claim 4, wherein, If the number of candidate points exceeds a first threshold and if the distance between them is less than a second threshold, then the candidate points form a cluster.
6. The method according to claim 4 or 5, wherein, The center of the cluster is calculated using the average value of the candidate points in the cluster.
7. The method according to any one of the preceding claims, wherein, The dimensions of the 3D bounding box of the computational traffic management object include: Multiple cross sections are calculated using the following steps: For each image frame including the 2D bounding box of the traffic management object, an angular projection line is drawn from the camera position associated with that image frame toward the 2D bounding box of the traffic management object; and the intersection of this line with a plane at the center of the 3D bounding box, the plane being parallel to the vertical axis and perpendicular to the horizontal component of the line-of-sight vector, is calculated to determine the cross section. The dimensions of the 3D bounding box are estimated using the plurality of cross-sections.
8. The method according to claim 7, further comprising: The size of the 3D bounding box is estimated using the average or maximum size of the plurality of cross sections.
9. The method according to any one of the preceding claims further comprises: Set a predefined value for the depth of the 3D bounding box or set the same value for its width and depth.
10. The method according to any one of the preceding claims, wherein, The orientation of the 3D bounding box of the computational traffic management object includes: Identify a reference frame in an image frame that includes the 2D bounding box of the traffic management object, wherein the reference frame has a position at a predetermined distance from the traffic management object; Determine the direction of movement at the position of the reference frame; and Calculate the orientation of the 3D bounding box, with the orientation being opposite to the direction of movement.
11. The method according to any one of claims 1 to 9, wherein, The orientation of the 3D bounding box of the computational traffic management object includes: The reference frame for identifying the 2D bounding box of the traffic management object with the maximum width, and The orientation is set to the opposite direction of the horizontal component of the line-of-sight vector of the traffic management object, based on the reference frame.
12. The method according to any one of the preceding claims, wherein, The orientation of the 3D bounding box of the computational traffic management object includes: The orientation of the 3D bounding box is determined using an image-based regression model.
13. The method according to any one of the preceding claims further comprises: The first model is trained to classify the state of traffic management objects in the plurality of 2D image crop blocks, wherein the state includes one or more of color and occlusion level.
14. The method of claim 13, further comprising: Based on one or more image frames associated with a traffic management object, the first model is used to infer the state of the traffic management object, including: for one of the one or more image frames, a majority vote is taken on the states inferred for a plurality of previous and subsequent image frames of that image frame to determine the state of the traffic management object.
15. The method according to any one of the preceding claims, further comprising: Obtain multiple 2D image crop blocks for each traffic management object, and A second model is trained to classify the mask type of traffic management objects using the multiple 2D image cropping blocks.
16. The method according to any one of the preceding claims, wherein, The traffic management object is a dynamic or static traffic management object, preferably including one or more of traffic signs, traffic lights, and dynamic message signs.
17. The method according to any one of the preceding claims further comprises: Provide the 3D bounding box in the map.
18. The method according to any one of the preceding claims, further comprising: Generate a training dataset, which includes the 3D bounding boxes of the traffic management objects, preferably including one or more of the state, occlusion, and mask type of each traffic management object.
19. A method for generating a training dataset, wherein, The training dataset is generated based on 3D bounding boxes of traffic management objects, and the training dataset is obtained by performing the method according to any one of the preceding claims.
20. At least one computer-readable medium having instructions stored thereon that, when executed by a computing device, configure the computing device to perform the method according to any one of the preceding claims.
21. An apparatus comprising: Memory, used to store image data; The processor is configured to access the memory and execute the method according to any one of claims 1 to 19.
Citation Information
Patent Citations
Determining traffic light labels and classification quality from infrastructure signals
EP4205025A1
AUTOMATED OBJECT ANNOTATION USING FUSED CAMERA / LiDAR DATA POINTS
KR1020230104841A
Urban environment labelling
US20230186499A1
