Traffic sign location mapping

Abstract

The present disclosure describes a method for estimating a traffic sign location using camera devices in multiple vehicles. The method includes detecting a traffic sign in numerous frames, selecting the last frame in which the traffic sign was detected for each pass by each of the multiple vehicles, and creating an oriented cluster of the determined last vehicle locations from each of the multiple observations of the same sign by different vehicles. The sign's location is estimated using the cluster's orientation and an estimated distance from at least one parameter corresponding to a subset of the determined last vehicle locations.

Description

[0001] TRAFFIC SIGN LOCATION MAPPING

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003]

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 562,644, filed March 07, 2024, the entirety of which is incorporated by reference herein.

[0004] TECHNICAL FIELD

[0005]

[0002] The present disclosure relates to systems and methods for location estimation of a traffic sign.

[0006] BACKGROUND

[0007]

[0003] Traffic signs are informative and prevalent components of road infrastructure that contribute to road safety and smooth traffic flow by providing necessary and / or helpful information to drivers. The accurate detection and recognition of these signs by automated driving systems or driver assistance systems, or the accurate mapping and these signs for retrieval, can significantly improve the safety and efficiency of vehicle navigation. However, the correct estimation of the location of traffic signs poses a significant challenge due to factors such as varying lighting conditions, different angles of view, occlusions, and sign degradation, as well as the dynamic nature of the road network and traffic regulation. Accurate mapping of these signs can be impaired due to errors or noise sources in the GPS localization of camera systems that detect these signs.

[0008]

[0004] Therefore, there is a need for an improved method for estimating the location of traffic signs that can overcome the limitations of the existing approaches.

[0009] SUMMARY

[0010]

[0005] Certain aspects of the present disclosure provide a method for accurately estimating the location of a traffic sign using multiple observations from different vehicles. The method involves receiving, by a cloud server, metadata related to an observation of a traffic sign in a plurality of frames from a camera device installed at a windshield of a vehicle; creating an oriented cluster from a first subset of observations from multiple observations of the traffic sign from different vehicles equipped with a similar camera device, wherein each observation of the multiple observations comprises a location estimate of the edge device for a final detection of the traffic sign in the plurality of frames; calculating an angle to the traffic sign relative to the oriented cluster based on the orientation of the cluster; and estimating a location of the traffic sign based on the angle, an estimated offset, and at least one parameter relating to the first subset of observations in the oriented cluster.

[0011]

[0006] Certain aspects of the present disclosure provide a computer program product for accurately estimating the location of a traffic sign using multiple observations from different vehicles. The computer program product includes at least one non-transitory computer- readable storage medium having stored thereon computer-executable program code instructions which, when executed by a computer, cause the computer to perform operations for: receiving metadata related to an observation of a traffic sign in a plurality of frames from a camera device installed at a windshield of a vehicle; creating an oriented cluster from a first subset of observations from multiple observations of the traffic sign from different vehicles equipped with a similar camera device, wherein each observation of the multiple observations comprises a location estimate of the edge device for a final detection of the traffic sign in the plurality of frames; calculating an angle to the traffic sign relative to the oriented cluster based on the orientation of the cluster; and estimating a location of the traffic sign based on the angle, an estimated offset, and at least one parameter relating to the first subset of observations in the oriented cluster.

[0012]

[0007] Certain aspects of the present disclosure provide a system for estimating the location of a traffic sign. The system includes at least one processor; at least one memory including computer program code; the at least one memory and the computer program code configured to, with the at least one processor, cause the system to at least: receive metadata related to an observation of a traffic sign in a plurality of frames from a camera device installed at a windshield of a vehicle; create an oriented cluster from a first subset of observations from multiple observations of the traffic sign from different vehicles equipped with a similar camera device, wherein each observation of the multiple observations comprises a location estimate of the edge device for a final detection of the traffic sign in the plurality of frames; calculate an angle to the traffic sign relative to the oriented cluster based on the orientation of the cluster; and estimate a location of the traffic sign based on the angle, an estimated offset, and at least one parameter relating to the first subset of observations in the oriented cluster.

[0013]

[0008] Certain aspects of the present disclosure provide a method for estimating traffic sign locations using multiple observations from various vehicles equipped with a camera device. The method involves detecting a traffic sign within the camera's field of view and calculating its relative angle. An oriented cluster is formed from multiple observations, and the centroid of this cluster is offset based on one or more of the detecting lane number, presence of a road shoulder, the type of sign. The sign’s location is estimated as the center of the observation cluster after an appropriate offset or offsets have been applied in the angle made by the oriented cluster relative to the direction of travel on the road.

[0014] BRIEF DESCRIPTION OF DRAWINGS

[0015]

[0009] Non-limiting embodiments of the present disclosure are described by way of example with reference to the accompanying figures, which are schematic and are not intended to be drawn to scale. Unless indicated as representing the background art, the figures represent aspects of the disclosure. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:

[0016]

[0010] FIG. 1 illustrates an environment for traffic sign location mapping according to various embodiments of the present disclosure.

[0017] [Oi l] FIG. 2 illustrates a block diagram of an edge device for traffic sign detection.

[0018]

[0012] FIG. 3 illustrates an example scenario for GPS and video frame synchronization.

[0019]

[0013] FIG. 4 illustrates an example scenario of an oriented cluster and a sub-cluster of GPS positions.

[0020]

[0014] FIG. 5 illustrates an example scenario of traffic sign location estimation.

[0021]

[0015] FIG. 6 illustrates a traffic sign location estimation based on a single observation.

[0022] DETAILED DESCRIPTION

[0023]

[0016] The detailed description set forth below, in connection with the appended drawings, is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0024]

[0017] Based on the teachings, one skilled in the art should appreciate that the scope of the disclosure is intended to cover any aspect of the disclosure, whether implemented independently of or combined with any other aspect of the disclosure. For example, an apparatus may be implemented, or a method may be practiced using any number of the aspects set forth. In addition, the scope of the disclosure is intended to cover such an apparatus or method practiced using other structure, functionality, or structure and functionality in addition to or other than the various aspects of the disclosure set forth. Any aspect of the disclosure disclosed may be embodied by one or more elements of a claim.

[0025]

[0018] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0026]

[0019] Although particular aspects are described herein, many variations and permutations of these aspects fall within the scope of the disclosure. Although some benefits and advantages of the preferred aspects are mentioned, the scope of the disclosure is not intended to be limited to particular benefits, uses or objectives. Rather, aspects of the disclosure are intended to be broadly applicable to different technologies, and system configurations, some of which are illustrated by way of example in the figures and in the following description of the preferred aspects. The detailed description and drawings are merely illustrative of the disclosure rather than limiting, the scope of the disclosure being defined by the appended claims and equivalents thereof.

[0027]

[0020] The terms “traffic sign” and “road sign” are both used throughout the description. Each term may refer to signs that are visible from the road. Traffic signs may be considered a subset of road signs that provide information to drivers and / or pedestrians with relevant information, such as about road conditions, directions, and rules. As explained below, the placement of traffic signs may be subject to an elevated level of regulations and / or guidance which may be leveraged in accordance with certain aspects of the present disclosure. In some contexts, local regulations, guidance, and or norms, may be inferred and incorporated into mapping of road sign locations. In addition, certain aspects of the present disclosure may be applied without consideration of whether a detectable road sign would be classified as a traffic sign

[0028]

[0021] Maintaining an accurate map of road signs, which may be any sign that is visible from a road, and which may include traffic signs, such as speed limit signs, presents a significant challenge due to the dynamic nature of the road network, road conditions, and traffic regulation. Speed limit signs frequently change due to alterations in traffic regulations, roadworks, temporary speed restrictions, and other factors. These changes can occur on both a micro and macro level, ranging from a single sign replacement in a residential area to widespread changes across a city or region. Traditional methods of map maintenance struggle to keep up with these frequent changes, leading to inaccuracies that can impede the performance of automated driving systems, driver assistance platforms, driver safety systems, and the like.

[0029]

[0022] To address these inaccuracies, the present disclosure proposes a method that leverages observations of road signs, or in some embodiments, more specifically traffic signs or even certain types of traffic signs, from a wide array of vehicles equipped with a camerabased driver safety device. The driver safety device (or, more generally, an “edge device”), can detect and recognize a variety of road signs, including but not limited to speed limit signs.

[0030]

[0023] In one embodiment, the edge device may have the ability to identify and record the position of traffic signs in a camera view, which may be used to estimate the location of the sign near the road. The edge device may detect a traffic sign in visual data captured by a camera included in the edge device. The edge device may include a GPS sensor to record the GPS coordinates of the vehicle installed with the edge device. The edge device may record the GPS coordinates of the vehicle when the edge device has lastly detected the traffic sign in the visual data (and may also record additional detections of the traffic sign location in earlier image frames). A metadata is generated including GPS coordinates of the vehicle and inference data related to the traffic sign. The metadata may be sent to a cloud server that aggregates similar data from edge devices in different vehicles that have passed the same traffic sign (which may include instances of the same vehicle passing the traffic sign at a different time). The cloud server may use the GPS coordinates of the multiple vehicles to form a cluster. The cluster may be oriented in the direction of the actual location of the traffic sign because of camera’s field of view and the selection of the GPS location associated with the last detection from each vehicle pass. If all vehicles use substantially the same camera, or use cameras with substantially the same horizontal field of view, then knowledge of the field of view need not be considered, since the orientation of the cluster may be considered a proxy of the angle from the camera focal point to the edge of the image in each observation. In some embodiments, the center of the cluster may be determined which is later added with an offset value to determine an estimate of the location of the traffic sign. In some embodiments, different offsets may be applied to different points in the oriented cluster, and then the estimate of the location of the traffic sign may be taken as the center of the offset points. The offset value may be determined by the cloud server at least based on the number of lanes of a road on which the multiple vehicles have travelled.

[0031]

[0024] Although the angle of the oriented cluster of the last detections will correspond to the camera’s field of view, to improve the accuracy of the estimate of the location of traffic sign, offset values may be calculated by considering road sign detections at more than one location within the camera’s field of view. For example, two oriented clusters may be obtained (i.e. for which at least one set of detections may correspond to the last detection before the sign crosses out of a predefined region of the field of view, even though the sign may continue to be detected in more eccentric image locations in subsequent frames). The first may comprise vehicle position estimates when the traffic sign was last visible within the camera field of view. This oriented cluster may be expected to align with one-half of the camera’s horizontal field of view. A second oriented cluster may comprise vehicle position estimates at a detection corresponding to an angular value below the FOV, such as 30 degrees from center. The second oriented cluster may be expected to form an oriented cluster having a 30-degree angle with the direction of travel of the road, since the traffic sign would appear at that angle in the camera frame if the camera were displaced anywhere along that 30-degree angle. In some embodiments, the two oriented clusters may each form the basis for rays that intersect at a location that may be taken as the estimated location of the sign by the side of the road. The above-mentioned method enhances navigation systems and autonomous driving technology by leveraging multiple observations of a traffic sign from vehicle-mounted camera systems to average out noise sources and collectively provide accurate locations of traffic signs.

[0032]

[0025] While the method has been described with a focus on speed limit signs, it is noteworthy that the method can be applied to map any type of road sign, or indeed, other types of objects that may be observed from a road in a collection of passes by vehicles with similar camera or similar sensor systems (for example, cameras mounted in a forward-facing location of a vehicle windshield and having the same or similar field of view). This ability to map diverse objects further enhances the versatility and utility of the proposed method in improving the accuracy and comprehensiveness of road mapping for enhanced vehicular navigation and safety.

[0033]

[0026] FIG. 1 depicts an illustration of an environment 100 in which one or more embodiments of the invention may be implemented. The environment 100 includes a cloud server 102, vehicles 104, 106, and a network 112 for communication between the cloud server and edge devices 104a, 106a installed in the vehicles 104, 106, respectively.

[0034]

[0027] The network 112 can be radio access networks such as GSM, LTE, 5G or the like. In one embodiment, the network can be Wi-Fi or other wireless broadband networks.

[0035]

[0028] The cloud server 102 provides configuration settings and software related services to the edge devices. The cloud server may store the data received from the edge devices. Further, the cloud server may provide supplemental computational capabilities to the edge device.

[0029] In one embodiment, each edge device 104a, 106a may be an advanced driver assistance system (ADAS) that includes one or more cameras and sensors. Further, each edge device 104a, 106a may include a processor and communication circuitry that includes radio interfaces and antennas. The radio interfaces may correspond to one or more of a plurality of radio access technologies including one or more of GSM, NB-IoT, LTE, 5G, WLAN, Bluetooth, BT-LE, NFC, radio frequency identifier (RFID), ultra-wideband (UWB), and the like. Each edge device 104a, 106a is installed in a vehicle 104, 106, respectively, to monitor driver behavior, which includes capturing a driving path of the vehicle with a forward-facing camera. Further, each edge device 104a, 106a may include a driver-facing camera that may capture an interior cabin of the vehicle, a microphone, an audio speaker or other output devices to provide notifications and alerts to the driver. Each edge device 104a, 106a may be a standalone device or a combination of devices. In one embodiment, the edge device installed in the vehicle may include a sensor device and a computing device, where the sensor device may include cameras and other inertial sensors, but not a processor. The computing device on the other hand may include one or more processors to process the information captured by the sensor device. The sensor module may include a GPS receiver in addition to other inertial sensors. The camera module may include one or more cameras to capture visual data related to the cabin of the vehicle and the path travelled by the vehicle. The sensor device may relay the captured data to the computing device via a wired or wireless connection. The distribution of functionalities among devices may be implemented to reduce the size of devices that are installed near the windshield. The various components of an exemplary edge device are explained next with reference to FIG. 2.

[0036]

[0030] FIG. 2 depicts a block diagram of the edge device 200 (which may be instantiated in edge devices 104a, 106a) including various components. The edge device 200 includes a processor 202, a communication module 204, input / output (I / O) module 206, memory 208, a camera module 210 and a sensor module 212. In one embodiment, the edge device 200 may not include a camera module 210 and in those scenarios, the camera module 210 may be connected to edge device 200 through wired or wireless connection. The edge device 200 may include input sensors (which may include a forward-facing camera, a driver facing camera, connections to other cameras that are not physically mounted to the device, inertial sensors, car OBD-II port sensor data (which may be obtained through a Bluetooth connection), and the like) and compute capability. The compute capability may be a CPU or an integrated System-on-a- chip (SOC), which may include a CPU and other specialized compute cores, such as a graphics processor (GPU), gesture recognition processor, and the like. In some embodiments, the edge device 200 for traffic sign location mapping may include wireless communication to cloud services, such as with Long Term Evolution (LTE) or Bluetooth communication to other devices nearby. For example, the cloud server 102 may provide real-time analytics assistance. In an embodiment involving cloud services, the cloud server 102 may facilitate aggregation and processing of data for offline analytics. The edge device 200 may also include a global positioning system (GPS) either as a separate module or integrated within a system-on-a-chip (e.g., included in sensor module 212).

[0037] GPS and Video Frame Synchronization

[0038]

[0031] The sensor module 212 may receive and / or compute GPS coordinates of the vehicle with a predefined frequency. For example, the sensor module may receive and / or compute GPS coordinates of the vehicle every one second. The edge device 200 may include a camera module 210 that further includes one or more cameras configured to capture visual data. A camera may be configured to capture visual data at a predefined frame rate. In some embodiments, the frame rate may be changed dynamicaly, such as whether the vehicle is presently in a location where a new traffic sign was recently detected, and therefore there is a need to construct an oriented cluster in accordance with the present disclosure. For example, the camera may capture a video at 15 frames per second in certain circumstances.

[0039]

[0032] In some embodiments, the edge device 200 may store and process high-resolution video data selectively, focusing on relevant parts of the image rather than the entire frame. For example, the edge device 200 may be configured with a neural network model that takes 360p visual data as input. The edge device 200 may yet collect image data at 1080p, or some other resolution. In normal operation, the edge device 200 may down sample the entire image frame to 360p (e.g. from 1080p to 360p). However, instead of down sampling the entire 1080p video data to 360p, the device 200 may identify and isolate the regions of the image that contain the detected sign. These regions may be cropped from the high-definition data at the original 1080p resolution, providing a 360p crop of the HD data that retains the detail and quality in the area of interest (which may be the area that contains the detected sign). This approach may reduce the computational resources needed to process the high-definition (1080p) video data, as the device only needs to process a small portion of each frame at the relatively high resolution. It may also maintain the quality and detail of the sign in the high-definition video data, which could improve the accuracy of the sign detection and location estimation.

[0033] One challenge in integrating GPS data with video frame data is the difference in sample rates. For example, GPS data may be sampled at a rate of one measurement per second, while video frames may be captured at a higher frequency such as 5 frames per second (fps) or 30 fps. The difference in rates at which visual data and GPS data are sampled can lead to inconsistencies and inaccuracies in aligning GPS data with corresponding video frames.

[0040]

[0034] The edge device 200 may overcome the above challenges by interpolating the GPS measurements to each video frame. The edge device 200 may calculate intermediate GPS coordinates (shown in FIG. 3) represented by ‘h, h ,h, U,...In’ for respective video frame 302 based on the known GPS coordinates at Li and L2 time instances. The calculation may enable each video frame, regardless of its capture frequency, to be associated with an accurate GPS position, such as by interpolating between the previous and next GPS readings. The calculation of the intermediate GPS coordinates may also be based on the direction and / or speed of the vehicle such as to account for a curved path. The intermediate GPS coordinates enable accurate estimation of the location of the vehicle at the time instance of the video frame in which the traffic sign was lastly detected.

[0041]

[0035] By overcoming the challenge of different sample rates between GPS and video frames, the above method may increase the accuracy and reliability of the location estimation process. This aspect significantly contributes to the robustness of the method and its effectiveness in maintaining an up-to-date and accurate map of road signs.

[0042] Cluster formation of GPS Coordinates

[0043]

[0036] In one embodiment, object detection algorithms may be deployed on the edge device, including but not limited to YOLO (You Only Look Once), R-CNN (Region-based Convolutional Neural Networks), and SSD (Single Shot Multi-Box Detector), which may be utilized to identify traffic signs within a video frame or a sequence of video frames. The positioning of a traffic sign within a video frame is dependent upon its actual location relative to a position on the road, as determined by the camera’s field of view based on the direction of travel of the vehicle on that road, and the camera being mounted on the vehicle, and pointed substantially in the direction of the vehicle’s forward direction of travel. For example, if the sign is situated on the left side of the road, it may appear on the leftmost portion of the frame, as captured within the camera’s field of view. Similarly, if the sign is located on the right side of the road, it may be depicted on the rightmost portion of the frame, as far out as the camera’s horizontal field of view.

[0037] The edge device 200 may determine GPS coordinates of the vehicle for the last frame in which the traffic sign was detected. The edge device 200 may determine an angle at which the traffic sign is located relative to the direction of travel of the vehicle. The edge device 200 may calculate the relative angle of the traffic sign’s location with respect to the vehicle’s direction of travel based on the field of view (FOV) value of the camera that captured the traffic sign. The calculation of the angle of the traffic sign may be avoided however, and the angle of the oriented cluster of similar observation points may instead be used to determine this angle. The GPS coordinates, and optionally, visual data related to the traffic sign in addition to other inference data, may be sent to the cloud server 102 from the edge device 200 of a vehicle. In some embodiments, the edge device may send an HD crop of the detected traffic sign and its pixel position in the captured image to the cloud server. The cloud server 102 may calculate the sign’s position in the full image based on the pixel position of the HD crop. Processing on the cloud server 102 may therefore contribute to or yield an estimate of the sign’s location relative to the vehicle, even when processing cropped regions of the video data. This would allow the system to leverage high-resolution data effectively, while minimizing computational resources and maintaining privacy (because the cropped images may be less likely to contain personal information associated with pedestrians or other vehicles on the road). The edge device 200 may encrypt the metadata before sending it to the cloud server 102 to ensure privacy and security of the original data, which may be considered in some instances to be personal data of the driver of the vehicle having the camera device. The edge device 200 may send inference data to the cloud server 102, where the inference data include sign type, pixel position, time stamp associated with video frame, GPS position. The edge device 200 may perform encoding techniques on the metadata to protect and ensure the privacy and security of the information. The inference data may include other inertial data inferences including direction of the vehicle, and speed of the vehicle.

[0044]

[0038] The cloud server 102 may receive metadata from multiple vehicles related to detection of same traffic sign. The cloud server 102 may collate all the metadata related to the traffic sign and form a cluster of GPS positions of the vehicles that have passed the traffic sign. Prior to collating, the cloud server 102 may determine the minimum number of observations (e.g., derived from metadata received from multiple edge devices) of the same traffic sign from vehicles required to form a reliable cluster. In some embodiments, the cloud server 102 may continue accumulating observations until a point that is determined based on statistics of the oriented cluster. For example, if the oriented cluster is wide, such that there is greater uncertainty about the true direction to the detected sign, the cluster may accumulate relatively more observation points to determine an estimated location of the detected traffic sign.

[0045] Conversely, if the oriented cluster is narrow, less observation points may be needed.

[0046]

[0039] An example cluster is shown in FIG. 4. When visually represented, the cluster 402 exhibits a directional orientation towards the actual geographical location of the traffic sign 404. In general, certain classes of traffic signs are placed on sides of a road and their applicability to the road depends on the direction of travel of a vehicle on the road. Other classes of traffic signs (such as overhead lane exit indicators) may tend to appear at locations above one or more lanes of road traffic. The road 406 may have one or more lanes 408 to ease the flow of traffic on the road. The number of lanes on a road may depend on the road type. In one embodiment, the cloud server 102 may consider lane information to create a cluster (which may be an oriented cluster itself). The cloud server 102 may create sub-clusters of the cluster based on lane information. For example, a sub-cluster 410 may be created for the GPS positions falling onto a lane of the road that has two or more lanes. The cloud server 102 may receive a lane number in which the vehicle was travelling when it has last seen the traffic sign. The lane number information may be included in the metadata sent from the edge device 200. The cloud server 102 may form a sub-cluster for a lane of the road based on the lane number information. The cloud server 102 may estimate the actual location of the traffic sign based on the created cluster and / or one or ore sub-clusters for that traffic sign.

[0047] Traffic sign location estimate

[0048]

[0040] The cloud server 102 may determine a centroid for the created cluster 402. Similarly, the cloud server 102 may determine a centroid for each of one or more sub-clusters 410 in the cluster 402. The one or more sub-clusters may correspond to some GPS positions that fall within the same lane of multiple lanes of a road. In one embodiment, the cloud server 102 may determine the number of lanes of the road by retrieving lane information from a road database. In another embodiment, the lane information may be determined by the edge device 200 and the edge device may send the lane information as part of the metadata or separately. For example, the edge device may contain a lane detection module which may detect visible lanes on the road and which may be able to infer the lane number of the vehicle at the time that the traffic sign was detected.

[0049]

[0041] In one embodiment, the cloud server 102 may calculate an offset value for the centroid based on road width (i.e., indicated by number of lanes of the road). The road width and number of lanes in the road may be determined based on road type information. For example, a road type “highway” may have at least 2 lanes. Further, the road type information may indicate whether there is a presence of a shoulder area to the road, where shoulder area is the portion of road to which the vehicle can pullover or stop. The road width (and / or offsets) may be calculated based on the presence or absence of a shoulder area. Further, the cloud server may determine a set of offset values for the cluster based on lane information and road width (or road width attributable to a shoulder). The number of offset values in the set may depend on the number of lanes in the road. The cloud server may determine an offset value for each lane based on a sub-cluster associated with that lane. The offset value for a sub-cluster may be based on a centroid of that sub-cluster and road width. The road width may be determined based on the number of lanes and any shoulder area, where number of lanes between the vehicle and the side of the road and presence of shoulder area may be derived from road type information and / or lane information. The cloud server may calculate an offset value for the entire cluster, or a range of offset values for the entire cluster based on computational resources and / or data points from vehicle passes available at the cloud server. The cloud server may calculate an offset value for the entire cluster when there are less computational resources and / or available data and may calculate a range of offset values for the entire cluster when there are adequate computational resources and / or available data, leading to effective utilization of the resources of the cloud server.

[0050]

[0042] In one embodiment, the cloud server 102 may calculate an angular distance value using offset value for the centroid of an oriented cluster or a sub-cluster and angle of the oriented cluster. For example, the angular distance value is calculated based on the offset value and sine function of halved camera’s field of view angle (or the angle of the oriented cluster). The angle of the oriented cluster is similar to the halved camera’s field of view when using the last frame in which the sign was detected to construct the oriented cluster. Similar to offset values, angular distance values may be calculated based on the range of offset values applicable for the oriented cluster.

[0051]

[0043] In one embodiment, the cloud server 102 may shift the entire cluster of GPS positions based on the calculated angular distance value for the entire cluster. In another embodiment, the cloud server may shift the entire cluster using a range of angular distance values. The GPS positions in the cluster may be added with an angular distance value from the range of angular distance values based on the lane on which GPS positions lie. The cloud server 102 may determine a centroid of the shifted cluster, where the points of the original cluster of GPS positions is either shifted based on an angular distance value or a range of angular distance values.

[0044] As an error-correcting mechanism, the cloud server 102 may determine a leadingedge point of the cluster in the direction of traffic sign (i.e. after excluding outliers). Upon shifting the entire cluster, the cloud server may determine whether the centroid of the shifted cluster is beyond the leading-edge point of the pre-shifted cluster. If the centroid of the shifted cluster is not beyond the leading-edge point (which may indicate that the applied offset(s) were too small leading to smaller angular distance values, and may for example, have estimated the sign to be at a location corresponding to a drivable lane or a road shoulder), then the cloud server 102 may increase the offset value leading to increase in angular distance value such that the centroid of the shifted cluster is beyond the leading-edge point of the original cluster. The cloud server 102 may determine an estimate of the location of the traffic sign as the shifted centroid. This error-correcting mechanism may be useful in refining estimated horizontal offsets for all signs detected on the same road and / or area. Further, this error-correcting mechanism may be more robustly implemented when multiple sub-clusters are offset to an estimated location, for example, each sub-cluster corresponding to a different lane number, and sub-clusters associated with lane number further from the road being initially offset in the direction of travel more than the sub-clusters from nearer lanes, when accounting for a field of view that is less than 180 degrees (90 degrees FOV from center to each side).

[0052]

[0045] In another embodiment, the cloud server 102 may shift a first set of GPS coordinates from the original cluster based on the determined angular distance value. The first set of GPS coordinates may be a sub-cluster corresponding to a lane. Alternatively, or in addition, the first sub-cluster may be GPS coordinates associated with a precision estimate above a threshold (i.e. which may correspond to a signal-to-noise ratio above a threshold), The cloud server 102 may calculate a centroid for the shifted sub-cluster and the centroid of the shifted sub-cluster may be considered as an estimate of the location of the traffic sign. The cloud server 102 may shift the entire cluster, or a sub-cluster based on heuristics regarding quality of each location estimate. This enables the cloud server to dynamically change the type of clustering to be used for estimation of the location of the traffic sign based on processing at the cloud server, which may include, for example, determining whether the oriented cluster exhibits multiple peaks, some of which correspond to individual lanes on which vehicles typically travel on a given road.

[0053]

[0046] The cloud server 102 may determine an estimate of the location of the traffic sign by adding an angular distance value to the centroid in the angular direction of the cluster. The angular direction of cluster may be determined by the cloud server, and may be compared with the camera’s field of view (or, similarly, compared with other oriented clusters in different locations, since each oriented cluster may be expected to have a similar angle). Oriented clusters having an angle that is substantially different from an expected angle given the camera’s field of view may be discarded, for example, or may be automatically re-calculated with newer data. In such exception cases, the cloud server 102 may, alternatively or in addition, determine the estimate the location of the traffic sign based on complementary approaches, such as by considering the dimensions of the road sign board and timestamp associated with video frame in which the traffic sign was detected. Further, such exception cases may be handled by deferring traffic sign estimation until at least an additional predefined number of observations are obtained. This may include selectively discarding previous observations based on age, which may handle cases where a sign has been moved.

[0054]

[0047] In one embodiment, the cloud server 102 may perform offset calculation incorporating standardized traffic sign placement parameters defined by transportation authorities to enhance location estimation accuracy. For example, in the US, the Manual on Uniform Traffic Control Devices (MUTCD) establishes specific lateral placement requirements that vary according to road classification, sign type, and roadway characteristics. These requirements specify different lateral offset ranges: 6-12 feet from travel lane edges for regulatory signs on conventional roads, minimum 12-foot offsets for highway installations, with additional modifications based on shoulder width and barrier placement.

[0055]

[0048] The cloud server 102 may process historical detection data to establish statistical correlations between vehicle detection positions and verified sign locations. Upon processing, the cloud server 102 may generate a structured offset reference database containing both standardized placement parameters and actual implementation variations across jurisdictions and installation periods. The offset parameters may comprise and be organized according to one or more factors: lane position (leftmost, center, rightmost), road configuration (conventional, highway, urban arterial), lane count, shoulder presence, and sign classification.

[0049] In one embodiment, the cloud server 102 may derive lane-specific offset parameters through statistical analysis of historical detection data correlated with verified sign locations. For each lane configuration, the cloud server may process detection metadata to extract statistical patterns between vehicle positions during detection events and ground-truth sign locations. The cloud server may organize detection data according to lane position indicators (leftmost, center, rightmost) and calculate the mean displacement vector between detection points and verified sign positions for each lane configuration. These displacement vectors may be further refined through regression analysis that incorporates road width parameters and / or standardized placement guidelines from transportation authorities. The resulting lane-specific offset parameters may be stored in a structured database indexed by lane position, road configuration type, and sign classification. When processing new detection data, the cloud server may query this reference database using the specific lane position from which the detection occurred to retrieve the statistically validated offset parameter appropriate for that particular lane configuration, thereby enabling more accurate sign placement estimates regardless of which lane position generated the detection data.

[0056]

[0050] In accordance with certain aspects, during location estimation, the cloud server 102 may query the reference database to retrieve appropriate offset values based on detection scenario parameters. The retrieved offset values that are converted to angular distance values may then be applied to transform raw detection coordinates into estimated sign positions. This integration of standardized placement specifications with derived measurement data may provide more accurate position estimates.

[0057]

[0051] FIG. 5 illustrates an example scenario of traffic sign location estimation. In FIG. 5, the cloud server 102 may transform an original cluster 508 of GPS coordinates from vehicle detection points into a shifted (i.e. offset) cluster 502 by applying a calculated angular distance value 506. The angular distance value is determined using both the lane-specific parameters from the reference database and the camera geometries 510 that define the field of view constraints. This transformation shifts the cluster in the direction indicated by the orientation vector, resulting in a shifted cluster centroid 504 that represents the predicted location of the traffic sign. The spatial relationship between the original cluster 508 and the shifted (i.e. offset) cluster 502 demonstrates how the cloud server 102 compensates for the positional difference between where signs are detected by vehicles and where they are physically located. The camera geometries 510 may contribute to determining both the magnitude and direction of the offset, as they establish the angular relationship between the vehicle trajectory and the sign visibility in the camera’s field of view. By applying this transformation consistently across different road configurations and sign types, the cloud server may achieve higher accuracy in sign location estimation than using raw detection coordinates alone.

[0058]

[0052] FIG. 6 depicts an example scenario for traffic sign location estimation based on a single observation of the traffic sign from an edge device installed in the vehicle. The edge device may determine the GPS location of the vehicle when the edge device detects a traffic sign in the captured visual data. The edge device may determine the type of traffic sign from the captured visual data by processing the visual data using a machine learning model. The edge device may estimate the location of the traffic sign based on road type and traffic sign type. In one embodiment, the edge device may send metadata including traffic sign type and GPS coordinates to the cloud server. The cloud server may calculate an estimate for the location of the traffic sign based on the GPS position of the vehicle, and in the case of singleobservation estimates, another factor, such as the dimension of the traffic sign board.

[0059]

[0053] In one embodiment, when the edge device detects a traffic sign within the camera’s field of view, the edge device extracts geometric parameters including angular position and relative dimensions of the sign within the image frame. The angular position parameter, denoted as “a” in FIG. 6, represents the angle between the vehicle’s forward trajectory and the detected sign. Parameters “dX” and “dY” represent the calculated horizontal and vertical displacement components, respectively, from the vehicle position to the estimated sign position.

[0060]

[0054] The edge device may implement a geometric transformation function utilizing the parameters “dX” and “dY” in conjunction with the camera’s field of view parameters. Position T1 represents the initial detection position of the vehicle shifted to the side of the road, while position TO represents the calculated true position after applying an offset along the direction of the road. The edge device may access stored sign dimensional parameters based on the detected sign type, enabling distance estimation through trigonometric relationships.

[0061]

[0055] The edge device may incorporate standard sign dimensions to calculate the lateral offset between the sign and the vehicle position. This lateral offset, when combined with the angular position parameter, enables the edge device to determine the estimated sign location through vector calculation. The edge device may adjust this calculation based on the detected sign type, as different categories of traffic signs follow different standardized dimensional specifications.

[0062]

[0056] The above single-observation estimation approach provides location data when the cloud server has insufficient multiple observations to form oriented clusters. The edge device may transmit the calculated sign location to the cloud server as supplementary data for subsequent aggregation with other vehicle observations. The single-observation estimation approach may be used when there is a preference to obscure the precise location of the vehicle and camera system from where the observation was made.

[0063]

[0057] In one embodiment, the cloud server may create a map based on the determined estimates of location of the traffic signs. The cloud server may update an existing map based on the determined location of the traffic signs. The map may be updated with sign type information and estimated location of the traffic signs. The cloud server may send the created map or the updated map to the edge devices to generate alerts based on the map and driving behavior. For example, the updated map may be used to assess whether a driver is speeding. The cloud server may store the shifted cluster data instead of the original cluster data, thereby disassociating the actual location of the vehicles from the data stored in the cloud server.

[0064]

[0058] In another embodiment, the cloud server may retrieve the original cluster from the shifted cluster using offset values. Further, the cloud server may receive an alert raised by the edge device, where the alert is related to applicability of a traffic sign to the vehicle installed with the edge device. The cloud server may determine the validity of the alert raised by the edge device based on the location of the vehicle and the cluster associated with the same traffic sign. The cloud server may determine whether the location of the vehicle (when it detected the traffic sign) falls within the cluster associated with that traffic sign. The cloud server may determine that the alert is invalid if the location of the vehicle is outside the boundary of the oriented cluster corresponding to the traffic sign.

[0065]

[0059] The computer program product may include a non-transitory computer-readable medium having instructions stored thereon, the instructions being executable by one or more processors configured to carry out any of the systems and / or methods described herein.

[0066]

[0060] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of this disclosure or the claims.

[0067]

[0061] Embodiments implemented in computer software may be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof. A code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0068]

[0062] The actual software code or specialized control hardware used to implement these systems and methods does not limit the claimed features or this disclosure. Thus, the operation and behavior of the systems and methods were described without reference to the specific software code being understood that software and control hardware can be designed to implement the systems and methods based on the description herein.

[0069]

[0063] When implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium. The steps of a method or algorithm disclosed herein may be embodied in a processorexecutable software module, which may reside on a computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another. Anon-transitory processor-readable storage media may be any available media that may be accessed by a computer. By way of example, and not limitation, such non-transitory processor-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer or processor. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which may be incorporated into a computer program product.

[0070]

[0064] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0065] The terms “data processing apparatus”, “data processing system”, “client device”, “client computing device”, “computing platform”, “computing device”, “computing system”, “user device”, or “device” can encompass all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA or an ASIC. The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them.

[0071]

[0066] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0072]

[0067] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer include a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a GPS receiver, a digital camera device, a video camera device, or a portable storage device (e.g., a universal serial bus (USB) flash drive), for example. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0073]

[0068] To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), plasma, or LCD monitor, for displaying information to the user; a keyboard; and a pointing device, e.g., a mouse, a trackball, or a touchscreen, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can include any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user.

[0074]

[0069] In certain circumstances, multitasking and parallel processing systems may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. For example, the computing devices described herein can each be a single module, a logic device having one or more processing modules, one or more servers, or an embedded computing device.

[0075]

[0070] Having now described some illustrative implementations and implementations, it is apparent that the foregoing is illustrative and not limiting, having been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, those acts and those elements may be combined in other ways to accomplish the same objectives. Acts, elements, and features discussed only in connection with one implementation are not intended to be excluded from a similar role in other implementations or implementations.

[0076]

[0071] The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” “having,” “containing,” “involving,” “characterized by,” “characterized in that,” and variations thereof herein, is meant to encompass the items listed thereafter, equivalents thereof, and additional items, as well as alternate implementations consisting of the items listed thereafter exclusively. In one implementation, the systems and methods described herein consist of one, each combination of more than one, or all of the described elements, acts, or components.

[0077]

[0072] Any references to implementations or elements or acts of the systems and methods herein referred to in the singular may also embrace implementations including a plurality of these elements, and any references in plural to any implementation or element or act herein may also embrace implementations including only a single element. References in the singular or plural form are not intended to limit the presently disclosed systems or methods, their components, acts, or elements to single or plural configurations. References to any act or element being based on any information, act, or element may include implementations where the act or element is based at least in part on any information, act, or element.

[0078]

[0073] Any implementation disclosed herein may be combined with any other implementation, and references to “an implementation,” “some implementations,” “an alternate implementation,” “various implementation,” “one implementation,” or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation may be included in at least one implementation. Such terms as used herein are not necessarily all referring to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

[0079]

[0074] References to “or” may be construed as inclusive so that any terms described using “or” may indicate any of a single, more than one, and all of the described terms.

[0080]

[0075] Where technical features in the drawings, detailed description or any claim are followed by reference signs, the reference signs have been included for the sole purpose of increasing the intelligibility of the drawings, detailed description, and claims. Accordingly, neither the reference signs nor their absence have any limiting effect on the scope of any claim elements.

[0081]

[0076] The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the embodiments described herein and variations thereof. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other embodiments without departing from the spirit or scope of the subject matter disclosed herein. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

[0082]

[0077] While various aspects and embodiments have been disclosed, other aspects and embodiments are contemplated. The various aspects and embodiments disclosed are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

Claims1. A method for estimating the location of a traffic sign, the method comprising: receiving, by a cloud server, metadata related to an observation of a traffic sign in a plurality of frames from a camera device installed at a windshield of a vehicle; creating an oriented cluster from a first subset of observations from multiple observations of the traffic sign from different vehicles equipped with a similar camera device, wherein each observation of the multiple observations comprises a location estimate of the edge device for a final detection of the traffic sign in the plurality of frames; calculating an angle to the traffic sign relative to the oriented cluster based on the orientation of the cluster; and estimating a location of the traffic sign based on the angle, an estimated offset, and at least one parameter relating to the first subset of observations in the oriented cluster.

2. The method of claim 1, further comprising: calculating an angular distance value based on the estimated offset and the angle; applying the angular distance value to the first subset of observations in the cluster in the direction of the calculated angle to produce an offset first subset of observations; and estimating the location of the traffic sign as a center of the offset first subset of observations.

3. The method of claim 1, further comprising: calculating the centroid of the oriented cluster based on the first subset; and wherein the first subset of observations is substantially all of the observations.

4. The method of claim 1, further comprising: determining the number of lanes in the road; and wherein estimating the offset is based on the number of lanes in the road.

5. The method of claim 4, further comprising: determining a road type for the road; and determining an average number of lanes for the determined road type; and wherein the determined number of lanes in the road is based on the determined average number of lanes.

6. The method of claim 1, further comprising: detecting a lane number corresponding to the lane that the vehicle was in during the observation; wherein the first subset of observations in the cluster consists of observations from vehicles in the lane number; and wherein the determination of the offset is further based on the lane number.

7. The method of claim 6, further comprising: determining an offset from a second subset of observations in the cluster to the location of the detected sign, wherein the second subset consists of observations from a second lane number.

8. The method of claim 1, further comprising: determining whether there is a shoulder on the road; and wherein the offset is estimated further based on a determined presence or absence of a shoulder on the road.

9. The method of claim 1, further comprising: determining a road type for the road; determining a likelihood that roads of the determined road type have a shoulder; and wherein determining whether there is a shoulder on the road is based on the determined likelihood.

10. The method of claim 1, further comprising: determining a leading edge of the oriented cluster; in response to a determination that the estimated location of the sign is not beyond the leading edge of the cluster, increasing the offset so that the estimated location of the sign is beyond the leading edge location.

11. The method of claim 1, further comprising: determining a sign type to which the detected traffic sign belongs; determining whether traffic signs of the sign type are posted on a side of a road, and wherein the estimated offset of the oriented cluster is further based on the sign type.

12. The method of claim 1, further comprising:querying a road database for lane information, and wherein the offset is estimated using a detected lane number for roads for which the road database has the lane information, and wherein the offset is calculated using a heuristic for roads without detectable lanes or without available lane information in the road database.

13. The method of claim 12, wherein the heuristic comprises calculating an average offset for a way type in a specific area for roads for which the road database does not have the lane information.

13. The method of claim 1, wherein the number of lanes for two directions of travel is used to calculate the offset for two-way streets by dividing the total number of detected lanes of the road by two.

14. The method of claim 1, wherein the offset considers the type of the detected sign due to differences in how signs are installed in a geographic region.

15. The method of claim 1, wherein the camera device is installed on the windshield of the vehicle.

16. The method of claim 1, wherein the observations are made during multiple passes by different vehicles.

17. The method of claim 1, further comprising: calculating an angular distance value based on the estimated offset and sine function of halved field of view angle of the camera device; wherein estimating the location of the traffic sign comprises applying the angular distance value along a vector in the angular direction of the oriented cluster.

18. The method of claim 1, further comprising: generating a reference database comprising derived offset values organized according to lane position, road configuration, and sign type; wherein the estimated offset is retrieved from the reference database based on detection parameters including at least one of: a detected lane position, a determined road configuration, and an identified sign type.

19. A system for estimating the location of a traffic sign, the system comprising: at least one processor; at least one memory including computer program code; the at least one memory and the computer program code configured to, with the at least one processor, cause the system to at least: receive metadata related to an observation of a traffic sign in a plurality of frames from a camera device installed at a windshield of a vehicle; create an oriented cluster from a first subset of observations from multiple observations of the traffic sign from different vehicles equipped with a similar camera device, wherein each observation of the multiple observations comprises a location estimate of the edge device for a final detection of the traffic sign in the plurality of frames; calculate an angle of the traffic sign relative to the oriented cluster based on the orientation of the cluster; and estimate a location of the traffic sign based on the angle, an estimated offset and at least one parameter relating to the first subset of observations in the oriented cluster.

20. A computer program product comprising at least one non-transitory computer-readable storage medium having stored thereon computer-executable program code instructions which, when executed by a computer, cause the computer to perform operations for: receiving meta data related to an observation of a traffic sign in a plurality of frames from a camera device installed at a windshield of a vehicle; creating an oriented cluster from a first subset of observations multiple observations of the traffic sign from different vehicles equipped with a similar camera device, wherein each observation of the multiple observations comprises a location estimate of the edge device for a final detection of the traffic sign in the plurality of frames; calculating an angle of the traffic sign relative to the oriented cluster based on the orientation of the cluster; and estimating a location of the traffic sign based on the angle, an estimated offset, and at least one parameter relating to the first subset of observations in the oriented cluster.

Images