Method and system for detection accuracy ranking and vehicle direction - Patents.com

Abstract

A system is disclosed that includes at least one hardware processor for updating a cloud-based road anomaly database, the system receiving information from a vehicle regarding objects detected by one or more sensors of the vehicle, where the information may include a location and / or a size of the detected object, comparing the location and / or size of a first detected object to data stored in a road feature database regarding the first detected object, determining accuracy scores associated with the first detected object and a second detected object based on comparing the locations and / or sizes, and updating the cloud-based road anomaly database with the received information about the second detected object based on the associated accuracy scores being higher than at least one of a threshold accuracy score or an accuracy score associated with information of a corresponding object stored in the cloud-based road anomaly database.

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION The embodiments described herein are generally directed to connected vehicle technology and the sharing of detected road hazards. [Background technology]

[0002] According to the National Highway Traffic Safety Administration (NHTSA), there are over 5 million crashes on American roads each year, resulting in over 30,000 deaths and many more serious injuries. Many deaths in automobile accidents are caused by poor road conditions, such as faulty road design, potholes, debris, cracks in the road, construction or work zones, and other such hazards. Over the years, many technologies have been developed to improve driving safety, such as airbags, antilock brakes, lane keeping, automatic emergency braking, and other such technologies. While these passive and semi-active systems and technologies may improve driving safety and reduce fatalities, their limited response time and detection range may limit the protection they provide to drivers by focusing on localized areas. Modern vehicles may be equipped with increasingly advanced sensors, such as stereo cameras, radar, and surround-view cameras, as well as numerous engine control units (ECUs) and onboard chips with ever-increasing computing power. These advanced and powerful ECUs allow for the processing of more sensor data and therefore provide additional safety features to avoid collisions, e.g., lane keeping, emergency braking, adaptive cruise control, etc. In addition, some information can also be shared to cloud-based platforms (e.g., vehicle-to-cloud (V2C)), other vehicles (V2V), pedestrians (V2P), road infrastructure (V2I), or in a broader definition, vehicle-to-everything (V2X).

[0003] Internet of Things (IoT)-based solutions for connected vehicles have been proposed to enable automakers to build and run applications that collect, process, and analyze connected vehicle data. Some IoT-based solutions may provide a low-latency, low-overhead, secure platform for managing vehicles and connecting them to a cloud-based infrastructure. Current methods and technologies for connected vehicles appear to focus on basic functions related to connectivity, data sharing, and combining data in a database. However, the methods and technologies may not consider detection accuracy.

[0004] Road hazards, such as potholes, debris, or other anomalies on the road, pose significant risks to driving safety and comfort. Connected vehicle technologies can be implemented to collect and share data about road anomalies detected from vehicles equipped with advanced sensors and algorithms, such as stereo cameras, light detection and ranging (lidar) systems, surround-view cameras, or other such sensors. During implementation, these traditional methods face drawbacks associated with low accuracy, erroneous detection results, and large amounts of similar data about the same road anomaly (e.g., from multiple vehicles).

[0005] The prior art describes performing the basic functions of 1) collecting data regarding hazard detection for incorporation into a platform for visualization or processing, and 2) discriminating between detected hazards using stored map data. However, issues remain regarding the accuracy of the data and mechanisms for sorting and selecting the data.

[0006] Connected vehicle technology has become crucial for improving driving safety and comfort by sharing detected road hazards with others. However, vehicles are equipped with various sets of sensors, and the performance of different sensor suites varies and biases will occur over time. As a result, road hazard information detected and shared from vehicles is mixed with low-accuracy or noisy data. A major challenge is how to ensure that the data shared from vehicles is accurate enough to be added to the road hazard database. Summary of the Invention

[0007] Accordingly, systems and methods are disclosed for updating a cloud-based road anomaly database with data collected from multiple vehicles. Additionally, the disclosed systems and methods may select specific data for specific road features (e.g., static features or dynamic features) from the data collected from the multiple vehicles based on the relative accuracy of the data received from the multiple vehicles.

[0008] In an embodiment, a system for updating a cloud-based road anomaly database is disclosed, the system comprising at least one hardware processor that receives information from a vehicle regarding objects detected by one or more sensors of the vehicle, the information including at least one of a position or a size of the detected object, compares at least one of the position or the size of a first detected object to data stored in a road feature database regarding the first detected object, determines accuracy scores associated with the first detected object and a second detected object based on comparing the at least one of the positions or the sizes, and updates the cloud-based road anomaly database with the received information about the second detected object based on the associated accuracy score being higher than at least one of a threshold accuracy score or an accuracy score associated with information of a corresponding object stored in the cloud-based road anomaly database.

[0009] In some embodiments, a method for updating a cloud-based road anomaly database is disclosed, the method including: receiving information from a vehicle regarding a first object and a second object detected by one or more sensors of the vehicle, the information including at least one of a position or a size of the detected objects; comparing at least one of the position or the size of the first detected object to data stored in a road feature database regarding the first detected object; determining an accuracy score associated with the first detected object and the second detected object based on comparing the at least one of the positions or the sizes; and updating the cloud-based road anomaly database with the received information about the second detected object based on the associated accuracy score being higher than at least one of a threshold accuracy score or an accuracy score associated with information of a corresponding object stored in the cloud-based road anomaly database.

[0010] In some embodiments, a non-transitory computer-readable medium storing a program for updating a cloud-based road anomaly database is disclosed, which, when executed by at least one hardware processor, receives information from a vehicle regarding a first object and a second object detected by one or more sensors of the vehicle, the information including at least one of a position or a size of the detected objects, compares at least one of the position or the size of the first detected object to data stored in a road feature database regarding the first detected object, determines accuracy scores associated with the first detected object and the second detected object based on comparing the at least one of the positions or the sizes, and updates the cloud-based road anomaly database with the received information about the second detected object based on the associated accuracy scores being higher than at least one of a threshold accuracy score or an accuracy score associated with information of a corresponding object stored in the cloud-based road anomaly database.

[0011] In some embodiments, a system for updating a cloud-based road anomaly database is disclosed, including: means for receiving, from a vehicle, information regarding a first object and a second object detected by one or more sensors of the vehicle, the information including at least one of a position or a size of the first detected object and at least one of a position or a size of the second detected object; means for comparing the position or the size of the first detected object with data stored in a road feature database regarding the first detected object; means for determining an accuracy score associated with the first detected object and the second detected object based on comparing the positions or the sizes; and means for updating the cloud-based road anomaly database with the received information about the second detected object based on the associated accuracy score being higher than at least one of a threshold accuracy score or an accuracy score associated with information of a corresponding object stored in the cloud-based road anomaly database.

[0012] In some aspects, the system, method, or computer-readable medium may further include, or be configured to: determine a route for the vehicle; identify a set of information of at least one object stored in the cloud-based road anomaly database associated with an object along the determined route; predict an accuracy score for information from the vehicle related to the object along the determined route; and send a sensor configuration instruction to the vehicle based on the predicted accuracy score. In some aspects, the instruction for sensor configuration includes a high-sensitivity configuration if the predicted accuracy score is higher than the accuracy score for the information stored in the cloud-based road anomaly database associated with the object along the determined route, and a low-sensitivity configuration if the predicted accuracy score is lower than the accuracy score for the information stored in the cloud-based road anomaly database associated with the object along the determined route.

[0013] In some embodiments, the system, method, or computer-readable medium may also include, or be configured to receive, from the vehicle, information regarding one or more additional objects detected by one or more sensors of the vehicle, where the information includes at least one of a position or a size of the detected one or more additional objects. Determining an accuracy score associated with the detected objects may further include determining an accuracy score based on the received information regarding the one or more additional objects, where an accuracy score may be associated with the detected object and each of the detected one or more additional objects. In some embodiments, the accuracy score includes an accuracy score for at least one of a first direction, a second direction, a third direction, a longitude, or a latitude. In some embodiments, the received information regarding the detected objects includes information regarding multiple instances of detecting the object, where the accuracy score is based on the multiple instances.

[0014] In some embodiments, the system, method, or computer-readable medium may further include, or be configured to: receive from the vehicle information regarding additional objects detected by the one or more sensors of the vehicle, the received information regarding the detected additional objects including information regarding a first number of instances of detecting the additional objects; determine that data regarding corresponding objects is not stored in the cloud-based road anomaly database; and add the detected additional objects to the cloud-based road anomaly database based on the first number of instances being greater than a threshold number of detection instances. In some embodiments, the system, method, or computer-readable medium may also include, or be configured to: determine an accuracy score associated with the first number of instances of detecting the additional objects, the adding of the detected additional objects to the cloud-based road anomaly database further based on an accuracy score indicating an accuracy greater than a threshold accuracy.

[0015] The system, method, or computer-readable medium may also, in some aspects, include, or be configured to: receive from the vehicle information regarding additional objects detected by the one or more sensors of the vehicle; determine additional accuracy scores associated with the detected additional objects; determine that data associated with corresponding objects is stored in the cloud-based road anomaly database, the stored data including a stored accuracy score; and not update the cloud-based road anomaly database with the received information about the detected additional objects based on the stored accuracy score being lower than the stored accuracy score.

[0016] In an embodiment, a system for providing data to a cloud-based route generation system to update a cloud-based road anomaly database of the cloud-based route generation system is disclosed, comprising at least one hardware processor that transmits information regarding a trajectory of the vehicle to the cloud-based route generation system, receives from the cloud-based route generation system an indication of at least one sensor configuration associated with at least one corresponding position associated with the trajectory of the vehicle, detects at least one object based on the at least one indicated sensor configuration, and transmits information regarding the at least one object detected by the indicated sensor configuration to the cloud-based route generation system.

[0017] In some embodiments, a method for updating a cloud-based road anomaly database is disclosed, the method including: transmitting information regarding a trajectory of the vehicle to the cloud-based route generation system; receiving from the cloud-based route generation system an indication of at least one sensor configuration associated with at least one corresponding position associated with the trajectory of the vehicle; detecting at least one object based on the at least one indicated sensor configuration; and transmitting information regarding the at least one object detected by the indicated sensor configuration to the cloud-based route generation system.

[0018] In some embodiments, a non-transitory computer-readable medium storing a program for updating a cloud-based road anomaly database is disclosed, which, when executed by at least one hardware processor, transmits information related to a trajectory of the vehicle to the cloud-based route generation system, receives from the cloud-based route generation system an indication of at least one sensor configuration associated with at least one corresponding position related to the trajectory of the vehicle, detects at least one object based on the at least one indicated sensor configuration, and transmits information related to the at least one object detected by the indicated sensor configuration to the cloud-based route generation system.

[0019] In some embodiments, a system for updating a cloud-based road anomaly database is disclosed, the system including: means for transmitting information regarding a trajectory of the vehicle to the cloud-based route generation system; means for receiving from the cloud-based route generation system an indication of at least one sensor configuration associated with at least one corresponding position associated with the trajectory of the vehicle; means for detecting at least one object based on the at least one indicated sensor configuration; and means for transmitting information regarding the at least one object detected by the indicated sensor configuration to the cloud-based route generation system. [Brief explanation of the drawings]

[0020] The details of the present invention, both as to its structure and operation, may be gleaned in part by study of the accompanying drawings, in which like reference numerals refer to like parts. [Figure 1] FIG. 1 illustrates a vehicle system according to an exemplary implementation. [Figure 2] FIG. 2 illustrates multiple vehicle systems and a management device according to an exemplary implementation. [Figure 3] FIG. 3 shows a system for updating a cloud-based road anomaly database. [Figure 4] FIG. 4 illustrates an exemplary implementation of the inputs and outputs of a data processing module. [Figure 5] FIG. 5 illustrates a set of fusion algorithms that may be used in some implementations of the present invention. [Figure 6] Figure 6 illustrates a set of deep learning-based fusion algorithms. [Figure 7] Figure 7 shows an exemplary object detection and road segmentation architecture via a fully convolutional neural network (FCNN) for camera and lidar data fusion. [Figure 8] Figure 8 shows the accuracy score calculation. [Figure 9] FIG. 9 illustrates an exemplary method for updating a dynamic cloud-based road anomaly (or hazard) database. [Figure 10] FIG. 10 illustrates an exemplary method for determining whether a detected hazard received from a vehicle is already included in the road hazard database. [Figure 11] FIG. 11 illustrates a comparison and sorting algorithm that may be applied to update the road hazard database for particular detected objects detected by multiple vehicles and provided to the cloud component. [Figure 12]FIG. 12 illustrates a method for improving accuracy associated with road hazards stored in a road hazard database by instructing one or more vehicles at locations where they should attempt to acquire more accurate data associated with the stored road hazards. [Figure 13] FIG. 13 illustrates a system for improving accuracy associated with road hazards stored in a road hazard database by instructing one or more vehicles at locations where they will attempt to acquire more accurate data associated with the stored road hazards. [Figure 14] FIG. 14 shows an exemplary route display and corresponding areas where minimum and optimum sensor configurations may be triggered by the system of FIG. 12 employing the method shown in FIG. [Figure 15] FIG. 15 illustrates a method for updating a cloud-based road anomaly (hazard) database. [Figure 16] FIG. 16 illustrates a method for updating road hazard data in a cloud-based road anomaly database. [Figure 17] FIG. 17 illustrates an exemplary computing environment having an exemplary computing device suitable for use with some exemplary implementations. DETAILED DESCRIPTION OF THE INVENTION

[0021] This application discloses an embodiment for updating a cloud-based road anomaly database using data collected from multiple vehicles. After reading this specification, it will become apparent to those skilled in the art how to implement cloud-based road anomaly database updating in various alternative embodiments and applications. However, while various embodiments and applications are described herein, it is understood that these embodiments and applications are presented by way of example and illustration only, and not limitation. Therefore, this detailed description should not be construed as limiting the scope or breadth of the invention as set forth in the appended claims. In addition, the exemplary features and functions described herein may be utilized alone or in combination with other features and functions in various embodiments and may be implemented through any means now known or developed in the future. Furthermore, while the processes described herein may be shown with a particular arrangement and order of subprocesses, each process may be implemented with fewer, more, or different subprocesses, and with a different arrangement and / or order of the subprocesses. It should also be understood that any subprocess that is not dependent on the completion of other subprocesses may be performed before, after, or in parallel with other independent subprocesses, even if the subprocesses are described or illustrated in a particular order.

[0022] 1 illustrates a vehicle system according to an exemplary implementation. Specifically, FIG. 1 illustrates an exemplary human-operated vehicle system configured to operate in human-operated and autonomous modes. An automated driving / advanced driver assistance system (AD / ADAS) ECU 1 is connected to and receives signals from a map positioning unit 6. These signals represent planned routes, map data, the vehicle's position on the map, the vehicle's direction, lane information such as the number of lanes, speed limits, and the type of road / vehicle location (e.g., highway and local road, service road, toll booth, parking lot, or garage, etc.).

[0023] The vehicle is provided with a driving parameter measurement unit for measuring the numerical values ​​of parameters indicative of the driving state of the vehicle, which may include a wheel speed measurement device 7 and a vehicle behavior measurement device 8. Signals provided by these devices are sent to the AD / ADAS ECU 1. The vehicle behavior measurement device 8 measures longitudinal acceleration, lateral acceleration, and yaw rate.

[0024] The vehicle is provided with environmental condition measuring devices for measuring the state of the environment around the vehicle, including a front camera 10f, a front radar 11f, a rear camera 10r, a rear radar 11r, a left front camera 12L, a right front camera 12R, a left rear camera 13L, and a right rear camera 13R. These environmental condition measuring devices transmit information about lane marks, obstacles, and asymmetric signs around the vehicle to the AD / ADAS ECU 1.

[0025] The vehicle camera may be a surround-eye camera or other type of camera depending on the desired implementation. In the vehicle camera system, the front camera 10f includes an imaging unit for capturing images of one or more asymmetrical signs around the vehicle and an output unit for providing a signal indicating the positional relationship between the vehicle and the one or more asymmetrical signs. The front radar 11f detects and locates other vehicles and pedestrians and provides a signal indicating the positional relationship between the vehicle and these objects. The rear camera 10r, the left front camera 12L, the right front camera 12R, the left rear camera 13L, and the right rear camera 13R are similar in function to the front camera 10f, the front radar 11f, and the rear radar 11r.

[0026] The vehicle is provided with an engine 21, an electronically controlled brake system 22, an electronically controlled differential 23, and an electronically controlled steering system 24. The AD / ADAS ECU 1 provides drive signals to actuators included in these systems 22, 23, and 24 based on the value of a control variable provided by the driver and / or environmental conditions, such as the detection of an asymmetrical sign or the activation of various autonomous modes for the vehicle systems described herein. When the vehicle needs to accelerate, the controller 1 provides an acceleration signal to the engine 21. When the vehicle needs to decelerate, the controller provides a deceleration signal to the electronically controlled brake system 22. When the vehicle needs to turn, the AD / ADAS ECU 1 provides a turn signal to at least one of the electronically controlled brake system 22, the electronically controlled differential 23, and the electronically controlled steering system 24.

[0027] The electronically controlled brake system 22 is a hydraulic brake system capable of controlling the individual braking forces applied to each wheel. The electronically controlled brake system applies braking forces to either the right or left wheels in response to a turning request, thereby applying a yawing moment to the vehicle. The electronically controlled differential mechanism 23 drives an electric motor or clutch in response to a turning request to generate a torque difference between the right and left axles, thereby applying a yawing moment to the vehicle. The electronically controlled steering system 24 is, for example, a steer-by-wire steering system capable of correcting the steering angle independently of the turning angle of the steering wheel in response to a turning request, thereby applying a yawing moment to the vehicle.

[0028] The vehicle is provided with an information output unit 26. The information output unit 26 displays an image showing information about the assistance operation, generates sound, and turns on a warning light depending on the type of driving assistance operation. The information output unit 26 is, for example, a monitor with a built-in speaker. Multiple information output units may be installed in the vehicle.

[0029] While the system shown in Figure 1 is an exemplary implementation of the vehicle systems described herein, other configurations are possible and are within the scope of exemplary implementations, and the present disclosure is not limited to the configuration shown in Figure 1. For example, a camera may be mounted on the top or roof of a vehicle for the purpose of detecting asymmetrical markings located on walls, road signs, billboards, etc.

[0030] 2 illustrates multiple vehicle systems and a management device according to an exemplary embodiment. One or more vehicle systems 101-1, 101-2, 101-3, and 101-4, as described with respect to FIG. 1, are communicatively coupled to a network 100 connected to a management device 102. The management device 102 manages a database 103 containing aggregated data feedback from the vehicle systems in the network 100. In an alternative exemplary implementation, data feedback from the vehicle systems 101-1, 101-2, 101-3, and 101-4 can be aggregated in a central repository or database, such as a proprietary database that aggregates data from systems such as an enterprise resource planning system, and the management device 102 can access or retrieve data from the central repository or database. Such vehicle systems can include human-operated vehicles, such as cars, trucks, tractors, vans, etc., depending on the desired implementation.

[0031] The management device 102 may be configured to receive position information from the vehicle systems 101-1, 101-2, 101-3, and 101-4, which transmit the positions of the corresponding vehicle systems relative to the markers, as described in FIG. 4(a), and may be further configured to transmit instructions to the vehicle systems 101-1, 101-2, 101-3, 101-4 indicating the vehicle trajectory to the next marker, and / or to adjust the operating mode, position, speed, and / or orientation.

[0032] FIG. 3 illustrates a system for updating a cloud-based road anomaly database. The system may include a vehicle side 310 mounted on a set of one or more vehicles (e.g., vehicle 318). In some embodiments, vehicle 318 may capture at least one image 311 from an associated camera. Vehicle 318 may also capture additional data from additional sensors (e.g., a stereo camera, a lidar system, a surround-view camera, or other such sensors) regarding one or more features (e.g., features 1-5). The captured image 311 (and other sensor data) may be provided to a data processing module 312 (e.g., using one or more of probabilistic, statistical, knowledge-based, evidential reasoning, or other data fusion algorithms or methods) to identify one or more road features or hazards (e.g., features 1-5). For example, vehicle 318 may capture image 315 and identify road features and hazards in a detection item list 314 (e.g., including pothole 316) and a set of associated data. The data associated with the identified road features and hazards may include size in one or more directions. The data associated with the identified road features and hazards may further include the location (e.g., latitude and longitude) of the road feature or hazard. The identified road features or hazards may be transmitted as results 319 to the cloud component 320.

[0033] The cloud component 320 may receive results 319 from a particular vehicle (e.g., vehicle 318). The results 319 may include detection results 321 for a set of road features (e.g., entries 1-4) and hazards (e.g., entry 5). The cloud component 320 may request 329 benchmark data points from a static road database 330 that stores data 332 for static road features (e.g., entries 1-4 of the detection results 321). In some aspects, accuracy (e.g., mean square error (MSE) 324) may be calculated for one or more components of the data (e.g., size, latitude, and longitude in each of three orthogonal directions). For example, for the first (“x”) direction component, the detection data points and corresponding set of benchmark data points 322 may be used to calculate the accuracy of the detection data (e.g., calculate an accuracy score). The MSE 324 (or other measure of accuracy) may be calculated for the x direction based on mathematical formula 323. The MSE 324 (or other accuracy measure / score) may be associated (e.g., in data structure 325) with a particular road feature or hazard (e.g., a dynamic road feature or hazard). In some aspects, the accuracy score is used to determine whether to add the identified road feature or hazard as an entry (or set of entries) 342 to dynamic map database 340 or whether to update an existing entry or set of entries in dynamic map database 340. If cloud component 320 determines to update dynamic map database 340, cloud component 320 may make an update request 339 to dynamic map database 340. In some aspects, dynamic map database 340 may associate the detected hazard with a map 344.

[0034] In some aspects, the primary function of the vehicle side (e.g., vehicle side 310 in FIG. 3 ) is to fuse information captured from various on-board sensors and provide detection results (e.g., results 319 in FIG. 3 ) to a cloud component (e.g., cloud component 320 in FIG. 3 ). The detection results may include results in different categories (e.g., road features and road hazards) with associated dimensions (and location-related GPS locations of results). To achieve this fusion, one or more different fusion algorithms (e.g., artificial intelligence (AI) or deep neural network (DNN) models) may be utilized by the data processing module, with data from the various sensors being fed into one or more different fusion models. The one or more different fusion models, in some aspects, may output a list of detection results with at least one of type, size, and / or location information associated with each detected object.

[0035] FIG. 4 is a diagram 400 illustrating an example implementation of inputs and outputs of a data processing module. Inputs 410 from sensors may include one or more of a stereo camera image input 411, a radar data input 412, a lidar data input 413, a 360° view camera image input 414, an inertial measurement unit (IMU) input 415, and a global positioning system (GPS) input 416. The inputs 410 from sensors may be provided to a data processing module 420. The data processing module 420 may include a fusion algorithm / method for incorporating different types of inputs. The fusion algorithm may include one or more of the probabilistic, statistical, knowledge-based, evidential reasoning, or other data fusion algorithms or methods shown in FIG. 5. Alternatively, the data processing module 420 may employ a deep learning-based fusion algorithm / method, including a recurrent neural network or a convolutional neural network (e.g., including a one-stage detector and a two-stage detector), as shown in FIG. 6.

[0036] The data processing module 420, in some embodiments, may generate (or identify) a set of road features 432 and a set of road hazards 434. The set of road features 432 and the set of road hazards 434 may be combined into an output list of detected features and hazards 440. The output list of detected features and hazards 440 may include a set of road features, such as license plates 441, lane markers 442, lane widths 443, road signs 444, and traffic lights 445. Some road features may be associated with a location (e.g., fixed features such as road signs 444, traffic lights 445, or other signs), while other road features may not be associated with a location (e.g., non-fixed features such as license plates 441 or features where location is not important, such as lane markers 442 or lane widths 443). In some embodiments, identified road hazards, such as potholes 446, debris 447, and bumps 448, may be associated with a location (e.g., longitude and latitude).

[0037] FIG. 5 is a diagram 500 illustrating a set of fusion algorithms 510 that may be used in some implementations of the present invention. The set of fusion algorithms 510 may include one or more of a probabilistic approach 520, a statistical approach 530, a knowledge-based approach 540, and an evidential reasoning approach 550. The probabilistic approach 520 may, in some embodiments, include a Bayesian network, least squares estimation, state space modeling, or other similar probabilistic approach. In some embodiments, the statistical approach 530 may include cross-covariance, covariance crossover, or other statistical approach. The knowledge-based approach 540 may, in some embodiments, include an artificial neural network (ANN), fuzzy logic, a genetic algorithm, a particle swarm, or other knowledge-based approach. In some embodiments, the evidential reasoning approach 550 may include a recursive operator, a combination rule, a Dempster-Shafer algorithm, or other evidential reasoning approach.

[0038] FIG. 6 is a diagram 600 illustrating a set of deep learning-based fusion algorithms 610. The set of deep learning-based fusion algorithms 610 may include one or more of a convolutional neural network (CNN) 620 or a recurrent neural network (RNN) 630. The CNN 620 may include a one-stage detector 640, such as a region-based CNN (R-CNN), a spatial pyramid pooling (SPP)-net, fast R-CNN, faster R-CNN, or other one-stage detector CNN. The CNN 620 may also include a two-stage detector 650, such as a You Only Look Once (YOLO), a single-shot detector (SSD), a deconvolutional SSD (DSSD), or other two-stage detector CNN. In some embodiments, the RNN 630 may be a long short-term memory (LSTM), a gated recurrent unit (GRU), or other RNN. In some embodiments, the deep learning-based fusion algorithm aims to mimic the function of the human brain. Deep learning-based fusion algorithms can be a subdivision of AI and ML, and can also be considered an improvement over neural networks.

[0039] In some embodiments, cameras may output bounding boxes, lane line positions, traffic light and sign status, etc., and can be the primary sensor for high-resolution tasks. However, cameras are 2D sensors that lack depth information. Stereo cameras utilize two or more lenses with separate image sensors for each lens to simulate human binocular vision, providing the ability to capture 3D dimensional images (e.g., disparity maps). Radar sensors use frequency-modulated continuous waves (FMCW) to reliably detect moving or stationary targets, and lidar sensors utilize pulsed lasers to calculate the variable distance of an object based on the return time of reflections.

[0040] Generally, various sensor combinations are employed to perform two main tasks: 1) environmental recognition—cameras, radar, and lidar; and 2) localization—IMUs, GPS, or other sensors. However, each type of sensor has its own advantages and disadvantages. Therefore, different sensor combinations can overcome or compensate for the disadvantages of a single sensor and produce more reliable results. To produce more reliable results, the data processing module may fuse data from multiple sensors using a fusion algorithm. For example, camera and lidar data fusion may be better for pedestrian and road detection, camera and radar data fusion may be better for vehicle detection and lane detection, camera and IMU may be better for SLAM (simultaneous localization and mapping), GPS and inertial navigation systems (INS) may be better for navigation, and fusion of maps, cameras, GPS, and INS may be better for self-positioning.

[0041] Based on the onboard sensors, different fusion algorithms can be selected and adjusted to obtain the best fusion results. FIG. 7 shows an example object detection and road segmentation architecture via a fully convolutional neural network (FCNN) 700 for camera and lidar data fusion. The FCNN 700 may receive as input camera image data 710 and lidar depth map 720, both of which have the same dimensionality (natively or based on preprocessing). The camera image data 710 and lidar depth map 720 may be processed through a set of one or more convolutional layers (e.g., convolutional layer 730) followed by a max-pooling layer 740. Additionally, the FCNN may include a skip convolutional layer 750 and a transposed convolutional layer 760. Based on the processing, the FCNN may generate an output 770.

[0042] Table 1 below shows examples of road features and road hazard detection results. [Table 1]

[0043] As described above in connection with FIG. 3 , each detected road feature or road hazard may be associated with measurements related to one or more directions (e.g., x, y, and z) and location. In some embodiments, a particular direction may be associated with data related to the type of detected road feature. For example, the z-direction of a detected license plate (or other sign) may indicate a type of license plate (or sign) that can identify a set of benchmark measurements for comparison with a set of measurements associated with a detected object. In some embodiments, vehicle plate dimensions vary from country to country and from small motorcycles to large trucks. For example, in the United States, a standard full-size regular vehicle plate measures 6 inches by 12 inches, while for small vehicle plates (e.g., for motorcycles or golf carts), the standard plate is 4 inches by 7 inches. Cells in the z-dimension may indicate a type of vehicle plate and / or a country (or a type of road feature or sign, such as a traffic light, stop sign, yield sign, or other road sign).

[0044] FIG. 8 illustrates accuracy score calculation 800. Accuracy score calculation 800 may start on the vehicle side 810 using a road feature and hazard detection module 812 to detect road features and hazards. The detected road features and hazards may be provided to a road feature classification module 814 to identify the nature of the detected road features and hazards (objects). For example, the road feature classification module 814 may identify the detected objects as either (1) road features such as vehicle license plates, traffic lights, road signs, or other non-hazardous objects, or (2) road hazards such as potholes, debris, bumps, or other objects that present (dynamic) road hazards (e.g., road hazards that may be temporary). The road feature classification module 814 may provide the detected and classified road features and hazards to a road feature size measurement module 816. The road feature size measurement module 816 may identify the measured size of the identified feature in one or more dimensions (e.g., direction) and associate the measured size with the identified feature.

[0045] For example, on-board sensors may be installed to capture surrounding environment information. The captured information may then be provided to one or more data processing algorithms (e.g., implemented by road feature and hazard detection module 812) to detect road features (e.g., road signs, traffic lights, lane markers, vehicle plates, and other such road features) and road hazards (e.g., potholes, debris, bumps, thin ice, objects on the road, etc.). The one or more data processing algorithms may utilize one or more different deep learning architectures (e.g., those shown in FIG. 6 ) or a combination of one or more traditional data processing algorithms (e.g., those shown in FIG. 5 ) and one or more deep learning algorithms. Although road feature and hazard detection module 812 is depicted in FIG. 8 as being deployed on vehicle-side 810, in some aspects, road feature and hazard detection module 812 may be deployed in a cloud component where input is sent with little or no preprocessing (e.g., compression without analysis). From one or more data processing algorithms, each road feature and hazard may be associated with a size in each direction (e.g., x, y, and z) and a GPS location. As shown in Figure 8, different road features may be associated with different size and location fields / data. For example, a traffic light may be associated with x, y, and z dimensions / fields and a GPS location field, a road sign may be associated with x and y dimensions / fields and a GPS location field, and a vehicle plate may be associated with x and y dimensions / fields but not a GPS location field.

[0046] The vehicle side 810 (more specifically, the road feature size measurement module 816) may provide a set 821 of detected road features, including size measurements and locations (e.g., latitude and / or longitude), to the cloud component 820 for accuracy score calculation. Different messaging protocols, such as MQ Telemetry Transport (MQTT), Constrained Application Protocol (CoAP), or other messaging protocols, may be used for data sharing between the vehicle (or edge) side and the cloud component. In some aspects, information about detected road hazards is not used to calculate the accuracy score, but the transmitted data may include information about the road hazards (e.g., their dimensions and location data) along with vehicle identification (VIN) numbers for identifying different vehicles (e.g., to identify an accuracy score associated with a particular vehicle).

[0047] The cloud component may obtain benchmark data from the road feature database 823 and use the received set of detected road features 821 and the obtained benchmark data to calculate an accuracy score using the accuracy score calculation module 824. For example, based on GPS location information associated with each road feature, road feature data associated with a corresponding (e.g., the same) road feature may be extracted or obtained from a static map database (e.g., static road database 330 of FIG. 3 ). The accuracy score (or calculated accuracy) may be calculated based on the MSE (e.g., Equation 1) associated with each direction. In some aspects, each detected feature in the set of detected road features 821 may include multiple data points associated with the same detected road feature, and a set of MSEs 825 (or other accuracy measure) may be calculated for each dimension associated with each detected road feature in the set of detected road features 821 based on the benchmark data obtained from the road feature database 823. The set of MSEs 825 may then be used to generate a set of accuracy scores for each dimension in the set of accuracy scores 827. In some aspects, the accuracy score for each dimension in the set of accuracy scores 827 is calculated as the average of the MSEs associated with that dimension.

[0048]

number

[0049] For example, Table 2 shows a set of detected features and associated benchmarks. Based on a comparison of the detected features to the benchmarks, the cloud component may calculate the MSE for each dimension (e.g., x, y, z, latitude, and longitude), as shown in the last row of Table 2. [Table 2]

[0050] The calculated accuracy score (e.g., MSE) may be associated with road hazards detected by the same vehicle. Table 3 shows a set of detected road hazards associated with calculated accuracy scores (e.g., MSE). [Table 3]

[0051] In some aspects, accuracy score calculation 800 may include a predictive analytics layer that generates a set of road hazards with associated size, location, and accuracy data based on the operations described above. In some aspects, the output of the predictive analytics layer (e.g., accuracy score calculation 800) may be provided to a prescriptive analytics layer for (1) updating a dynamic map database and (2) generating a vehicle trigger signal.

[0052] 9 shows an example method 900 for updating a dynamic cloud-based road anomaly (or hazard) database. Method 900 may be performed by a prescriptive analytics layer of a cloud component, e.g., a prescriptive analytics program implemented on a processor associated with the cloud component, or more generally, by a cloud component (e.g., cloud component 320 or 820). At 902, the cloud component may receive new hazard information from a particular vehicle (e.g., identified by a VIN number) that detects (or does not detect) a hazard. The received new hazard information may include the type of hazard, location (e.g., latitude and longitude or other location identifier), and size associated with the detected hazard. The new hazard information may further include the number of times the hazard was detected.

[0053] At 904, the cloud component may determine whether the number of detections of the detected hazard is greater than a set threshold number of detections. In some embodiments, the determination at 904 identifies false positives. For example, a vehicle typically detects a hazard within a certain distance, e.g., 15 meters, and continues to track or detect the road hazard approximately 5-10 times per second until the vehicle passes the hazard or the hazard moves out of the detection range of an onboard sensor. However, this continuous detection pattern may be missed due to false positives. Therefore, by setting a threshold number of detections, false positives may be significantly reduced.

[0054] At 906, the cloud component may determine an accuracy (or accuracy score) associated with the new hazard information. For example, with reference to FIG. 8, the cloud component 820 may calculate an accuracy score for the detected hazard based on other road feature information or data. The accuracy score may be calculated based on other road features detected contemporaneously or historically by the same vehicle and benchmark data in a static map database (e.g., road feature database 823). The calculated accuracy may include the accuracy associated with each type of data (e.g., size, latitude, or longitude in one or more directions) for the detected hazard.

[0055] At 908, the cloud component may determine whether the detected hazard received from the vehicle is already included in the road hazard database. FIG. 10 shows an example method 1000 for determining whether the detected hazard received from the vehicle is already included in the road hazard database. At 1002, the cloud component may extract location data for the received new hazard information. The location may be GPS data (e.g., including latitude and longitude) or other location data used to identify a location in the road hazard database. Based on the location data extracted at 1002, the cloud component may determine a location range for searching the road hazard database at 1004. The location range for searching the road hazard database may include a first range of latitude values ​​and a second range of longitude values ​​based on the location data extracted at 1002, and a threshold distance set for each of the latitude and longitude such that the first (second) range is from the latitude (longitude) of the detected hazard minus a set threshold to the latitude (longitude) of the detected hazard plus a set threshold.

[0056] To determine 908 whether the detected hazard is already included in the road hazard database, the cloud component may access the road hazard database (e.g., dynamic map database 340) to determine 1006 whether any hazard data is stored at the same location as the location of the detected hazard received from the vehicle (e.g., whether any hazard data is stored at a latitude and longitude within a first and second range, respectively). If the cloud component determines 1006 that hazard data is stored at the same location as the location of the detected hazard, the cloud component may determine 1008 whether the stored road hazard is of the same type as the detected hazard, e.g., a pothole, debris, bump, or other road hazard type. If the stored road hazard is determined 1008 to not be of the same type, the cloud component may determine 908 that the detected hazard is a new hazard not stored in the road hazard database, and if the stored road hazard is determined 1008 to be of the same type, the cloud component may determine 908 that the detected hazard is already stored in the road hazard database.

[0057] If it is determined at 908, 1006, or 1008 that the new hazard does not already exist in the road hazard database, the cloud component may determine at 910 whether the accuracy (or accuracy score) is higher than a set threshold accuracy. In some aspects, the set threshold accuracy may be determined based on a level of accuracy that may be useful when attempting to avoid the hazard (e.g., accurately within a lane of a road). The accuracy associated with the detected hazard on which the determination at 910 is based may be the lowest accuracy (e.g., highest associated MSE) associated with a particular type of information (e.g., size in any one or more directions, or location). In some aspects, the accuracy associated with the detected hazard is an average of the accuracies associated with each of different dimensions or components of the location data. The accuracy associated with the detected hazard may, in some aspects, be the accuracy associated with a particular type of data that is deemed more relevant to identifying the hazard (e.g., identifying the location of a hazard within one lane may be more important than determining the size of the hazard within one foot). In some aspects, different set accuracy thresholds may be configured for different types of data to account for different usefulness thresholds for different types of data.

[0058] If the accuracy associated with the detected hazard is determined to be less than the threshold accuracy at 910 (e.g., if the MSE associated with the detected hazard is higher than the threshold MSE), the cloud component may discard information about the detected hazard at 912. Alternatively, if the accuracy associated with the detected hazard is determined to be greater than the threshold accuracy at 910 (e.g., if the MSE associated with the detected hazard is less than the threshold MSE), the cloud component may add information about the detected hazard to the hazard database as a new road hazard entry at 914. For example, with reference to FIG. 3 , the cloud component 320 may perform a prescriptive analysis of the detected hazard information received from the vehicle side 310 to determine whether it meets a minimum threshold accuracy, and make an update request 339 to the dynamic map database 340 if the detected hazard information meets the minimum accuracy threshold.

[0059] If the cloud component determines at 908 (or 1008) that the detected hazard received from the vehicle is already included in the road hazard database, the cloud component may determine at 920 whether the accuracy associated with the detected hazard is higher than the accuracy associated with the stored data. The accuracy associated with the detected hazard on which the determination at 920 is based may be the lowest accuracy (e.g., highest associated MSE) associated with a particular type of information (e.g., size in any one or more directions, or location). In some aspects, the accuracy associated with the detected hazard (and stored information) is an average of the accuracy associated with each of different dimensions or components of the location data. The accuracy associated with the detected hazard may, in some aspects, be the accuracy associated with a particular type of data deemed more relevant to identifying the hazard (e.g., identifying the location of the hazard within one lane may be more important than determining the size of the hazard within one foot). In some aspects, the cloud component may determine at 920 whether the accuracy associated with each component of the detected hazard (e.g., each direction of size, latitude, or longitude) is higher than the accuracy associated with the corresponding component of the stored hazard data.

[0060] If the cloud component determines at 920 that the accuracy associated with the detected hazard is less than the accuracy associated with the stored hazard data, then at 922 the cloud component may discard the information about the detected hazard and maintain the stored data about the hazard. On the other hand, if the cloud component determines at 920 that the accuracy associated with the detected hazard (or associated with one or more components of the information about the detected hazard) is higher than the accuracy associated with the stored hazard data (or associated with one or more components of the stored information corresponding to the detected hazard), then the cloud component may update the road hazard database at 924 with the detected hazard data (or the components associated with the higher accuracy). By comparing the accuracy (or accuracy scores), the cloud component can update the road hazard database with the best available information (in real time). In some aspects, updating the road hazard data at 924 may include deleting data from the road hazard database based on new hazard information received at 902 that identifies the absence of a hazard at the location of the hazard in the road hazard database (e.g., identifies an object in the environment that indicates the absence of a hazard). For example, referring to FIG. 3 , the cloud component 320 may perform a prescriptive analysis of the detected hazard received from the vehicle side 310 to determine whether it is associated with greater accuracy than existing data stored in the dynamic map database 340, and if the detected hazard is associated with greater accuracy than existing data stored in the dynamic map database 340, may make an update request 339 to the dynamic map database 340 to update the data in the dynamic map database 340.

[0061] FIG. 11 illustrates a comparison and sorting algorithm 1100 that may be applied to update a road hazard database for particular detected objects detected by multiple vehicles and provided to a cloud component. It should be understood that the process described in connection with the determination at 920 of FIG. 9 may be a special case of the comparison and sorting algorithm 1100 for detected hazard data received from a single vehicle. The comparison and sorting algorithm 1100 may begin with a data collection phase 1110, during which the cloud component may receive road hazard data from multiple vehicles corresponding to particular road hazards stored in the road hazard database and extract associated new hazard data 1114, as described in connection with the data reception at 902 of FIG. 9 and the data extraction at 1002 of FIG. 10. The data collection phase 1110 may also include querying the road hazard database to obtain database hazard data 1112 for road hazards corresponding to the received new (or newly detected) hazards.

[0062] The comparison and sorting algorithm 1100 may be executed when detected hazard data is received in online mode. Alternatively, the comparison and sorting algorithm 1100 may be executed in offline or batch mode (e.g., periodically or based on a trigger event). For example, the comparison and sorting algorithm 1100 may be configured to run daily, hourly, or every 15 minutes (e.g., daily for potholes, hourly for construction or work zones, and every 15 minutes for debris) depending on the type of road hazard and its expected duration. The trigger event may include, for example, the location of new and stored road hazard locations on which the comparison and sorting algorithm 1100 may be executed, and may include the vehicle querying a road hazard database for road hazards within range of the vehicle's current location.

[0063] In the data aggregation stage 1120, the comparison and sorting algorithm 1100 may aggregate information from two data sets (e.g., database risk data 1112 and new risk data 1114) into three groups. For example, the first group may aggregate data from the x and y dimensions 1122, the second group may aggregate data from the z dimension 1124, and the third group may aggregate location (e.g., GPS) data 1126. While FIG. 11 illustrates aggregating the x and y dimensions together, in other embodiments, each dimension may be aggregated separately. The aggregated data, in some embodiments, includes ordered pairs for each component (e.g., x and y, z, or latitude and longitude) that include a value and accuracy (or accuracy score).

[0064] Once the data is aggregated, the comparison and sorting algorithm 1100 may perform a sorting operation during a data sorting stage 1130. As shown, each group of aggregated data is independently sorted so that data associated with different vehicles can be selected for each group. The sorting algorithm used during the data sorting stage 1130 may include a bubble sort, a merge sort, a binary sort, or any other algorithm capable of outputting elements in an ordered list. For example, sorted x- and y-dimension data 1132 identifies stored data in the x- and y-dimensions (e.g., (x, A_x)(y, A_y)), sorted z-dimension data 1134 identifies z-dimension values ​​associated with a third vehicle (e.g., (z_3, Az_3)), and sorted location data 1136 identifies location data associated with the first vehicle (e.g., (lat_1, Alat_1), (lon_1, Alon_1)). When using multi-component data, such as x- and y-dimensions or location data (e.g., including latitude and longitude), ranking may be based on the lowest (or highest) accuracy value of the multi-component data. In some embodiments, the average accuracy (or accuracy score) may be used to sort the multi-component data. Alternatively, the accuracy of one component may be used for the multi-component data without regard to the accuracy of the other components (e.g., based on the expectation that the x- and y-dimensions or latitude and longitude will have similar accuracy scores).

[0065] During data selection 1140, comparison and sorting algorithm 1100 may select the data identified as having the highest accuracy for inclusion in the road hazard database. For example, the road hazard database may be updated to include the z-dimension value (e.g., (z_3, Az_3)) associated with the third vehicle identified by sorted z-dimension data 1134 and the location (e.g., latitude and longitude) data (e.g., (lat_1, Alat_1), (lon_1, Alon_1)) identified by sorted location data 1136, while the x- and y-dimension values ​​identify the stored data as associated with the highest accuracy and may not be updated based on sorted x- and y-dimension data 1132. Thus, in the illustrated embodiment, the updated stored road hazard data 1142 may be {(x, A_x), (y, A_y), (z_3, Az_3), (lat_1, Alat_1), (lon_1, Alon_1)}.

[0066] FIG. 12 illustrates a method 1200 for improving accuracy associated with road hazards stored in a road hazard database by directing one or more vehicles to locations where they should attempt to acquire more accurate data associated with the stored road hazards. The method may be performed by a cloud component (or a processor executing a program of the cloud component) of a system similar to system 300 of FIG. 3. Additionally, the method may be performed by a system such as that illustrated in FIG. 13. FIG. 13 illustrates a system 1300 for improving accuracy associated with road hazards stored in a road hazard database by directing one or more vehicles to locations where they should attempt to acquire more accurate data associated with the stored road hazards. FIG. 14 illustrates an exemplary route display 1440 and corresponding areas where a minimum sensor configuration (locations 1410 and 1430) and an optimal sensor configuration (location 1420) may be triggered by the system of FIG. 12 employing the comparison and sorting algorithm 1100. Accordingly, method 1200 is described in conjunction with FIGS. 13 and 14, as the case may be.

[0067] At 1202, a cloud component (e.g., cloud component 320 or 1320) may receive trajectory information from a vehicle (e.g., vehicle 1310). The trajectory information may include a current location, a desired destination, and route or driving mode preferences (e.g., no highways, no tolls, or other such user preferences) from which the cloud component may generate a route. In some aspects, the trajectory information may identify a route and may be received in a dynamic road hazard data request for display to a user in a route display (e.g., in route display 1440).

[0068] At 1204, the cloud component may obtain a road hazard list based on the received trajectory information (e.g., route) and a road hazard database. In some aspects, the trajectory information may be used by the cloud component (e.g., vehicle trigger algorithm 1324) to query a road hazard database, such as a dynamic real-time road hazard database 1330 similar to dynamic map database 340, to obtain the road hazard list. The road hazard list may include a list of road hazards within a certain distance of the route, as identified by locations (e.g., latitude and longitude values) associated with the road hazards in the road hazard database.

[0069] At 1206, the cloud component (e.g., vehicle trigger algorithm 1324) may determine whether the road hazard list is empty or, for each road hazard in the road hazard list, whether the accuracy (or accuracy score) associated with the road hazard exceeds a set threshold accuracy (e.g., whether stored road hazard characteristics, such as size and location, have been identified with sufficient accuracy for application). In some aspects, the threshold accuracy is based on a predicted accuracy for information from the vehicle. For example, based on data previously received from the vehicle, the cloud component may determine an accuracy associated with the vehicle and use that accuracy as the threshold accuracy value. If the cloud component (e.g., vehicle trigger algorithm 1324) determines at 1206 either (1) that the road hazard list is empty or (2) that each road hazard in the road hazard list is associated with an accuracy that exceeds a set threshold accuracy, the cloud component may determine at 1208 that a minimal (e.g., low-sensitivity) sensor configuration is sufficient. A minimal sensor configuration may save power in the vehicle (e.g., vehicle 1310) and may save processing power in both the vehicle and cloud components, as less data can be processed in the vehicle and communicated to the cloud component for further processing.

[0070] If the cloud component (e.g., vehicle trigger algorithm 1324) determines at 1206 that the road hazard list is not empty and that at least one component related to at least one road hazard in the road hazard list is associated with an accuracy less than the set threshold accuracy, the cloud component may determine at 1210 the location of the at least one road hazard relative to the vehicle's location. The cloud component may further identify available vehicle sensors to determine which sensors should be triggered at the locations associated with the at least one road hazard. After determining the relative locations of the at least one road hazard relative to the vehicle's location at 1210, the cloud component (e.g., vehicle trigger algorithm 1324) may determine at 1212 that an optimal (e.g., highly sensitive) sensor configuration should be used at the set of locations associated with the at least one road hazard. Determining at 1212 that an optimal sensor configuration should be used may also include identifying an optimal sensor configuration for updating specific road hazard data that is not associated with an accuracy greater than the set threshold accuracy. For example, the location of a road hazard may indicate that an optimal sensor configuration applies to the right (or left) side sensors but not the opposite side sensors, so the vehicle may be instructed to trigger the right side sensors to obtain more accurate information without having to trigger all of the vehicle's sensors, which may conserve power while attempting to capture more accurate data.

[0071] Regardless of whether the cloud component determined at 1208 that a minimum sensor configuration is sufficient or that an optimal sensor configuration should be used at 1212, the cloud component may transmit the determined sensor configuration associated with a different location along the vehicle's trajectory at 1214. For example, with reference to Figures 13 and 14, the cloud component 1320 may transmit a trigger alert 1312 to the vehicle 1310 or 1412 to identify a location, e.g., a trigger region 1420, where an optimal sensor configuration should be used to collect data regarding an object 1421 associated with at least one accuracy below a threshold accuracy. Figure 14 further illustrates that for regions / locations 1410 and 1430, a minimum sensor configuration may be used because no object about which information should be collected has been identified.

[0072] After transmitting the determined sensor configurations for different regions along the trajectory in 1214, trigger processing may end, and the cloud component may receive new road hazard data 1314 and use descriptive analytics module 1326 and predictive analytics module 1328 to determine whether to update a road hazard database, e.g., dynamic real-time road hazard database 1330, as described in connection with FIG. 9. Updating the road hazard database, in some aspects, includes any of adding a road hazard to the road hazard database, updating data of an existing road hazard in the road hazard database, or deleting a road hazard from the road hazard database, as described in connection with operations 914 and 924 of FIG. 9. For example, if data received from one or more vehicles based on the determined sensor configurations for a particular region transmitted in 1214 indicates that a particular road hazard no longer exists, data about the particular road hazard may be deleted from the data in the road hazard database. If fewer hazards are identified in the received data than are identified in the road hazard database (e.g., a set of hazards that does not include the particular hazard), the received data may indicate that the particular hazard no longer exists.

[0073] FIG. 15 illustrates a method 1500 for updating a cloud-based road anomaly (hazard) database. The method may be performed by a cloud component (e.g., cloud component 320 or 1320) of a system for collecting dynamic road hazard data. While the present application refers to a cloud component, the component performing method 1500 or the methods of FIGS. 8-8 or 12 may be any system including a processor capable of executing a program to perform the operations described above. At 1502, the cloud component may receive information from a vehicle regarding a first object and a second object detected by one or more sensors of the vehicle. In some embodiments, as described above, the information regarding each of the first and second objects includes at least one of the position or size of the detected object. For example, referring to Table 1, the received information may include data regarding the size in one or more of three directions and location data, including latitude and longitude, for a traffic light (first detected object) and a pothole (second detected object). In some aspects, the received information regarding the first and second detected objects includes information regarding multiple instances of detecting the first and second objects, and the accuracy score is based on the multiple instances.

[0074] At 1504, the cloud component may compare at least one of the location or size of the first detected object to data stored in a road feature database regarding the first detected object. For example, with reference to FIG. 8, the accuracy score calculation module 824 may compare at least the first road feature to a benchmark stored for that road feature in the road feature database 823. The road feature database may be a static feature database with precision information obtained from an external source.

[0075] At 1506, the cloud component may determine an accuracy score associated with the second detected object based on comparing at least one of the location or size. In some aspects, as shown in FIG. 3, an accuracy score may be calculated based on the first detected object (e.g., a static road feature) and applied to the second detected object (e.g., a dynamic road hazard). In some aspects, multiple detected static road features may be used to determine an accuracy score associated with the detected dynamic road hazard. For example, with reference to FIG. 8, the accuracy score calculation module 824 may determine the accuracy (or accuracy score) of the dataset by comparing at least the first road feature to a stored benchmark for that road feature and applying it to the second detected object.

[0076] At 1508, the cloud component may update the cloud-based road anomaly database with the received information about the second detected object based on the associated accuracy score being higher than at least one of a threshold accuracy score or an accuracy score associated with information about the corresponding object stored in the cloud-based road anomaly database. As described in connection with FIG. 9 and FIG. 11 , the cloud component may identify a corresponding road hazard stored in the road hazard database, compare the accuracy score associated with the detected road hazard to the accuracy score associated with the corresponding road hazard, and determine to update the data stored for the road hazard. In some embodiments, different components of the data about the detected object and the stored corresponding object are compared individually, and the determination to update the road hazard database may include a determination to update fewer than all components of the data stored in the road hazard database, as described in connection with the comparison and sorting algorithm 1100 of FIG. 11 .

[0077] 16 shows a method 1600 for updating road hazard data in a cloud-based road anomaly database. The method may be performed by a vehicle (e.g., vehicle 318, 1310, or 1412). At 1602, the vehicle may transmit information regarding the vehicle's trajectory. As described above, the information regarding the trajectory may include a current location and a destination. The trajectory information may be transmitted to a cloud-based route generation system (e.g., system 1300 of FIG. 13) that includes a cloud component in communication with the static road feature database and the dynamic road hazard database.

[0078] Based on the information transmitted in 1602, the vehicle may receive, from the cloud-based route generation system, an indication of at least one sensor configuration associated with at least one corresponding location in 1604. As described in connection with FIG. 14, the sensor configuration may be one of a minimum sensor configuration or an optimal sensor configuration for detecting road hazards. The sensor configuration may indicate a set of sensors that the vehicle will enable or disable at different locations along the trajectory.

[0079] At 1606, the vehicle may sense at least one object based on the at least one commanded sensor configuration. The at least one detected object may be detected using a minimum sensor configuration or an optimal sensor configuration. Detecting the at least one object at 1606 may include performing a first set of processes using a data processing module, as described in connection with FIGS. 3 through 8, or utilizing other data fusion algorithms if multiple types of sensors are used.

[0080] Finally, at 1608, the vehicle may transmit information about at least one object detected by the indicated sensor configuration to the cloud-based route generation system. The transmitted information may include data regarding size in one or more of three directions and location data including latitude and longitude. The transmitted information may include data for multiple detected objects and multiple instances of detecting the detected object.

[0081] 17 illustrates an exemplary computing environment having an exemplary computing device suitable for use with some exemplary implementations. The computing device 1705 in the computing environment 1700 can include one or more processing units, cores, or processors 1710, memory 1715 (e.g., RAM, ROM, and / or the like), internal storage 1720 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or IO interface 1725, any of which can be coupled to a communication mechanism or bus 1730 for communicating information or can be embedded in the computing device 1705. The IO interface 1725 can also be configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0082] The computing device 1705 can be communicatively coupled to an input / user interface 1735 and an output device / interface 1740. Either or both of the input / user interface 1735 and the output device / interface 1740 can be wired or wireless interfaces and can be detachable. The input / user interface 1735 can include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touchscreen interfaces, keyboards, pointing / cursor control, microphones, cameras, Braille, motion sensors, accelerometers, optical readers, and / or the like). The output device / interface 1740 can include displays, televisions, monitors, printers, speakers, Braille, etc. In some example implementations, the input / user interface 1735 and the output device / interface 1740 can be embedded in or physically connected to the computing device 1705. In other example implementations, other computing devices may function as or provide the functionality of input / user interface 1735 and output device / interface 1740 for computing device 1705 .

[0083] Examples of computing devices 1705 may include, but are not limited to, highly mobile devices (e.g., smartphones, devices in vehicles and other machines, and devices carried by humans and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions, radios with one or more processors embedded and / or connected to one or more processors, etc.).

[0084] Computing device 1705 can be communicatively coupled (e.g., via IO interface 1725) to external storage 1745 and a network 1750 to communicate with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 1705 or any connected computing device can function as, provide services to, or be referred to as, a server, a client, a thin server, a general-purpose machine, a special-purpose machine, or another label.

[0085] IO interface 1725 can include, without limitation, wired and / or wireless interfaces using any communication or IO protocol or standard (e.g., Ethernet, 1502.11x, Universal System Bus, WiMax, modem, cellular network protocols, etc.) for communicating information between at least all connected components, devices, and networks in computing environment 1700. Network 1750 can be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0086] The computing device 1705 can use and / or communicate using computer-usable or computer-readable media, including transitory and non-transitory media. Transitory media include transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, etc. Non-transitory media include magnetic media (e.g., disks and tape), optical media (e.g., CD-ROMs, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage devices), and other non-volatile storage or memory.

[0087] In some exemplary computing environments, the computing device 1705 can be used to implement techniques, methods, applications, processes, or computer-executable instructions. The computer-executable instructions can be obtained from a transitory medium, stored on a non-transitory medium, or obtained from a non-transitory medium. The executable instructions can originate from one or more of any programming language, scripting language, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0088] The processor 1710 can run under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, including a logic unit 1760, an application programming interface (API) unit 1765, an input unit 1770, an output unit 1775, and an inter-unit communication mechanism 1795 through which different units communicate with each other, with the OS, and with other applications (not shown). The described units and elements may vary in design, function, configuration, or implementation and are not limited to the provided description. The processor 1710 can be in the form of a hardware processor, such as a central processing unit (CPU), or a combination of hardware and software units.

[0089] In some example implementations, when information or instructions for execution are received by API unit 1765, it may be communicated to one or more other units (e.g., logic unit 1760, input unit 1770, output unit 1775). In some cases, in some example implementations described above, logic unit 1760 may be configured to control the flow of information between units and direct the services provided by API unit 1765, input unit 1770, output unit 1775. For example, the flow of one or more processes or implementations may be controlled by logic unit 1760 alone or in combination with API unit 1765. Input unit 1770 may be configured to obtain inputs for the calculations described in the example implementations, and output unit 1775 may be configured to provide outputs based on the calculations described in the example implementations.

[0090] The processor 1710 can be configured to obtain material properties and modal properties of the physical system. The processor 1710 can be configured to receive information from the vehicle regarding a first object and a second object detected by one or more sensors of the vehicle, the information including at least one of a position or a size of the detected objects, compare at least one of the position or the size of the first detected object to data stored in a road feature database regarding the first detected object, determine accuracy scores associated with the first detected object and the second detected object based on comparing the at least one of the positions or sizes, and update the cloud-based road anomaly database with the received information about the second detected object based on the associated accuracy scores being higher than at least one of a threshold accuracy score or an accuracy score associated with information of a corresponding object stored in the cloud-based road anomaly database.

[0091] The processor 1710 may also be configured to transmit information regarding the vehicle's trajectory, receive an indication of at least one sensor configuration associated with at least one corresponding position related to the vehicle's trajectory, sense at least one object based on the at least one indicated sensor configuration, and transmit information regarding the at least one object detected by the indicated sensor configuration, the information including at least one of a position or a size of the detected object.

[0092] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the essence of their innovations to others skilled in the art. An algorithm is a sequence of defined steps leading to a desired end state or result. In exemplary implementations, the steps performed require tangible physical manipulations of quantities to achieve a tangible result.

[0093] Unless otherwise indicated, and as will be apparent from the description, throughout the description, descriptions utilizing terms such as "processing," "calculating," "calculating," "determining," "displaying," and the like will be understood to include operations and processes of a computer system or other information processing device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of the computer system into other data similarly represented as physical quantities in the memory or registers of the computer system or other information storage, transmission, or display device.

[0094] Exemplary implementations may also relate to apparatuses for performing the operations herein. This apparatus may be specially configured for the required purposes, or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. Computer-readable storage media may include tangible media, such as, but not limited to, optical disks, magnetic disks, read-only memory, random-access memory, solid-state devices and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. A computer program may include a pure software implementation containing instructions for performing the operations of a desired implementation.

[0095] Various general-purpose systems may be used with the programs and modules according to the examples herein, or it may prove convenient to construct more specialized apparatus to perform the desired method steps. Moreover, the example implementations are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​can be used to implement the teachings of the example implementations described herein. The instructions of the programming language may be executed by one or more processing units, such as a central processing unit (CPU), processor, or controller.

[0096] As is known in the art, the operations described above can be performed by hardware, software, or some combination of software and hardware. Various aspects of the exemplary implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software), which, when executed by a processor, cause the processor to perform methods that implement the implementations of the present application. Furthermore, some exemplary implementations of the present application may be performed solely in hardware, while other exemplary implementations may be performed solely in software. Furthermore, the various functions described may be performed in a single unit or distributed among multiple components in any number of ways. When performed by software, the methods may be executed by a processor, such as a general-purpose computer, based on instructions stored on a computer-readable medium. If necessary, the instructions may be stored on the medium in compressed and / or encrypted format.

[0097] Additionally, other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. Various aspects and / or components of the described exemplary implementations may be used alone or in any combination. It is intended that the specification and exemplary embodiments be considered exemplary only, with the true scope and spirit of the present application being indicated by the following claims.

[0098] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles described herein may be applied to other embodiments without departing from the spirit or scope of the present invention. Thus, it should be understood that the description and drawings presented herein represent presently preferred embodiments of the present invention and, therefore, represent the subject matter broadly contemplated by the present invention. It should be further understood that the scope of the present invention fully encompasses other embodiments that may become apparent to those skilled in the art, and therefore, the scope of the present invention is not limited.

[0099] Combinations such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof," described herein, include any combination of A, B, and / or C, and may include multiple As, multiple Bs, or multiple Cs. Specifically, combinations such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof" may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, and such combinations may include one or more members of its components A, B, and / or C. For example, a combination of A and B may include one A and multiple Bs, multiple A and one B, or multiple A and multiple Bs.

Claims

1. 1. A method for updating a cloud-based road anomaly database, comprising: receiving information from the vehicle regarding a first object and a second object detected by one or more sensors of the vehicle, the information including at least one of a position or a size of the detected objects; comparing at least one of the location or the size of the detected first object with data stored in a road feature database relating to the detected first object; determining an accuracy score associated with the detected second object based on comparing at least one of the location or the size; and updating the cloud-based road anomaly database with the received information about the detected second object based on the associated accuracy score being higher than at least one of a threshold accuracy score or an accuracy score associated with information of a corresponding object stored in the cloud-based road anomaly database.

2. determining a route for the vehicle; identifying at least one set of object information stored in the cloud-based road anomaly database that is associated with an object along the determined route; and predicting an accuracy score for information from the vehicle regarding the object along the determined path; and The method of claim 1 , further comprising: sending sensor configuration instructions to the vehicle based on the predicted accuracy score.

3. The indication of the sensor configuration may include: a high sensitivity configuration if the predicted accuracy score is higher than the accuracy score of the information stored in the cloud-based road anomaly database related to the object along the determined route; 3. The method of claim 2, comprising a low sensitivity configuration if the predicted accuracy score is lower than the accuracy score of the information stored in the cloud-based road anomaly database related to the object along the determined path.

4. 2. The method of claim 1 , wherein determining the accuracy score associated with the detected second object comprises determining an accuracy score associated with the detected first object; and associating the determined accuracy score for the detected first object with the detected second object.

5. 5. The method of claim 4, wherein determining the accuracy score associated with the detected first object further comprises determining the accuracy score based on a magnitude of a difference between the received information regarding the detected first object and the data stored in the road feature database regarding the detected first object.

6. The method of claim 1 , wherein the accuracy scores include an accuracy score for at least one of a first direction, a second direction, a third direction, a longitude, or a latitude.

7. The method of claim 1 , wherein the received information about the detected first and second objects includes information about a plurality of instances of detecting the first and second objects, and the accuracy score is based on the plurality of instances.

8. receiving, from the vehicle, information regarding additional objects detected by the one or more sensors of the vehicle, the received information regarding the detected additional objects including information regarding a first number of instances of detecting the additional objects; determining that data relating to a corresponding object is not stored in the cloud-based road anomaly database; 10. The method of claim 1, further comprising: adding the detected additional object to the cloud-based road anomaly database based on the first number of instances being greater than a threshold number of detection instances.

9. 10. The method of claim 8, further comprising determining an accuracy score associated with the first number of instances of detecting the additional object, wherein adding the detected additional object to the cloud-based road anomaly database is further based on the accuracy score being greater than a threshold accuracy score.

10. receiving information from the vehicle regarding additional objects detected by the one or more sensors of the vehicle; determining additional accuracy scores associated with the additional detected objects; determining that data associated with a corresponding object is stored in the cloud-based road anomaly database, the stored data including a stored accuracy score; 2. The method of claim 1, further comprising: not updating the cloud-based road anomaly database with the received information about the detected additional object based on the determined additional accuracy score being lower than the stored accuracy score.

11. 1. A method in which a vehicle provides data to a cloud-based route generation system to update a cloud-based road anomaly database of the cloud-based route generation system, the method comprising: transmitting information about the vehicle's trajectory to the cloud-based route generation system; receiving, from the cloud-based route generation system, an indication of at least one sensor configuration associated with at least one corresponding position associated with the trajectory of the vehicle; detecting at least one object based on the at least one indicated sensor configuration; and transmitting information about the at least one object detected by the indicated sensor configuration to the cloud-based route generation system; The indication of the at least one sensor configuration may include: a high sensitivity configuration for at least one location associated with a road hazard for which the cloud-based route generation system does not store data associated with an accuracy above a threshold accuracy; a low sensitivity configuration for at least one location associated with at least one of an absence of road hazards or a road hazard for which the cloud-based route generation system stores data associated with an accuracy above a threshold accuracy.

12. The method of claim 11 , further comprising determining at least one of a size or a position associated with the at least one detected object.

13. The method of claim 12 , wherein determining at least one of the size or the location associated with the at least one detected object is based on fusing data from multiple sensor types.

14. The method of claim 12 , further comprising determining an accuracy score associated with the at least one detected object, wherein the transmitted information regarding the at least one object includes the determined accuracy score.

15. 1. A method in which a vehicle provides data to a cloud-based route generation system to update a cloud-based road anomaly database of the cloud-based route generation system, the method comprising: transmitting information about the vehicle's trajectory to the cloud-based route generation system; receiving, from the cloud-based route generation system, an indication of at least one sensor configuration associated with at least one corresponding position associated with the trajectory of the vehicle; detecting at least one object based on the at least one indicated sensor configuration; and transmitting information about the at least one object detected by the indicated sensor configuration to the cloud-based route generation system; determining at least one of a size or a position associated with the at least one detected object; and determining an accuracy score associated with the at least one detected object, wherein the transmitted information regarding the at least one object includes the determined accuracy score.

16. The method of claim 14 or 15, wherein the accuracy scores include an accuracy score for at least one of a first direction, a second direction, a third direction, a longitude, or a latitude.

17. 16. The method of claim 11 or 15, wherein the indicated sensor configuration is a high sensitivity configuration and the at least one detected object is a road hazard for which the cloud-based route generation system does not store data associated with a threshold accuracy.

18. 1. An apparatus for updating a cloud-based road anomaly database, comprising: means for receiving information from a vehicle regarding a first object and a second object detected by one or more sensors of the vehicle, the information including at least one of a position or a size of the detected objects; means for comparing at least one of the position or the size of the detected first object with data stored in a road feature database relating to the detected first object; means for determining an accuracy score associated with the detected second object based on comparing at least one of the location or the size; means for updating the cloud-based road anomaly database with the received information about the detected second object based on the associated accuracy score being higher than at least one of a threshold accuracy score or an accuracy score associated with information of a corresponding object stored in the cloud-based road anomaly database.

19. means for determining a route for the vehicle; means for identifying at least one set of object information stored in the cloud-based road anomaly database that is associated with an object along the determined route; means for determining that an accuracy score associated with at least one object along the determined path is associated with an accuracy score below a threshold accuracy score; 20. The apparatus of claim 18, further comprising: means for transmitting to the vehicle an indication of a sensor configuration associated with the object along the determined route and associated with a position along the determined route, for use by the vehicle to detect the object along the determined route, based on determining that the accuracy score associated with the at least one object along the determined route is associated with an accuracy score that is below the threshold accuracy score.

20. means for receiving from the vehicle information relating to the location associated with the at least one object along the determined path, the information being collected by the one or more sensors of the vehicle; means for determining an additional accuracy score associated with the received information; means for determining that the at least one object is no longer at the location; 20. The apparatus of claim 19, further comprising: means for updating the cloud-based road anomaly database to remove the at least one object based on the additional accuracy score associated with the received information indicating the at least one object is no longer at the location being higher than the stored accuracy score.

Images