Vehicle traffic sign prediction

Abstract

A method of traffic sign prediction includes determining a plurality of road features. The method also includes determining a plurality of road user information. The method also includes determining a plurality of weather-related information. The method further includes determining a predicted traffic sign based at least in part on the plurality of road features, the plurality of road user information, and the plurality of weather-related information. The method further includes performing an action based at least in part on the predicted traffic sign.

Description

Technical Field

[0001] The present disclosure relates to advanced driver assistance and automated driving systems and methods for vehicles, and more particularly to traffic sign prediction systems and methods for vehicles. Background Art

[0002] To enhance occupant awareness and convenience, vehicles may be equipped with advanced driver assistance systems (ADAS) and / or automated driving systems (ADS). ADAS systems can use various sensors such as cameras, radars, and LiDAR (light imaging, detection, and ranging) to detect and identify objects around the vehicle, including other vehicles, pedestrians, road configurations, and traffic signs. ADS systems can take actions based on environmental conditions around the vehicle, such as controlling the vehicle to navigate toward a destination in the environment. However, the performance of current ADS systems may be affected by traffic sign recognition and interpretation. For example, in some cases, traffic signs may be missing, outdated, invalid, or inapplicable. In other cases, traffic signs may be obstructed or blocked due to weather conditions, etc., which prevents accurate recognition and / or identification.

[0003] Therefore, while ADAS and ADS systems and methods achieve their intended purposes, a need remains for a new and improved system and method for vehicle traffic sign prediction. Summary of the invention

[0004] According to several aspects, a method for traffic sign prediction is provided. The method includes determining a plurality of road features. The method also includes determining a plurality of road user information. The method also includes determining a plurality of weather-related information. The method also includes determining a predicted traffic sign based at least in part on the plurality of road features, the plurality of road user information, and the plurality of weather-related information. The method also includes performing an action based at least in part on the predicted traffic sign.

[0005] In another aspect of the present disclosure, determining a plurality of road features may also include using at least one of a perception sensor and data retrieved from a remote server to determine a plurality of road features. The plurality of road features include at least one road geometry feature, at least one road condition feature, and at least one roadside environment feature.

[0006] In another aspect of the present disclosure, determining the plurality of road user information may further include detecting the plurality of road users. Determining the plurality of road user information may further include detecting the plurality of roadside users. Determining the plurality of road user information may further include determining the plurality of predicted paths for each of the plurality of road users and the plurality of roadside users. Determining the plurality of road user information may further include determining the plurality of road user information. The plurality of road user information includes at least the number of the plurality of road users, the number of the plurality of roadside users, and the plurality of predicted paths.

[0007] In another aspect of the present disclosure, determining a plurality of weather-related information may also include using at least one of a sensory sensor and data retrieved from a remote server to determine a plurality of weather-related information. The plurality of weather-related information includes at least one precipitation weather condition and at least one visibility weather condition.

[0008] In another aspect of the present disclosure, determining the predicted traffic sign may further include using a rule-based algorithm to determine the predicted traffic sign. The rule-based algorithm takes a plurality of road features, a plurality of road user information, and a plurality of weather-related information as inputs and generates a predicted traffic sign as output.

[0009] In another aspect of the present disclosure, determining the predicted traffic sign may further include using a fuzzy logic algorithm to determine the predicted traffic sign. The fuzzy logic algorithm takes as input a plurality of road features, a plurality of road user information, and a plurality of weather-related information, and generates as output a plurality of possible predicted traffic signs and a plurality of corresponding confidence values. Determining the predicted traffic sign may further include determining the predicted traffic sign as one of the plurality of possible predicted traffic signs based at least in part on the plurality of corresponding confidence values.

[0010] In another aspect of the present disclosure, determining the predicted traffic sign may also include using a machine learning algorithm to determine the predicted traffic sign. The machine learning algorithm takes as input a plurality of road features, a plurality of road user information, and a plurality of weather-related information, and generates as output a plurality of possible predicted traffic signs and a plurality of corresponding confidence values. Determining the predicted traffic sign may also include determining the predicted traffic sign as one of the plurality of possible predicted traffic signs based at least in part on the plurality of corresponding confidence values.

[0011] In another aspect of the present disclosure, determining the predicted traffic sign may further include validating the predicted traffic sign using a predetermined rule set.

[0012] In another aspect of the present disclosure, validating the predicted traffic sign may further include evaluating the predicted traffic sign using a predetermined rule set; and in response to determining that the predicted traffic sign violates the predetermined rule set, modifying the predicted traffic sign.

[0013] In another aspect of the present disclosure, performing an action based at least in part on the predicted traffic sign may further include transmitting the predicted traffic sign to at least one of a remote vehicle and a remote server. Performing an action based at least in part on the predicted traffic sign may further include adjusting a path planning and control algorithm of the autonomous driving system based at least in part on the predicted traffic sign.

[0014] According to several aspects, a system for vehicle traffic sign prediction is provided. The system includes a plurality of vehicle sensors. The system also includes a controller in electrical communication with the plurality of vehicle sensors. The controller is programmed to determine a plurality of road features using at least one of the plurality of vehicle sensors. The controller is also programmed to determine a plurality of road user information using at least one of the plurality of vehicle sensors. The controller is also programmed to determine a plurality of weather-related information using at least one of the plurality of vehicle sensors. The controller is also programmed to determine a predicted traffic sign based at least in part on the plurality of road features, the plurality of road user information, and the plurality of weather-related information. The controller is also programmed to perform an action based at least in part on the predicted traffic sign.

[0015] In another aspect of the present disclosure, the plurality of vehicle sensors further comprises at least one perception sensor. To determine the plurality of road features, the controller is further programmed to perform a plurality of measurements of the vehicle surroundings using the at least one perception sensor. To determine the plurality of road features, the controller is further programmed to determine the plurality of road features based at least in part on the plurality of measurements. The plurality of road features comprises at least one road geometry feature, at least one road condition feature, and at least one roadside environment feature.

[0016] In another aspect of the present disclosure, the plurality of vehicle sensors further include a vehicle communication system. To determine the plurality of road features, the controller is further programmed to receive at least one of a V2V message and a V2X message including the plurality of road features using the vehicle communication system. The plurality of road features include at least one road geometry feature, at least one road condition feature, and at least one roadside environment feature.

[0017] In another aspect of the present disclosure, in order to determine multiple road user information, the controller is further programmed to detect multiple road users using multiple vehicle sensors. In order to determine multiple road user information, the controller is further programmed to detect multiple roadside users using multiple vehicle sensors. In order to determine multiple road user information, the controller is further programmed to determine multiple predicted paths for each of the multiple road users and multiple roadside users. In order to determine multiple road user information, the controller is further programmed to determine multiple road user information. The multiple road user information includes at least the number of multiple road users, the number of multiple roadside users, and the multiple predicted paths.

[0018] In another aspect of the present disclosure, the plurality of vehicle sensors further comprises at least one of a perception sensor and a vehicle communication system. To determine the plurality of weather-related information, the controller is further programmed to use at least one of the perception sensor and the vehicle communication system to determine the plurality of weather-related information. The plurality of weather-related information comprises at least one precipitation weather condition and at least one visibility weather condition.

[0019] In another aspect of the present disclosure, to determine the predicted traffic sign, the controller is further programmed to determine the predicted traffic sign using a prediction algorithm. The prediction algorithm takes as input a plurality of road features, a plurality of road user information, and a plurality of weather-related information, and generates as output a plurality of possible predicted traffic signs and a plurality of corresponding confidence values. The prediction algorithm is at least one of a rule-based algorithm, a fuzzy logic algorithm, and a machine learning algorithm. To determine the predicted traffic sign, the controller is further programmed to determine the predicted traffic sign as one of a plurality of possible predicted traffic signs based at least in part on a plurality of corresponding confidence values. To determine the predicted traffic sign, the controller is further programmed to verify the predicted traffic sign by evaluating the predicted traffic sign using a predetermined set of rules. To determine the predicted traffic sign, the controller is further programmed to modify the predicted traffic sign in response to determining that the predicted traffic sign is inconsistent with a predetermined set of rules.

[0020] In another aspect of the present disclosure, the system further includes an autonomous driving system in electrical communication with the controller. The plurality of vehicle sensors further includes a vehicle communication system. To perform the action, the controller is further programmed to transmit the predicted traffic sign to at least one of a remote vehicle and a remote server using the vehicle communication system. To perform the action, the controller is further programmed to adjust a path planning and control algorithm of the autonomous driving system based at least in part on the predicted traffic sign.

[0021] According to several aspects, a vehicle traffic sign prediction system is provided. The system includes a plurality of vehicle sensors. The plurality of vehicle sensors include at least one perception sensor. The plurality of vehicle sensors also include at least a vehicle communication system. The system also includes an autonomous driving system. The system also includes a controller in electrical communication with the plurality of vehicle sensors and the autonomous driving system. The controller is programmed to determine a plurality of road features using at least one of the plurality of vehicle sensors. The plurality of road features include at least one road geometry feature, at least one road condition feature, and at least one roadside environment feature. The controller is also programmed to determine a plurality of road user information using at least one of the plurality of vehicle sensors. The controller is also programmed to determine a plurality of weather-related information using at least one of the plurality of vehicle sensors. The plurality of weather-related information includes at least one precipitation weather condition and at least one visibility weather condition. The controller is also programmed to determine a predicted traffic sign based at least in part on the plurality of road features, the plurality of road user information, and the plurality of weather-related information. The controller is also programmed to transmit the predicted traffic sign to at least one of a remote vehicle and a remote server using the vehicle communication system. The controller is also programmed to adjust the path planning and control algorithms of the autonomous driving system based at least in part on the predicted traffic sign.

[0022] In another aspect of the present disclosure, in order to determine multiple road user information, the controller is further programmed to detect multiple road users using multiple vehicle sensors. In order to determine multiple road user information, the controller is further programmed to detect multiple roadside users using multiple vehicle sensors. In order to determine multiple road user information, the controller is further programmed to determine multiple predicted paths for each of the multiple road users and multiple roadside users. In order to determine multiple road user information, the controller is further programmed to determine multiple road user information. The multiple road user information includes at least the number of multiple road users, the number of multiple roadside users, and the multiple predicted paths.

[0023] In another aspect of the present disclosure, to determine the predicted traffic sign, the controller is further programmed to determine the predicted traffic sign using a prediction algorithm. The prediction algorithm takes as input a plurality of road features, a plurality of road user information, and a plurality of weather-related information, and generates as output a plurality of possible predicted traffic signs and a plurality of corresponding confidence values. The prediction algorithm is at least one of a rule-based algorithm, a fuzzy logic algorithm, and a machine learning algorithm. To determine the predicted traffic sign, the controller is further programmed to determine the predicted traffic sign as one of a plurality of possible predicted traffic signs based at least in part on a plurality of corresponding confidence values. To determine the predicted traffic sign, the controller is further programmed to verify the predicted traffic sign by evaluating the predicted traffic sign using a predetermined set of rules. To determine the predicted traffic sign, the controller is further programmed to modify the predicted traffic sign in response to determining that the predicted traffic sign is inconsistent with a predetermined set of rules.

[0024] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0026] Figure 1 is a schematic diagram of a vehicle traffic sign prediction system according to an exemplary embodiment; and

[0027] Figure 2 is a flowchart of a vehicle traffic sign prediction method according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses.

[0029] Traffic sign recognition and identification may be very important for accurate operation of an automated driving system. In some cases, traffic signs may be missing or obscured. In other cases, dynamic road conditions such as road construction, weather conditions, traffic volume, and / or the like may render posted traffic signs invalid and / or irrelevant. Therefore, the present disclosure provides a new and improved vehicle traffic sign prediction system and method that can predict traffic signs when actual traffic signs are missing, obscured, or otherwise unreadable.

[0030] refer to Figure 1 , a vehicle traffic sign prediction system is shown, generally indicated by the reference numeral 10. The system 10 is shown with an exemplary vehicle 12. Although a passenger vehicle is shown, it should be understood that the vehicle 12 can be any type of vehicle without departing from the scope of the present disclosure. The system 10 generally includes a controller 14, a plurality of vehicle sensors 16, an autonomous driving system 18, and a display 20.

[0031] As will be described below, the controller 14 is used to implement the traffic sign prediction method 100. The controller 14 includes at least one processor 22 and a non-transitory computer readable storage device or medium 24. The processor 22 can be a custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor of several processors associated with the controller 14, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a device for executing instructions. The computer readable storage device or medium 24 can include volatile and non-volatile storage devices such as read-only memory (ROM), random access memory (RAM), and keep alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the processor 22 is powered off. The computer readable storage device or medium 24 may be implemented using a plurality of storage devices, such as a PROM (Programmable Read Only Memory), an EPROM (Electrically Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read Only Memory), a flash memory, or other electrical, magnetic, optical, or combination storage device capable of storing data, some of which represent executable instructions used by the controller 14 to control the various systems of the vehicle 12. The controller 14 may also be comprised of multiple controllers in electrical communication with each other. The controller 14 may be interconnected with additional systems and / or controllers of the vehicle 12, thereby allowing the controller 14 to access data, such as the speed, acceleration, braking, and steering angles of the vehicle 12.

[0032] The controller 14 is in electrical communication with a plurality of vehicle sensors 16, an autonomous driving system 18, and a display 20. In an exemplary embodiment, the electrical communication is established using, for example, a CAN network, a FLEXRAY network, a local area network (e.g., WiFi, Ethernet, etc.), a serial peripheral interface (SPI) network, etc. It should be understood that various other wired and wireless technologies and communication protocols for communicating with the controller 14 are within the scope of the present disclosure.

[0033] The plurality of vehicle sensors 16 are used to acquire information about the environment surrounding the vehicle 12 . In the exemplary embodiment, the plurality of vehicle sensors 16 include at least a camera system 26 (ie, a perception sensor) and a vehicle communication system 28 .

[0034] In another exemplary embodiment, the plurality of vehicle sensors 16 further include sensors for determining performance data regarding the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 further include at least one of a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a brake position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor.

[0035] In another exemplary embodiment, the plurality of vehicle sensors 16 also include sensors for determining information about the environment within the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 also include at least one of a seat occupancy sensor, a cabin air temperature sensor, a cabin motion detection sensor, a cabin camera, a cabin microphone, and the like.

[0036] In another exemplary embodiment, the plurality of vehicle sensors 16 also include sensors for determining information about the environment surrounding the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 also include at least one of an ambient air temperature sensor, an air pressure sensor, a global navigation satellite system (GNSS), and / or a camera and / or a video camera positioned to observe the environment in front of the vehicle 12.

[0037] In another exemplary embodiment, at least one of the plurality of vehicle sensors 16 is a perception sensor capable of measuring distances in the environment surrounding the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 include a stereo camera with distance measurement capabilities. In an example, at least one of the plurality of vehicle sensors 16 is fixed to the interior of the vehicle 12, such as in the roof lining of the vehicle 12, and can be viewed through the windshield of the vehicle 12. In another example, at least one of the plurality of vehicle sensors 16 is fixed to the exterior of the vehicle 12, such as on the roof of the vehicle 12, and can observe the environment surrounding the vehicle 12. It should be understood that various other types of vehicle sensors, such as LiDAR sensors, ultrasonic ranging sensors, radar sensors, time of flight sensors, and / or other types of perception sensors are within the scope of the present disclosure. As described above, the plurality of vehicle sensors 16 are in electrical communication with the controller 14.

[0038] The camera system 26 is a perception sensor for capturing images and / or video of the environment surrounding the vehicle 12. In an exemplary embodiment, the camera system 26 includes a camera and / or a camera positioned to observe the environment surrounding the vehicle 12. In a non-limiting example, the camera system 26 includes a camera fixed to the interior of the vehicle 12 (e.g., in the headliner of the vehicle 12) that can observe through the windshield. In another non-limiting example, the camera system 26 includes a camera fixed to the exterior of the vehicle 12 (e.g., fixed to the roof of the vehicle 12) that can observe the environment in front of the vehicle 12.

[0039] In another exemplary embodiment, the camera system 26 is a surround view camera system including a plurality of cameras (also referred to as satellite cameras) configured to observe the environment adjacent to all sides of the vehicle 12. In a non-limiting example, the camera system 26 includes a front camera (e.g., mounted in the front grille of the vehicle 12), a rear camera (e.g., mounted on the rear tailgate of the vehicle 12), and two side cameras (e.g., mounted respectively under the two side mirrors of the vehicle 12). In another non-limiting example, the camera system 26 also includes an additional rear view camera mounted near the center high-mounted brake light of the vehicle 12.

[0040] It should be understood that camera systems with other cameras and / or other mounting locations are within the scope of the present disclosure. It should also be understood that cameras with various sensor types (including, for example, charge coupled device (CCD) sensors, complementary metal oxide semiconductor (CMOS) sensors, and / or high dynamic range (HDR) sensors) are within the scope of the present disclosure. In addition, cameras with various lens types including, for example, wide angle lenses and / or narrow angle lenses are also within the scope of the present disclosure.

[0041] The controller 14 uses the vehicle communication system 28 to communicate with other systems external to the vehicle 12. For example, the vehicle communication system 28 includes functionality for communicating with vehicles ("V2V" communication), infrastructure ("V2I" communication), remote systems of remote call centers (e.g., General Motors' ON-STAR), and / or personal devices. In general, the term "vehicle-to-everything communication" ("V2X" communication) refers to communication between the vehicle 12 and any remote system (e.g., vehicle, infrastructure, and / or remote system). In some embodiments, the vehicle communication system 28 is a wireless communication system that is configured to communicate using a wireless local area network (WLAN) of the IEEE 802.11 standard or by using cellular data communications (e.g., using GSMA standards such as SGP.02, SGP.22, SGP.32, etc.). Therefore, the vehicle communication system 28 may also include an embedded universal integrated circuit card (eUICC) that is configured to store at least one cellular connection configuration profile, such as an embedded user identity module (eSIM) profile. The vehicle communication system 28 is also configured to communicate via a personal area network (e.g., Bluetooth) and / or near field communication (NFC). However, other or alternative communication methods such as dedicated short-range communication (DSRC) channels and / or mobile telecommunication protocols based on the third generation partnership project (3GPP) standard are also considered to be within the scope of the present disclosure. DSRC channels refer to unidirectional or bidirectional short-range to medium-range wireless communication channels designed specifically for automotive use and a set of corresponding protocols and standards. 3GPP refers to a partnership between multiple standard organizations that develop protocols and standards for mobile telecommunications. The 3GPP standard is structured in "versions". Therefore, communication methods based on 3GPP versions 14, 15, 16 and / or future 3GPP versions are considered to be within the scope of the present disclosure. Therefore, the vehicle communication system 28 may include one or more antennas and / or communication transceivers for receiving and / or sending signals, such as collaborative sensing messages (CSM). The vehicle communication system 28 is configured to perform wireless information communication between the vehicle 12 and another vehicle. In addition, the vehicle communication system 28 is configured to perform wireless information communication between the vehicle 12 and infrastructure or other vehicles. It should be appreciated that the vehicle communication system 28 may be integrated with the controller 14 (eg, integrated on the same circuit board as the controller 14 or otherwise be a part of the controller 14 ) without departing from the scope of the present disclosure.

[0042] The automated driving system 18 is used to provide assistance to the occupants to increase the occupants' awareness and / or control of the behavior of the vehicle 12. Within the scope of the present disclosure, the occupants include the driver and / or passengers of the vehicle 12. Within the scope of the present disclosure, the automated driving system 18 encompasses systems that provide any level of assistance to the occupants (e.g., blind spot warnings, lane departure warnings, etc.) as well as systems that are capable of automatically driving the vehicle 12 under certain or all conditions. It should be understood that all driving automation levels defined by, for example, SAE J3016 (i.e., SAE LEVEL 0, SAE LEVEL 1, SAE LEVEL 2, SAE LEVEL 3, SAE LEVEL 4, and SAE LEVEL 5) are within the scope of the present disclosure.

[0043] In an exemplary embodiment, the autopilot system 18 is configured to detect and / or receive information about the environment surrounding the vehicle 12 and process the information to provide assistance to the occupants. In some embodiments, the autopilot system 18 is a software module executed on the controller 14. In other embodiments, the autopilot system 18 includes a separate autopilot system controller similar to the controller 14 that is capable of processing information about the environment surrounding the vehicle 12. In an exemplary embodiment, the autopilot system 18 can operate in a manual operation mode, a partially automated operation mode, and / or a fully automated operation mode.

[0044] Within the scope of the present disclosure, manual operation mode means that the autonomous driving system 18 provides warnings or notifications to the occupants, but does not directly intervene or control the vehicle 12. In a non-limiting example, the autonomous driving system 18 receives information from a plurality of vehicle sensors 16. Through technology such as computer vision, the autonomous driving system 18 can understand the environment around the vehicle 12 and provide assistance to the occupants. For example, if the autonomous driving system 18 identifies that the vehicle 12 is likely to collide with a distant vehicle based on data from the plurality of vehicle sensors 16, the autonomous driving system 18 can use the display 20 to issue a warning to the occupants.

[0045] Within the scope of the present disclosure, a partially automated mode of operation means that the automated driving system 18 provides warnings or notifications to the occupants and may directly intervene or control the vehicle 12 in certain circumstances. In a non-limiting example, the automated driving system 18 is additionally in electrical communication with components of the vehicle 12, such as the braking system, propulsion system, and / or steering system of the vehicle 12, so that the automated driving system 18 can control the behavior of the vehicle 12. In a non-limiting example, the automated driving system 18 may control the behavior of the vehicle 12 by applying the brakes of the vehicle 12 to avoid an imminent collision. In another non-limiting example, the automated driving system 18 may control the braking system, propulsion system, and steering system of the vehicle 12 to temporarily drive the vehicle 12 to a predetermined destination. However, intervention by the occupants may be required at any time. In an exemplary embodiment, the automated driving system 18 may include additional components, such as an eye tracking device configured to monitor the occupant's attention level and ensure that the occupant is ready to take over control of the vehicle 12.

[0046] Within the scope of this disclosure, a fully automated mode of operation refers to the automated driving system 18 using data from the plurality of vehicle sensors 16 to understand the environment and control the vehicle 12 to drive the vehicle 12 to a predetermined destination without the need for occupant control or intervention.

[0047] The automated driving system 18 operates using a path planning and control algorithm that is configured to generate a safe and efficient trajectory for the vehicle 12 to navigate the environment surrounding the vehicle 12. In an exemplary embodiment, the path planning and control algorithm is a machine learning algorithm that is trained to output control signals for the vehicle 12 based on input data collected from the plurality of vehicle sensors 16. In another exemplary embodiment, the path planning and control algorithm is a deterministic algorithm that has been programmed to output control signals for the vehicle 12 based on data collected from the plurality of vehicle sensors 16.

[0048] In a non-limiting example, the path planning and control algorithm performs perception and mapping tasks to interpret data collected from multiple vehicle sensors 16 and create a detailed map of the environment. The detailed map may include information on lane boundaries, road geometry, speed limits, traffic signs, and / or other relevant features. Based on the detailed map and the current state of the vehicle 12 (i.e., the location, speed, and direction of the vehicle 12), the path planning and control algorithm generates a series of waypoints or a continuous path that the vehicle 12 should follow to reach the destination while complying with rules, regulations, and safety restrictions. It should be understood that the autonomous driving system 18 may include any software and / or hardware modules configured to operate in the above-mentioned manual operation mode, partially automated operation mode, or fully automated operation mode.

[0049] The display 20 is used to provide information to the occupants of the vehicle 12. Figure 1In the exemplary embodiment shown, the display 20 is a human-machine interface (HMI) that is located within the occupant's field of vision and is capable of displaying text, graphics, and / or images. It should be understood that HMI display systems including LCD displays, LED displays, and the like are within the scope of the present disclosure. Other exemplary embodiments in which the display 20 is disposed in a rearview mirror are also within the scope of the present disclosure. In another exemplary embodiment, the display 20 includes a head-up display (HUD) that is configured to provide information to the occupant by projecting text, graphics, and / or images on the windshield of the vehicle 12. The text, graphics, and / or images are reflected by the windshield of the vehicle 12, and the occupant can see them without moving their eyes away from the road in front of the vehicle 12. In another exemplary embodiment, the display 20 includes an augmented reality head-up display (AR-HUD). AR-HUD is a HUD that is configured to enhance the occupant's view of the road in front of the vehicle 12 by overlaying text, graphics, and / or images on physical objects in the surrounding environment of the vehicle 12 within the occupant's field of vision. In an exemplary embodiment, an occupant may interact with the display 20 using a human interface device (HID) including, for example, a touch screen, electromechanical switches, capacitive switches, knobs, etc. It should be appreciated that other systems for displaying information to an occupant of the vehicle 12 are within the scope of the present disclosure.

[0050] refer to Figure 2 , a flow chart of a traffic sign prediction method 100 is shown. The method 100 starts at box 102 and proceeds to boxes 104, 106, 108 and 110.

[0051] At box 104, the controller 14 uses the plurality of vehicle sensors 16 to perform a plurality of measurements of the environment around the vehicle 12. In an exemplary embodiment, the plurality of measurements are performed using a perception sensor (e.g., a camera system 26, a LiDAR system, a radar system, etc.) of the plurality of vehicle sensors 16. In a non-limiting example, the plurality of measurements include distance measurements from the vehicle 12 to a plurality of points in the environment around the vehicle 12. In another non-limiting example, the plurality of measurements include images and / or videos of the environment around the vehicle 12 captured by the camera system 26. In another non-limiting example, the plurality of measurements include data about the environment around the vehicle 12 received using the vehicle communication system 28. For example, the controller 14 may use the vehicle communication system 28 to transmit the location of the vehicle 12 (as determined using GNSS) to a remote server. The remote server may respond by transmitting information about the environment around the vehicle 12 (e.g., using at least one of a V2V message and / or a V2X message) to the vehicle communication system 28, including, for example, information about the road geometry, road conditions, and / or roadside environment of the environment around the vehicle 12. After frame 104 , the method 100 proceeds to frame 112 .

[0052] At box 112, the controller 14 determines a plurality of road features based at least in part on the plurality of measurements performed at box 104. Within the scope of the present disclosure, the plurality of road features are features of a road in the environment surrounding the vehicle. In an exemplary embodiment, the plurality of road features include at least one road geometry feature (i.e., a feature of the road geometry), at least one road condition feature (i.e., a feature of the road surface condition), and at least one roadside environment feature (i.e., a feature of the road environment background). In a non-limiting example, the at least one road geometry feature includes, for example, lane width, horizontal curve radius, sight distance, vertical curve radius, vertical elevation, presence of intersections, etc. In a non-limiting example, the at least one road condition feature includes, for example, road surface condition, etc. In a non-limiting example, the at least one roadside environment feature includes, for example, road function (e.g., urban, rural, highway), presence of lanes along the road, presence of development projects (e.g., commercial buildings and / or residential buildings) on the roadside, shoulder width, shoulder road surface condition, presence of pedestrian infrastructure (e.g., sidewalks) along the road, and / or the like. Following frame 112 , the method 100 proceeds to frame 114 , as will be discussed in greater detail below.

[0053] At box 106, the controller 14 detects multiple road users. Within the scope of the present disclosure, multiple road users are users of the road other than the vehicle 12, such as remote vehicles, construction workers, pedestrians, cyclists, wild animals, etc. In an exemplary embodiment, the controller 14 uses one or more of the multiple vehicle sensors 16 to detect multiple road users. In a non-limiting example, the controller 14 uses the vehicle communication system 28 to receive V2V and / or V2X messages from remote vehicles and / or infrastructure. The V2V and / or V2X messages include information about one or more of the multiple road users, such as the location, speed, and direction of travel of the remote vehicle on the road. In another non-limiting example, the controller 14 uses the camera system 26 to capture multiple images of the environment around the vehicle 12, and uses a computer vision algorithm to analyze the multiple images and identify multiple road users. In yet another non-limiting example, the controller 14 uses another type of perception sensor (e.g., LiDAR sensor, radar sensor, etc.) to perform multiple measurements of the environment around the vehicle 12 and identify multiple road users. Following frame 106 , the method 100 proceeds to frame 116 , as will be discussed in greater detail below.

[0054] At box 108, the controller 14 detects multiple roadside users. Within the scope of the present disclosure, multiple roadside users are users of the road surroundings (i.e., roadsides). Multiple roadsides may include, for example, pedestrians on the sidewalk and / or shoulders, cyclists and animals, vehicles parked / parked on the shoulders, etc. In an exemplary embodiment, the controller 14 uses one or more of the multiple vehicle sensors 16 to detect multiple roadside users. In a non-limiting example, the controller 14 uses the vehicle communication system 28 to receive V2V and / or V2X messages from remote vehicles and / or infrastructure. The V2V and / or V2X messages include information about one or more of the multiple roadside users, such as the location of a remote vehicle parked on the roadside. In another non-limiting example, the controller 14 uses the camera system 26 to capture multiple images of the environment around the vehicle 12, and uses a computer vision algorithm to analyze the multiple images and identify multiple roadside users. In yet another non-limiting example, the controller 14 uses another type of perception sensor (e.g., LiDAR sensor, radar sensor, etc.) to measure the environment around the vehicle 12 multiple times and identify multiple roadside users. Following frame 108 , the method 100 proceeds to frame 116 , as will be discussed in greater detail below.

[0055] At box 116, the controller 14 determines a plurality of predicted paths for each of the plurality of road users and the plurality of roadside users identified at boxes 108 and 108. In an exemplary embodiment, the plurality of predicted paths are determined using a statistical algorithm such as a Kalman filter algorithm. The Kalman filter algorithm is a recursive algorithm that combines measurements from the plurality of vehicle sensors 16 with system dynamics to estimate the state of a dynamic system. In the case of path prediction, the Kalman filter considers measurements such as position, velocity, and acceleration to estimate the current and future states of the plurality of road users and the plurality of roadside users.

[0056] In a non-limiting example, the Kalman filter algorithm operates in two main steps: prediction and update. In the prediction step, the Kalman filter algorithm uses the previous state estimate and the system dynamics model to predict the expected state of multiple road users and multiple roadside users at the next time step. The prediction combines information about the multiple road users and multiple roadside users, such as speed, acceleration, driving direction, etc. In the update step, the Kalman filter algorithm combines the predicted state with real-time measurements from multiple vehicle sensors 16. By iteratively repeating the prediction and update steps, the Kalman filter algorithm continuously improves and predicts the future paths of multiple road users and multiple roadside users. After box 116, method 100 proceeds to box 118.

[0057] At box 118, the controller 14 determines a plurality of road user information based at least in part on the plurality of road users detected at box 106, the plurality of roadside users detected at box 108, and the plurality of predicted paths determined at box 116. In an exemplary embodiment, the plurality of road user information includes at least the number of road users (i.e., the number of remote vehicles, construction workers, pedestrians, cyclists, wild animals, etc. on the road), the number of roadside users (i.e., many pedestrians, cyclists and animals on the sidewalk and / or shoulder, vehicles parked / parked on the shoulder, etc.), and the plurality of predicted paths determined in box 116. In an exemplary embodiment, the plurality of road user information may include more details, such as the type of remote vehicles on the road / roadside (e.g., construction vehicles, emergency vehicles, semi-trailer trucks, etc.). In addition, the plurality of road user information may be filtered based on relevance. For example, information related to a remote vehicle traveling on the other side of a divided highway with the vehicle 12 may not be relevant, and therefore such information may be deleted from the plurality of road user information. After box 118, the method 100 proceeds to box 114, as will be discussed in more detail below.

[0058] At box 110, the controller 14 determines a plurality of weather-related information. Within the scope of the present disclosure, the plurality of weather-related information is information about the weather in the environment surrounding the vehicle 12. In an exemplary embodiment, the plurality of weather-related information includes at least one precipitation weather condition and at least one visibility weather condition. In a non-limiting example, the at least one precipitation weather condition includes, for example, whether there is rain, snow, sleet, hail, sunny, etc. At least one visibility weather condition includes, for example, whether there is fog and / or density, visibility distance, etc. In an exemplary embodiment, in order to determine a plurality of weather-related information, the controller 14 uses a plurality of perception sensors (e.g., a camera system 26) in the vehicle sensor 16 to capture an image of the environment surrounding the vehicle 12. The image is then analyzed using, for example, a computer vision algorithm to determine a plurality of weather-related information. In another exemplary embodiment, the controller 14 uses the vehicle communication system 28 of the plurality of vehicle sensors 16 to retrieve data from a remote server (e.g., an Internet application programming interface that provides weather data), wherein the data includes a plurality of weather-related information. After box 110, the method 100 proceeds to box 114.

[0059] At box 114, the controller 14 determines a predicted traffic sign based at least in part on the plurality of road features determined at box 112, the plurality of road user information determined at box 118, and the plurality of weather-related information determined at box 110. Within the scope of the present disclosure, a predicted traffic sign is a traffic sign (e.g., a stop sign, a yield sign, a speed limit sign, etc.) and / or a traffic signal that is predicted to be applicable to road conditions in the environment surrounding the vehicle 12. The predicted traffic sign is determined using a prediction algorithm.

[0060] In a first exemplary embodiment, the prediction algorithm is a rule-based algorithm. Within the scope of the present disclosure, a rule-based algorithm is an algorithm that operates on a predefined set of rules and / or conditions to determine a predicted traffic sign. In a non-limiting example, the predefined set of rules is defined by a standard, such as the Manual on Uniform Traffic Control Devices. In a non-limiting example, the rule-based algorithm is implemented using a lookup table (LUT) that maps a plurality of road features determined at box 112, a plurality of road user information determined at box 118, and a plurality of weather-related information determined at box 110 to a predicted traffic sign.

[0061] The LUT has three key columns (i.e., one key column for each of a plurality of road features, a plurality of road user information, and a plurality of weather-related information) and one value column (i.e., one value column for predicted traffic signs). In an exemplary embodiment, the LUT includes a plurality of rows, each of which maps a unique combination of a plurality of road features, a plurality of road user information, and a plurality of weather-related information in the three key columns to a value in the value column (i.e., a predicted traffic sign). The LUT is stored in a medium 24 of the controller 14. In an exemplary embodiment, the plurality of rows of the LUT are predetermined. In another exemplary embodiment, the plurality of rows of the LUT may be modified by an occupant using a human-machine interface device, etc. In another exemplary embodiment, the plurality of rows of the LUT may be updated via an over-the-air download (OTA) using the vehicle communication system 28. It should be understood that any rule-based algorithm (e.g., a programming data structure, a logic equation, a mathematical function, etc.) is within the scope of the present disclosure.

[0062] In a second exemplary embodiment, the prediction algorithm is a fuzzy logic algorithm. Within the scope of the present disclosure, a fuzzy logic algorithm is an extension of classical (Boolean) logic that allows for the representation and manipulation of partial truth values ​​or degrees of membership. For example, in fuzzy logic, a variable is not a true or false value in the strict sense, but a degree of truth or falsehood between 0 and 1, where 0 represents absolute falsehood, 1 represents absolute truth, and values ​​in between represent different degrees of truth or falsehood. In a non-limiting example, the fuzzy logic algorithm takes as input a plurality of road features determined at box 112, a plurality of road user information determined at box 118, and a plurality of weather-related information determined at box 110, and generates a plurality of possible predicted traffic signs and a plurality of confidence values ​​as outputs. Each of the plurality of confidence values ​​corresponds to one of the plurality of possible predicted traffic signs. Based at least in part on the plurality of confidence values, a predicted traffic sign is selected from the plurality of possible predicted traffic signs. In a non-limiting example, one of the plurality of possible predicted traffic signs having the highest confidence value is determined as the predicted traffic sign.

[0063] In a third exemplary embodiment, the prediction algorithm is a machine learning algorithm. In a non-limiting example, the machine learning algorithm takes as input a plurality of road features determined at box 112, a plurality of road user information determined at box 118, and a plurality of weather-related information determined at box 110, and generates as output a plurality of possible predicted traffic signs and a plurality of confidence values. Each of the plurality of confidence values ​​corresponds to one of the plurality of possible predicted traffic signs. A predicted traffic sign is selected from the plurality of possible predicted traffic signs based at least in part on the plurality of confidence values. In a non-limiting example, one of the plurality of possible predicted traffic signs having the highest confidence value is determined as the predicted traffic sign.

[0064] In a non-limiting example, the machine learning algorithm is a deep neural network having multiple layers, including an input layer and an output layer, and one or more hidden layers. The input layer receives as input a plurality of road features determined at box 112, a plurality of road user information determined at box 118, and a plurality of weather-related information determined at box 110. The input is then passed to the hidden layer. Each hidden layer transforms the data (e.g., a nonlinear transformation) and passes the result to the next hidden layer until the last hidden layer. The output layer generates a plurality of possible predicted traffic signs and a plurality of confidence values.

[0065] To train the machine learning algorithm, an input data set of corresponding traffic signs is used. The algorithm is trained by adjusting the internal weights between the nodes in each hidden layer to minimize the prediction error. During the training process, optimization techniques such as gradient descent are used to adjust the internal weights to reduce the prediction error. The training process is repeated for the entire data set until the prediction error is minimized, and the resulting trained model is then used to predict traffic signs based on new input data.

[0066] After the machine learning algorithm is sufficiently trained, the algorithm is able to accurately and precisely predict traffic signs based on the plurality of road features determined at box 112, the plurality of road user information determined at box 118, and the plurality of weather-related information determined at box 110. By adjusting the weights between nodes in each hidden layer during the training process, the algorithm "learns" to recognize patterns in the data that indicate different traffic signs.

[0067] It should be understood that the prediction algorithm may include any inductive reasoning method to determine the predicted traffic sign based at least in part on the plurality of road features determined at box 112, the plurality of road user information determined at box 118, and the plurality of weather-related information determined at box 110, within the scope of the present disclosure. After box 114, the method 100 proceeds to box 120.

[0068] At box 120, the controller 14 verifies the predicted traffic sign determined at box 114. In an exemplary embodiment, the controller 14 uses computer vision and / or scene text recognition (STR) technology to evaluate data from multiple vehicle sensors 16 and verify the predicted traffic sign. In another exemplary embodiment, in order to verify the predicted traffic sign, the controller 14 uses a predetermined rule set to evaluate the predicted traffic sign. In a non-limiting example, the predetermined rule set includes rules such as maximum speed limit = 75 miles / hour, minimum speed limit = 15 miles / hour, maximum speed limit on urban roads = 45 miles / hour, etc. In a non-limiting example, the predetermined rule set includes rules from traffic laws, regulations, standards, etc., such as the rules in the Manual on Uniform Traffic Control Devices. If the predicted traffic sign violates the predetermined rule set, the method 100 proceeds to box 122. If the predicted traffic sign does not violate the predetermined rule set, the method 100 proceeds to boxes 124 and 126.

[0069] At box 122, the controller 14 modifies the predicted traffic sign in response to determining at box 120 that the predicted traffic sign is contrary to the predetermined rule set. For example, if the predicted traffic sign is a speed limit sign indicating an 80 mile per hour speed limit, then at box 122, the predicted traffic sign is modified to indicate a speed limit of 75 miles per hour according to the exemplary predetermined rule set discussed above. After box 112, the method 100 proceeds to boxes 124 and 126.

[0070] At box 124, the controller 14 transmits the predicted traffic sign using the vehicle communication system 28. In an exemplary embodiment, the predicted traffic sign is transmitted to a remote vehicle using, for example, a V2V message. In another exemplary embodiment, the predicted traffic sign is transmitted to a remote server, such as an Internet-connected map database containing traffic sign information. After box 124, the method 100 enters a standby state at box 128.

[0071] At box 126, the controller 14 adjusts the path planning and control algorithms of the automated driving system 18 based at least in part on the predicted traffic sign. As described above, the path planning and control algorithms generate a series of waypoints or a continuous path that the vehicle 12 should follow to reach the destination based at least in part on the traffic signs in the environment surrounding the vehicle 12. Therefore, after the predicted traffic sign is determined and verified, the predicted traffic sign is transmitted to the automated driving system 18, and the automated driving system 18 recalculates the path planning and control algorithms to account for the predicted traffic sign. After box 126, the method 100 enters a standby state at box 128.

[0072] In an exemplary embodiment, the controller 14 repeatedly exits the standby state 128 and restarts the method 100 at block 102. In a non-limiting example, the controller 14 exits the standby state 128 and restarts the method 100 on a timer, such as every three hundred milliseconds.

[0073] The system 10 and method 100 of the present disclosure have several advantages. If multiple vehicle sensors 16 are unable to identify traffic signs along a road, or the sign recognition method is unable to detect and / or identify the sign message, the system 10 and method 100 can be used to predict traffic signs, thereby improving the performance of the autonomous driving system 18. In addition, dynamic road conditions such as construction, weather, traffic volume, etc. may make posted traffic signs and / or traffic sign information stored in a map database invalid or suboptimal. Therefore, the system 10 and method 100 can be used to predict traffic signs based on dynamic conditions. After the traffic sign is predicted, the system 10 and method 100 can be used to transmit the predicted traffic sign to other vehicles and / or remote systems, thereby achieving crowdsourcing of traffic sign predictions.

[0074] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to fall within the scope of the present disclosure. These variations should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. A method for traffic sign prediction, the method comprising: determining multiple road features; Determine multiple road user information; Determine multiple weather-related information; determining a predicted traffic sign based at least in part on the plurality of road features, the plurality of road user information, and the plurality of weather-related information; as well as An action is performed based at least in part on the predicted traffic sign.

2. The method according to claim 1, wherein: Determining the plurality of road features further comprises: The plurality of road features are determined using at least one of: a perception sensor and data retrieved from a remote server, wherein the plurality of road features include at least one road geometry feature, at least one road condition feature, and at least one roadside environment feature.

3. The method according to claim 1, wherein: Determining the plurality of road user information further includes: Detecting multiple road users; Detect multiple roadside users; determining a plurality of predicted paths for each of the plurality of road users and the plurality of roadside users; and The plurality of road user information is determined, wherein the plurality of road user information includes at least the number of the plurality of road users, the number of the plurality of roadside users, and the plurality of predicted paths.

4. The method according to claim 1, wherein: Determining the plurality of weather-related information further includes: The plurality of weather-related information is determined using at least one of: a sensory sensor and data retrieved from a remote server, wherein the plurality of weather-related information includes at least one precipitation weather condition and at least one visibility weather condition.

5. The method according to claim 1, wherein: Determining the predicted traffic sign further includes: The predicted traffic sign is determined using a rule-based algorithm, wherein the rule-based algorithm takes the plurality of road features, the plurality of road user information, and the plurality of weather-related information as inputs and generates the predicted traffic sign as output.

6. The method according to claim 1, wherein: Determining the predicted traffic sign further includes: determining the predicted traffic sign using a fuzzy logic algorithm, wherein the fuzzy logic algorithm takes the plurality of road features, the plurality of road user information, and the plurality of weather-related information as inputs and generates a plurality of possible predicted traffic signs and a plurality of corresponding confidence values ​​as outputs; and The predicted traffic sign is determined to be one of the plurality of possible predicted traffic signs based at least in part on the plurality of corresponding confidence values.

7. The method according to claim 1, wherein: Determining the predicted traffic sign further includes: determining the predicted traffic sign using a machine learning algorithm, wherein the machine learning algorithm takes the plurality of road features, the plurality of road user information, and the plurality of weather-related information as inputs and generates a plurality of possible predicted traffic signs and a plurality of corresponding confidence values ​​as outputs; and The predicted traffic sign is determined to be one of the plurality of possible predicted traffic signs based at least in part on the plurality of corresponding confidence values.

8. The method according to claim 1, wherein: Determining the predicted traffic sign further includes: The predicted traffic sign is validated using a predetermined set of rules.

9. The method according to claim 8, wherein: Verifying the predicted traffic signs also includes: evaluating the predicted traffic sign using the predetermined rule set; and In response to determining that the predicted traffic sign is contrary to the predetermined rule set, the predicted traffic sign is modified.

10. The method according to claim 1, wherein: Performing an action based at least in part on the predicted traffic sign further comprises: transmitting the predicted traffic sign to at least one of: a remote vehicle and a remote server; and Path planning and control algorithms of the automated driving system are adjusted based at least in part on the predicted traffic sign.

Images