TRAFFIC SIGN FORECAST FOR A VEHICLE

Abstract

A traffic sign prediction method involves determining a variety of road surface properties. The method also involves determining a variety of road user information. The method also involves determining a variety of weather-related information. The method also involves determining a predicted traffic sign, at least partially, based on the variety of road surface properties, the variety of road user information, and the variety of weather-related information. The method also involves performing an action, at least partially, based on the predicted traffic sign.

Description

INTRODUCTION

[0001] The present disclosure relates to advanced driver assistance and automated driving systems and methods for vehicles, and more particularly to traffic sign prediction systems and methods for a vehicle.

[0002] To increase occupant alertness and comfort, vehicles can be equipped with advanced driver assistance systems (ADAS) and / or automated driving systems (ADS). ADAS systems can use various sensors such as cameras, radar, and LiDAR (Laser Imaging, Detection, and Ranging) to detect and identify objects in the area around the vehicle, including other vehicles, pedestrians, road configurations, and traffic signs. ADS systems can take actions based on environmental conditions surrounding the vehicle, such as steering the vehicle to navigate it through the environment to a destination. However, the performance of current ADS systems can be compromised by the detection and interpretation of traffic signs. For example, in some cases, traffic signs may be missing, outdated, or otherwise invalid or inapplicable.In other cases, traffic signs or traffic signals may be obscured by weather conditions or the like, making accurate identification and / or recognition difficult.

[0003] Although ADAS and ADS systems and methods serve their purpose, there is a need for a new and improved system and method for traffic sign prediction for a vehicle. SUMMARY

[0004] According to several aspects, a method for traffic sign prediction is provided. The method includes determining a plurality of roadway characteristics. The method also includes determining a plurality of road user information. The method also includes determining a plurality of weather-related information. The method also includes determining a predicted traffic sign based at least in part on the plurality of roadway characteristics, the plurality of road user information, and the plurality of weather-related information. The method also includes performing an action based at least in part on the predicted traffic sign.

[0005] In another aspect of the present disclosure, determining the plurality of roadway characteristics may further comprise determining the plurality of roadway characteristics using a sensor from a perception sensor and data retrieved from a remote server. The plurality of roadway characteristics includes at least one roadway geometry characteristic, at least one roadway condition characteristic, and at least one roadside environmental characteristic.

[0006] In another aspect of the present disclosure, determining the plurality of road user information may further comprise detecting a plurality of road users. Determining the plurality of road user information may further comprise detecting a plurality of roadside users. Determining the plurality of road user information may further comprise determining a plurality of predicted paths of each of the plurality of road users and the plurality of roadside users. Determining the plurality of road user information may further comprise determining the plurality of road user information. The plurality of road user information comprises at least a set of the plurality of road users, a set of the plurality of roadside users, and the plurality of predicted paths.

[0007] In another aspect of the present disclosure, determining the plurality of weather-related information items may further comprise determining the plurality of weather-related information items using a sensor from a perception sensor and data retrieved from a remote server. The plurality of weather-related information items includes at least one precipitation weather condition and at least one visibility weather condition.

[0008] In another aspect of the present disclosure, determining the predicted traffic sign may further comprise determining the predicted traffic sign using a rule-based algorithm. The rule-based algorithm takes the plurality of roadway characteristics, the plurality of road user information, and the plurality of weather-related information as inputs and produces the predicted traffic sign as output.

[0009] In another aspect of the present disclosure, determining the predicted traffic sign may further comprise determining the predicted traffic sign using a fuzzy logic algorithm. The fuzzy logic algorithm takes the plurality of roadway characteristics, the plurality of road user information, and the plurality of weather-related information as inputs and produces a plurality of possible predicted traffic signs and a corresponding plurality of confidence values ​​as output. Determining the predicted traffic sign may further comprise determining the predicted traffic sign as one of the plurality of possible predicted traffic signs based at least in part on the corresponding plurality of confidence values.

[0010] In another aspect of the present disclosure, determining the predicted traffic sign may further comprise determining the predicted traffic sign using a machine learning algorithm. The machine learning algorithm takes the plurality of roadway characteristics, the plurality of road user information, and the plurality of weather-related information as inputs and produces a plurality of possible predicted traffic signs and a corresponding plurality of confidence values ​​as output. Determining the predicted traffic sign may further comprise determining the predicted traffic sign as one of the plurality of possible predicted traffic signs based at least in part on the corresponding plurality of confidence values.

[0011] In another aspect of the present disclosure, determining the predicted traffic sign may further comprise validating the predicted traffic sign using a predetermined set of rules.

[0012] In another aspect of the present disclosure, validating may further comprise evaluating the predicted traffic sign with the predetermined rule set; modifying the predicted traffic sign in response to determining that the predicted traffic sign conflicts with the predetermined rule set.

[0013] In another aspect of the present disclosure, performing the action based at least in part on the predicted traffic sign may further comprise transmitting the predicted traffic sign to at least one of a remote vehicle and a remote server. Performing the action based at least in part on the predicted traffic sign may further comprise adjusting a path planning and control algorithm of an automated driving system based at least in part on the predicted traffic sign.

[0014] According to several aspects, a traffic sign prediction system for a vehicle is provided. The system includes a plurality of vehicle sensors. The system also includes a controller in electrical communication with the plurality of vehicle sensors. The controller is programmed to determine a plurality of roadway characteristics using at least one of the plurality of vehicle sensors. The controller is further programmed to determine a plurality of road user information using at least one of the plurality of vehicle sensors. The controller is further programmed to determine a plurality of weather-related information using at least one of the plurality of vehicle sensors.The controller is further programmed to determine a predicted traffic sign based at least in part on the plurality of roadway characteristics, the plurality of road user information, and the plurality of weather-related information. The controller is further programmed to perform an action based at least in part on the predicted traffic sign.

[0015] In another aspect of the present disclosure, the plurality of vehicle sensors further includes at least one perception sensor. To determine the plurality of road characteristics, the controller is further programmed to perform a plurality of measurements of an environment surrounding the vehicle using the at least one perception sensor. To determine the plurality of road characteristics, the controller is further programmed to determine the plurality of road characteristics based at least in part on the plurality of measurements. The plurality of road characteristics includes at least one road geometry characteristic, at least one road condition characteristic, and at least one roadside environmental characteristic.

[0016] In another aspect of the present disclosure, the plurality of vehicle sensors further comprises a vehicle communication system. To determine the plurality of road characteristics, the controller is further programmed to receive at least one of a V2V message and a V2X message comprising the plurality of road characteristics using the vehicle communication system. The plurality of road characteristics comprises at least one road geometry characteristic, at least one road condition characteristic, and at least one roadside environmental characteristic.

[0017] In another aspect of the present disclosure, to determine the plurality of road user information items, the controller is programmed to detect a plurality of road users using the plurality of vehicle sensors. To determine the plurality of road user information items, the controller is further programmed to detect a plurality of roadside users using the plurality of vehicle sensors. To determine the plurality of road user information items, the controller is further programmed to determine a plurality of predicted paths of each of the plurality of road users and the plurality of roadside users. To determine the plurality of road user information items, the controller is further programmed to determine the plurality of road user information items.The plurality of road user information includes at least a set of the plurality of road users, a set of the plurality of roadside users, and the plurality of predicted paths.

[0018] In another aspect of the present disclosure, the plurality of vehicle sensors further includes at least one perception sensor and a vehicle communication system. To determine the plurality of weather-related information items, the controller is further programmed to determine the plurality of weather-related information items using at least one of the perception sensor and the vehicle communication system. The plurality of weather-related information items includes at least one precipitation weather condition and at least one visibility weather condition.

[0019] In another aspect of the present disclosure, the controller is programmed to determine the predicted traffic sign using a prediction algorithm. The prediction algorithm takes the plurality of roadway characteristics, the plurality of road user information, and the plurality of weather-related information as inputs and produces a plurality of possible predicted traffic signs and a corresponding plurality of confidence values ​​as outputs. The prediction algorithm is at least one of a rule-based algorithm, a fuzzy logic algorithm, and a machine learning algorithm.To determine the predicted traffic sign, the controller is further programmed to determine the predicted traffic sign as one of the plurality of possible predicted traffic signs based at least in part on the corresponding plurality of confidence values. To determine the predicted traffic sign, the controller is further programmed to validate the predicted traffic sign by evaluating the predicted traffic sign against a predetermined set of rules. To determine the predicted traffic sign, the controller is further programmed to modify the predicted traffic sign in response to determining that the predicted traffic sign conflicts with the predetermined set of rules.

[0020] In another aspect of the present disclosure, the system further comprises an automated driving system in electrical communication with the controller. The plurality of vehicle sensors further comprises a vehicle communication system. To perform the action, the controller is further programmed to transmit the predicted traffic sign to at least one of a remote vehicle and a remote server. To perform the action, the controller is further programmed to adjust a path planning and control algorithm of the automated driving system based at least in part on the predicted traffic sign.

[0021] According to several aspects, a traffic sign prediction system for a vehicle is provided. The system includes a plurality of vehicle sensors. The plurality of vehicle sensors includes at least one perception sensor. The plurality of vehicle sensors further includes at least one vehicle communication system. The system also includes an automated driving system. The system also includes a controller in electrical communication with the plurality of vehicle sensors and the automated driving system. The controller is programmed to determine a plurality of roadway characteristics using at least one of the plurality of vehicle sensors. The plurality of roadway characteristics includes at least one roadway geometry characteristic, at least one roadway condition characteristic, and at least one roadside environment characteristic.The controller is further programmed to determine a plurality of road user information items using at least one of the plurality of vehicle sensors. The controller is further programmed to determine a plurality of weather-related information items using at least one of the plurality of vehicle sensors. The plurality of weather-related information items includes at least one precipitation weather condition and at least one visibility weather condition. The controller is further programmed to determine a predicted traffic sign based at least in part on the plurality of roadway characteristics, the plurality of road user information items, and the plurality of weather-related information items. The controller is further programmed to transmit the predicted traffic sign to at least one of a remote vehicle and a remote server.The control unit is further programmed to adjust a path planning and control algorithm of the automated driving system based at least in part on the predicted traffic sign.

[0022] In another aspect of the present disclosure, to determine the plurality of road user information items, the controller is programmed to detect a plurality of road users using the plurality of vehicle sensors. To determine the plurality of road user information items, the controller is further programmed to detect a plurality of roadside users using the plurality of vehicle sensors. To determine the plurality of road user information items, the controller is further programmed to determine a plurality of predicted paths of each of the plurality of road users and the plurality of roadside users. To determine the plurality of road user information items, the controller is further programmed to determine the plurality of road user information items.The plurality of road user information includes at least a set of the plurality of road users, a set of the plurality of roadside users, and the plurality of predicted paths.

[0023] In another aspect of the present disclosure, the controller is programmed to determine the predicted traffic sign using a prediction algorithm. The prediction algorithm takes the plurality of roadway characteristics, the plurality of road user information, and the plurality of weather-related information as inputs and produces a plurality of possible predicted traffic signs and a corresponding plurality of confidence values ​​as outputs. The prediction algorithm is at least one of a rule-based algorithm, a fuzzy logic algorithm, and a machine learning algorithm.To determine the predicted traffic sign, the controller is further programmed to determine the predicted traffic sign as one of the plurality of possible predicted traffic signs based at least in part on the corresponding plurality of confidence values. To determine the predicted traffic sign, the controller is further programmed to validate the predicted traffic sign by evaluating the predicted traffic sign against a predetermined set of rules. To determine the predicted traffic sign, the controller is further programmed to modify the predicted traffic sign in response to determining that the predicted traffic sign conflicts with the predetermined set of rules.

[0024] Further areas of applicability will become apparent from the description provided herein. It should be understood that the description and specific examples are for purposes of illustration only and are not intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. Fig. 1 is a schematic diagram of a traffic sign prediction system for a vehicle, according to an exemplary embodiment; and Fig. 2 is a flowchart of a method for traffic sign prediction, according to an exemplary embodiment. DETAILED DESCRIPTION

[0026] The following description is merely exemplary and is not intended to limit the present disclosure, application, or uses.

[0027] The identification and recognition of traffic signs can be important for the accurate operation of automated driving systems. In some cases, traffic signs may be missing or obscured. In other cases, dynamic road conditions, such as road construction, weather conditions, traffic volume, and / or the like, may render posted traffic signs invalid and / or irrelevant. Accordingly, the present disclosure provides a new and improved system and method for traffic sign prediction for a vehicle that enables traffic sign prediction in situations where current traffic signs are missing, obscured, or otherwise unreadable.

[0028] With reference to Fig. 1, a system for interpreting colors is illustrated and generally designated by reference numeral 10. System 10 is shown with an exemplary vehicle 12. Although a passenger car is illustrated, vehicle 12 may be any type of vehicle without departing from the scope of the present disclosure. System 10 generally includes a controller 14, a plurality of vehicle sensors 16, an automated driving system 18, and a display 20.

[0029] The controller 14 is used to implement a traffic sign prediction method 100, described below. The controller 14 includes at least one processor 22 and a non-transitory storage device or media 24. The processor 22 may be any custom or off-the-shelf processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among multiple processors associated with the controller 14, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a device for executing instructions. The computer-readable storage device or media 24 may include volatile and non-volatile memory, such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM).A KAM is a persistent or non-volatile memory that can be used to store various operating variables while the processor 22 is powered off. The computer-readable storage device or media 24 can be implemented using a number of storage devices, such as PROMs (programmable read-only memories), EPROMs (erasable PROMs), EEPROMs (electrically erasable PROMs), flash memory, or other electrical, magnetic, optical, or combination storage devices capable of storing data, some of which represent executable instructions used by the controller 14 in controlling various systems of the vehicle 12. The controller 14 may also consist of multiple controllers that are in electrical communication with one another.The controller 14 can be connected to other systems and / or controls of the vehicle 12 so that the controller 14 can access data such as speed, acceleration, braking and steering angle of the vehicle 12.

[0030] The controller 14 is in electrical communication with the plurality of vehicle sensors 16, the automated driving system 18, and the display 20. In an exemplary embodiment, the electrical communication is established using, for example, a CAN network, a FLEXRAY network, a local area network (e.g., Wi-Fi, Ethernet, etc.), a Serial Peripheral Interface (SPI) network, or the like. It is understood that various additional wired and wireless technologies and communication protocols for connecting to the controller 14 are within the scope of the present disclosure.

[0031] The plurality of vehicle sensors 16 are used to collect information about an environment surrounding the vehicle 12. In an exemplary embodiment, the plurality of vehicle sensors 16 includes at least one camera system 26 (i.e., a perception sensor) and a vehicle communication system 28.

[0032] In another exemplary embodiment, the plurality of vehicle sensors 16 further includes sensors for determining performance data of the vehicle 12. In one non-limiting example, the plurality of vehicle sensors 16 further includes at least one of an engine speed sensor, an engine torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a brake position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor.

[0033] In another exemplary embodiment, the plurality of vehicle sensors 16 further includes sensors for determining information about the environment within the vehicle 12. In one non-limiting example, the plurality of vehicle sensors 16 further includes at least one seat occupancy sensor, a cabin air temperature sensor, a cabin motion detection sensor, a cabin camera, a cabin microphone, and / or the like.

[0034] In another exemplary embodiment, the plurality of vehicle sensors 16 further includes sensors for determining information about the environment surrounding the vehicle 12. In one non-limiting example, the plurality of vehicle sensors 16 further includes at least one of an ambient temperature sensor, a barometric pressure sensor, a global navigation satellite system (GNSS), and / or a still and / or video camera positioned to view the environment in front of the vehicle 12.

[0035] In another exemplary embodiment, at least one of the plurality of vehicle sensors 16 is a perception sensor capable of measuring distances in the environment surrounding the vehicle 12. In one non-limiting example, the plurality of vehicle sensors 16 includes a stereoscopic camera with distance measurement capabilities. In one example, at least one of the plurality of vehicle sensors 16 is mounted inside the vehicle 12, for example, in a headliner of the vehicle 12, with a view through a windshield of the vehicle 12. In another example, at least one of the plurality of vehicle sensors 16 is mounted outside the vehicle 12, for example, on a roof of the vehicle 12, with a view of the environment in front of the vehicle 12.It is understood that various additional types of vehicle sensors, such as LiDAR sensors, ultrasonic sensors, radar sensors, time-of-flight sensors, and / or other types of perception sensors, are within the scope of the present disclosure. The plurality of vehicle sensors 16 are in electrical communication with the controller 14 as described above.

[0036] The camera system 26 is a perception sensor capable of capturing images and / or videos of the environment surrounding the vehicle 12. In an exemplary embodiment, the camera system 26 includes a still and / or video camera positioned to view the environment surrounding the vehicle 12. In one non-limiting example, the camera system 26 includes a camera mounted inside the vehicle 12, for example, in a headliner of the vehicle 12, having a view through a windshield. In another non-limiting example, the camera system 26 includes a camera mounted outside the vehicle 12, for example, on a roof of the vehicle 12, having a view of the environment in front of the vehicle 12.

[0037] In another exemplary embodiment, the camera system 26 is a surround-view camera system that includes a plurality of cameras (also known as satellite cameras) arranged to provide a view of the surroundings on all sides of the vehicle 12. In one non-limiting example, the camera system 26 includes a forward-facing camera (e.g., mounted in a grille of the vehicle 12), a rearward-facing camera (e.g., mounted on a tailgate of the vehicle 12), and two side-facing cameras (e.g., mounted below each of the two side view mirrors of the vehicle 12). In another non-limiting example, the camera system 26 further includes an additional rearview camera mounted near a high-mounted center brake light of the vehicle 12.

[0038] It is understood that camera systems having additional cameras and / or additional mounting locations are within the scope of the present disclosure. It is understood that cameras having different sensor types, for example, charge-coupled device (CCD) sensors, complementary metal oxide semiconductor (CMOS) sensors, and / or high dynamic range (HDR) sensors, are within the scope of the present disclosure. Furthermore, cameras having different lens types, for example, wide-angle and / or narrow-angle lenses, are also within the scope of the present disclosure.

[0039] The vehicle communication system 28 is used by the control unit 14 to communicate with other systems outside the vehicle 12. The vehicle communication system 28 includes, for example, functions for communication with vehicles ("V2V" communication), with the infrastructure ("V2I" communication), remote systems in a remote call center (e.g., ON-STAR from GENERAL MOTORS), and / or personal devices. In general, the term vehicle-to-everything communication ("V2X" communication) refers to communication between the vehicle 12 and any remote system (e.g., vehicles, infrastructure, and / or remote systems). In certain embodiments, the vehicle communication system 28 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using IEEE 802.11 standards or via cellular data communication (e.g.,using GSMA standards, such as SGP.02, SGP.22, SGP.32, and the like). Accordingly, the vehicle communication system 28 may further include an embedded universal integrated circuit card (eUICC) configured to store at least one configuration profile for cellular connectivity, for example, an embedded subscriber identity module (eSIM). The vehicle communication system 28 is further configured to communicate via a personal area network (e.g., BLUETOOTH) and / or near-field communication (NFC). Additional or alternative communication methods, such as a dedicated short-range communications (DSRC) channel and / or mobile telecommunications protocols based on the 3rd Generation Partnership Project standards, may be used.However, communication methods based on the 3rd Generation Partnership Project (3GPP) are also considered within the scope of the present disclosure. DSRC channels refer to short- to medium-range, one-way or two-way wireless communication channels specifically designed for use in vehicles, as well as a corresponding set of protocols and standards. 3GPP refers to a partnership between several standards organizations that develop protocols and standards for mobile telecommunications. 3GPP standards are structured as "releases." Therefore, communication methods based on 3GPP versions 14, 15, 16, and / or future 3GPP versions are within the scope of the present disclosure.Accordingly, the vehicle communication system 28 may include one or more antennas and / or communication transceivers for receiving and / or transmitting signals, such as Cooperative Sensing Messages (CSMs). The vehicle communication system 28 is configured to wirelessly communicate information between the vehicle 12 and another vehicle. Furthermore, the vehicle communication system 28 is configured to wirelessly communicate information between the vehicle 12 and infrastructure or other vehicles. It is understood that the vehicle communication system 28 may be integrated with the control unit 14 (e.g., on the same circuit board as the control unit 14 or otherwise as part of the control unit 14) without departing from the scope of the present disclosure.

[0040] The automated driving system 18 is used to provide assistance to an occupant to increase occupant alertness and / or control the behavior of the vehicle 12. Within the scope of the present disclosure, the occupant includes a driver and / or a passenger of the vehicle 12. Within the scope of the present disclosure, the automated driving system 18 includes systems that provide any level of assistance to the occupant (e.g., blind spot warning, lane departure warning, and / or the like), as well as systems capable of autonomously controlling the vehicle 12 under some or all conditions. It is understood that all levels of driving automation, such as those defined in SAE J3016 (i.e., SAE LEVEL 0, SAE LEVEL 1, SAE LEVEL 2, SAE LEVEL 3, SAE LEVEL 4, and SAE LEVEL 5), are within the scope of the present disclosure.

[0041] In an exemplary embodiment, the automated driving system 18 is configured to detect and / or receive information about the environment surrounding the vehicle 12 and process the information to provide assistance to the occupant. In some embodiments, the automated driving system 18 is a software module executing on the controller 14. In other embodiments, the automated driving system 18 includes a separate automated driving system controller, similar to the controller 14, that is capable of processing the information about the environment surrounding the vehicle 12. In an exemplary embodiment, the automated driving system 18 can operate in a manual mode of operation, a partially automated mode of operation, and / or a fully automated mode of operation.

[0042] Within the scope of the present disclosure, the manual mode of operation means that the automated driving system 18 warns or notifies the occupant, but does not intervene or directly control the vehicle 12. In one non-limiting example, the automated driving system 18 receives information from the plurality of vehicle sensors 16. Using techniques such as, for example, computer vision, the automated driving system 18 understands the environment surrounding the vehicle 12 and provides assistance to the occupant. For example, if the automated driving system 18 identifies, based on data from the plurality of vehicle sensors 16, that the vehicle 12 is likely to collide with a distant vehicle, the automated driving system 18 may use the display 20 to provide a warning to the occupant.

[0043] Within the scope of the disclosure, the partially automated mode of operation means that the automated driving system 18 warns or notifies the occupant and may intervene or directly control the vehicle 12 in certain situations. In one non-limiting example, the automated driving system 18 is additionally in electrical communication with components of the vehicle 12, such as a braking system, a propulsion system, and / or a steering system of the vehicle 12, so that the automated driving system 18 may control the behavior of the vehicle 12. In one non-limiting example, the automated driving system 18 may control the behavior of the vehicle 12 by applying the brakes of the vehicle 12 to avoid an impending collision.In another non-limiting example, the automated driving system 18 may control the braking system, the propulsion system, and the steering system of the vehicle 12 to temporarily drive the vehicle 12 to a predetermined destination. However, occupant intervention may be required at any time. In an exemplary embodiment, the automated driving system 18 may include additional components, such as an eye monitoring device configured to monitor the occupant's level of alertness and ensure the occupant is ready to assume control of the vehicle 12.

[0044] Within the scope of the present disclosure, the fully automated mode of operation means that the automated driving system 18 uses data from the plurality of vehicle sensors 16 to understand the environment and control the vehicle 12 such that the vehicle 12 travels to a predetermined destination without requiring control or intervention from the occupant.

[0045] The automated driving system 18 operates using a path planning and control algorithm configured to generate a safe and efficient trajectory for the vehicle 12 to navigate the environment surrounding the vehicle 12. In an exemplary embodiment, the path planning and control algorithm is a machine learning algorithm trained to output control signals for the vehicle 12 based on input data collected from the plurality of vehicle sensors 16.

[0046] In another exemplary embodiment, the path planning and control algorithm is a deterministic algorithm programmed to output control signals to the vehicle 12 based on data collected from the plurality of vehicle sensors 16.

[0047] As one non-limiting example, the path planning and control algorithm performs perception and mapping tasks to interpret the data collected from the plurality of vehicle sensors 16 and generate a detailed map of the surrounding environment. The detailed map may include information about lane boundaries, road geometry, speed limits, traffic signs, and / or other relevant features. Based on the detailed map and a current state of the vehicle 12 (i.e., position, speed, and orientation of the vehicle 12), the path planning and control algorithm generates a sequence of waypoints or a continuous path that the vehicle 12 should follow to reach a destination while adhering to rules, regulations, and safety restrictions.It is understood that the automated driving system 18 may include any software and / or hardware module configured to operate in manual mode of operation, semi-automated mode of operation, or fully automated mode of operation as described above.

[0048] The display 20 is used to provide information to the occupant of the vehicle 12. In the Fig. 1, the display 20 is a human-machine interface (HMI) located in the occupant's field of view and capable of displaying text, graphics, and / or images. It is understood that HMI display systems including LCD displays, LED displays, and the like are within the scope of the present disclosure. Other exemplary embodiments in which the display 20 is disposed in a rearview mirror are also within the scope of the present disclosure. In another exemplary embodiment, the display 20 includes a head-up display (HUD) configured to provide information to the occupant by projecting text, graphics, and / or images onto the windshield of the vehicle 12.The text, graphics, and / or images are reflected from the windshield of the vehicle 12 and are visible to the occupant without having to look away from the roadway in front of the vehicle 12. In another exemplary embodiment, the display 20 includes an augmented reality head-up display (AR-HUD). The AR-HUD is a type of HUD configured to enhance the occupant's view of the roadway in front of the vehicle 12 by overlaying text, graphics, and / or images on physical objects in the environment surrounding the vehicle 12 within a field of view of the occupant. In an exemplary embodiment, the user may interact with the display 20 via a human-interface device (HID), such as a touchscreen, an electromechanical switch, a capacitive switch, a rotary knob, and the like.It will be appreciated that additional systems for displaying information to the occupants of the vehicle 12 are also within the scope of the present disclosure.

[0049] With reference to Fig. Figure 2 shows a flowchart of the traffic sign prediction method 100. The method 100 begins at block 102 and continues with blocks 104, 106, 108, and 110.

[0050] In block 104, the controller 14 performs a plurality of measurements of the environment surrounding the vehicle 12 using the plurality of vehicle sensors 16. In an exemplary embodiment, the plurality of measurements are performed using a perception sensor of the plurality of vehicle sensors 16, such as the camera system 26, a LiDAR system, a radar system, and / or the like. In one non-limiting example, the plurality of measurements includes distance measurements from the vehicle 12 to a plurality of points in the environment surrounding the vehicle 12. In another non-limiting example, the plurality of measurements includes images and / or videos of the environment surrounding the vehicle 12 captured by the camera system 26.In another non-limiting example, the plurality of measurements includes data about the environment surrounding the vehicle 12 received using the vehicle communication system 28. For example, the controller 14 may use the vehicle communication system 28 to transmit the location of the vehicle 12 (as determined using GNSS) to a remote server. The remote server may respond by transmitting information about the environment surrounding the vehicle 12 to the vehicle communication system 28 (e.g., using at least one of a V2V message and / or a V2X message), including, for example, information about the roadway geometry, roadway conditions, and / or roadside environment of the environment surrounding the vehicle 12. After block 104, the method 100 proceeds to block 112.

[0051] In block 112, the controller 14 determines a plurality of road surface properties based at least in part on the plurality of measurements taken in block 104. Within the scope of the present disclosure, the plurality of road surface properties are properties of a road surface in the environment surrounding the vehicle. In an exemplary embodiment, the plurality of road surface properties include at least one road surface geometry property (i.e., properties of a geometry of the road surface), at least one road surface condition property (i.e., properties of a surface condition of the road surface), and at least one road surface property (i.e., properties of an environmental context of the road surface).In one non-limiting example, the at least one roadway geometry characteristic includes, for example, a roadway width, a horizontal curve radius, a sight distance, a vertical curve radius, a vertical elevation, the presence of an intersection, and / or the like. In one non-limiting example, the at least one roadway condition characteristic includes, for example, a pavement condition of the roadway. In one non-limiting example, the at least one roadside environment characteristic includes, for example, a road function (e.g., urban, rural, highway), the presence of driveways along the roadway, the presence of roadside development (e.g., commercial buildings and / or residential buildings), the width of the shoulder pavement, the condition of the shoulder pavement, the presence of pedestrian infrastructure along the roadway (e.g., sidewalks), and / or the like.After block 112, the method 100 continues to block 114, which is explained in more detail below.

[0052] In block 106, the controller 14 detects a plurality of road users. Within the scope of the present disclosure, the plurality of road users are users of the roadway adjacent to the vehicle 12, for example, remote vehicles, construction workers, pedestrians, cyclists, wildlife, and / or the like. In an exemplary embodiment, the controller 14 uses one or more of the plurality of vehicle sensors 16 to detect the plurality of road users. In one non-limiting example, the controller 14 uses the vehicle communication system 28 to receive V2V and / or V2X messages from remote vehicles and / or infrastructure. The V2V and / or V2X messages include information about one or more road users, for example, the location, speed, and heading of a remote vehicle on the roadway.In another non-limiting example, controller 14 uses camera system 26 to capture a plurality of images of the environment surrounding vehicle 12 and a computer vision algorithm to analyze the plurality of images and identify the plurality of road users. In another non-limiting example, controller 14 uses another type of perception sensor, such as a LiDAR sensor, a radar sensor, and / or the like, to take a plurality of measurements of the environment surrounding vehicle 12 and identify the plurality of road users. After block 106, method 100 proceeds to block 116, which is explained in more detail below.

[0053] In block 108, the controller 14 detects a plurality of roadside users. Within the scope of the present disclosure, the plurality of roadside users are users of the vicinity of the roadway (i.e., the roadside). The plurality of roadside users may include, for example, pedestrians, cyclists, and animals on a sidewalk and / or shoulder of the roadway, stopped / parked vehicles on the shoulder of the roadway, and / or the like. In an exemplary embodiment, the controller 14 uses one or more of the plurality of vehicle sensors 16 to detect the plurality of roadside users. In one non-limiting example, the controller 14 uses the vehicle communication system 28 to receive V2V and / or V2X messages from remote vehicles and / or infrastructure.The V2V and / or V2X messages include information about one or more roadside users, such as the location of a remote vehicle parked on the roadside. In another non-limiting example, the controller 14 uses the camera system 26 to capture a plurality of images of the environment surrounding the vehicle 12 and a computer vision algorithm to analyze the plurality of images and identify the plurality of roadside users. In another non-limiting example, the controller 14 uses another type of perception sensor, such as a LiDAR sensor, a radar sensor, and / or the like, to take a plurality of measurements of the environment surrounding the vehicle 12 and identify the plurality of roadside users. After block 108, the method 100 proceeds to block 116, which is explained in more detail below.

[0054] In block 116, the controller 14 determines a plurality of predicted paths of each of the plurality of road users and the plurality of roadside users identified in blocks 108 and 108. In an exemplary embodiment, the plurality of predicted paths is determined using a statistical algorithm, such as a Kalman filter algorithm. The Kalman filter algorithm is a recursive algorithm that combines measurements from the plurality of vehicle sensors 16 with system dynamics to estimate the state of a dynamic system. In the context of path prediction, the Kalman filter considers measurements such as position, velocity, and acceleration to account for the current and future state of the plurality of road users and the plurality of roadside users.

[0055] As a non-limiting example, the Kalman filter algorithm operates in two main steps: prediction and update. During the prediction step, the Kalman filter algorithm uses a prior state estimate and a system dynamics model to project the expected state of the plurality of road users and the plurality of roadside users in the next time step. This prediction incorporates information about the plurality of road users and the plurality of roadside users, such as speed, acceleration, direction, and / or the like. In the update step, the Kalman filter algorithm combines the predicted state with real-time measurements from the plurality of vehicle sensors 16.By iteratively repeating the prediction and update steps, the Kalman filter algorithm continuously refines and predicts the future path of the plurality of road users and the plurality of roadside users. After block 116, the method 100 proceeds to block 118.

[0056] In block 118, the controller 14 determines a plurality of road user information items based at least in part on the plurality of road users detected in block 106, the plurality of roadside users detected in block 108, and the plurality of predicted paths determined in block 116. In an exemplary embodiment, the plurality of road user information items includes at least a set of road users (i.e., a set of remotely operated vehicles, construction workers, pedestrians, cyclists, wildlife, and / or the like on the roadway), a set of roadside users (i.e., a set of pedestrians, cyclists, and animals on the sidewalk and / or shoulder of the roadway, stopped / parked vehicles on the shoulder of the roadway, and / or the like), and the plurality of predicted paths determined in block 116.In an exemplary embodiment, the plurality of road user information items may include further details, such as the type of remote vehicle on the roadway / roadside (e.g., construction vehicle, emergency vehicle, semi-trailer truck, and / or the like). Furthermore, the plurality of road user information items may be filtered by relevance. For example, information related to remote vehicles traveling on an opposite side of a divided highway from vehicle 12 may not be relevant, and therefore, such information may be removed from the plurality of road user information items. After block 118, the method 100 proceeds to block 114, which is explained in more detail below.

[0057] In block 110, the controller 14 determines a plurality of weather-related information items. Within the scope of the present disclosure, the plurality of weather-related information items is information about the weather in the environment surrounding the vehicle 12. In an exemplary embodiment, the plurality of weather-related information items includes at least one precipitation weather condition and at least one visibility weather condition. In one non-limiting example, the at least one precipitation weather condition includes, for example, the presence of rain, snow, sleet, hail, clear skies, and the like. The at least one visibility weather condition includes, for example, the presence and / or density of fog, a visibility range, and / or the like.In one exemplary embodiment, to determine the plurality of weather-related information, the controller 14 uses a perception sensor of the plurality of vehicle sensors 16 (e.g., the camera system 26) to capture an image of the environment surrounding the vehicle 12. The image is then analyzed, for example, with a computer vision algorithm, to determine the plurality of weather-related information. In another exemplary embodiment, the controller 14 uses the vehicle communication system 28 of the plurality of vehicle sensors 16 to retrieve data from a remote server (e.g., an Internet application programming interface that provides weather data), the data comprising the plurality of weather-related information. After block 110, the method 100 proceeds to block 114.

[0058] In block 114, controller 14 determines a predicted traffic sign based at least in part on the plurality of roadway characteristics determined in block 112, the plurality of road user information determined in block 118, and the plurality of weather-related information determined in block 110. Within the scope of the present disclosure, the predicted traffic sign is a traffic sign (e.g., stop sign, yield sign, speed limit sign, and / or the like) and / or a traffic signal predicted for the road conditions in the environment surrounding vehicle 12. The predicted traffic sign is determined using a prediction algorithm.

[0059] In a first exemplary embodiment, the prediction algorithm is a rule-based algorithm. Within the scope of the present disclosure, the rule-based algorithm is an algorithm that operates according to a predefined set of rules and / or conditions to determine the predicted traffic sign. In one non-limiting example, the predefined set of rules is defined by standards, such as, for example, the Manual on Uniform Traffic Control Devices. In one non-limiting example, the rule-based algorithm is implemented using a lookup table (LUT) that associates the plurality of roadway characteristics determined in block 112, the plurality of road user information determined in block 118, and the plurality of weather-related information determined in block 110 with a predicted traffic sign.

[0060] The LUT has three key columns (i.e., one key column for each of the plurality of road surface characteristics, the plurality of road user information items, and the plurality of weather-related information items) and one value column (i.e., a value column for the predicted traffic sign). In an exemplary embodiment, the LUT includes a plurality of rows, each of the plurality of rows mapping a unique combination of the plurality of road surface characteristics, the plurality of road user information items, and the plurality of weather-related information items in the three key columns to a value in the value column (i.e., a predicted traffic sign). The LUT is stored on the media 24 of the controller 14. In an exemplary embodiment, the number of rows of the LUT is predetermined.In another exemplary embodiment, the plurality of rows of the LUT may be modified by the user, for example, using a user interface. In another exemplary embodiment, the plurality of rows of the LUT may be updated wirelessly (OTA, over-the-air) using the vehicle communication system 28. It is understood that any rule-based algorithm (e.g., programmatic data structure, logical equation, mathematical function, and / or the like) is within the scope of the present disclosure.

[0061] In a second exemplary embodiment, the prediction algorithm is a fuzzy logic algorithm. Within the scope of the present disclosure, the fuzzy logic algorithm is an extension of classical (Boolean) logic that allows for the representation and manipulation of partial truth or degrees of membership. For example, in fuzzy logic, variables can assume degrees of truth between 0 and 1 instead of strictly true or false values, where 0 represents absolute falsehood, 1 represents absolute truth, and values ​​in between represent various degrees of truth.In one non-limiting example, the fuzzy logic algorithm takes the plurality of roadway characteristics determined in block 112, the plurality of road user information determined in block 118, and the plurality of weather-related information determined in block 110 as inputs and produces a plurality of possible predicted traffic signs and a plurality of confidence values ​​as outputs. Each of the plurality of confidence values ​​corresponds to one of the plurality of possible predicted traffic signs. The predicted traffic sign is selected from the plurality of possible predicted traffic signs based at least in part on the plurality of confidence values. In one non-limiting example, one of the plurality of possible predicted traffic signs with the highest confidence value is determined as the predicted traffic sign.

[0062] In a third exemplary embodiment, the prediction algorithm is a machine learning algorithm. In one non-limiting example, the machine learning algorithm takes the plurality of roadway characteristics determined in block 112, the plurality of road user information determined in block 118, and the plurality of weather-related information determined in block 110 as inputs and produces a plurality of possible predicted traffic signs and a plurality of confidence values ​​as outputs. Each of the plurality of confidence values ​​corresponds to one of the plurality of possible predicted traffic signs. The predicted traffic sign is selected from the plurality of possible predicted traffic signs based at least in part on the plurality of confidence values.In one non-limiting example, one of the plurality of possible predicted traffic signs with the highest confidence value is determined as the predicted traffic sign.

[0063] As a non-limiting example, the machine learning algorithm is a deep neural network with multiple layers, including an input layer and an output layer, and one or more hidden layers. The input layer receives as inputs the plurality of roadway characteristics determined in block 112, the plurality of road user information determined in block 118, and the plurality of weather-related information determined in block 110. The inputs are then passed to the hidden layers. Each hidden layer applies a transformation (e.g., a non-linear transformation) to the data and passes the result to the next hidden layer until the last hidden layer is reached. The output layer produces the plurality of possible predicted traffic signs and the plurality of confidence values.

[0064] To train the machine learning algorithm, a dataset containing inputs and their corresponding traffic signs is used. The algorithm is trained by adjusting the internal weights between nodes in each hidden layer to minimize prediction errors. During training, an optimization technique (e.g., gradient descent) is used to adjust the internal weights and reduce the prediction error. The training process is repeated with the entire dataset until the prediction error is minimized, and the resulting trained model is then used for traffic sign prediction based on new input data.

[0065] After sufficient training of the machine learning algorithm, the algorithm is able to accurately and precisely predict traffic signs based on the multitude of road surface characteristics determined in block 112, the multitude of road user information determined in block 118, and the multitude of weather-related information determined in block 110. By adjusting the weights between the nodes in each hidden layer during training, the algorithm "learns" to recognize patterns in the data that indicate different traffic signs.

[0066] It is understood that the prediction algorithm may include any method of abductive reasoning to determine the predicted traffic sign based at least in part on the plurality of roadway characteristics determined in block 112, the plurality of road user information determined in block 118, and the plurality of weather-related information determined in block 110, all within the scope of the present disclosure. After block 114, the method 100 proceeds to block 120.

[0067] In block 120, the controller 14 validates the predicted traffic sign determined in block 114. In one exemplary embodiment, the controller 14 uses computer vision and / or scene text recognition (STR) techniques to evaluate data from the plurality of vehicle sensors 16 and validate the predicted traffic sign. In another exemplary embodiment, to validate the predicted traffic sign, the controller 14 evaluates the predicted traffic sign with the predetermined rule set. In one non-limiting example, the predetermined rule set includes rules such as, for example, maximum speed = 75 mph, minimum speed = 15 mph, maximum urban roadway speed = 45 mph, and / or the like.In one non-limiting example, the predetermined rule set includes rules from traffic laws, regulations, standards, and / or the like, such as rules from the Manual on Uniform Traffic Control Devices. If the predicted traffic sign does not conform to the predetermined rule set, the method 100 proceeds to block 122. If the predicted traffic sign does not conflict with the predetermined rule set, the method 100 proceeds to blocks 124 and 126.

[0068] In block 122, the controller 14 changes the predicted traffic sign in response to determining that the predicted traffic sign conflicts with the predetermined rule set in block 120. For example, if the predicted traffic sign is a sign indicating a speed limit of 80 mph, the predicted traffic sign is changed in block 122 to indicate a speed limit of 75 mph according to the exemplary predetermined rule set discussed above. After block 112, the method 100 continues with blocks 124 and 126.

[0069] In block 124, the controller 14 uses the vehicle communication system 28 to transmit the predicted traffic sign. In one exemplary embodiment, the predicted traffic sign is transmitted to a remote vehicle, for example, using a V2V message. In another exemplary embodiment, the predicted traffic sign is transmitted to a remote server, such as an internet-connected map database that includes traffic sign information. After block 124, the method 100 continues to enter a standby state in block 128.

[0070] In block 126, the controller 14 adjusts the path planning and control algorithm of the automated driving system 18 based at least in part on the predicted traffic sign. As described above, the path planning and control algorithm generates a sequence of waypoints or a continuous path that the vehicle 12 should follow to reach a destination based at least in part on traffic signs in the environment surrounding the vehicle 12. Therefore, after determining and validating the predicted traffic sign, the predicted traffic sign is transmitted to the automated driving system 18, and the automated driving system 18 recalculates the path planning and control algorithm to account for the predicted traffic sign. After block 126, the method 100 continues to enter the standby state in block 128.

[0071] In an exemplary embodiment, the controller 14 repeatedly exits the standby state 128 and restarts the method 100 at block 102. In one non-limiting example, the controller 14 exits the standby state 128 and restarts the method 100 after a timer, for example, every three hundred milliseconds.

[0072] The system 10 and method 100 of the present disclosure provide several advantages. When traffic signs along a roadway cannot be identified from the plurality of vehicle sensors 16, or sign recognition methods are unable to detect and / or recognize sign messages, the system 10 and method 100 may be used to predict traffic signs, thereby improving the performance of the automated driving system 18. Further, dynamic road conditions, such as, for example, construction, weather, traffic volume, and / or the like, may cause traffic signs and / or traffic sign information stored in map databases to be invalid or less than ideal. Therefore, the system 10 and method 100 may be used to predict traffic signs based on dynamic conditions.After predicting the traffic sign, the system 10 and method 100 may be used to transmit the predicted traffic sign to other vehicles and / or remote systems, enabling crowd-sourcing of traffic sign predictions.

[0073] The description of the present disclosure is merely exemplary, and variations that do not depart from the gist of the present disclosure are to be considered within the scope of the present disclosure. Such variations should not be considered a departure from the spirit and scope of the present disclosure.

Claims

[1] A method for traffic sign prediction, the method comprising: Determining a variety of road surface properties; Determining a variety of road user information; Determining a variety of weather-related information; Determining a predicted traffic sign based at least in part on the plurality of roadway characteristics, the plurality of road user information, and the plurality of weather-related information; and Performing an action based at least in part on the predicted traffic sign. [2] The method of claim 1, wherein determining the plurality of roadway characteristics further comprises: Determining the plurality of roadway characteristics using at least one of: a perception sensor and data retrieved from a remote server, wherein the plurality of roadway characteristics comprises at least one roadway geometry characteristic, at least one roadway condition characteristic, and at least one roadside environment characteristic. [3] The method of claim 1, wherein determining the plurality of road user information further comprises: Detecting a variety of road users; Detecting a variety of roadside users; Determining a plurality of predicted paths of each of the plurality of road users and the plurality of roadside users; and Determining the plurality of road user information, wherein the plurality of road user information comprises at least a set of the plurality of road users, a set of the plurality of roadside users, and the plurality of predicted paths. [4] The method of claim 1, wherein determining the plurality of weather-related information further comprises: Determining the plurality of weather-related information using at least one of: a perception sensor and data retrieved from a remote server, wherein the plurality of weather-related information comprises at least one precipitation weather condition and at least one visibility weather condition. [5] The method of claim 1, wherein determining the predicted traffic sign further comprises: Determining the predicted traffic sign using a rule-based algorithm, wherein the rule-based algorithm takes the plurality of roadway characteristics, the plurality of road user information, and the plurality of weather-related information as inputs and produces the predicted traffic sign as output. [6] The method of claim 1, wherein determining the predicted traffic sign further comprises: Determining the predicted traffic sign using a fuzzy logic algorithm, wherein the fuzzy logic algorithm takes the plurality of road surface properties, the plurality of road user information, and the plurality of weather-related information as inputs and produces a plurality of possible predicted traffic signs and a corresponding plurality of confidence values as outputs; and Determining the predicted traffic sign as one of the plurality of possible predicted traffic signs based at least in part on the corresponding plurality of confidence values. [7] The method of claim 1, wherein determining the predicted traffic sign further comprises: Determining the predicted traffic sign using a machine learning algorithm, wherein the machine learning algorithm takes the plurality of roadway characteristics, the plurality of road user information, and the plurality of weather-related information as inputs and produces a plurality of possible predicted traffic signs and a corresponding plurality of confidence values as outputs; and Determining the predicted traffic sign as one of the plurality of possible predicted traffic signs based at least in part on the corresponding plurality of confidence values. [8] The method of claim 1, wherein determining the predicted traffic sign further comprises: Validate the predicted traffic sign using a predetermined rule set. [9] The method of claim 8, wherein validating the predicted traffic sign further comprises: Evaluating the predicted traffic sign with the predetermined rule set; and Changing the predicted traffic sign in response to determining that the predicted traffic sign conflicts with the predetermined rule set. [10] The method of claim 1, wherein performing the action based at least in part on the predicted traffic sign further comprises: Transmitting the predicted traffic sign to at least one of: a remote vehicle and a remote server; and Adjusting a path planning and control algorithm of an automated driving system based at least in part on the predicted traffic sign.

Images