VEHICLE SYSTEM AND METHOD FOR INFORMING AUTHORISATIONS ABOUT ROAD CONDITIONS

Abstract

A system for notifying an authority about a vehicle's road condition can comprise a variety of vehicle sensors, a vehicle communication system, and an electronic control unit (ECU) that communicates electrically with the vehicle sensors and the communication system. The ECU is programmed to identify a measurement trigger. The ECU is further programmed to perform a measurement of the vehicle's surroundings using the vehicle sensors in response to the identification of the measurement trigger. The ECU is further programmed to determine a measurement classification based, at least in part, on the measurement. The ECU is further programmed to transmit the measurement and the measurement classification to a remote server system via the vehicle communication system.

Description

INTRODUCTION

[0001] This disclosure relates to systems and methods for notifying an authority of road conditions.

[0002] To increase occupant awareness and comfort, vehicles may be equipped with perception and sensing systems configured to monitor and measure aspects of a vehicle's surroundings. Perception and sensing systems may include, for example, camera systems, light detection and ranging (LiDAR) systems, and / or the like. Perception and sensing systems may be used as part of a vehicle system to enable features such as advanced driver assistance systems (ADAS), automated driving systems (ADS), vehicle safety systems, and / or the like. However, current perception and sensing systems may not be able to collect information about road conditions in the vehicle's surroundings and transmit road condition information to authorities. Therefore, authorities may require manual collection of road condition information.

[0003] While perception and detection systems and procedures serve their purpose, there is a need for a new and improved system and procedure for notifying an authority about road conditions. SUMMARY

[0004] According to several aspects, a system for notifying an authority of road conditions for a vehicle is provided. The system may include a plurality of vehicle sensors, a vehicle communication system, and a controller in electrical communication with the plurality of vehicle sensors and the vehicle communication system. The controller is programmed to identify a measurement trigger. The controller is further programmed to perform a measurement of an environment of the vehicle using the plurality of vehicle sensors in response to identifying the measurement trigger. The controller is further programmed to determine a measurement classification based at least in part on the measurement. The controller is further programmed to communicate the measurement and the measurement classification to a remote server system via the vehicle communication system.

[0005] In another aspect of the present disclosure, the plurality of vehicle sensors includes at least one global navigation satellite system (GNSS). To identify the measurement trigger, the controller is further programmed to determine a location of the vehicle using the GNSS. To identify the measurement trigger, the controller is further programmed to compare the location of the vehicle to a predetermined geographically defined location area. To identify the measurement trigger, the controller is further programmed to identify the measurement trigger in response to determining that the location of the vehicle is within the predetermined geographically defined location area.

[0006] In another aspect of the present disclosure, to identify the measurement trigger, the controller is further programmed to receive a trigger request from a vehicle occupant and / or an authority. To identify the measurement trigger, the controller is further programmed to identify the measurement trigger based at least in part on the trigger request.

[0007] In another aspect of the present disclosure, the plurality of vehicle sensors includes at least one camera system. To perform the measurement of the vehicle's surroundings, the controller is further programmed to capture one or more images of a remote vehicle using the camera system based at least in part on the trigger request. The trigger request includes a license plate number of a remote vehicle, a model of a remote vehicle, and / or a color of a remote vehicle.

[0008] In another aspect of the present disclosure, the measurement classification determination controller is further programmed to determine the measurement classification of the one or more images as a law enforcement measurement classification in response to determining that the remote vehicle in the one or more images includes the license plate number of the remote vehicle, the model of the remote vehicle, and / or the color of the remote vehicle.

[0009] In another aspect of the present disclosure, to identify the measurement trigger, the controller is further programmed to perform a trigger measurement using the plurality of vehicle sensors. To identify the measurement trigger, the controller is further programmed to detect a road condition using a machine learning algorithm during the trigger measurement. The road condition includes one of a normal road condition and an abnormal road condition. To identify the measurement trigger, the controller is further programmed to identify the measurement trigger in response to determining that the road condition is the abnormal road condition.

[0010] In another aspect of the present disclosure, the plurality of vehicle sensors includes at least one camera system. To perform the measurement of the vehicle's surroundings, the controller is further programmed to capture one or more images using the camera system. To perform the measurement of the vehicle's surroundings, the controller is further programmed to store the one or more images in a non-transitory memory of the controller.

[0011] In another aspect of the present disclosure, the plurality of vehicle sensors includes at least one global navigation satellite system (GNSS). To determine the measurement classification, the controller is further programmed to determine a location of the vehicle using the GNSS. To determine the measurement classification, the controller is further programmed to compare the location of the vehicle to a predetermined plurality of point-of-interest (POI) locations. To determine the measurement classification, the controller is further programmed to determine the measurement classification of the one or more images as a tourist measurement classification in response to determining that the location of the vehicle is within a predetermined distance of at least one of the plurality of POI locations.

[0012] In another aspect of the present disclosure, to determine the measurement classification, the controller is further programmed to identify a traffic sign in the one or more images using a computer vision algorithm. To determine the measurement classification, the controller is further programmed to calculate a correlation value between the one or more images and a reference image using the computer vision algorithm. To determine the measurement classification, the controller is further programmed to determine the measurement classification of the one or more images as a damaged traffic sign measurement classification in response to identifying the traffic sign in the one or more images and in response to determining that the correlation value is less than a predetermined correlation threshold.

[0013] In another aspect of the present disclosure, to determine the measurement classification, the controller is further programmed to detect a poor road condition in the one or more images using a machine learning algorithm. The poor road condition includes at least one of a foreign object on a roadway, an animal on the roadway, and a damaged road surface. To determine the measurement classification, the controller is further programmed to determine a risk score for the poor road condition. To determine the measurement classification, the controller is further programmed to determine the measurement classification of the one or more images as a poor road condition measurement classification based at least in part on the poor road condition and the risk score.

[0014] According to several aspects, a method for notifying an authority of road conditions for a vehicle is provided. The method may include performing a measurement of the vehicle's surroundings using a plurality of vehicle sensors. The method may further include determining a measurement classification based at least in part on the measurement. The method may further include communicating the measurement and the measurement classification to an authority.

[0015] In another aspect of the present disclosure, performing the measurement and determining the measurement classification may further comprise capturing one or more images of the surroundings of the vehicle using a camera system. Performing the measurement and determining the measurement classification may further comprise detecting debris in the one or more images using a machine learning algorithm. Performing the measurement and determining the measurement classification may further comprise determining a risk score for the debris. Performing the measurement and determining the measurement classification may further include determining the measurement classification of the one or more images as a debris measurement classification based at least in part on the debris and the risk score.

[0016] In another aspect of the present disclosure, performing the measurement and determining the measurement classification may further comprise performing one or more measurements of the environment of the vehicle using an infrared sensor configured to detect thermal radiation. Performing the measurement and determining the measurement classification may further comprise detecting a fire in the environment of the vehicle based at least in part on the one or more measurements. Performing the measurement and determining the measurement classification may further comprise determining the measurement classification of the one or more measurements as a measurement classification for a fire in response to detecting a fire in the environment of the vehicle.

[0017] In another aspect of the present disclosure, performing the measurement and determining the measurement classification may further comprise capturing one or more images of the surroundings of the vehicle using a camera system. Performing the measurement and determining the measurement classification may further comprise detecting graffiti in the one or more images using a machine learning algorithm. Performing the measurement and determining the measurement classification may further comprise determining the measurement classification of the one or more images as a graffiti measurement classification in response to detecting graffiti in the one or more images.

[0018] In another aspect of the present disclosure, performing the measurement and determining the measurement classification may further comprise capturing one or more images of the surroundings of the vehicle using a camera system. Performing the measurement and determining the measurement classification may further comprise detecting a crime in the one or more images using a machine learning algorithm. Performing the measurement and determining the measurement classification may further comprise determining the measurement classification of the one or more images as a crime measurement classification in response to detecting a crime in the one or more images.

[0019] In another aspect of the present disclosure, performing the measurement and determining the measurement classification may further comprise capturing one or more images of the surroundings of the vehicle using a camera system. Performing the measurement and determining the measurement classification may further comprise detecting water on a roadway in the one or more images using a machine learning algorithm. Performing the measurement and determining the measurement classification may further comprise determining the measurement classification of the one or more images as a flood measurement classification in response to detecting water on the roadway in the one or more images.

[0020] In another aspect of the present disclosure, performing the measurement and determining the measurement classification may further comprise capturing a plurality of images of the surroundings of the vehicle using a camera system. Performing the measurement and determining the measurement classification may further comprise detecting one or more points of interest (POIs) in the plurality of images using a machine learning algorithm. Performing the measurement and determining the measurement classification may further comprise determining the measurement classification of the plurality of images as a tourism measurement classification in response to detecting the one or more POIs in the plurality of images. Performing the measurement and determining the measurement classification may further comprise creating a movie with the plurality of images.Performing the measurement and determining the measurement classification may further include displaying the film to a vehicle occupant.

[0021] According to several aspects, a system for notifying an authority of road conditions for a vehicle is provided. The system may include a camera system. The system may further include a global navigation satellite system (GNSS). The system may further include a vehicle communication system. The system may further include a controller in electrical communication with the camera system, the GNSS, and the vehicle communication system. The controller is programmed to identify a measurement trigger. The controller is further programmed to capture one or more images of an area surrounding the vehicle using the camera system in response to identifying the measurement trigger. The controller is further programmed to determine a location of each of the one or more images using the GNSS.The control unit is further programmed to determine a measurement classification for each of the one or more images. The control unit is further programmed to transmit the one or more images, the measurement classification of each of the one or more images, and the location of each of the one or more images to a remote server system via the vehicle communication system. The remote server system is configured for access by the agency.

[0022] In another aspect of the present disclosure, the measurement trigger includes at least one of a predetermined geographically defined location area, a trigger request initiated by a vehicle occupant, and a trigger request sent by the authority.

[0023] In another aspect of the present disclosure, to determine the measurement classification, the controller is further programmed to identify a road condition in the one or more images. The road condition includes at least one of: a point of interest (POI), a damaged traffic sign, a foreign object on the roadway, an animal on the roadway, a damaged road surface, a fire in the vicinity of the vehicle, graffiti, a crime, and water on the roadway. To determine the measurement classification, the controller is further programmed to determine the measurement classification based at least in part on the road condition in the one or more images.

[0024] Further areas of applicability will become apparent from the present description. It should be understood that the description and specific examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. Fig. 1 is a schematic diagram of a system for notifying an authority of road conditions for a vehicle, according to an exemplary embodiment; Fig. 2 is a flowchart of the method for notifying an authority of road conditions for a vehicle, according to an exemplary embodiment; Fig. 3 is a flowchart of a first exemplary embodiment of a method for identifying a measurement trigger, according to an exemplary embodiment; Fig. 4 is a flowchart of a second exemplary embodiment of a method for identifying a measurement trigger, according to an exemplary embodiment; and Fig. 5 is a flowchart of a third exemplary embodiment of a method for identifying a measurement trigger, according to an exemplary embodiment. DETAILED DESCRIPTION

[0026] The following description is merely exemplary and is not intended to limit the present disclosure, application, or uses.

[0027] Various conditions encountered on or near roadways may require the attention and / or resolution of civil authorities, such as law enforcement, fire departments, highway maintenance agencies, and / or the like. However, civil authorities may be responsible for serving a large and / or complex jurisdiction and may rely on manual surveying and / or reporting methods to identify problems requiring attention. Therefore, the present disclosure provides a new and improved system and method for notifying an agency of road conditions.

[0028] With reference to Fig. 1, a system for notifying an authority of a vehicle's road conditions is illustrated and generally designated by reference numeral 10. The system 10 is shown with an example vehicle 12. Although a passenger car is illustrated, the vehicle 12 may be any type of vehicle without departing from the scope of the present disclosure. The system 10 generally includes a controller 14, a plurality of vehicle sensors 16, and a vehicle communication system 18.

[0029] The controller 14 is used to implement a method 100 for notifying an authority of road conditions for a vehicle, as described below. The controller 14 includes at least one processor 20 and a non-transitory computer-readable storage device or medium 22. The processor 20 may be a custom or off-the-shelf processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among multiple processors connected to the controller 14, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally any device for executing instructions. The computer-readable storage devices or media 22 may include, for example, volatile and non-volatile memory such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM).KAM is persistent or non-volatile memory that can be used to store various operating variables while the processor 20 is powered off. The computer-readable storage device or medium 22 can be implemented using a variety of storage devices such as PROMs (programmable read-only memories), EPROMs (electrical PROMs), EEPROMs (electrically erasable PROMs), flash memory, or other electrical, magnetic, optical, or combination storage devices capable of storing data, some of which represent executable instructions used by the controller 14 to control various systems of the vehicle 12. The controller 14 can also consist of multiple controllers that communicate electrically with each other.The control unit 14 can be connected to other systems and / or control units of the vehicle 12 so that the control unit 14 can access data such as speed, acceleration, braking and steering angle of the vehicle 12.

[0030] The controller 14 is in electrical communication with the plurality of vehicle sensors 16 and the vehicle communication system 18. In an exemplary embodiment, the electrical communication is established, for example, via a CAN network, a FLEXRAY network, a local area network (e.g., Wi-Fi, Ethernet, etc.), a Serial Peripheral Interface (SPI) network, or the like. It should be understood that various additional wired and wireless technologies and communication protocols for communicating with the controller 14 are within the scope of the present disclosure.

[0031] The plurality of vehicle sensors 16 are used to collect information relevant to the vehicle 12. In an exemplary embodiment, the plurality of vehicle sensors 16 includes at least one camera system 24, a global navigation satellite system (GNSS) 26, and an infrared sensor 28.

[0032] In another exemplary embodiment, the plurality of vehicle sensors 16 further includes sensors for determining performance data about the vehicle 12. In one non-limiting example, the plurality of vehicle sensors 16 further includes at least one of an engine speed sensor, an engine torque sensor, a voltage and / or current sensor for the electric drive motor, an accelerator pedal position sensor, a brake position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor.

[0033] In another exemplary embodiment, the plurality of vehicle sensors 16 further includes sensors for determining information about an environment within the vehicle 12. In one non-limiting example, the plurality of vehicle sensors 16 further includes a seat occupancy sensor, a cabin air temperature sensor, a cabin motion detection sensor, a cabin camera, a cabin microphone, and / or the like.

[0034] In another exemplary embodiment, the plurality of vehicle sensors 16 further includes sensors for determining information about an environment of the vehicle 12. In one non-limiting example, the plurality of vehicle sensors 16 further includes at least one of an ambient air temperature sensor and / or a barometric pressure sensor.

[0035] In another exemplary embodiment, at least one of the plurality of vehicle sensors 16 is a perception sensor capable of sensing objects and / or measuring distances in the environment of the vehicle 12. In one non-limiting example, the plurality of vehicle sensors 16 includes a stereo camera with distance measurement capabilities. In one example, at least one of the plurality of vehicle sensors 16 is mounted inside the vehicle 12, e.g., in a headliner of the vehicle 12, viewing through a windshield of the vehicle 12. In another example, at least one of the plurality of vehicle sensors 16 is mounted outside the vehicle 12, e.g., on a roof of the vehicle 12, providing a view of the environment of the vehicle 12. It should be understood that various additional types of perception sensors, such asLiDAR sensors, ultrasonic sensors, radar sensors, and / or time-of-flight sensors fall within the scope of the present disclosure. The plurality of vehicle sensors 16 communicate electrically with the controller 14, as discussed above.

[0036] The camera system 24 is a perception sensor capable of capturing images and / or videos of the surroundings of the vehicle 12. In an exemplary embodiment, the camera system 24 includes a still and / or video camera positioned to capture the surroundings of the vehicle 12. In one non-limiting example, the camera system 24 includes a camera mounted inside the vehicle 12, e.g., on a headliner of the vehicle 12, looking through the windshield. In another non-limiting example, the camera system 24 includes a camera mounted outside the vehicle 12, e.g., on the roof of the vehicle 12, capturing the surroundings in front of the vehicle 12.

[0037] In another exemplary embodiment, the camera system 24 is a surround-view camera system having a plurality of cameras (also called satellite cameras) arranged to provide a view of the nearby surroundings on all sides of the vehicle 12. In one non-limiting example, the camera system 24 includes a forward-facing camera (e.g., mounted in a grille of the vehicle 12), a rear-facing camera (e.g., mounted on a tailgate of the vehicle 12), and two side-facing cameras (e.g., mounted below each of the two side view mirrors of the vehicle 12). In another non-limiting example, the camera system 24 further includes an additional rearview camera mounted near a top-mounted center brake light of the vehicle 12.

[0038] It is understood that camera systems with additional cameras and / or additional mounting locations fall within the scope of the present disclosure. It is further understood that cameras with various sensor types, e.g., charge-coupled device (CCD) sensors, complementary metal oxide semiconductor (CMOS) sensors, and / or high dynamic range (HDR) sensors, fall within the scope of the present disclosure. Furthermore, cameras with various lens types, e.g., wide-angle and / or narrow-angle lenses, also fall within the scope of the present disclosure.

[0039] The GNSS 26 is used to determine the geographic location of the vehicle 12. In one exemplary embodiment, the GNSS 26 is a global positioning system (GPS). In one non-limiting example, the GPS includes a GPS receiving antenna (not shown) and a GPS controller (not shown) that electrically communicates with the GPS receiving antenna. The GPS receiving antenna receives signals from a plurality of satellites, and the GPS controller calculates the geographic position of the vehicle 12 based on the signals received by the GPS receiving antenna. In one exemplary embodiment, the GNSS 26 additionally includes a map. The map contains information about infrastructure such as municipal boundaries, road layouts, railroads, sidewalks, buildings, and the like. Therefore, the geographic location of the vehicle 12 is contextualized using the map information.In one non-limiting example, the map is retrieved from a remote source via a wireless connection. In another non-limiting example, the map is stored in a database of the GNSS 26. It is understood that various additional types of satellite-based radio navigation systems, such as the Global Positioning System (GPS), Galileo, GLONASS, and the BeiDou Navigation Satellite System (BDS), are within the scope of the present disclosure. It is understood that the GNSS 26 may be integrated into the controller 14 (e.g., on the same circuit board as the controller 14 or otherwise as part of the controller 14) without departing from the scope of the present disclosure.

[0040] The infrared sensor 28 is used to detect thermal radiation in the surroundings of the vehicle 12. In one exemplary embodiment, the infrared sensor 28 determines the temperature of objects in the surroundings of the vehicle 12 by measuring the thermal radiation emitted by objects in the surroundings of the vehicle 12. In one non-limiting example, the infrared sensor 28 includes an infrared sensor element and a signal processing unit. In some embodiments, the infrared sensor 28 may further include one or more lenses to focus the infrared radiation onto the infrared sensor element. The infrared sensor element detects infrared radiation and converts it into electrical signals. The electrical signals are then sent to the signal processing unit. The signal processing unit evaluates the data and calculates the temperature of the objects in the surroundings of the vehicle 12 based on the intensity of the detected infrared radiation.It is understood that other devices for non-contact temperature measurement of objects in the environment of the vehicle 12 fall within the scope of the present disclosure.

[0041] The vehicle communication system 18 is used by the controller 14 to communicate with other systems external to the vehicle 12. The vehicle communication system 18 includes, for example, functions for communication with vehicles ("V2V" communication), with the infrastructure ("V2I" communication), with remote systems at a remote call center (e.g., ON-STAR from GENERAL MOTORS), and / or with personal devices. Generally, the term vehicle-to-everything ("V2X" communication) refers to communication between the vehicle 12 and any remote system (e.g., vehicles, infrastructure, and / or remote systems). In certain embodiments, the vehicle communication system 18 is a wireless communication system configured to communicate over a wireless local area network (WLAN) using IEEE 802.11 standards or using cellular data communication (e.g.,using GSMA standards, such as SGP.02, SGP.22, SGP.32, and the like). Accordingly, the vehicle communication system 18 may further include an embedded universal integrated circuit card (eUICC) configured to store at least one cellular connectivity configuration profile, such as an embedded subscriber identity module (eSIM) profile. The vehicle communication system 18 is further configured to communicate via a personal area network (e.g., BLUETOOTH), near field communication (NFC), and / or an additional type of radio frequency communication. However, additional or alternative communication methods, such as Dedicated Short Range Communication (DSRC) and / or mobile telecommunications protocols based on the 3rd Generation Partnership Project (3GPP) standards, are also contemplated within the scope of the present disclosure.DSRC channels refer to short- to medium-range, one-way or two-way wireless communication channels specifically designed for use in vehicles, along with a corresponding set of protocols and standards. 3GPP refers to a partnership between several standards organizations that develop protocols and standards for mobile telecommunications. 3GPP standards are structured as "releases." Therefore, communication methods based on 3GPP releases 14, 15, 16, and / or future 3GPP releases are within the scope of the present disclosure. Accordingly, the vehicle communication system 18 may include one or more antennas and / or communication transceivers for receiving and / or transmitting signals, such as cooperative acquisition messages (CSMs). The vehicle communication system 18 is configured to wirelessly communicate information between the vehicle 12 and another vehicle.Furthermore, the vehicle communication system 18 is configured to wirelessly communicate information between the vehicle 12 and the infrastructure or other vehicles. It is understood that the vehicle communication system 18 may be integrated into the control unit 14 (e.g., on the same circuit board as the control unit 14 or otherwise as part of the control unit 14) without departing from the scope of the present disclosure. The vehicle communication system 18 is in electrical communication with the control unit 14, as discussed above.

[0042] With further reference to Fig. 1, a remote server system is illustrated and generally designated by reference numeral 30. The remote server system 30 includes a server controller 32 in electrical communication with a server database 34 and a server communications system 36. In one non-limiting example, the remote server system 30 is located in a server farm, data center, or the like and is connected to the Internet.

[0043] The server controller 32 includes at least one server processor 38 and a non-transitory computer-readable server storage device or server medium 40. The description of type and configuration provided above for the controller 14 also applies to the server controller 32. In some examples, the server controller 32 may differ from the controller 14 in that the server controller 32 may achieve higher processing speeds and include more memory, more inputs / outputs, and / or the like. In one non-limiting example, the server processor 38 and the server media 40 of the server controller 32 are similar in structure and / or function to the processor 20 and the media 22 of the controller 14, as described above.

[0044] The server database 34 is used to store measurements and measurement classifications, as explained in more detail below. In an exemplary embodiment, the server database 34 includes one or more mass storage devices, such as hard disk drives, magnetic tape drives, magneto-optical disk drives, optical disks, solid-state drives, and / or additional devices capable of storing data in a persistent and machine-readable manner. In some examples, the one or more mass storage devices may be configured to provide redundancy in the event of hardware failure and / or data corruption, e.g., by using a redundant array of independent disks (RAID). In one non-limiting example, the server controller 32 may implement software, such asrun a database management system (DBMS) that enables the organization of and access to the data stored on the one or more mass storage devices.

[0045] The server communication system 36 is used to communicate with external systems, such as the control unit 14, via the vehicle communication system 18. In one non-limiting example, the server communication system 36 is similar in structure and / or function to the vehicle communication system 18, as described above. In some examples, the server communication system 36 may differ from the vehicle communication system 18 in that the server communication system 36 is capable of higher-power signal transmission, more sensitive signal reception, higher-bandwidth transmission, additional transmission / reception protocols, and / or the like.

[0046] With reference to Fig. 2 shows a flowchart of the method 100 for notifying an authority of road conditions for a vehicle. The method 100 begins at block 102 and proceeds to block 104. At block 104, the controller 14 identifies a measurement trigger. Within the scope of the present disclosure, the measurement trigger is an event that causes the controller 14 to perform a measurement using the plurality of vehicle sensors 16, as explained in more detail below. The identification of the measurement trigger is discussed in more detail below. After block 104, the method 100 proceeds to block 106.

[0047] With reference to Fig. 3 illustrates a first exemplary embodiment 104a of block 104. The first exemplary embodiment 104a begins at block 302. In block 302, the controller 14 determines a location of the vehicle 12 using the GNSS 26. After block 302, the first exemplary embodiment 104a proceeds to block 304. In block 304, the location of the vehicle 12 determined in block 302 is compared to a predetermined geographically defined location area. Within the scope of the present disclosure, the predetermined geographically defined location area is a geographical area that has been predetermined as an area of ​​interest for conducting measurements. In one non-limiting example, the predetermined geographically defined location area includes one or more street segments, a neighborhood, an urban block, and / or the like.In an exemplary embodiment, the predetermined geographically defined location may be updated by the remote server system 30 and received by the controller 14 using the vehicle communication system 18, for example, using a learn and update (SLU) data collection system. If the location of the vehicle 12 determined in block 302 is not within the predetermined geographically defined location area, the first exemplary embodiment 104a proceeds to block 306. If the location of the vehicle 12 determined in block 302 is within the predetermined geographically defined location area, the first exemplary embodiment 104a proceeds to block 308.

[0048] In block 306, the measurement trigger is not identified. After block 306, the first exemplary embodiment 104a is completed, and the method 100 continues as described below. In block 308, the measurement trigger is identified. After block 308, the first exemplary embodiment 104a is completed, and the method 100 continues as described below.

[0049] With reference to Fig. 4, a second exemplary embodiment 104b of block 104 is illustrated. The second exemplary embodiment 104b begins in block 402. In block 402, the controller 14 receives a trigger request. Within the scope of the present disclosure, the trigger request is a request for the system 10 to perform measurements in response to one or more conditions being met. The trigger request includes the one or more conditions. In an exemplary embodiment, the trigger request is received from an occupant of the vehicle 12. In one non-limiting example, the occupant of the vehicle 12 uses a human-machine interface (HMI) to input the trigger request. The trigger request may include, for example, a user request to capture images of a particular point of interest (POI).Within the scope of the present disclosure, the POI includes, for example, a store, a restaurant, a vehicle service center, a hospital, a police station, a gas station, a vehicle charging station, a recreation area, a nature reserve, a museum, a theater, a tourist attraction, a statue, a monument, a memorial, and / or the like. In another exemplary embodiment, the trigger request is received from an authority. Within the scope of the present disclosure, the authority includes organizations and / or individuals responsible for maintaining public road infrastructure, e.g., a law enforcement agency, a fire department, a highway maintenance department, and / or the like. In one non-limiting example, the authority provides a trigger request to the remote server system 30. After block 402, the second exemplary embodiment 104b proceeds to block 404.

[0050] In block 404, the controller 14 uses the plurality of vehicle sensors 16 to perform a measurement of the surroundings of the vehicle 12. In an exemplary embodiment, the controller 14 uses the camera system 24 to capture an image of the surroundings of the vehicle 12. After block 404, the second exemplary embodiment 104b proceeds to block 406.

[0051] In block 406, the measurement taken in block 404 is compared to the trigger request received in block 402. For example, a law enforcement agency may provide a trigger request including a license plate number of a remote vehicle, a model of a remote vehicle, and a color of a remote vehicle. If the measurement taken in block 404 (i.e., the image captured in block 404) includes the license plate number of the remote vehicle, the model of the remote vehicle, and / or the color of the remote vehicle, it is determined that the measurement meets the trigger request. As a non-limiting example, the determination of the license plate number of the remote vehicle is performed according to the methods discussed in "Automatic Number Plate Recognition: A Detailed Survey of Relevant Algorithms" by Lubna et al. (Sensors, 21, 3028, Apr. 2021), the entire contents of which are hereby incorporated by reference.

[0052] In another example, a road maintenance agency may provide a trigger request that includes all types of traffic signs. Thus, if the measurement taken in block 404 (i.e., the image captured in block 404) includes a traffic sign, the measurement is determined to match the trigger request. If it is determined that the measurement does not match the trigger request, the second exemplary embodiment 104b returns to block 404 to perform a new measurement. If it is determined that the measurement matches the trigger request, the second exemplary embodiment 104b proceeds to block 408.

[0053] In block 408, the measurement trigger is identified. After block 408, the second exemplary embodiment 104b is completed, and the method 100 continues as described below.

[0054] With reference to Fig. 5 illustrates a third exemplary embodiment 104c of block 104. The third exemplary embodiment 104c begins in block 502. In block 502, the controller 14 uses the plurality of vehicle sensors 16 to perform a trigger measurement of the surroundings of the vehicle 12. Within the scope of the present disclosure, a trigger measurement is a measurement used to identify the measurement trigger. In an exemplary embodiment, the controller 14 uses the camera system 24 to capture an image of the surroundings of the vehicle 12. After block 502, the third exemplary embodiment 104c proceeds to block 504.

[0055] In block 504, the controller 14 evaluates the trigger measurement using a machine learning algorithm. In an exemplary embodiment, the machine learning algorithm is configured to identify a road condition in the trigger measurement. Within the scope of the present disclosure, the road condition includes a normal road condition and an abnormal road condition. Within the scope of the present disclosure, the abnormal road condition includes, for example, a foreign object on the roadway, an animal on the roadway, water on the roadway, a damaged road surface, a damaged traffic sign, a fire, a crime, and / or graffiti.

[0056] As a non-limiting example, the machine learning algorithm comprises multiple layers, including an input layer and an output layer, as well as one or more hidden layers. The input layer receives the trigger measurement as input. The input is then passed to the hidden layers. Each hidden layer applies a transformation (e.g., a nonlinear transformation) to the data and passes the result to the next hidden layer, up to the last hidden layer. The output layer provides the road condition.

[0057] To train the machine learning algorithm, a dataset containing inputs and their corresponding road conditions is used. The algorithm is trained by adjusting the internal weights between nodes in each hidden layer to minimize the prediction error. During training, an optimization technique (e.g., gradient descent) is used to adjust the internal weights to reduce the prediction error. The training process is repeated with the entire dataset until the prediction error is minimized, and the trained model is then used to classify new input data.

[0058] After sufficient training of the machine learning algorithm, it is able to accurately and precisely determine the road condition based on the trigger measurement. By adjusting the weights between the nodes in each hidden layer during training, the algorithm "learns" to recognize patterns in the data that indicate the road condition. After block 504, the third exemplary embodiment 104c proceeds to block 506.

[0059] In block 506, the third exemplary embodiment 104c returns to block 502 to perform another measurement if the abnormal road condition is not identified in block 504. If the abnormal road condition is identified in block 504, the third exemplary embodiment 104c proceeds to block 508. In block 508, the measurement trigger is identified. After block 508, the third exemplary embodiment 104c is complete, and the method 100 continues as described below.

[0060] With further reference to Fig. 2, in block 106, if the measurement trigger is not identified in block 104, the method 100 transitions to a standby state in block 108. If the measurement trigger is identified in block 104, the method 100 continues to block 110.

[0061] In block 110, the controller 14 uses the plurality of vehicle sensors 16 to perform a measurement of the surroundings of the vehicle 12. In one exemplary embodiment, performing the measurement comprises capturing one or more images of the surroundings of the vehicle 12 using the camera system 24. In one non-limiting example, the one or more images are stored in the media 22 of the controller 14 for later retrieval. In another exemplary embodiment, performing the measurement further comprises determining a location of the vehicle 12 corresponding to each of the one or more images using the GNSS 26. The location of the vehicle 12 is stored along with the one or more images in the media 22 of the controller 14. In another exemplary embodiment, performing the measurement further comprises performing one or more measurements with the infrared sensor 28.After block 110, the method 100 proceeds to block 112.

[0062] In block 112, the controller 14 extracts important information from the one or more images captured in block 110. Within the scope of the present disclosure, the important information includes poor road conditions (i.e., a foreign object on the roadway, an animal on the roadway, water on the roadway, and / or a damaged road surface), a damaged traffic sign, a fire, a crime, graffiti, and / or a point of interest (POI). In an exemplary embodiment, the controller 14 uses a computer vision algorithm to extract the important information from the one or more images captured in block 110. In one non-limiting example, the computer vision algorithm uses machine learning techniques to analyze pixel-level information of an input image to detect and classify objects or patterns of interest.As a non-limiting example, the computer vision algorithm begins by preprocessing the input image using techniques such as image resizing, normalization, and / or filtering for noise reduction. The computer vision algorithm then extracts relevant features from the input image using methods such as edge detection, corner detection, texture analysis, and / or the like. The computer vision algorithm may then use a machine learning model, such as a convolutional neural network (CNN), to classify and label relevant features of the input image based on learned patterns and associations. After block 112, the method 100 proceeds to block 114.

[0063] In block 114, the controller 14 determines a measurement classification of the environmental measurement performed in block 110. In an exemplary embodiment, the measurement classification is determined at least in part based on the trigger request. For example, if the environmental measurement performed in block 110 corresponds to a trigger request received from a law enforcement agency, as described above with reference to Fig. 4, the measurement classification is determined as a law enforcement measurement classification. In another example, if the environmental measurement taken in block 110 corresponds to a trigger request received from the inmate, as discussed above with reference to Fig. 4, the measurement classification is determined as an occupant-requested classification. In another example, if the environmental measurement performed in block 110 corresponds to a trigger request received from a highway maintenance agency, as discussed above with reference to Fig. 4, the measurement classification is determined as the measurement classification for road maintenance.

[0064] In another exemplary embodiment, the measurement classification is determined based at least in part on the content of the environment measurement taken in block 110. In one non-limiting example, the environment measurement taken in block 110 is analyzed with a computer vision algorithm to determine whether the environment measurement taken in block 110 includes a traffic sign. If the environment measurement taken in block 110 includes a traffic sign, the controller 14 then calculates a correlation value between the environment measurement taken in block 110 (i.e., one or more images captured by the camera system 24) and a reference image of an intact traffic sign using a computer vision algorithm.Within the scope of the present disclosure, the correlation value quantifies the degree of similarity between the environment measurement performed in block 110 and the reference image. Controller 14 then compares the correlation value to a predefined correlation threshold (e.g., 60%). If the correlation value is less than the predefined correlation threshold, the traffic sign in the environment measurement performed in block 110 is determined to be a damaged traffic sign. Therefore, the measurement classification is determined to be a damaged traffic sign measurement classification.

[0065] In another non-limiting example, broken traffic signs are detected according to methods discussed in "Broken Detection of the Traffic Sign by using the Location Histogram Matching" by Yang et al. (Journal of Korea Multimedia Society, Vol. 15, No. 3 (pp. 312-322), March 2012), the entire contents of which are hereby incorporated by reference.

[0066] In another non-limiting example, the environmental measurement taken in block 110 is analyzed using a machine learning algorithm to determine whether the environmental measurement taken in block 110 includes a poor road condition. Within the scope of the present disclosure, the poor road condition includes at least one of: a foreign object on a roadway, an animal on the roadway, water on the roadway, and / or a damaged road surface. If the environmental measurement taken in block 110 includes a poor road condition, the controller 14 then determines a risk score for the poor road condition. Within the scope of the present disclosure, the risk score quantifies a level of risk that the poor road condition poses to the vehicles on the roadway.

[0067] In an exemplary embodiment, controller 14 uses a machine learning algorithm configured to determine risk values ​​based on adverse road conditions to determine the risk value. In an exemplary embodiment, the risk value is determined based on the type and severity of the adverse road conditions. As a non-limiting example, the risk value is relatively higher when the adverse road conditions involve complete obstruction of the roadway than when the adverse road conditions involve minimal obstruction of the roadway.

[0068] The control unit 14 then compares the risk value with a predefined risk value threshold. If the risk value exceeds the predefined risk value threshold, the measurement classification is determined as a poor road condition measurement classification.

[0069] It is understood that the measurement classification may be further specified depending on the type of poor road condition. For example, if the environmental measurement taken in block 110 includes debris, such as garbage or litter, the measurement classification may be determined as a debris measurement classification. As a non-limiting example, debris detection follows the procedures discussed in the "FRAMEWORK for Roadside Litter Identification and Face Recognition using Convolutional Neural Networks" by Abiga Sansuri et al. (Proceedings of ACM / CSI / IEEECS Research & Industry Symposium on IoT Cloud For Societal Applications (IoTCloud'21)), the entire contents of which are hereby incorporated by reference.

[0070] If the environmental measurement taken in block 110 includes an animal in the roadway, the measurement classification may be determined as a roadkill measurement classification. If the environmental measurement taken in block 110 includes water in the roadway, the measurement classification may be determined as a flood measurement classification. As a non-limiting example, water in the roadway is detected according to the methods discussed in "Water Detection with Segmentation Guided Dynamic Texture Recognition" by Santana et al. (2012 IEEE International Conference on Robotics and Biomimetics, ROBIO 2012 - Conference Digest, Dec. 2012), the entire contents of which are hereby incorporated by reference.

[0071] If the environmental measurement performed in block 110 includes a damaged road surface (e.g., a pothole), the measurement classification may be determined as a damaged road surface measurement classification. It should be understood that the debris measurement classification, the road animal measurement classification, the flood measurement classification, and the damaged road surface measurement classification are determined in a process analogous to the poor road condition measurement classification discussed above.

[0072] In another non-limiting example, the ambient measurement taken in block 110 is analyzed using a machine learning algorithm to determine whether the ambient measurement taken in block 110 is indicative of a fire in the environment of the vehicle 12. In a non-limiting example, if the ambient measurement taken in block 110 includes thermal radiation measurements taken using the infrared sensor 28, the thermal radiation measurements are compared to a predetermined thermal radiation threshold. If one or more of the thermal radiation measurements are greater than or equal to the thermal radiation threshold, the measurement classification is determined to be a fire measurement classification.In another non-limiting example, if the environmental measurement performed in block 110 includes one or more images captured by camera system 24, a machine learning algorithm is used to detect a fire in the one or more images. In another non-limiting example, a fire is detected in the environmental measurement performed in block 110 using the methods discussed in "A Review on Early Forest Fire Detection Systems Using Optical Remote Sensing" by Barmpoutis et al. (Sensors, 20, 6442, Nov. 2020), the entire contents of which are hereby incorporated by reference.

[0073] In another non-limiting example, the environmental measurement taken in block 110 is analyzed using a machine learning algorithm to determine whether the environmental measurement taken in block 110 includes graffiti. In another non-limiting example, if the environmental measurement taken in block 110 includes one or more images captured by camera system 24, a machine learning algorithm is used to detect graffiti in the one or more images. In another non-limiting example, graffiti is detected in the environmental measurement taken in block 110 using the methods discussed in "Deep Learning-Based Graffiti Detection: A Study Using Images from the Streets of Lisbon" by Fogaça et al. (Applied Sciences, 13, 2249, Feb. 2023), the entire contents of which are hereby incorporated by reference.If the measurement of the environment performed in block 110 determines that it contains graffiti, the measurement classification is determined as a graffiti measurement classification.

[0074] In another non-limiting example, the environmental measurement taken in block 110 is analyzed with a machine learning algorithm to determine whether the environmental measurement taken in block 110 contains a crime (e.g., theft, vandalism, violence, and / or the like). In a non-limiting example, if the environmental measurement taken in block 110 includes one or more images captured by camera system 24, a machine learning algorithm is used to detect a crime in the one or more images. If the environmental measurement taken in block 110 is determined to contain a crime, the measurement classification is determined to be a crime measurement classification.

[0075] In another exemplary embodiment, the measurement classification is determined at least in part based on the location of the environmental measurement performed in block 110, which was determined using the GNSS 26. In one non-limiting example, the measurement classification is determined as a tourist measurement classification if the location of the environmental measurement performed in block 110 is determined to be within a predetermined distance threshold (e.g., ten meters) from a POI (e.g., a museum, statue, monument, memorial, nature reserve, tourist attraction, and / or the like). In another non-limiting example, the measurement classification is determined as the tourist measurement classification if the location of the environmental measurement performed in block 110 is within a predetermined distance from one of a predetermined plurality of POI locations (e.g.,stored in the media 22 of the controller 14). In one non-limiting example, the measurement of the environment performed in block 110 (e.g., one or more images captured by the camera system 24) is analyzed using a machine learning algorithm to detect one or more points of interest (POIs) in the measurement of the environment performed in block 110. If one or more POIs are detected in the measurement of the environment performed in block 110, the measurement classification is determined to be the tourist measurement classification.

[0076] It is understood that additional measurement classifications and subclassifications useful for providing information to authorities and / or increasing occupant comfort and convenience are within the scope of the present disclosure. After block 114, the method 100 proceeds to block 116.

[0077] In block 116, the controller 14 uses the vehicle communication system 18 to transmit the environmental measurement taken in block 110 (e.g., the one or more images captured by the camera system 24), the location of the environmental measurement taken in block 110, and the measurement classification determined in block 114 to the remote server system 30. In an exemplary embodiment, the server controller 32 of the remote server system 30 uses the server communication system 36 to receive the environmental measurement taken in block 110, the location of the environmental measurement taken in block 110, and the measurement classification determined in block 114. The server controller 32 then stores the environmental measurement taken in block 110, the location of the environmental measurement taken in block 110, and the measurement classification determined in block 114 in the server database 34 for later retrieval.

[0078] In one exemplary embodiment, the environmental measurement taken in block 110, the location of the environmental measurement taken in block 110, and the measurement classification determined in block 114 are transmitted over a cellular data connection. In another exemplary embodiment, the environmental measurement taken in block 110, the location of the environmental measurement taken in block 110, and the measurement classification determined in block 114 are transmitted using a Wi-Fi connection.

[0079] In another exemplary embodiment, the controller 14 compiles all measurements (e.g., images and / or videos) with the tourist measurement classification into a multimedia presentation (e.g., photo collage, movie, slideshow, and / or the like) and displays the multimedia presentation to the occupant, e.g., via an HMI of the vehicle 12. After block 116, the method 100 proceeds to block 118.

[0080] In block 118, the server controller 32 provides the environmental measurement taken in block 110, the location of the environmental measurement taken in block 110, and the measurement classification determined in block 114 and stored in the server database 34 to the authorities in block 116. In one exemplary embodiment, the authorities can connect to the server controller 32 using the server communication system 36 and request measurements with specific measurement classifications and / or specific locations. In another exemplary embodiment, the server controller 32 uses the server communication system 36 to automatically transmit measurements with specific measurement classifications to specific authorities.As a non-limiting example, measurements are submitted to a law enforcement agency using the Law Enforcement Measurement Classification, the Graffiti Measurement Classification, and / or the Crime Measurement Classification.

[0081] In another non-limiting example, measurements with the road maintenance measurement classification, the damaged traffic sign measurement classification, the poor road condition measurement classification, the debris measurement classification, the roadkill measurement classification, the flood measurement classification, and / or the damaged road surface measurement classification are transmitted to a road maintenance department. In another non-limiting example, measurements with the fire measurement classification are transmitted to the fire department. After block 118, the method 100 transitions to the standby state in block 108.

[0082] In another non-limiting example, the server controller 32 uses the server communication system 36 to provide an application programming interface (API) that enables agencies to send trigger requests and receive information stored in the server database 34.

[0083] In an exemplary embodiment, the controller 14 repeatedly exits the standby state 108 and restarts the method 100 in block 102. In a non-limiting example, the controller 14 exits the standby state 108 and restarts the method 100 after a timer, e.g., every three hundred milliseconds.

[0084] The system 10 and method 100 of the present disclosure offer several advantages. By collecting and aggregating road condition data, authorities can be promptly notified of problems requiring resolution. Using methods 100, the system 10 can utilize additional computing capacity to provide information to authorities. By recording images and / or videos of POIs, the system 10 and methods 100 can also be used to enhance occupant comfort and convenience.

[0085] The description of the present disclosure is merely exemplary, and variations that do not depart from the gist of the present disclosure are intended to be included within the scope of the present disclosure. Such variations should not be considered a departure from the spirit and scope of the present disclosure. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Automatic Number Plate Recognition: A Detailed Survey of Relevant Algorithms" by Lubna et al. (Sensors, 21, 3028, Apr. 2021

[0051] Broken Detection of the Traffic Sign by using the Location Histogram Matching" by Yang et al. (Journal of Korea Multimedia Society, Vol. 15, No. 3 (pp. 312-322), March 2012

[0065] FRAMEWORK for Roadside Litter Identification and Face Recognition using Convolutional Neural Networks" by Abiga Sansuri et al. (Proceedings of ACM / CSI / IEEECS Research & Industry Symposium on IoT Cloud For Societal Applications (IoTCloud'21

[0069] Water Detection with Segmentation Guided Dynamic Texture Recognition“ von Santana et al. (2012 IEEE International Conference on Robotics and Biomimetics, ROBIO 2012 - Conference Digest, Dez. 2012

[0070] A Review on Early Forest Fire Detection Systems Using Optical Remote Sensing“ von Barmpoutis et al. (Sensors, 20, 6442, Nov. 2020

[0072] Deep Learning-Based Graffiti Detection: A Study Using Images from the Streets of Lisbon“ von Fogaça et al. (Applied Sciences, 13, 2249, Feb. 2023

[0073]

Claims

[1] A system for notifying an authority of road conditions for a vehicle, the system comprising: a variety of vehicle sensors; a vehicle communication system; and a control unit in electrical communication with the plurality of vehicle sensors and the vehicle communication system, the control unit being programmed to: Identifying a measurement trigger; performing a measurement of an environment of the vehicle using the plurality of vehicle sensors in response to identifying the measurement trigger; Determining a measurement classification based at least in part on the measurement; and Transmitting the measurement and measurement classification to a remote server system via the vehicle communication system. [2] The system of claim 1, wherein the plurality of vehicle sensors includes at least one global navigation satellite system (GNSS), and wherein to identify the measurement trigger, the controller is further programmed to: Determining a location of the vehicle using GNSS; Comparing the location of the vehicle with a predefined geographically defined location area; and Identifying the measurement trigger in response to determining that the vehicle's location is within the specified geographically defined location area. [3] The system of claim 1, wherein to identify the measurement trigger, the controller is further programmed to: Receiving a trigger request from a vehicle occupant and / or an authority; and Identifying the measurement trigger based at least in part on the trigger requirement. [4] The system of claim 3, wherein the plurality of vehicle sensors comprise at least one camera system, and wherein, to perform the measurement of the surroundings of the vehicle, the controller is further programmed to: Capturing one or more images of a remote vehicle using the camera system based at least in part on the trigger request, wherein the trigger request includes a license plate number of a remote vehicle, a model of a remote vehicle, and / or a color of a remote vehicle. [5] The system of claim 4, wherein to determine the measurement classification, the controller is further programmed to: Determining the measurement classification of the one or more images as a law enforcement measurement classification in response to determining that the remote vehicle in the one or more images includes the license plate number of the remote vehicle, the model of the remote vehicle, and / or the color of the remote vehicle. [6] The system of claim 1, wherein to identify the measurement trigger, the controller is further programmed to: Performing a trigger measurement using the plurality of vehicle sensors; Identifying a road condition in the trigger measurement using a machine learning algorithm, wherein the road condition includes one of the following: a normal road condition and an abnormal road condition; and Identifying the measurement trigger in response to determining that the road condition is the abnormal road condition. [7] The system of claim 1, wherein the plurality of vehicle sensors comprise at least one camera system, and wherein, to perform the measurement of the environment of the vehicle, the controller is further programmed to: Taking one or more images with the camera system; and Storing the one or more images in a non-transitory memory of the control unit. [8] The system of claim 7, wherein the plurality of vehicle sensors comprises at least one global navigation satellite system (GNSS), and wherein the control device for determining the measurement classification is further programmed to: Determining a location of the vehicle using GNSS; Comparing the location of the vehicle with a predetermined plurality of point-of-interest (POI) locations; and Determining the measurement classification of the one or more images as a tourism measurement classification in response to determining that the location of the vehicle is within a predetermined distance of at least one of the plurality of POI locations. [9] The system of claim 7, wherein to determine the measurement classification, the controller is further programmed to: identifying a traffic sign in the one or more images using a computer vision algorithm; Calculating a correlation value between the one or more images and a reference image using the computer vision algorithm; and Determining the measurement classification of the one or more images as a measurement classification for a damaged traffic sign in response to identifying the traffic sign in the one or more images and in response to determining that the correlation value is less than a predetermined correlation threshold. [10] The system of claim 7, wherein to determine the measurement classification, the controller is further programmed to: Detecting a poor road condition in the one or more images using a machine learning algorithm, wherein the poor road condition includes a foreign object on a roadway, an animal on the roadway, and / or a damaged road surface; Determining a risk value for the poor road condition; and Determining the measurement classification of the one or more images as a poor road condition measurement classification based at least in part on the poor road condition and the risk score.

Images