Road defect detection

Abstract

A system for identifying a road defect within a road ahead of a moving vehicle includes at least one camera in communication with a system controller and adapted to capture a plurality of time series images of the road ahead of the moving vehicle, and at least one motion sensor in communication with the system controller and adapted to detect when the vehicle encounters a road defect, the system controller is adapted to associate a road defect with at least one image of a plurality of time-series images of the road, analyze the at least one image of the plurality of time-series images using a computer vision algorithm, identify a road defect within the at least one image of the plurality of time-series images by the computer vision algorithm, and identify a road defect within the at least one image of the plurality of time-series images by the computer vision algorithm. And marking the identified road defect within at least one image of the plurality of time series images.

Description

Technical Field

[0001] The present disclosure relates to a system and method for detecting defects in a roadway prior to impacting the defects. Background Art

[0002] Vehicles are equipped with many sensors that monitor the environment around the vehicle and detect vehicle motion when the vehicle hits imperfections in the road. Motion sensors are able to measure the acceleration and inertia changes of the vehicle and vehicle suspension components to detect whether the vehicle is driving over a rough road surface (hitting many small imperfections) or hitting a pothole (hitting one large imperfection). However, such systems rely on the sensors' "feeling" of the imperfections. It would be best if they could be detected before the vehicle hits these imperfections. Vision systems using cameras can "see" road imperfections before the vehicle hits them, but such systems require manual marking of road imperfections, and many road imperfections are too subtle to be visually identified by a person using a camera image. In addition, such systems rely on global positioning to determine the location of road imperfections for future reference, and cannot provide precise information about the location of such imperfections and the specific lateral location of such road imperfections within the road.

[0003] Therefore, while current systems and methods achieve their intended purposes, there is a need for a new and improved system and method for detecting defects within a road by correlating motion sensor data collected when a vehicle strikes a road defect with images of the road defect captured and creating a machine learning model to predict the road defect before the vehicle strikes the road defect. Summary of the invention

[0004] According to several aspects of the present disclosure, a method for identifying road defects in a road ahead of a moving vehicle includes: capturing multiple time-series images of the road ahead of the moving vehicle through at least one camera that communicates with a system controller; detecting when the vehicle encounters a road defect through at least one motion sensor located in the vehicle and communicating with the system controller; associating, through the system controller, the road defect detected by the at least one motion sensor with at least one image of the multiple time-series images of the road collected by the at least one camera; analyzing, through the system controller, at least one image of the multiple time-series images related to the road defect detected by the at least one motion sensor using a computer vision algorithm; identifying, through the system controller and the computer vision algorithm, a road defect within at least one image of the multiple time-series images related to the road defect detected by the at least one motion sensor; and marking, through the system controller, the road defect identified in at least one image of the multiple time-series images related to the road defect detected by the at least one motion sensor.

[0005] According to another aspect, identifying a road defect within at least one of a plurality of time-series images associated with a road defect detected by at least one motion sensor via a system controller and a computer vision algorithm further includes: receiving, via the system controller, data of an image previously captured by the vehicle from a local database in communication with the system controller, wherein the road defect has been marked by the system controller; and predicting, via the system controller, a location of the road defect within at least one of a plurality of time-series images associated with a road defect detected by at least one motion sensor using a machine learning model in communication with the system controller and the local database.

[0006] According to another aspect, the method further includes updating the machine learning model using data of a road defect detected by at least one motion sensor and at least one image from a plurality of time series images associated with the road defect detected by the at least one motion sensor.

[0007] According to another aspect, identifying a road defect within at least one of a plurality of time-series images associated with a road defect detected by at least one motion sensor via a system controller and a computer vision algorithm further includes: receiving, via the system controller, data of images previously captured by other vehicles from a remote database in communication with the system controller, in which the road defect has been identified and marked; and predicting, via the system controller, a location of the road defect within at least one of a plurality of time-series images associated with a road defect detected by at least one motion sensor using a global machine learning model that communicates with the system controller and the remote database and is trained using data from other vehicles stored in the remote database.

[0008] According to another aspect, the method further includes updating the global machine learning model using data of a road defect detected by at least one motion sensor and at least one image from a plurality of time series images associated with the road defect detected by the at least one motion sensor.

[0009] According to another aspect, identifying a road defect within at least one of a plurality of time-series images associated with a road defect detected by at least one motion sensor via a system controller and a computer vision algorithm further includes: receiving, via the system controller, data of images previously captured by other vehicles having contextual features matching the vehicle from a remote database in communication with the system controller, wherein the road defect has been identified and marked; and predicting, via the system controller, a location of the road defect within at least one of a plurality of time-series images associated with a road defect detected by at least one motion sensor using a contextual global machine learning model that communicates with the system controller and the remote database and is trained using data from other vehicles stored in the remote database.

[0010] According to another aspect, associating, by the system controller, a road defect detected by the at least one motion sensor with at least one image of a plurality of time-series images of the road collected by the at least one camera further comprises: at time t n At a location, detecting a road defect by at least one motion sensor; and calculating by a position algorithm communicating with a system controller: at least one camera at time t n The posture at time t n relative to at least one motion sensor and at least one camera; at time t n-x at time t, a pose of at least one camera for at least one image in a plurality of time-series images associated with a road defect detected by at least one motion sensor; and at time t n-x At the location, a relative position of a road defect within at least one image of a plurality of time-series images associated with the road defect detected by at least one motion sensor.

[0011] According to another aspect, the method further includes determining, by the system controller, a lateral location of the road defect within the road.

[0012] According to another aspect, determining, by the system controller, the lateral position of a road defect within the road further includes: receiving, by the system controller, data related to the relative severity of the road defect at different lateral positions across the road from at least one motion sensor, wherein the at least one motion sensor includes a plurality of motion sensors located at different lateral positions within the vehicle; and calculating, by the system controller, a lateral severity distribution of the road defect.

[0013] According to another aspect, determining, by a system controller, a lateral position of a road defect within a road further comprises: receiving, by the system controller, data related to relative severity of road defects at different lateral positions across the road previously collected by other vehicles from a remote database in communication with the system controller; predicting, by the system controller, a lateral severity distribution of road defects using a global machine learning model that communicates with the system controller and the remote database and is trained using data stored in the remote database and related to relative severity of road defects at different lateral positions across the road previously collected by other vehicles; and updating the global machine learning model using data received from a plurality of motion sensors located at different lateral positions within the vehicle.

[0014] According to several aspects of the present disclosure, a system for identifying road defects in a road ahead of a moving vehicle includes at least one camera that communicates with a system controller and is suitable for capturing multiple time-series images of the road ahead of the moving vehicle, and at least one motion sensor that communicates with the system controller and is suitable for detecting when the vehicle encounters a road defect, wherein the system controller is suitable for: associating a road defect detected by the at least one motion sensor with at least one image of a plurality of time-series images of the road collected by the at least one camera, analyzing at least one image of the plurality of time-series images related to the road defect detected by the at least one motion sensor using a computer vision algorithm, identifying a road defect within at least one image of the plurality of time-series images related to the road defect detected by the at least one motion sensor using a computer vision algorithm, and marking a road defect identified within at least one image of the plurality of time-series images related to the road defect detected by the at least one motion sensor.

[0015] According to another aspect, when identifying a road defect within at least one of a plurality of time-series images associated with a road defect detected by at least one motion sensor, the system controller is further adapted to: receive data of an image previously captured by the vehicle from a local database in communication with the system controller, wherein the road defect has been marked by the system controller; predict a location of the road defect within at least one of a plurality of time-series images associated with the road defect detected by at least one motion sensor using a machine learning model in communication with the system controller and the local database; and update the machine learning model using the data of the road defect detected by the at least one motion sensor and at least one of the plurality of time-series images associated with the road defect detected by the at least one motion sensor.

[0016] According to another aspect, when a road defect is identified within at least one of a plurality of time-series images associated with a road defect detected by at least one motion sensor, the system controller is further adapted to: receive data of images previously captured by other vehicles from a remote database in communication with the system controller, wherein the road defect has been identified and marked; predict the location of the road defect within at least one of a plurality of time-series images associated with the road defect detected by at least one motion sensor using a global machine learning model that communicates with the system controller and the remote database and is trained using data of other vehicles stored in the remote database; and update the global machine learning model using the data of the road defect detected by the at least one motion sensor and at least one of the plurality of time-series images associated with the road defect detected by the at least one motion sensor.

[0017] According to another aspect, when identifying a road defect within at least one of a plurality of time-series images associated with a road defect detected by at least one motion sensor, the system controller is further adapted to: receive data of images previously captured by other vehicles having contextual features matching the vehicle from a remote database in communication with the system controller, wherein the road defect has been identified and marked; predict the location of the road defect within at least one of a plurality of time-series images associated with the road defect detected by at least one motion sensor using a contextual global machine learning model that communicates with the system controller and the remote database and is trained using data of other vehicles stored in the remote database; and update the contextual global machine learning model using the data of the road defect detected by the at least one motion sensor and at least one of the plurality of time-series images associated with the road defect detected by the at least one motion sensor.

[0018] According to another aspect, when the road defect detected by the at least one motion sensor is associated with at least one image of the plurality of time-series images of the road collected by the at least one camera, the system controller is further adapted to: at time t n At a location, detecting a road defect by at least one motion sensor; and calculating by a position algorithm communicating with a system controller: at least one camera at time t n The posture at time t n relative to at least one motion sensor and at least one camera; at time t n-x at time t, a pose of at least one camera for at least one image in a plurality of time-series images associated with a road defect detected by at least one motion sensor; and at time t n-x At the location, a relative position of a road defect within at least one image of a plurality of time-series images associated with the road defect detected by at least one motion sensor.

[0019] According to another aspect, the system controller is further adapted to determine a lateral location of the road defect within the road.

[0020] According to another aspect, when determining the lateral position of a road defect within the road, the system controller is further adapted to: receive data related to the relative severity of the road defect at different lateral positions across the road from at least one motion sensor, wherein the at least one motion sensor includes a plurality of motion sensors located at different lateral positions within the vehicle; and calculate a lateral severity distribution of the road defect.

[0021] According to another aspect, when determining the lateral position of a road defect within the road, the system controller is further adapted to: receive data related to the relative severity of the road defect at different lateral positions across the road previously collected by other vehicles from a remote database in communication with the system controller; predict the lateral severity distribution of the road defect using a global machine learning model that is in communication with the system controller and the remote database and trained using data related to the relative severity of the road defect at different lateral positions across the road previously collected by other vehicles stored in the remote database; and update the global machine learning model by data received from multiple motion sensors located at different lateral positions within the vehicle.

[0022] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0024] Figure 1 is a schematic diagram of a vehicle having a system for identifying a road defect within a road according to an exemplary embodiment;

[0025] Figure 2 is a schematic diagram of a system according to an exemplary embodiment;

[0026] Figure 3 is a schematic diagram of a vehicle approaching a road defect on a road;

[0027] Figure 4 yes Figure 3 A schematic diagram of a road defect on a road struck by a vehicle shown;

[0028] Figure 5A is a schematic diagram showing a timeline of a vehicle approaching a road defect within a road;

[0029] Figure 5B is at least one camera at time t n-1 A schematic diagram of the captured image;

[0030] Figure 5C is at least one camera at time t n-2 A schematic diagram of the captured image;

[0031] Figure 6 is a schematic diagram showing a coordinate system for illustrating the severity of a road defect that crosses the road laterally;

[0032] Figure 7 yes Figure 6A schematic diagram of a vehicle including inputs from motion sensors on the vehicle;

[0033] Figure 8 is a schematic diagram of the probability distribution of lateral severity;

[0034] Fig. 9 is a schematic diagram of how a vehicle uses the lateral severity probability distribution to change the vehicle's route; and

[0035] Fig.10 is a flow chart illustrating a method according to an exemplary embodiment.

[0036] The drawings are not necessarily drawn to scale, and certain features may be enlarged or reduced, for example, to show details of a particular component. In some cases, well-known components, systems, materials, or methods are not described in detail to avoid obscuring the present disclosure. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to adopt the present disclosure in different ways. DETAILED DESCRIPTION

[0037] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application or use. In addition, it is not intended to be constrained by any express or implied theory presented in the aforementioned technical field, background technology, summary of the invention or the following specific embodiments. It should be understood that throughout the drawings, corresponding reference numerals represent similar or corresponding components and features. The term module used herein refers to any hardware, software, firmware, electronic control component, processing logic and / or processor device, alone or in any combination, including but not limited to: application specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated or groups) and memories that execute one or more software or firmware programs, combinational logic circuits and / or other suitable components that provide the described functions. Although the drawings shown herein depict examples with certain element arrangements, additional intermediate elements, devices, features or components may exist in actual embodiments. It should also be understood that the drawings are merely illustrative and may not be drawn to scale.

[0038] The term "vehicle" as used herein is not limited to automobiles. Although the present technology is described primarily in conjunction with automobiles herein, the present technology is not limited to automobiles. These concepts can be used in a variety of applications, such as aircraft, ships, other vehicles, and consumer electronic components.

[0039] According to an exemplary embodiment, Figure 1A vehicle 10 is shown according to various embodiments having an associated system 11 for detecting road defects in a road ahead of the vehicle 10 as the vehicle 10 moves along the road. Generally speaking, the system 11 works in conjunction with other systems within the vehicle 10. The vehicle 10 generally includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is disposed on the chassis 12 and substantially surrounds components of the vehicle 10. The body 14 and the chassis 12 may together form a vehicle frame. The front wheels 16 and the rear wheels 18 are each rotatably connected to the chassis 12 near respective corners of the body 14.

[0040] In various embodiments, the vehicle 10 is an autonomous vehicle and the system 11 is incorporated into the autonomous vehicle 10. For example, the autonomous vehicle 10 is a vehicle 10 that is automatically controlled to transport passengers from one location to another. The vehicle 10 is depicted as a passenger car in the illustrated embodiment, but it should be understood that any other vehicle may also be used, including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), etc. In an exemplary embodiment, the vehicle 10 is equipped with a so-called level 4 or level 5 automation system. The level 4 system represents "high automation" and refers to the driving mode-specific performance of the autonomous driving system in all aspects of dynamic driving tasks, even if the human driver does not respond appropriately to the intervention request. The level 5 system represents "full automation" and refers to the full-time performance of the autonomous driving system in all aspects of dynamic driving tasks under all road and environmental conditions that the human driver can manage. The novel aspects of the present disclosure are also applicable to non-autonomous driving vehicles.

[0041] As shown, the vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, a vehicle controller 34, and a wireless communication module 36. In an embodiment where the vehicle 10 is an electric vehicle, the transmission system 22 may not be present. In various embodiments, the propulsion system 20 may include an internal combustion engine, an electric motor such as a traction motor, and / or a fuel cell propulsion system. The transmission system 22 is configured to transmit power from the propulsion system 20 to the front wheels 16 and rear wheels 18 of the vehicle according to a selectable speed ratio. According to various embodiments, the transmission system 22 may include a stepped automatic transmission, a continuously variable transmission, or other suitable transmissions. The braking system 26 is configured to provide braking torque to the front wheels 16 and rear wheels 18 of the vehicle. In various embodiments, the braking system 26 may include friction brakes, wire control brakes, regenerative braking systems (e.g., motors), and / or other suitable braking systems. The steering system 24 affects the position of the front wheels 16 and rear wheels 18. Although depicted as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of the present disclosure, the steering system 24 may not include a steering wheel.

[0042] The sensor system 28 includes one or more sensing devices 40a-40n that sense observable conditions of the external environment and / or the internal environment of the autonomous vehicle 10. The sensing devices 40a-40n may include, but are not limited to, radar, lidar, global positioning system, optical camera, thermal camera, ultrasonic sensor, and / or other sensors. The camera may include two or more digital cameras spaced at a selected distance from each other, wherein the two or more digital cameras are used to obtain stereo images of the surrounding environment in order to obtain a three-dimensional image or map. The plurality of sensing devices 40a-40n are used to determine information about the surrounding environment of the vehicle 10. In an exemplary embodiment, the plurality of sensing devices 40a-40n include at least one of a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor. In another exemplary embodiment, the plurality of sensing devices 40a-40n also include sensors for determining information about the surrounding environment of the vehicle 10, such as an ambient air temperature sensor, an atmospheric pressure sensor, and / or a camera and / or a video camera, which is positioned to observe the environment in front of the vehicle 10. In another exemplary embodiment, at least one of the plurality of sensing devices 40 a - 40 n is capable of measuring distances in the environment surrounding the vehicle 10 .

[0043] In a non-limiting example where the plurality of sensing devices 40a-40n include cameras, the plurality of sensing devices 40a-40n measure distance using an image processing algorithm configured to process images from the cameras and determine distances between objects. In another non-limiting example, the plurality of vehicle sensors 40a-40n include stereo cameras with distance measurement capabilities. In one example, at least one of the plurality of sensing devices 40a-40n is fixed to the interior of the vehicle 10, for example, fixed in the roof lining of the vehicle 10 and visible through the windshield of the vehicle 10. In another example, at least one of the plurality of sensing devices 40a-40n is a camera fixed to the exterior of the vehicle 10 (e.g., fixed on the roof of the vehicle 10) that has a field of view of the environment surrounding the vehicle 10 and is suitable for collecting information (images) related to the environment outside the vehicle 10. It should be understood that various additional types of sensing devices, such as LiDAR sensors, ultrasonic ranging sensors, radar sensors, and / or time-of-flight sensors are within the scope of the present disclosure. The actuator system 30 includes one or more actuator devices 42 a - 42 n that control one or more functions of the vehicle 10 , such as, but not limited to, the propulsion system 20 , the transmission system 22 , the steering system 24 , and the braking system 26 .

[0044] The vehicle controller 34 includes at least one processor 44 and a computer readable storage device or medium 46. The at least one data processor 44 can be any custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor in a plurality of processors associated with the vehicle controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or any device generally used to execute instructions. The computer readable storage device or medium 46 can include volatile and non-volatile storage devices such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the at least one data processor 44 is powered off. The computer readable storage device or medium 46 can be implemented using any of a variety of known storage devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or any other electrical, magnetic, optical, or combination storage device capable of storing data used by the controller 34 in controlling the vehicle 10, some of which represents executable instructions.

[0045] The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. When executed by at least one processor 44, the instructions receive and process signals from the sensor system 28, execute logic, calculations, methods and / or algorithms for automatically controlling components of the vehicle 10, and generate control signals to the actuator system 30 to automatically control components of the vehicle 10 based on the logic, calculations, methods and / or algorithms. Although Figure 1 Only one controller 34 is shown, but embodiments of the vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication mediums and collaborate to process sensor signals, execute logic, calculations, methods and / or algorithms, and generate control signals to automatically control functional components of the autonomous vehicle 10.

[0046] In various embodiments, one or more instructions of the vehicle controller 34 are embodied in the trajectory planning system and, when executed by at least one data processor 44, generate trajectory outputs that address the kinematic and dynamic constraints of the environment. For example, the instructions receive process sensor and map data as input. The instructions perform a graph-based approach using a customized cost function to handle different road scenarios for cities and highways.

[0047] The wireless communication module 36 is configured to wirelessly communicate information with other remote entities 48, such as, but not limited to, other vehicles ("V2V" communication), infrastructure ("V2I" communication), remote systems, remote servers, cloud computers, and / or personal devices. In an exemplary embodiment, the communication system 36 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communications. However, other or alternative communication methods such as dedicated short-range communication (DSRC) channels are also considered to be within the scope of the present disclosure. A DSRC channel refers to a one-way or two-way short-range to medium-range wireless communication channel designed specifically for automotive use and a set of corresponding protocols and standards.

[0048] The vehicle controller 34 is a non-universal electronic control device having a pre-programmed digital computer or processor, a memory or non-transitory computer-readable medium for storing data such as control logic, software applications, instructions, computer code, data, look-up tables, etc., and a transceiver [or input / output port]. Computer-readable media include any type of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard drive, compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer-readable media excludes wired, wireless, optical or other communication links that transmit temporary electrical or other signals. Non-transitory computer-readable media include media that can store data permanently and media that can store data and then be rewritten, such as rewritable optical disks or erasable storage devices. Computer code includes any type of program code, including source code, object code, and executable code.

[0049] refer to Figure 2 , a schematic diagram of the system 11 is shown. The system 11 includes a system controller 34A in communication with a plurality of sensing devices (onboard sensors) 40a-40n. The system controller 34A may be the vehicle controller 34, or the system controller 34A may be a separate controller in communication with the vehicle controller 34. The plurality of onboard sensors 40a-40n include at least one camera 50 in communication with the system controller 34A and adapted to capture a plurality of time-series images 52 of a road 54 in front of the moving vehicle 10, and at least one motion sensor 56 in communication with the system controller 34A and adapted to detect when the vehicle 10 encounters a road defect 58.

[0050] When the vehicle 10 travels along the road 54, at least one camera 50 captures a time series of images 52 of the road 54. Figure 3, camera 50 captures an image of a road 54 in front of a moving vehicle 10. As shown, camera 50 captures an image 52 of road 54, which includes road defects 58 in the road 54 in front of vehicle 10, such as potholes as shown. Camera 50 can be any type of camera known in the industry to be suitable for capturing images near vehicle 10. There are systems that use cameras to capture images of the road 54 in front of vehicle 10 and use computer vision algorithms to identify objects such as other vehicles, pedestrians, and road defects in the path of vehicle 10. However, such systems can only identify road defects that have been manually marked. In addition, such systems rely on the vehicle's global positioning system to obtain inaccurate location information about road defects 56.

[0051] refer to Figure 4 , at least one motion sensor 56 detects when the vehicle 10 encounters a road defect 58 (pothole). The use of motion sensors to detect and measure road defects and road roughness is well known in the industry. Various vehicle sensors can be used to sense road defects 58. For example, wheel rotation differences can provide data related to road roughness or the impact of road defects such as potholes, but do not provide precise information related to the lateral location of the road defect. Inertial sensors (longitudinal, lateral and vertical) can provide acceleration data to enhance the readings of other sensors and enable event type differentiation (e.g. potholes vs. speed bumps) without providing precise lateral location. Semi-active and active suspension systems based on magnetorheological, electrorheological and electromagnetic actuators can improve ride comfort, report shock absorber displacement measurements for each wheel, and can more accurately indicate the road defects and road roughness experienced by each wheel. Such systems normalize various sensor readings to account for differences in wheel size / tire profile and tire pressure, vehicle type (sports sedan / SUV, etc.) and surrounding environment.

[0052] Thus, the at least one camera 50 helps provide visual identification of objects (including the road defect 58) in the road 54 ahead of the vehicle 10, in part by “seeing” the road defect 58 in the images 52 captured by the at least one camera 50, while the at least one motion sensor 56 helps “feel” the impact when the vehicle 10 actually encounters the road defect 58. In an exemplary embodiment, the system controller 34A is adapted to associate the road defect 58 detected by the at least one motion sensor 56 with at least one of the plurality of time-series images 52 of the road 54 collected by the at least one camera 50.

[0053] At least one camera 50 captures a plurality of time-series images 52 leading to a road defect 58 , some of the plurality of time-series images 52 including the road defect 58 , and when at least one motion sensor 56 detects that the vehicle 10 collides with the road defect 58 , the system controller 34A reviews the plurality of time-series images 52 to identify at least one of the plurality of time-series images 52 including the road defect 58 .

[0054] The system controller 34A further uses a computer vision algorithm 60 to analyze at least one of the multiple time-series images 52 associated with a road defect 58 detected by at least one motion sensor 56 to identify the road defect 58 within at least one of the multiple time-series images 52 associated with the road defect 58 detected by at least one motion sensor 56 through the computer vision algorithm 60, and mark the road defect 58 identified within at least one of the multiple time-series images 52 associated with the road defect 58 detected by at least one motion sensor 56.

[0055] refer to Figure 5A In an exemplary embodiment, when the road defect 58 detected by the at least one motion sensor 56 is associated with at least one of the plurality of time-series images 52 of the road 54 collected by the at least one camera 50, the system controller 34A is further adapted to detect the road defect 58 detected by the at least one motion sensor 56 at a time t n At time t, a road defect 58 is detected by at least one motion sensor 56. n is the time when the vehicle 10 actually encounters the road defect 58. Figure 4 , t n is the moment when the front tire 16 of the vehicle 10 physically impacts the road defect 58 (pothole). Figure 5A and Figure 5B , time t n-1 5A represents the time at which at least one camera 50 captures an image 52A of a road 54 before the vehicle 10 impacts the road defect 58, the image 52A including the road defect 58 within the image 52A. Figure 5A and Figure 5C , time t n-2 Indicates that at time t n-1 At least one camera 50 captures an image 52B of a road 54, the image 52B including a road defect within the image. Thus, in this example, before the vehicle 10 encounters the road defect 58, Figure 4 As shown, two images 52A, 52B are captured by at least one camera 50, such as Figure 3 As shown, it includes road defects 58, such as Figure 5B and Figure 5C shown.

[0056] The system controller 34A uses a position algorithm 62 in communication with the system controller 34A to calculate the position of at least one camera 50 at time t n The posture at t is given by n (x, y, z, α, β, γ), where x, y, and z represent conventional three-dimensional coordinates, α represents the roll of the vehicle 10 (and therefore the camera 50), β represents the pitch of the vehicle 10 (and therefore the camera 50), and γ represents the yaw of the vehicle (and therefore the camera 50). The pose of the camera 50 is the six-dimensional position of the camera based on the quantitative values ​​of the variables x, y, z, α, β, γ, which are measured by various sensors 40a-40n within the vehicle 10 and provided to the system controller 34A.

[0057] The system controller 34A is further adapted to calculate the relative position of the road defect 58 with respect to the at least one camera 50 through the position algorithm 62, and, since the at least one camera 50 and the at least one motion sensor 56 are mounted on the same rigid structure (the vehicle 10), at time t n The relative position of the at least one motion sensor 56. Therefore, the system controller 34A can determine the relative position of the at least one motion sensor 56 at time t. n The position of the camera 50 is translated from the position of the road defect 58 back to the position of the at least one camera 50, as shown by the line 64 at the collision point. In addition, the system controller 34A calculates the position of the camera 50 at time t by the position algorithm 62. n-x The six-dimensional pose of at least one camera 50 of at least one image in a plurality of time-series images 52A, 52B associated with a road defect 58 detected by at least one motion sensor 56, wherein t n-x represents the time (t n-1 ,t n-2 ).

[0058] In this example, at least one of the plurality of time-series images 52A, 52B associated with the road defect 58 detected by the at least one motion sensor 56 includes a time t n-1 The image 52A captured at time t n-2 At least one camera 50 captures an image 52B at time t n-1 The six-dimensional posture at is given as t n-1 (x, y, z, α, β, γ), at least one camera 50 at time t n-2 The six-dimensional posture of n-2 Finally, the system controller 34A calculates the position of the road defect 58 in the multiple time series images 52 (at time t n-1 The image 52A captured at time t n-2The relative position in each of at least one of the images 52B) captured at the time t n-x The position of the road defect 58 detected at time t is correlated with the position of the road defect 58 detected at time t, as indicated by lines 66 and 68, respectively. Since the pose of each of the correlated images 52A, 52B is known, the position of the road defect 58 can be converted back to the position at time t. n-1 and time t n-2 Images 52A, 52B captured at.

[0059] In an exemplary embodiment, when identifying a road defect 58 within at least one of the plurality of time-series images 52 associated with a road defect 58 detected by at least one motion sensor 56, the system controller 34A is further adapted to receive data from previously captured images of the vehicle 10, the data being stored in a local database 70 in communication with the system controller 34A. The images include past events in which the road defect 58 was detected, identified, and marked, and the system controller 34A and the machine learning model 72 then use these images to predict the location of the road defect 58 within at least one of the plurality of time-series images 52 associated with the road defect 58 detected by the at least one motion sensor 56.

[0060] The machine learning model 72 uses past instances of the vehicle 10 encountering a road defect 58 and the labeled images captured during such past instances to support the real-time data collected by the at least one motion sensor 56 and more accurately correlate the detection of the "felt" road defect 58 back to the multiple time-series images 52 captured by the at least one camera 50. Therefore, in the future, when the vehicle 10 approaches a road defect 58, the system controller 34A and the machine learning model 72 will be able to access more accurately labeled images that include the road defect 58. Based on what has happened in the past, the machine learning model 72 is taught to identify the road defect 58 based on the real-time images 52A, 52B captured by the vehicle 10, thereby allowing the system controller 34A and the machine learning model 72 to accurately identify the road defect 58 in the images 58 in front of the vehicle 10 before the vehicle 10 actually encounters the road defect 58. Each time the vehicle 10 encounters a road defect 58 , the system controller 34A updates the machine learning model 72 using data of the road defect 58 detected by at least one motion sensor 56 and at least one image from a plurality of time series images 52A, 52B associated with the road defect 58 detected by the at least one motion sensor 56 to continuously improve the accuracy of the machine learning model 72 .

[0061] In another exemplary embodiment, when identifying a road defect 58 within at least one of the plurality of time-series images 52 associated with a road defect 58 detected by at least one motion sensor 56, the system controller 34A is further adapted to receive data of images previously captured by other vehicles, the data being stored in a remote database 74 in communication with the system controller 34A. These images include past events where the road defect 58 was detected, identified, and marked by other vehicles encountering the road defect 58, and then the system controller 34A and the global machine learning model 76 predict the location of the road defect 58 within at least one of the plurality of time-series images 52 associated with the road defect 58 detected by at least one motion sensor 56 through these images.

[0062] The global machine learning model 76 uses past instances of other vehicles encountering the road defect 58 and the labeled images captured by such other vehicles during such past instances to support the real-time data collected by the at least one motion sensor 56 and more accurately associate the at least one motion sensor 56's "feeling" detection of the road defect 58 with the plurality of time-series images 52 captured by the at least one camera 50. Based on past events, the global machine learning model 76 is taught to identify the road defect 58 based on the real-time images captured by the vehicle 10, thereby allowing the system controller 34A and the global machine learning model 76 to accurately identify the road defect 58 in front of the vehicle 10 before the vehicle 10 actually encounters the road defect 58. Each time the vehicle 10 encounters the road defect 58, the system controller 34A updates the global machine learning model 76 with the data of the road defect 58 detected by the at least one motion sensor 56 and at least one of the plurality of time-series images 52A, 52B associated with the road defect 58 detected by the at least one motion sensor 56 to continuously improve the accuracy of the global machine learning model 76. Data from the global machine learning model 76 can be uploaded to a map application, allowing a user in the vehicle to make routing decisions by identifying road defects 58 in advance through a human-machine interface 78 or through a vehicle control module 80 in an autonomous vehicle.

[0063] Various techniques are used to extract meaningful features from the sensor readings and data, including time series analysis, frequency domain analysis, and spatiotemporal patterns. The machine learning model or global machine learning model can be one of but not limited to long short-term memory (LSTM), gated recurrent unit (GRU), decision tree, random forest, support vector machine (SVM), neural network (NN), K nearest neighbor (KNN), gradient boosting, and recurrent neural network (RNN). The global machine learning model is trained using data collected from multiple different vehicles76. A diverse dataset is collected from vehicles equipped with sensors such as GPS, accelerometers, cameras, radars, and lidars. The data covers various driving scenarios including urban, highway, and off-road driving. Before the data is fed into the global machine learning model, a pre-processing step is required to remove noise, handle missing values, and standardize features.

[0064] An important step is to extract relevant features from the raw data. As described above, various techniques are used to extract meaningful features from the sensor readings, including time series analysis, frequency domain analysis, and spatiotemporal patterns. Different types of machine learning algorithms can be used to identify road defects within at least one image in a plurality of time series images captured by at least one camera. The global machine learning model 76 is trained on a labeled dataset and evaluated using various performance metrics such as accuracy, precision, recall, F1 score, and confusion matrix. The hyperparameters of the model are adjusted to obtain optimal results. The global machine learning model 76 is further trained on the training data and learns to map input features to corresponding probabilities.

[0065] Real-time data from events where the system controller 34A identifies and marks the road defect 58 is saved to the remote database 74 for continuous updating of the global machine learning model 76. Thus, the system controller 34A, through the global machine learning model 76 and machine learning techniques, more accurately identifies and marks the road defect 58 within the relevant images 52A, 52B based on analysis of real-time data collected by at least one camera 50 and at least one motion sensor 56, and analysis of captured images 52A, 52B and data of past events in the remote database 74 using computer vision and image recognition techniques.

[0066] In another exemplary embodiment, the global machine learning model 76 is a contextual global machine learning model 76A. In general, the global machine learning model 76 is created and taught with data from models obtained from all other participating vehicles. This data is "aggregated" into a single global machine learning model 76. Traditional federated learning techniques simply combine locally trained models from individual vehicles into a cloud global machine learning model 76. However, this may lose knowledge learned by the locally trained models and reduce the performance of the aggregated global machine learning model 76. When the "aggregated" global machine learning model 76 is sent back to the vehicle, it destroys the local learning.

[0067] In another exemplary embodiment, when identifying a road defect within at least one of the plurality of time-series images 52 associated with a road defect 58 detected by at least one motion sensor 56, the system controller 34A is further adapted to receive data of images previously captured by other vehicles from a remote database 74A in communication with the system controller 34A. These images include past events where the road defect 58 was detected, identified, and marked by other vehicles that encountered the road defect 58, and then the system controller 34A and the contextual global machine learning model 76A predict the location of the road defect 58 within at least one of the plurality of time-series images 52 associated with the road defect 58 detected by at least one motion sensor 56 through these images.

[0068] Local machine learning model corruption is reduced by grouping different models based on contextual features and maintaining a separate contextual global machine learning model 76A for each contextual scenario. The contextual features can be based on road type, vehicle model, contextual features extracted using vehicle sensors (e.g., lighting conditions), and / or other user-provided information. When the locally trained model parameters are transmitted to the cloud server for aggregation in the global machine learning model, the server first uses the contextual information to search (create a new one if one does not exist) for an appropriate contextual global machine learning model 76A, and then performs global aggregation with the contextual global machine learning model 76A.

[0069] In an exemplary embodiment, the system controller 34A is further adapted to determine the lateral position of the road defect 58 within the road 54. Typically, for example, with potholes, the road defect 58 does not extend across the entire width of the lane of the road 54, or, if the road defect 58 extends fully across the lane, the severity across the width is different. When a human driver sees a pothole, they can maneuver the vehicle 10 to avoid the pothole, so it is useful, particularly in an autonomous vehicle 10, for the system controller 34A to be able to detect changes in the lateral position and / or severity of the road defect 58 laterally across the road 54.

[0070] In an exemplary embodiment, when determining the lateral position of the road defect 58 within the road 54, the system controller 34A is further adapted to receive data related to the relative severity of the road defect 58 at different lateral positions across the road 54 from at least one motion sensor 56, wherein the at least one motion sensor 56 includes a plurality of motion sensors located at different lateral positions within the vehicle 10.

[0071] For example, refer to Figure 6 , a plurality of sensors 40a-40n in the vehicle 10 detect a first lane line 82 and a second lane line 84. For example, a coordinate system 86 may be created with an origin at the first lane line 82, and the relative positions of the left front wheel 16A and the right front wheel 16B may be calculated within the coordinate system 86. Figure 7 , the inputs 88A, 88B from the motion sensors of the suspension components mounted at the left front wheel 16A and the right front wheel 16B are added to the coordinate system 86. Figure 8 When multiple vehicles encounter a road defect 58, their readings may be added to a coordinate system 86, where, as a non-limiting example, a kernel density estimation algorithm is used to generate a lateral severity probability distribution 90 of the road defect 58. Fig. 9 When the vehicle approaches a road defect, the system controller accesses the lateral severity probability distribution 90 of the road defect 58 by communicating with the remote database 74B, and changes the vehicle route, as shown by line 92, to avoid the most severe portion of the road defect 58, or to avoid collision with the road defect 58 altogether.

[0072] In another exemplary embodiment, when determining the lateral position of the road defect 58 within the road 54, the system controller 34A is further adapted to receive data stored in a remote database 74B in communication with the system controller 34A and related to the relative severity of the road defect 58 at different lateral positions across the road 54 collected from other vehicles. The data includes past events where the road defect 58 was detected and the relative severity across the road 54 was measured and recorded by motion sensors in other vehicles that encountered the road defect 58, which is then used by the system controller 34A and the global machine learning model 76B to predict the longitudinal and lateral position of the road defect 58.

[0073] The global machine learning model 76B uses past instances of other vehicles encountering the road defect 58 and data previously collected by such other vehicles stored in the remote database 74B relating to the relative severity of the road defect 58 at different lateral locations across the road 54 to support the real-time data collected by the at least one motion sensor 56 and more accurately calculate the lateral severity probability distribution 90 of the road defect 58. Based on past occurrences, the global machine learning model 76B is taught to identify the road defect 58 based on the real-time images 52A, 52B captured by the vehicle 10, thereby allowing the system controller 34A and the global machine learning model 76B to accurately identify the road defect 58 in front of the vehicle 10 before the vehicle 10 actually encounters the road defect 58, and to assess the relative severity of the road defect 58 laterally across the lane, thereby allowing the autonomous vehicle system (vehicle control module 80) to strategically reroute the vehicle 10. The system controller 34A updates the global machine learning model 76B each time the vehicle 10 encounters a road defect 58 to continuously improve the accuracy of the global machine learning model 76B and the calculated lateral severity probability distribution 90. Data from the global machine learning model 76B may be uploaded to a mapping application, thereby allowing a user or navigation system to identify road defects 58 and the relative lateral severity of road defects 58 in advance.

[0074] refer to Fig.10 , a method 100 for identifying a road defect 58 in a road 54 in front of a moving vehicle 10 includes: starting at box 102, capturing a plurality of time-series images 52 of the road 54 in front of the moving vehicle 10 by at least one camera 50 in communication with a system controller 34A; moving to box 104, detecting when the vehicle 10 encounters the road defect 58 by at least one motion sensor 56 located in the vehicle 10 and in communication with the system controller 34A; moving to box 106, associating, by the system controller 34A, the road defect 58 detected by the at least one motion sensor 56 with at least one image in the plurality of time-series images 52 of the road 54 collected by the at least one camera 50; moving to box 107, In box 108, the system controller 34A is used to analyze at least one image of the multiple time-series images 52 associated with the road defect 58 detected by the at least one motion sensor 56 using the computer vision algorithm 60; moving to box 110, the system controller 34A and the computer vision algorithm 60 are used to identify the road defect 58 within at least one image of the multiple time-series images 52 associated with the road defect 58 detected by the at least one motion sensor 56; and moving to box 112, the road defect 58 identified within at least one image of the multiple time-series images 52 associated with the road defect 58 detected by the at least one motion sensor 56 is marked by the system controller 34A.

[0075] In an exemplary embodiment, identifying, by the system controller 34A and the computer vision algorithm 60, a road defect 58 within at least one of the plurality of time-series images 52 associated with a road defect 58 detected by at least one motion sensor 56 at box 110 further includes: moving from box 108 to box 114, receiving, by the system controller 34A, data of an image previously captured by the vehicle 10 from a local database 70 in communication with the system controller 34A, wherein the road defect 58 has been marked by the system controller 34A; and moving to box 116, predicting, by the system controller 34A, a location of the road defect 58 within at least one of the plurality of time-series images 52 associated with the road defect 58 detected by at least one motion sensor 56 using a machine learning model 72 in communication with the system controller 34A and the local database 70.

[0076] In an exemplary embodiment, the method 100 further includes, moving from box 116 to box 118, updating the machine learning model 72 using data of the road defect 58 detected by the at least one motion sensor 56 and at least one image in the plurality of time-series images 52 associated with the road defect 58 detected by the at least one motion sensor 56.

[0077] In an exemplary embodiment, identifying a road defect 58 within at least one of the multiple time-series images 52 associated with a road defect 58 detected by at least one motion sensor 56 via the system controller 34A and the computer vision algorithm 60 at box 110 further includes: moving from box 108 to box 120, receiving, via the system controller 34A, data of images previously captured by other vehicles from a remote database 74 in communication with the system controller 34A, in which the road defect 58 has been identified and marked; and moving to box 122, predicting, via the system controller 34A, a location of the road defect 58 within at least one of the multiple time-series images 52 associated with the road defect 58 detected by at least one motion sensor 56 using a global machine learning model 76 in communication with the system controller 34A and trained using data from other vehicles stored in the remote database 74.

[0078] In an exemplary embodiment, the method 100 further includes moving to box 124 and updating the global machine learning model 76 using data of the road defect 58 detected by the at least one motion sensor 56 and at least one image in the plurality of time-series images 52 associated with the road defect 58 detected by the at least one motion sensor 56 .

[0079] In another exemplary embodiment, identifying the road defect 58 within at least one of the multiple time-series images 52 associated with the road defect 58 detected by the at least one motion sensor 56 by the system controller 34A and the computer vision algorithm 60 at box 110 further includes: moving from box 108 to box 126, receiving, by the system controller 34A, data of images previously captured by other vehicles having contextual features matching the vehicle 10 from a remote database 74A in communication with the system controller 34A, wherein the road defect 58 has been identified and marked; and moving to box 128, predicting, by the system controller 34A, a location of the road defect 58 within at least one of the multiple time-series images 52 associated with the road defect 58 detected by the at least one motion sensor 56 using a contextual global machine learning model 76A in communication with the system controller 34A and trained using data of other vehicles stored in the remote database 74A.

[0080] In an exemplary embodiment, the method 100 further includes, moving to box 130, updating the global machine learning model 76A using data of the road defect 58 detected by the at least one motion sensor 56 and at least one image in the plurality of time-series images 52 associated with the road defect 58 detected by the at least one motion sensor 56.

[0081] In an exemplary embodiment, associating, at block 106 by the system controller 34A, the road defect 58 detected by the at least one motion sensor 56 with at least one of the plurality of time-series images 52 of the road 54 collected by the at least one camera 50 further includes: moving from block 104 to block 132, at time t n At the location, at least one motion sensor 56 detects a road defect 58; and moves to block 134, calculating by a position algorithm 62 in communication with the system controller 34A: at least one camera 50 at time t n The road defect 58 at time t n At a relative position relative to at least one motion sensor 56 and at least one camera 50; at time t n-x t , a pose of at least one camera 50 for at least one image 52A, 52B of a plurality of time-series images 52 associated with a road defect 58 detected by at least one motion sensor 56; and at time t n-x At the position of the road defect 58 detected by the at least one motion sensor 56 , a relative position of the road defect 58 within at least one image of the plurality of time-series images 52 is determined.

[0082] In another exemplary embodiment, the method 100 further includes, moving from frame 112 to frame 136 , determining, by the system controller 34A, a lateral location of the road defect 58 within the road 34 .

[0083] In another exemplary embodiment, determining the lateral position of the road defect 58 within the road 54 by the system controller 34A at box 136 further includes: moving from box 112 to box 138, receiving, by the system controller 34A, data related to the relative severity of the road defect 58 at different lateral positions across the road 54 from at least one motion sensor 56, wherein the at least one motion sensor 56 includes a plurality of motion sensors located at different lateral positions within the vehicle 10; and moving to box 140, calculating, by the system controller 34A, a lateral severity distribution 90 of the road defect 58.

[0084] In yet another exemplary embodiment, determining the lateral position of the road defect 58 within the road 54 by the system controller 34A at box 136 further includes: moving from box 112 to box 142, receiving, by the system controller 34A, data related to the relative severity of the road defect 58 at different lateral positions across the road 54 previously collected by other vehicles from a remote database 74B in communication with the system controller 34A; moving to box 144, predicting the lateral severity distribution 90 of the road defect 58 by the system controller 34A using a global machine learning model 76B that is in communication with the system controller 34A and trained using data related to the relative severity of the road defect 58 at different lateral positions across the road 54 previously collected by other vehicles stored in the remote database 74B; and moving to box 146, updating the global machine learning model 76B by data received from multiple motion sensors 56 located at different lateral positions within the vehicle 10.

[0085] An advantage of the system 10 and method 100 of the present disclosure is that when the vehicle 10 actually encounters a road defect 58, the motion sensor 56 "senses" the road defect 58 and associates it with the images 52A, 52B of the road 54 taken before the vehicle 10 actually encountered the road defect 58, and the images 52A, 52B include the road defect 58. In this way, fine-grained position information can be obtained to accurately locate the road defect 58, teach the machine learning model 76, 76A, 76B to enable the system controller 34A to identify the road defect 58 before the vehicle 10 actually encounters the road defect 58, and calculate the lateral severity probability distribution 90 so that the system controller 34A can assess the severity of the road defect 58 that crosses the lane laterally within the road 54. The system provides more accurate information about the road defect 58, which enables the autonomous vehicle controller to more accurately identify the road defect 58 and its relative severity, and more effectively react to and plan routes for the road defect 58 identified within the road.

[0086] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to fall within the scope of the present disclosure. These variations should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. A method for identifying a road defect in a road ahead of a moving vehicle, comprising: capturing a plurality of time-series images of a road ahead of the moving vehicle via at least one camera in communication with a system controller; detecting, by at least one motion sensor located within the vehicle and in communication with the system controller, when the vehicle encounters a road imperfection; associating, by the system controller, the road defect detected by the at least one motion sensor with at least one of the plurality of time-series images of the road collected by the at least one camera; analyzing, by the system controller, using a computer vision algorithm, the at least one image of the plurality of time-series images associated with the road defect detected by the at least one motion sensor; identifying, by the system controller and the computer vision algorithm, a road defect within the at least one image of the plurality of time-series images associated with the road defect detected by the at least one motion sensor; as well as The road defect identified within the at least one image of the plurality of time-series images is marked, by the system controller, in association with the road defect detected by the at least one motion sensor.

2. The method according to claim 1, wherein: Identifying, by the system controller and the computer vision algorithm, a road defect within the at least one image of the plurality of time-series images associated with the road defect detected by the at least one motion sensor further comprises: receiving, by the system controller, data of an image previously captured by the vehicle from a local database in communication with the system controller, wherein the road defect has been marked by the system controller; and A machine learning model in communication with the system controller and the local database is utilized, by the system controller, to predict a location of the road defect within the at least one image of the plurality of time-series images associated with the road defect detected by the at least one motion sensor.

3. The method according to claim 2 further includes updating the machine learning model through data of the road defect detected by the at least one motion sensor and the at least one image in the multiple time series images related to the road defect detected by the at least one motion sensor.

4. The method according to claim 1, wherein: Identifying, by the system controller and the computer vision algorithm, a road defect within the at least one image of the plurality of time-series images associated with the road defect detected by the at least one motion sensor further comprises: receiving, by the system controller, data of images previously captured by other vehicles from a remote database in communication with the system controller, wherein the road defect has been identified and marked; and The system controller predicts, by a global machine learning model that communicates with the system controller and the remote database and is trained using data of other vehicles stored in the remote database, a location of the road defect within the at least one image of the plurality of time-series images associated with the road defect detected by the at least one motion sensor.

5. The method according to claim 4 further includes updating the global machine learning model through data of the road defect detected by the at least one motion sensor and the at least one image in the multiple time series images related to the road defect detected by the at least one motion sensor.

6. The method according to claim 1, wherein: Identifying, by the system controller and the computer vision algorithm, a road defect within the at least one image of the plurality of time-series images associated with the road defect detected by the at least one motion sensor further comprises: receiving, by the system controller, from a remote database in communication with the system controller, data of images previously captured by other vehicles having contextual features matching the vehicle, wherein the road defect has been identified and marked; and The system controller predicts, by the system controller, a location of the road defect within the at least one image of the plurality of time-series images associated with the road defect detected by the at least one motion sensor using a contextual global machine learning model that communicates with the system controller and the remote database and is trained using data of other vehicles stored in the remote database.

7. The method according to claim 1, wherein: Associating, by the system controller, the road defect detected by the at least one motion sensor with at least one of the plurality of time-series images of the road collected by the at least one camera further comprises: At time t n At, detecting the road defect by the at least one motion sensor; and The position algorithm communicates with the system controller to calculate: The at least one camera at time t n The posture of the place; The road defect at time t n a relative position relative to the at least one motion sensor and the at least one camera; At time t n-x at least one camera for at least one image in the plurality of time-series images associated with the road defect detected by the at least one motion sensor; and At time t n-x At the position of the road defect detected by the at least one motion sensor, a relative position of the road defect within at least one image of the plurality of time-series images is determined.

8. The method of claim 1, further comprising determining, by the system controller, a lateral location of the roadway defect within the roadway.

9. The method according to claim 8, wherein: Determining, by the system controller, the lateral position of the road defect within the road further comprises: receiving, by the system controller, data related to relative severity of the roadway imperfections at different lateral locations across the roadway from the at least one motion sensor, wherein the at least one motion sensor comprises a plurality of motion sensors located at different lateral locations within the vehicle; and A lateral severity distribution of the road defect is calculated by the system controller.

10. The method according to claim 9, wherein: Determining, by the system controller, the lateral position of the road defect within the road further comprises: receiving, by the system controller, from a remote database in communication with the system controller, data relating to relative severity of the roadway imperfections at different lateral locations across the roadway previously collected by other vehicles; predicting, by the system controller, a lateral severity distribution of the roadway defect using a global machine learning model in communication with the system controller and the remote database and trained using data stored in the remote database and relating to relative severity of the roadway defect at different lateral locations across the roadway previously collected by other vehicles; and The global machine learning model is updated using data received from the plurality of motion sensors located at different lateral locations within the vehicle.

Images