Intelligent vehicle system and control logic for accident prediction and assistance

Abstract

An intelligent vehicle system for off-road driving incident prediction and assistance, methods for manufacturing / operating such systems, and vehicles networked with such systems are presented. A method for operating a motor vehicle includes a system controller receiving geographic location data indicative of the vehicle being in or entering an off-road terrain. In response to the vehicle geographic location data, the controller receives camera-generated images from cameras mounted on the vehicle, each camera-generated image containing the drive wheel(s) of the vehicle and / or the surface of the off-road terrain. The controller receives vehicle operating characteristic data and vehicle dynamics data of the motor vehicle from a controller area network bus. The camera data, vehicle operating characteristic data, and vehicle dynamics data are processed via a convolutional neural network backbone to predict the occurrence of a driving incident on the off-road terrain within a prediction time horizon. The system controller, in response to the predicted occurrence of a driving incident, commands a resident vehicle system to perform a control operation.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to control systems for motor vehicles. More specifically, aspects of the present disclosure relate to intelligent vehicle navigation systems and control logic for predicting driving incidents in off-road situations. BACKGROUND

[0002] Currently produced motor vehicles, such as modern automobiles, can be equipped with in-vehicle electronics networks and wireless communication capabilities that provide automated driving capabilities and navigation assistance. As vehicle processing, communication, and sensing capabilities have increased, manufacturers have pressed to provide more automated driving functionality, while desiring to produce fully autonomous “self-driving” vehicles capable of navigating between heterogeneous vehicle types in both urban and rural scenarios. Original equipment manufacturers (OEMs) are moving toward talking car-to-car (V2V) and car-to-infrastructure (V2I) with higher levels of driving automation that employ autonomous control systems to enable vehicle routing with steering, lane changes, scenario planning, and the like. For example, automated path planning systems utilize vehicle state and dynamics sensors, geolocation information, map and road condition data, and path prediction algorithms to provide route derivation with automated lane centering and lane change prediction.

[0003] Many automobiles now come equipped with in-vehicle computer navigation systems that utilize global positioning system (GPS) transceivers in cooperation with navigation software and geolocation map services to obtain road topography, traffic, and speed limit data associated with the vehicle’s current location. For example, self-organizing network-based driver assistance systems can use GPS and map data in conjunction with multi-hop geographic broadcast V2V and V2I data exchange to facilitate automated vehicle maneuvering and powertrain control. During vehicle operation, the resident navigation system can identify a recommended travel route based on an estimated shortest travel time or an estimated shortest travel distance between a route origin and a route destination for a given trip. This recommended travel route can then be displayed as a map trace, or as a route-by-route driving direction on a geocoded and annotated map with optional voice commands output by the vehicle’s audio system. SUMMARY

[0004] Presented herein are intelligent vehicle navigation systems with companion control logic for off-road driving incident prediction and assistance, methods of manufacturing such systems and methods for using such systems, and motor vehicles networked with such systems. By way of example, off-road driving can involve "emergency" situations in which the vehicle becomes inoperable or inoperable to function due to unique track conditions of a given route (e.g., the host vehicle getting stuck on a boulder or in mud on a forest track). The occurrence of these situations can be due to the host vehicle having breached its off-road envelope capabilities (e.g., driving at higher than rated approach or departure angles). Presented are intelligent vehicle systems and control logic for automatically predicting an impending emergency situation by employing, for example, vehicle geolocation data, telemetry data, topography data, and vehicle sensing data (e.g., a camera focused on the bottom of the vehicle body). Using this data, a machine learning (ML) based model is implemented to estimate the probability of an impending emergency situation. Upon the prediction of an impending emergency situation, an off-road driving expert can utilize off-road driving techniques to provide remote assistance to the driver of the host vehicle to improve the situation. The expert remote assistant can receive a detailed 3D topography mesh of the terrain under and around the vehicle from one or more data sources. To prevent or correct the situation, the remote assistant can provide real-time instructions to the user or will remotely operate the vehicle without driver input.

[0005] Incidental benefits of at least some of the disclosed concepts include intelligent vehicle navigation systems that accurately predict driving incidents in off-road areas, such as breakdown or rollover incidents on off-road tracks or remote roads. To facilitate driving incident prediction and minimize false positives, a 3D mesh rendering of the surface topography under the host vehicle can be generated in real-time using onboard camera and ultrasonic sensor data. Many modern cars with advanced driver assistance systems (ADAS) and autonomous driving capabilities are equipped with front, rear, and side-facing cameras; however, to enable automatic prediction of impending off-road issues, the disclosed host vehicle can employ a networked sensor array with a dedicated underbody camera to capture real-time images of the vehicle chassis. In addition to accurate incident prediction, features are disclosed for resolving emergency events in off-road situations using an expert remote assistant or intelligent virtual assistant.

[0006] Aspects of the present disclosure relate to system control logic, closed loop feedback control techniques, and computer readable media (CRM) for manufacturing and / or for operating any disclosed vehicle sensor network, navigation system, and / or host vehicle. In one example, a method is presented for controlling operation of a motor vehicle equipped with a sensor array comprising a network of video cameras mounted at discrete locations on the vehicle. This representative method includes, in any order and in any combination with the above and below disclosed options and features: receiving, e.g., via a wired / wireless communication device from an on-board telematics unit, a smart phone, or a GPS-based satellite service via a resident or remote system controller, vehicle geolocation data indicative of the motor vehicle being in or entering an off-road terrain; responsive to the vehicle entering the off-road terrain / traveling across the off-road terrain, receiving, e.g., via the system controller from the sensor array, camera data indicative of camera-generated images captured by the cameras and containing one or more of a drive wheel of the motor vehicle and / or a terrain surface of the off-road terrain; receiving, e.g., via the controller from a controller area network (CAN) bus of the vehicle, vehicle operating characteristic data and vehicle dynamics data of the motor vehicle; processing, via a convolutional neural network (CNN) backbone, the camera data, the vehicle operating characteristic data, and the vehicle dynamics data to predict an occurrence of a driving incident on the off-road terrain within a prediction time horizon; and transmitting, via the system controller, one or more command signals to one or more resident vehicle systems to perform one or more control operations in response to the predicted occurrence of the driving incident.

[0007] A non-transitory CRM storing instructions executable by one or more processors operable to control a system controller of a motor vehicle is also presented. The vehicle includes a sensor array having a network of video cameras mounted at discrete locations on a vehicle body. When executed by the one or more processors, the instructions cause the controller to perform operations including: receiving, via a wireless communication device, vehicle geolocation data indicative of the motor vehicle being in or entering an off-road terrain; receiving, from the sensor array, camera data indicative of camera-generated images captured by the cameras and each containing a drive wheel of the vehicle and / or a terrain surface; receiving, from a CAN bus of the motor vehicle, vehicle operating characteristic data and vehicle dynamics data of the motor vehicle; processing, via a CNN backbone, the camera data, the vehicle operating characteristic data, and the vehicle dynamics data to predict whether a driving incident will occur on the off-road terrain within a prediction time horizon; and transmitting, to a resident vehicle system, a command signal to perform a control operation in response to the predicted occurrence of the driving incident.

[0008] Additional aspects of the present disclosure relate to an intelligent vehicle navigation system that provides navigation and emergency services for a motor vehicle. As used herein, the terms "vehicle" and "motor vehicle" can be used interchangeably and synonymously to include any relevant vehicle platform, such as passenger cars (ICE, HEV, FEV, fuel cell, fully autonomous and partially autonomous, etc.), commercial vehicles, industrial vehicles, tracked vehicles, off-road and all-terrain vehicles (ATVs), motorcycles, agricultural equipment, etc. In one example, the motor vehicle includes a vehicle body having a plurality of road wheels, a passenger cabin, and other standard original equipment. A prime mover, such as an electric traction motor and / or an internal combustion engine assembly, drives one or more road wheels to propel the vehicle. A sensor array is also mounted on the vehicle, including a network of video cameras mounted at discrete locations on the vehicle body (e.g., front, rear, port, starboard, and underbody cameras).

[0009] Continuing the discussion of the above example, one or more resident or remote electronic controllers receive, via wired or wireless communication means, vehicle geolocation data indicating that the vehicle is in or entering an off-road terrain. In response to the geolocation data indicating that the vehicle is in / entering the off-road terrain, the controller(s) communicate with the vehicle sensor array to receive camera data indicative of camera-generated images captured by the video cameras. Each camera-generated image contains at least one of a drive wheel of the motor vehicle and / or a surface of the off-road terrain. The system controller(s) also communicate with the vehicle CAN bus to receive vehicle operating characteristic data and vehicle dynamics data of the motor vehicle. The CNN backbone processes the camera data, the vehicle operating characteristic data, and the vehicle dynamics data to predict the occurrence of a driving incident on the off-road terrain within a prediction time horizon. If the system controller predicts that a driving incident will occur, it transmits at least one command signal to at least one resident vehicle system to perform at least one control operation in response to the predicted occurrence of the driving incident.

[0010] For any disclosed vehicle, system, and method, a region of interest (ROI) is defined for each camera; the ROI is embedded and fixed in a predefined location of the camera view of the camera. In this case, a cropped camera image is generated by cropping each camera-generated image to remove image data outside the ROI. The CNN backbone can analyze the cropped image to determine one or more wheel characteristics of one or more of the vehicle drive wheels. The wheel characteristics can include a loss of ground contact state, a stuck in mud, sand, water, and / or other obstruction state, and / or a loss of tire tread, pressure, traction, and / or other normal operating feature state. Predicting the occurrence of a driving incident can be based at least in part on the wheel characteristic(s) of the drive wheel(s).

[0011] For any disclosed vehicle, system, and method, the CNN backbone can analyze the cropped images to determine one or more terrain properties of the terrain surface of the off-road terrain. The terrain properties can include terrain types (e.g., sand, mud, asphalt, rock, etc.) and / or terrain conditions (e.g., wet, dry, icy, snowy, loose / tight, etc.). As another option, the CNN backbone can analyze the cropped images to determine one or more obstacle properties of the obstacle impeding the path of the motor vehicle. The obstacle properties can include an obstacle height relative to a wheel height of the drive wheel(s) and / or a body height of the vehicle body. Predicting the occurrence of the driving incident can be based at least in part on the terrain property(s) of the off-road terrain and / or the obstacle property(s) of the obstacle.

[0012] For any disclosed vehicle, system, and method, processing the camera data can include, for each camera-generated image independent of other images, analyzing a single frame of the ROI to assign properties to the drive wheels and / or terrain surface contained in the image, and based on the ROI analysis, generating, via the CNN backbone, situation cues indicative of the properties and assessable via a situation classifier module operable to predict the occurrence of the driving incident using multiple task heads. As yet another option, processing the camera data can include analyzing, via the CNN backbone, consecutive frames of the ROI for all camera-generated images to extract features of the drive wheels and / or terrain surface contained in the images, linking the camera-generated images into a sequence or chain using a concatenation module, and extracting temporal information of each camera-generated image via a recurrent neural network (RNN).

[0013] For any disclosed vehicle, system, and method, the camera network can include a vehicle underbody camera mounted proximate to the vehicle chassis. The vehicle underbody camera can be operable to capture an outward-facing downward view from the vehicle body and generate a signal indicative of the view. As yet another option, the vehicle dynamics data retrieved from the CAN bus can include vehicle roll data, vehicle pitch data, vehicle yaw data, vehicle lateral / longitudinal velocity data, vehicle lateral / longitudinal acceleration data, wheel speed data, steering angle data, etc. In this regard, the vehicle operating property data retrieved from the CAN bus can include throttle position data, brake force data, park brake status, powertrain mode data, suspension height data, etc.

[0014] For any disclosed vehicle, system, and method, the resident vehicle system commanded by the system controller includes an autonomous driving control (ADC) module operable to automatically drive the motor vehicle. In this case, the control operation can include automating one or more driving maneuvers of the motor vehicle based at least in part on the predicted occurrence of the driving incident. As another option, the resident vehicle system can include a vehicle steering system, a vehicle braking system, and / or a vehicle powertrain. In this case, the control operation can include controlling a virtual or human assistance of steering maneuvers, braking operations, and / or powertrain torque output based at least in part on the predicted occurrence of the driving incident. Optionally, the vehicle system can include a vehicle navigation system having a display device. In this case, the display device of the navigation system can display an alert with a remedial driving maneuver to prevent the occurrence of the driving incident.

[0015] The present invention provides the following technical solutions:

[0016] 1. A method for controlling operation of a motor vehicle with a sensor array comprising a network of video cameras mounted at discrete locations on the motor vehicle, the method comprising:

[0017] receiving, via an electronic system controller, vehicle geolocation data indicative of the motor vehicle being in or entering an off-road terrain via a wireless communication device;

[0018] in response to the vehicle geolocation data, receiving, via the system controller, camera data from the sensor array, the camera data indicative of camera-generated images captured by the video cameras and including a drive wheel of the motor vehicle and a terrain surface of the off-road terrain;

[0019] receiving, via the system controller, vehicle operating characteristic data and vehicle dynamics data of the motor vehicle from a controller area network (CAN) bus;

[0020] processing the camera data, the vehicle operating characteristic data, and the vehicle dynamics data via a convolutional neural network (CNN) backbone to predict an occurrence of a driving incident on the off-road terrain within a prediction time horizon; and

[0021] in response to the predicted occurrence of the driving incident, transmitting, via the system controller, a command signal to a resident vehicle system to perform a control operation.

[0022] The method according to technical solution 1, further comprising:

[0023] determining a region of interest (ROI) of a predefined location of a camera view embedded and fixed in each of the video cameras; and

[0024] generating cropped camera images by cropping each of the images generated by the camera to remove image data outside of the ROI.

[0025] The method of technical solution 2, wherein processing the camera data comprises the CNN backbone analyzing the cropped images to determine wheel characteristics of drive wheels of the motor vehicle, wherein predicting the occurrence of the driving incident is based on the wheel characteristics of the drive wheels.

[0026] The method of technical solution 3, wherein the wheel characteristics comprise a loss of ground contact state, a stopping in an obstruction state, and / or a loss of normal operating characteristics state.

[0027] The method of technical solution 2, wherein processing the camera data comprises the CNN backbone analyzing the cropped images to determine terrain characteristics of a terrain surface of the off-road terrain, wherein predicting the occurrence of the driving incident is based on the terrain characteristics of the terrain surface.

[0028] The method of technical solution 5, wherein the terrain characteristics comprise a terrain type and / or a terrain condition.

[0029] The method of technical solution 2, wherein processing the camera data comprises the CNN backbone analyzing the cropped images to determine obstacle characteristics of an obstruction obstructing driving of the motor vehicle, wherein predicting the occurrence of the driving incident is based on the obstacle characteristics of the obstruction.

[0030] The method of technical solution 7, wherein the obstacle characteristics comprise an obstacle height relative to a wheel height of the drive wheels and / or a body height of a body of the motor vehicle.

[0031] The method of technical solution 2, wherein processing camera data comprises:

[0032] for each of the images generated by the cameras independently of one another, analyzing individual frames of the ROI to assign characteristics to drive wheels and / or terrain surfaces contained in the camera generated images; and

[0033] based on the ROI analysis, generating, via the CNN backbone using the plurality of task heads, situation cues indicative of the characteristics and assessable via a situation classifier module operable to predict the occurrence of the driving incident.

[0034] The method of technical solution 2, wherein processing camera data comprises:

[0035] analyzing, via the CNN backbone, consecutive frames of the ROI for all of the camera generated images to extract features of drive wheels and / or terrain surfaces contained in the camera generated images;

[0036] linking the camera-generated images into a series or chain via a concatenation module; and

[0037] extracting temporal information for each camera-generated image via a recurrent neural network (RNN).

[0038] The method of claim 1, wherein the network of cameras includes a vehicle underbody camera mounted to a body of the motor vehicle proximate to a chassis of the body, the vehicle underbody camera operable to capture an outwardly facing downward view from the body and generate a signal indicative of the view.

[0039] The method of claim 1, wherein the vehicle dynamics data includes vehicle roll data, vehicle pitch data, vehicle yaw data, vehicle lateral / longitudinal speed data, vehicle lateral / longitudinal acceleration data, wheel speed data, and / or steering angle data, and wherein the vehicle operating characteristic data includes throttle position data, brake force data, park brake status, powertrain mode data, and / or suspension height data.

[0040] The method of claim 1, wherein the resident vehicle system includes an autonomous driving control module operable to autonomously drive the motor vehicle, the control operation including automating driving maneuvers of the motor vehicle based on the predicted occurrence of the driving incident.

[0041] The method of claim 1, wherein the resident vehicle system includes a vehicle navigation system having a display device, the control operation including displaying, via the display device, an alert with a remedial driving maneuver to prevent the occurrence of the driving incident.

[0042] A non-transitory computer-readable medium storing instructions executable by a system controller operable to control operation of a motor vehicle including a sensor array having a network of cameras mounted at discrete locations on the motor vehicle, the instructions, when executed, causing the system controller to perform operations comprising:

[0043] receiving, via a wireless communication device, vehicle geolocation data indicative of the motor vehicle being at or entering an off-road terrain;

[0044] receiving, from the sensor array, camera data indicative of camera-generated images captured by the cameras, and each camera-generated image containing a drive wheel of the motor vehicle and / or a terrain surface of the off-road terrain;

[0045] receiving, from a controller area network (CAN) bus of the motor vehicle, vehicle operating characteristic data and vehicle dynamics data of the motor vehicle;

[0046] processing the camera data, the vehicle operating characteristic data, and the vehicle dynamics data via a convolutional neural network (CNN) backbone to predict occurrence of a driving incident on the off-road terrain within a prediction time horizon; and

[0047] in response to the predicted occurrence of the driving incident, transmitting a command signal to a resident vehicle system to perform a control operation.

[0048] A motor vehicle comprising:

[0049] a vehicle body;

[0050] a plurality of drive wheels mounted to the vehicle body;

[0051] a prime mover mounted to the vehicle body and operable to drive one or more of the drive wheels to propel the motor vehicle;

[0052] a sensor array comprising a network of cameras mounted at discrete locations on the vehicle body; and

[0053] an electronic system controller programmed to:

[0054] receive, via a wireless communication device, vehicle geolocation data indicative of the motor vehicle being on or entering an off-road terrain;

[0055] in response to the vehicle geolocation data, receive, from the sensor array, camera data indicative of camera-generated images captured by the cameras, and each camera-generated image containing one or more of the drive wheels and / or a terrain surface of the off-road terrain;

[0056] receive, from a controller area network (CAN) bus, vehicle operating characteristic data and vehicle dynamics data of the motor vehicle;

[0057] process the camera data, the vehicle operating characteristic data, and the vehicle dynamics data via a convolutional neural network (CNN) backbone to predict occurrence of a driving incident on the off-road terrain within a prediction time horizon; and

[0058] in response to the predicted occurrence of the driving incident, transmitting a command signal to a resident vehicle system to perform a control operation.

[0059] The motor vehicle of claim 16, wherein the electronic system controller is further programmed to:

[0060] define a fixed region of interest (ROI) within a camera view of each of the cameras embedded in the cameras; and

[0061] generate cropped camera images by cropping each of the camera-generated images to remove image data outside the ROI.

[0062] The motor vehicle of claim 16, wherein processing the camera data includes a CNN backbone analyzing the cropped images to determine off-road driving characteristics, including: wheel characteristics of drive wheels of the motor vehicle; terrain characteristics of a terrain surface on which the plurality of drive wheels of the motor vehicle are located; and / or obstacle characteristics of obstacles impeding driving of the motor vehicle, wherein the occurrence of a driving incident is predicted based on the off-road driving characteristics.

[0063] The motor vehicle of claim 16, wherein the electronic system controller is further programmed to:

[0064] for each of the camera-generated images independent of one another, analyze individual frames of the ROI to assign respective characteristics to drive wheels and / or terrain surfaces contained in the camera-generated images; and

[0065] based on the ROI analysis, generate, via the CNN backbone, situation cues indicative of the characteristics and assessable via a situation classifier module operable to predict the occurrence of a driving incident.

[0066] The motor vehicle of claim 16, wherein the electronic system controller is further programmed to:

[0067] analyze, via the CNN backbone, consecutive frames of the ROI for all camera-generated images to extract features of drive wheels and / or terrain surfaces contained in the camera-generated images;

[0068] link the camera-generated images into a series or chain via a cascading module; and

[0069] extract temporal information of each of the camera-generated images via a recurrent neural network (RNN).

[0070] The above summary is not intended to represent each embodiment or every aspect of the present disclosure. Rather, the above summary merely provides some examples of the novel concepts and features set forth herein. The above summary, as well as the following detailed description of the illustrated examples and representative modes of practicing the present disclosure, will be better understood in conjunction with the accompanying drawings and the appended claims. Furthermore, the present disclosure expressly encompasses all possible combinations and sub-combinations of the elements and features presented herein above and below. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is a partial schematic side view illustration of a representative motor vehicle having a network of on-board controllers, sensing devices, and communication devices for exchanging data with an intelligent vehicle navigation system for driving incident prediction in off-road areas in accordance with aspects of the present disclosure.

[0072] Figure 2 illustrates a front, side, and downward perspective view of a motor vehicle captured by a vehicle body front, left side, right side, and vehicle body bottom camera in accordance with aspects of the present disclosure.

[0073] Figure 3 is a schematic diagram illustrating a representative intelligent vehicle navigation system providing off-road incident prediction and assistance for a motor vehicle in accordance with aspects of the disclosed concept.

[0074] Figure 4 is a flowchart illustrating a representative driving incident prediction protocol (image classifier + situation classifier) for a motor vehicle, which can correspond to memory-stored instructions executable by a resident or remote controller, control logic circuit, programmable control unit, or other integrated circuit (IC) device or network of devices in accordance with aspects of the disclosed concept.

[0075] Figure 5 is a flowchart illustrating another representative driving incident prediction protocol (end-to-end training) for a motor vehicle, which can correspond to memory-stored instructions executable by a resident or remote controller, control logic circuit, programmable control unit, or other integrated circuit (IC) device or network of devices in accordance with aspects of the disclosed concept.

[0076] The present disclosure can take form in various modifications and alternative forms, and some representative embodiments are shown in the drawings and will be described in detail herein. It should be understood, however, that the novel aspects of the present disclosure are not limited to the particular forms disclosed herein. Rather, the disclosure will cover all modifications, equivalents, combinations, sub-combinations, permutations, and alternatives falling within the scope of the present disclosure as encompassed by the appended claims. DETAILED DESCRIPTION

[0077] The present disclosure is susceptible to various modifications and alternative forms, specific embodiments of which are shown by way of example in the drawings and described in detail herein. It should be understood that the novel aspects of the present disclosure are not limited to the particular

[0078] For purposes of this detailed description, unless specifically stated otherwise, as applicable: the singular includes the plural and vice versa; the words "and" and "or" shall be both conjunctive and disjunctive; the words "any" and "all" shall mean "any and all"; and the words "comprising," "containing," "including," "having" and the like shall mean "including, but not limited to." Further, approximate language, such as "about," "substantially," "generally," "approximately," and the like, can be used herein to express uncertainty or inexactitude with respect to a particular measurement, such as when a value is not exact, but is close to an exact value. Finally, directional adjectives and adverbs, such as front, rear, inner, outer, starboard, port, vertical, horizontal, upward, downward, forward, aft, left, right, and the like, can be with respect to a motor vehicle, such as the forward driving direction of the motor vehicle when the vehicle is operably oriented on a horizontal driving surface.

[0079] Reference is now made to the drawings, wherein like reference numerals designate identical features throughout the several views, and wherein Figure 1 A representative automobile is shown in FIG. 1, which is indicated generally at 10, and is depicted herein as a passenger car for purposes of discussion, although the illustrated automobile 10 - also referred to herein simply as a "motor vehicle" or "vehicle" - is merely an exemplary application in which the novel aspects of the present disclosure can be practiced. Likewise, incorporation of the present concepts into an all-electric vehicle powertrain should also be appreciated as a non-limiting implementation of the disclosed features. As such, it should be understood that aspects and features of the present disclosure can be applied to other powertrain configurations, can be implemented for any logically relevant type of vehicle, and can be provided by other intelligent system architectures. Moreover, only selected components of the motor vehicle and vehicle control system are shown and described herein in additional detail. However, the vehicle and vehicle systems discussed below can include many additional and alternative features, as well as other available peripheral components, for performing the various methods and functions of the present disclosure.

[0080] Figure 1 The representative vehicle 10 is initially equipped with a vehicle telematics and information ("telematics") unit 14 that wirelessly communicates with a remotely-located or "off-board" cloud computing host service 24 (e.g., ONSTAR®), for example via cell towers, base stations, mobile switching centers, satellite services, etc. As a non-limiting example, Figure 1Some other vehicle hardware components 16 generally shown include an electronic video display device 18, a microphone 28, an audio speaker 30, and various user input controls 32 (e.g., buttons, knobs, pedals, switches, touchpads, joysticks, touchscreens, etc.). These hardware components 16 serve in part as a human-machine interface (HMI) to enable a user to communicate with the telematics unit 14 and other system components within the vehicle 10. The microphone 28 provides a means for a vehicle passenger to input oral or other audible commands; the vehicle 10 can be equipped with an embedded speech processing unit that utilizes audio filtering, editing, and analysis modules. Conversely, the speaker 30 provides audible output to a vehicle passenger and can be a standalone speaker dedicated for use with the telematics unit 14 or can be part of the audio system 22. The audio system 22 is operably connected to the network connection interface 34 and the audio bus 20 to receive analog information for presentation as sound via one or more speaker components.

[0081] The network connection interface 34 is communicatively coupled to the telematics unit 14, suitable examples of which include twisted pair / fiber optic Ethernet switches, parallel / serial communication buses, local area network (LAN) interfaces, controller area network (CAN) interfaces, media oriented system transfer (MOST) interfaces, local interconnect network (LIN) interfaces, etc. Other suitable communication interfaces can include those compliant with ISO, SAE, and / or IEEE standards and specifications. The network connection interface 34 enables the vehicle hardware 16 to send and receive signals with each other and with various systems and subsystems within or “resident” within the vehicle body 12 and outside or “remote” from the vehicle body 12. This allows the vehicle 10 to perform various vehicle functions, such as modulating powertrain output, managing operation of the vehicle transmission, selectively engaging friction and regenerative braking systems, controlling vehicle steering, regulating charging and discharging of the vehicle battery module, and other autonomous driving functions. For example, the telematics unit 14 receives and transmits signals and data from / to a powertrain control module (PCM) 52, an advanced driver assistance system (ADAS) module 54, an electronic battery control module (EBCM) 56, a steering control module (SCM) 58, a brake system control module (BSCM) 60, and other various vehicle ECUs, such as a transmission control module (TCM), an engine control module (ECM), a sensor system interface module (SSIM), etc.

[0082] With continued reference to Figure 1, the telematics unit 14 is an on-board computing device that provides hybrid services both independently and through its communication with other networked devices. The telematics unit 14 is generally comprised of one or more processors 40, each of which can be embodied as a discrete microprocessor, an application specific integrated circuit (ASIC), or a dedicated control module. The vehicle 10 can provide centralized vehicle control via a central processing unit (CPU) 36, which is operably coupled to a real-time clock (RTC) 42 and one or more electronic memory devices 38, each of which can take the form of a CD-ROM, a disk, an IC device, a flash memory, a semiconductor memory (e.g., various types of RAM or ROM), etc.

[0083] Long-range vehicle communication capabilities with remote off-board networked devices can be provided via one or more or all of a cellular chipset / component, a navigation and location chipset / component (e.g., a global positioning system (GPS) transceiver), or a wireless modem, all of which are collectively represented at 44. Short-range wireless connectivity can be provided via a short-range wireless communication device 46 (e.g., a BLUETOOTH® unit or a near field communication (NFC) transceiver), a dedicated short-range communication (DSRC) component 48, and / or a dual antenna 50. It should be appreciated that the vehicle 10 can be implemented without one or more of the above-listed components, or optionally, can include additional components and functionality as desired for a particular end-use. The various communication devices described above can be configured to exchange data as part of a periodic broadcast in a vehicle-to-vehicle (V2V) communication system or a vehicle-to-everything (V2X) communication system, vehicle-to-everything (V2X) such as vehicle-to-infrastructure (V2I), vehicle-to-pedestrian (V2P), vehicle-to-device (V2D), etc.

[0084] The CPU 36 receives sensor data from one or more sensing devices that use, for example, light detection, radar, laser, ultrasonic, optical, infrared, or other suitable technologies, including short-range communication technologies (e.g., DSRC) or ultra-wideband (UWB) radio technologies, for performing autonomous driving operations or vehicle navigation services. According to the illustrated example, the automobile 10 can be equipped with one or more digital video cameras 62, one or more distance sensors 64, one or more vehicle speed sensors 66, one or more vehicle dynamics sensors 68, and any necessary filtering, classification, fusion, and analysis hardware and software for processing raw sensor data. The type, placement, number, and interoperability of the distributed on-board sensor array can be individually or collectively adapted to a given vehicle platform for achieving a desired level of autonomous vehicle operation.

[0085] The digital video camera(s) 62 can use charge-coupled device (CCD) sensors or other suitable optical sensors to generate images indicative of the field of view of the vehicle 10, and can be configured for continuous image generation, e.g., at least about 35+ images per second. In comparison, the distance sensor(s) 64 can emit and detect reflected radio, infrared, light-based, or other electromagnetic signals (e.g., short-range radar, long-range radar, EM induction sensing, light detection and ranging (LIDAR), etc.) to detect, e.g., the presence, geometric dimensions, and / or proximity of target objects. The vehicle speed sensor(s) 66 can take various forms, including wheel speed sensors that measure wheel speed, which is then used to determine real-time vehicle speed. Further, the vehicle dynamics sensor(s) 68 can have the nature of single- or three-axis accelerometers, angular velocity sensors, inclinometers, etc., for detecting longitudinal and lateral acceleration, yaw, roll, and / or pitch rates, or other dynamics-related parameters. Using data from the sensing devices 62, 64, 66, 68, the CPU 36 identifies the surrounding driving conditions, determines road characteristics and surface conditions, identifies target objects within the vehicle's detectable range, determines attributes of the target objects such as size, relative positioning, orientation, distance, approach angle, relative velocity, etc., and performs automatic control maneuvers based on these performed operations.

[0086] These sensors can be distributed throughout the motor vehicle 10 at unobstructed locations operable with respect to the vehicle's front or rear or port or starboard side view. Each sensor generates an electrical signal indicative of a characteristic or condition of the host vehicle or one or more target objects, typically as an estimated value with a corresponding standard deviation. While the maneuvering characteristics of these sensors are generally complementary, some sensors are more reliable than others in estimating certain parameters. Most sensors have different maneuvering ranges and coverage areas, and are capable of detecting different parameters within their operating range. For example, radar-based sensors can estimate the distance, rate of distance change, and azimuthal position of an object, but can not be robust in estimating the extent of a detected object. On the other hand, video cameras with optical processing can be more robust in estimating the shape and azimuthal positioning of an object, but can be less efficient in estimating the distance and rate of distance change of a target object. Scanning LIDAR-based sensors can efficiently and accurately perform estimation of distance and azimuthal positioning, but can not accurately estimate rate of distance change, and thus can not be accurate for new object acquisition / identification. In contrast, ultrasonic sensors are capable of estimating distance, but generally cannot accurately estimate rate of distance change and azimuthal positioning. Further, the performance of many sensor technologies can be affected by different environmental conditions. Thus, the sensors generally exhibit parameter variation, the operation of which overlaps to provide an opportunity for sensory fusion.

[0087] The sensor fusion control module can execute a fusion algorithm in conjunction with calibration information stored by the associated memory to receive sensor data from available sensors, cluster the data into available estimates and measurements, and fuse the clustered observations to determine, for example, lane geometry and relative target positioning estimates. The fusion algorithm can utilize any suitable sensor fusion method, such as a Kalman Filter (KF) fusion application. The KF application can be used to explore temporal correlations of each target (e.g., assuming that a tracked target moves smoothly over a predefined period of time). Likewise, the KF application can capture spatial correlations, i.e., the relative positioning of each target object observed by multiple sensors with respect to the host vehicle. Additional information regarding sensor data fusion can be found in U.S. Patent No. 7,460,951 to Osman D. Altan et al., the entirety of which is incorporated herein by reference for all purposes.

[0088] To propel the electrically driven vehicle 10, the electrified powertrain is operable to generate and deliver tractive torque to one or more road wheels 26 of the vehicle. In Figure 1 In general, the powertrain is represented by a rechargeable energy storage system (RESS), which can have the nature of a chassis-mounted traction battery pack 70, which is operably connected to an electric traction motor 78. The traction battery pack 70 is generally composed of one or more battery modules 72, each having a stack of battery cells 74, such as lithium-ion, lithium-polymer, or pouch, can, or prismatic type nickel-metal hydride battery cells. One or more electric machines, such as traction motor / generator (M) units 78, draw power from, and optionally deliver power to, the battery pack 70 of the RESS. A dedicated power inverter module (PIM) 80 electrically connects the battery pack 70 to the motor / generator (M) unit(s) 78 and modulates the current transfer therebetween. The disclosed concepts are similarly applicable to HEV and ICE based powertrain architectures.

[0089] The battery pack 70 can be configured such that module management, battery cell sensing, and module-to-module or module-to-host communication functions are directly integrated into each battery module 72 and performed wirelessly via a wireless-enabled cell monitoring unit (CMU) 76. The CMU 76 can be a microcontroller-based sensor array mounted on a printed circuit board (PCB). Each CMU 76 can have GPS transceiver and RF capabilities and can be packaged on or in the battery module housing. The battery module cell 74, CMU 76, housing, coolant lines, busbars, etc., collectively define a battery cell module assembly.

[0090] As noted above, Figure 1The vehicle 10 can be equipped with an on-board sensor array that continuously outputs instantaneous, "real-time" sensor data for use in performing assisted or automated vehicle operations. By way of non-limiting example, Figure 1 The digital camera(s) 62 can be embodied as a distributed network of digital cameras 102 mounted about the periphery of the vehicle body 12 (also referred to herein as a "camera sensor system" or "CSS"). According to the illustrated example, the camera network 102 is depicted as having a front (first) camera 120 mounted near the front end of the vehicle body 12 (e.g., on the front grille cover), and a rear (second) camera mounted near the rear end of the vehicle body (e.g., on the rear liftgate, tailgate, or trunk lid). The vehicle camera sensor system 102 can also employ a port (third) camera 122 mounted near the left lateral side of the vehicle body 12 (e.g., integrated into the driver's side rearview mirror assembly), and a starboard (fourth) camera 124 mounted near the right lateral side of the vehicle body 12 (e.g., integrated into the passenger's side rearview mirror assembly). A vehicle underbody (fifth) camera 126 is mounted near the undercarriage of the vehicle body 12 (e.g., on the undercarriage side rails or cross members). Figure 2 The cameras 102 can be composed of any number, type, and arrangement of image capture devices, each of which can be fabricated with a complementary metal-oxide-semiconductor (CMOS) sensor, a charge-coupled device (CCD) sensor, or other suitable active-pixel sensor (APS). The type, location, placement, and interoperation of the distributed on-board sensor array can be individually or collectively adapted to a given vehicle platform to achieve a desired level of target object acquisition accuracy, vehicle underbody hazard detection, or autonomous vehicle operation.

[0091] With continued reference to the example set forth in Figure 2 According to the example set forth in

[0092] Defined within each respective camera field of view of the cameras 120, 122, 124, 126 is a fixed region of interest (ROI) that is used to improve underbody hazard detection and driving incident prediction. The ROI can be characterized as a depicted camera frame region that is embedded within one of the vehicle's camera-generated views and is fixed at a predefined location within that camera view. By way of example and not limitation, the front (first) ROI 121 is embedded within the forward vehicle view captured by the front camera 120, while the left side (second) ROI 123 is embedded within the left-side facing vehicle view captured by the port side camera 122. In contrast, the right side (third) ROI 125 is embedded within the right-side facing vehicle view captured by the starboard side camera 124, while the underbody (fourth) ROI 127 is embedded within the downward facing vehicle view captured by the underbody camera 126. These ROIs can be defined as vehicle calibration features that are specific to the subject host vehicle. For example, the ROIs can be based on the vehicle's make / model / trim, the respective mounting locations of the cameras, the view area / angle of each camera, and / or a predefined safety zone around the vehicle (e.g., a geo-fence that is 20 cm from all exterior surfaces). The ROIs can be "defined" by retrieving parameters from a lookup table stored in resident memory, by calculating parameters during an initial vehicle calibration procedure, by model-based technology estimates that are available, or by any other suitable determination procedure.

[0093] Figure 3 Presented in FIG. 1 is a schematic diagram of an exemplary intelligent vehicle navigation (IVN) system 200 that is used to provide navigation and emergency services for a distributed network of vehicles, among other features. While a single cloud computing host service 224 system is illustrated in communication with a single motor vehicle 210 and a single remote assistant 250, it is contemplated that any number of host computing services (e.g., cloud, edge, distributed, serverless, etc.) can be in communication with any number of vehicles and any number of third-party entities and their associated computing nodes that are suitably equipped for wireless exchange of data. Although differing in appearance, it is contemplated that any of the features and options described above with reference to the automobile 10 and the host service 24 of Figure 1 may be incorporated into the host vehicle 210 and the cloud computing host service 224 of Figure 3 respectively, either individually or in any combination, and vice versa.

[0094] Figure 3 The cloud computing host service 224 of FIG. 1 is communicatively connected to each motor vehicle 210 and each third-party entity 250 via a wireless communication network 252 (e.g., as described above with reference to the host service 24 of FIG. 1). The cloud computing host service 224 is configured to provide a variety of services to the motor vehicles 210 and the third-party entities 250, including, but not limited to, navigation services, emergency services, and other services. Figure 1The network 252 can be any available type of network, including a combination of public distributed computing networks (e.g., the Internet) and secure private networks (e.g., local area networks, wide area networks, virtual private networks, etc.). It can also include wireless and wired transmission systems (e.g., satellite, cellular tower networks, terrestrial networks, etc.). Wireless data exchange between the vehicle 210 and the host service 224 can be conducted directly, in configurations where the vehicle 210 is equipped as a standalone wireless device, or indirectly by pairing and piggybacking the vehicle 210 onto a wireless-enabled device such as a smartphone, smartwatch, handheld GPS transceiver, laptop, etc. It is also contemplated that the host service 224 can communicate directly with a personal computing device of the driver or passenger of the vehicle 210, and thus forego or supplement direct communication with the vehicle 210. As a further option, many of the services provided by the host service 224 can be loaded onto the motor vehicle 210, and vice versa.

[0095] With continued reference to Figure 3 The cloud computing host service 224 system can be implemented by high-speed, server-class computing devices 254 or mainframe computers capable of handling large amounts of data processing, resource planning, and transaction processing. For example, the host service 224 can operate as a host in a client-server interface for any necessary data exchange and communication with one or more “third-party” servers or devices to complete a particular transaction. Alternatively, the cloud computing host service 224 can operate as middleware for IoT (Internet of Things), WoT (Web of Things), vehicle-to-everything (V2X), and / or M2M (machine-to-machine) services, e.g., connecting various devices with a service-oriented architecture (SOA). As one example, the host service 224 can be implemented as a middleware node to provide different functionalities for dynamically loading devices, multiplexing data from each device, and routing data through reconfigurable processing logic for processing and transmission to one or more destination applications.

[0096] The IVN system 200 provides driving incident prediction and assistance for one or more vehicles 210 in off-road driving scenarios. Prediction of the occurrence of off-road driving incidents is performed through an AI-driven model learning process that is self-supervised and trained with data from vehicles that participate in driving across off-road terrain. The model can be executed by the host server 254 or a resident vehicle control module, as discussed above with respect to the remote information processing unit 14. The network 252 can be any available type of network, including a combination of public distributed computing networks (e.g., the Internet) and secure private networks (e.g., local area networks, wide area networks, virtual private networks, etc.). It can also include wireless and wired transmission systems (e.g., satellite, cellular tower networks, terrestrial networks, etc.). Wireless data exchange between the vehicle 210 and the host service 224 can be conducted directly, in configurations where the vehicle 210 is equipped as a standalone wireless device, or indirectly by pairing and piggybacking the vehicle 210 onto a wireless-enabled device such as a smartphone, smartwatch, handheld GPS transceiver, laptop, etc. It is also contemplated that the host service 224 can communicate directly with a personal computing device of the driver or passenger of the vehicle 210, and thus forego or supplement direct communication with the vehicle 210. As a further option, many of the services provided by the host service 224 can be loaded onto the motor vehicle 210, and vice versa. Figure 3The new classifier 258 in the model is represented in FIG. 2 - and the background database (DB) 256 aggregates and annotates data from customer vehicles to continually improve / train the model. During off-road driving, the host vehicle can record data once every N seconds and transmit the data to the central database if no accident occurred (non-urgent samples). Conversely, if a distress call is made to an emergency number through the vehicle infotainment system, the vehicle can record accident data and transmit the data to the central database (urgent samples). Likewise, if a call is made through the vehicle’s infotainment system and the words “stuck” or “help” are spoken (or any words from a predefined dictionary that imply a driving accident), the vehicle can record accident data and transmit the data to the central database. In Figure 3 FIG. 2, the non-urgent samples and urgent samples, along with any corresponding behavior data, are collectively represented by the end off-road trip data 201.

[0097] The model learns to predict driving accidents, including both emergency and non- emergency events, before such accidents occur and helps provide ameliorating actions to avoid the accident within a predicted timeframe. As explained in the discussion of the ML techniques outlined below in Figure 4 and 5 , the model inputs can include: (1) serial (CAN) data: throttle, braking force, parking brake, vehicle dynamics (roll, pitch, yaw, lateral / longitudinal velocity, lateral / longitudinal acceleration), wheel speed, vehicle load, suspension height, steering angle, etc.; (2) camera data: lateral, longitudinal, and underbody image frames, 360° view, etc.; (3) driver history data: driving performance in off-road scenarios, propensity to violate off-road envelope capabilities, etc.; (4) ranging sensor data: RADAR, LiDAR, and / or ultrasonic sensor data; and (5) other data: telemetry, weather, topography maps, location-based history (likelihood of accidents in related coordinates), etc. In Figure 3 FIG. 2, these inputs can be collectively represented as downloaded trip data 203 (e.g., downloaded to the vehicle prior to an off-road trip), online trip data 205 (e.g., collected during an off-road trip), DB data retrieved from the host service database 256, and vehicle driving history data collected by the driver behavior module 260. From these inputs, a deep learning neural network within the model can infer: (1) terrain surface type: sand, mud, asphalt, rock, water, etc.; (2) terrain surface condition: wet, snowy, dry, icy, loose / tight, etc.; (3) tire state: tire stuck causing the chassis to contact the ground, tire lifted off the ground, etc.; and (4) tire condition: clogged with mud or other obstructions, low tire tread, low air pressure, loss of other normal operating characteristics, etc. It is contemplated that more, fewer, or alternative inputs and outputs can be incorporated into the model learning process of FIG. 2. Figures 3-5 ​

[0098] After the ML-based off-road incident predictor analyzes the input data and infers the data, it can output a driving incident score indicative of a likelihood of a driving incident. For incident prediction, a binary model can output only a yes or no response, such as a system alert indicating a future emergency or a memory flag indicating no emergency. When a driving incident is predicted to occur, the driver can be prompted to take a series of mitigation actions to avoid the predicted incident from occurring. If the system has learned these mitigation actions (decision block 207 = yes), the mitigation steps can be retrieved from resident memory and presented to the driver via the HMI 262 or automatically executed by the on-board ADAS or ADC modules. If the actions have not yet been learned by the system (decision block 207 = no), the mitigation steps can be provided by the host service 224 or the remote assistant 250. Alternatively, a remotely located third-party intervener, whether a virtual assistant or a human assistant, can take temporary control of the operation of the vehicle, or can provide computer-executable instructions to the vehicle to take assisted or autonomous ameliorating actions.

[0099] To facilitate expert remote assistance, the remote assistant 250 can be provided with telemetry data, camera data, terrain data (e.g., a 3D mesh of the surface beneath the host vehicle 210), etc., which is uploaded to the remote assistant 250 via the uplink data packets 209. The surface terrain can be characterized as a map representation specifying the relative distribution and three-dimensional (3D) characteristics of natural and man-made features of a surface area. Such terrain data can be retrieved from resident or remote memory for areas that have been previously mapped, or for areas that have not been previously mapped, can be generated in real-time. By way of example and not limitation, a simultaneous localization and mapping (SLAM) computational application can be used to construct a 3D map (e.g., a topological model, a digital elevation model, or a mesh model) of an environment as well as geo-location information using computational geometry algorithms and computer vision data in a tractable amount of time. The camera data provided to the remote assistant 250 can include single frames, successive frames, spherical camera views, or any other camera data described herein. For example, for a host vehicle 10 ( Figure 2 Figure 1 ) having a multi-device camera sensor system 102 (

[0100] Referring next to the flowcharts of Figure 4 and Figure 5 , the use of a distributed sensor array, such as the multi-device camera sensor system 102, is generally described at 300 and 400, in accordance with aspects of the present disclosure. Figure 2 ​CSS 102) and intelligent navigation systems (such as Figure 3 improved methods or control strategies for ML-based driving incident detection for host vehicles (e.g. Figure 1 vehicles 10) of the type illustrated in Figure 4 and Figure 5 Some or all of the operations illustrated in and described in further detail below can represent algorithms corresponding to processor-executable instructions stored in, for example, a primary or secondary or remote memory (e.g., memory device 38 of Figure 1 and / or database 256 of the type illustrated in Figure 3 and executed, for example, by an electronic controller, processing unit, logic circuit, or other module or device or network of modules / devices (e.g., CPU 36 and / or cloud computing service 224) to perform any one or all of the functions associated with the disclosed concepts described above and below. It will be recognized that the order of execution of the illustrated blocks of operations can be changed, that additional blocks of operations can be added, and that some of the described operations can be modified, combined or eliminated.

[0101] Methods 300 and 400 begin, respectively, at start terminal blocks 301 and 401, with memory-stored processor-executable instructions for a programmable controller or control module or similar suitable processor to invoke an initializer routine for an off-road emergency prediction protocol. This routine can be executed in real-time, near real-time, continuously, systematically, occasionally, and / or at regular intervals, for example, every 10 or 100 milliseconds during normal and ongoing operation of the motor vehicle 10. As yet another option, terminal block 301 can be initialized in response to a user command prompt, a resident vehicle controller prompt, or a broadcast prompt signal received from a “non-vehicle-mounted” central vehicle service system (e.g., host cloud computing service 24). For example, methods 300, 400 can be automatically initialized in response to an electronic system controller receiving vehicle geolocation data indicating that the subject host vehicle is about to enter or is currently traversing a wildland terrain. Upon completion of the control operations presented in Figure 4 and Figure 5 Methods 300, 400 can proceed, respectively, to end terminal blocks 315 and 415 and terminate temporarily, or optionally, can loop back to terminal blocks 301, 401 and operate in a continuous loop fashion, upon completion of the control operations presented in

[0102] Figure 4 Method 300 of the type illustrated in Figure 1 and Figure 2For the example of a vehicle 10, the vehicle CPU 36 can prompt the CSS 102 to begin polling the cameras 120, 122, 124, 126 to capture real-time image data of their respective fields of view and transmit signals indicative of said data. Each camera image can contain one or more of a view of the vehicle's drive wheels (e.g., the road wheels 26), a surface view of the terrain below, a perspective view of an obstacle target object (e.g., boulder, log, mud pit, etc.), and / or any other desired content.

[0103] After any necessary sensor data is collected at the sensor data input block 303, the method 300 executes a computer automated image cropping procedure 305, for example, to eliminate any extraneous or unimportant subject matter from the camera generated images. As noted above with respect to the discussion of Figure 2 As noted above with respect to the discussion of Figure 4 Two representative cropped camera images are illustrated: a driver side front drive wheel and contact surface image 305a, and a passenger side front drive wheel and contact surface image 305b. An uncropped underbody camera image 305c of the front axle, front drive wheels, and terrain surface below is also evaluated.

[0104] The method 300 proceeds from the image cropping procedure 305 to an image classifier module 307, which implements a convolutional neural network (CNN) backbone to analyze the vehicle sensor data and infer therefrom one or more situational characteristics of the off-road driving scenario of the host vehicle. As used herein, the term "backbone" can be defined to include a pre-trained deep learning neural network that is responsible for taking a selected image data as input and extracting a feature map therefrom, upon which the rest of the network is based. For the computer vision DeepLab model, the feature extractor network computes pre-defined features from the input image data and then "upsamples" these features through a simple decoder module to generate a segmented mask for classifying the image. The image classifier module 307 can employ a single common backbone (e.g., a Resnet CNN backbone) and multiple task heads to perform the image classification task. Implementing multiple outputs in a single backbone network helps the ML training process to focus on features that contain identifiable "strong visual" information, rather than simply overfitting the model to the training dataset. The architecture of the network can consist of a CNN backbone that extracts general features and CNN heads that perform classification tasks (e.g., each head classifies a different class). The CNN backbone can contain a set of layers at the beginning of the network; the architecture of the backbone can be built as an open question that performs well for the image classification task. The camera images can be analyzed independently for each round of ROIs and the entire vehicle body frame (e.g., the image classifier can take a single frame as input and use the CNN backbone and multiple task heads to characterize the situation of that image); the resulting data is then passed as a clue to the situation classification.

[0105] With continued reference to Figure 4 , the image classifier module 307 outputs situation clues 309 to a situation classifier module 311, which is trained to predict whether a driving incident will occur within a predicted time horizon. As above in Figure 3As explained in the discussion of method 300, image classifier module 307 implements a CNN backbone to infer one or more drive wheel characteristics (e.g., wheel lift off the ground head 309b, wheel steering angle head 309e, etc.) of the drive wheel(s) of the host vehicle, one or more terrain characteristics (e.g., terrain type head 309a, mud / water height relative to wheels head 309d) of the off-road terrain being evaluated, and / or one or more obstacle characteristics (e.g., obstacle height relative to wheels head 309c) of the obstacle impeding movement of the host vehicle. Other inputs to the situation classifier module 311 for the drive incident prediction case can include other vehicle sensor data 309f and serial (CAN) data 309g. At command prompt 313, method 300 outputs a future emergency / non-emergency determination 313a, and for a predicted drive incident, one or more "best" actions 313b (e.g., steering angle, gas pedal input, brake pedal input, etc.) to help prevent the drive incident.

[0106] With Figure 4 In contrast to the illustrated image classifier + situation classifier ML technique, method 400 can represent an end-to-end training ML technique that trains an ML-based NN system using, for example, consecutive frames of all available data. In particular, a vehicle camera can produce a temporal data stream; this temporal data includes consecutive image frames and consecutive serial (CAN) data. Each camera frame from all available cameras can be passed through a convolutional neural network as a feature extraction backbone. Using a recurrent neural network (RNN) 411, the system maintains and updates state information while sequentially accepting new frames and serial data. Similar to Figure 4 Method 300 of Figure 5 Method 400 employs a CNN backbone to extract features from camera images; an RNN receives and analyzes this data set for incident prediction. However, in this example, training "full system end-to-end" means that the CNN backbone is not pre-trained on a different task and is frozen; rather, the training set and stochastic gradient descent are used to train the CNN backbone 407 and RNN 411 together.

[0107] After the off-road emergency prediction protocol is initialized at start terminal block 401, the method 400 executes a feature extraction module 405 whereby sensor data is first received from the vehicle CSS 102 at sensor data input block 403 (e.g., as described above with respect to sensor data input block 303). The received camera sensor data is passed through the CNN backbone 407 on a frame-by-frame basis. Since each image can be considered to be "very high dimensional" (e.g., a 3 megapixel image has 3 million values), a portion of which can contain "meaningful information", the classifier attempts to isolate only the meaningful information while the rest is ignored as "noise". The CNN backbone 407 uses a self-learning neural network to extract information features from each image frame.

[0108] The method 400 proceeds to a concatenation module 409 to link the camera generated images into a series or chain so that the RNN 411 can extract temporal information for each camera generated image. The output of the CNN backbone 407 from the images from the CSS 102 can be represented by camera j with an associated vector of size N [cj1, cj2, cj3,..., cjN] while the serial CAN data is a vector of size M [s1, a2, s3,..., sM]. During concatenation, the module 409 merges the vectors into a single long vector. If we have K cameras, then the output of the concatenation is a single vector of size (K*N) + M in the form [c11, c12,..., c1N, c21, c22,..., c2N,..., cK1, cK2,..., cKN, s1, s2,..., sM]. At command prompt block 413, the method 400 outputs a future emergency indicator or no emergency detected indicator at block 413a. If the system predicts that a driving incident will occur within a predicted time frame (e.g., the next 5-8 seconds), the method 300 can determine one or more "best" actions 413b for helping to prevent the driving incident.

[0109] In some embodiments, aspects of the present disclosure can be implemented by way of computer executable instructions, such as program modules, generally referred to as software applications or application programs, executed by any of the controllers or controller variants described herein. In non-limiting examples, the software can include routines, programs, objects, components, and data structures that perform particular tasks or implement particular data types. The software can form an interface to allow a computer to react to a source of input. The software can also cooperate with other code segments to initiate a variety of tasks in response to data received in conjunction with the source of the received data. The software can be stored on any of a variety of memory media, such as CD-ROM, diskette, and semiconductor memory (e.g., various types of RAM or ROM).

[0110] Moreover, aspects of the disclosure can be practiced with various computer system and computer network configurations, including multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. Additionally, aspects of the disclosure can be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices. As such, aspects of the disclosure can be implemented in a computer system or other processing system that includes a variety of hardware, software, or combinations thereof.

[0111] Any of the methods described herein can include machine readable instructions for execution by (a) a processor, (b) a controller, and / or (c) any other suitable processing device. Any of the algorithms, software, control logic, protocols, or methods disclosed herein can be embodied in software stored on a tangible media such as, for example, flash memory, solid state memory (SSD), hard disk drive (HDD) memory, CD-ROM, digital versatile disk (DVD), or other memory devices. The entire algorithm, control logic, protocol, or method and / or portions thereof can alternatively be embodied in devices other than controllers and / or in firmware or hard-wired logic (e.g., by an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable logic device (FPLD), discrete logic, etc.). Moreover, although specific algorithms can be described herein with reference to flow and / or work flow diagrams, many other methods of implementing the example machine readable instructions can be used.

[0112] Aspects of the disclosure have been described in detail with reference to the illustrated embodiments; however, it will be appreciated that various modifications can be made to the disclosure without departing from its scope. The disclosure is not limited to the precise construction and composition disclosed herein; any and all modifications, changes, and variations that are evident from the foregoing description and that are within the scope of the disclosure are included in the disclosure. Moreover, the disclosure expressly includes any and all combinations and subcombinations of the foregoing elements and features.

Claims

1. A method for controlling operation of a motor vehicle with a sensor array comprising a network of cameras mounted at discrete locations on the motor vehicle, the method comprising: receiving, via an electronic system controller, vehicle geo-location data indicating that the motor vehicle is at or entering an off-road terrain via a wireless communication device; in response to the vehicle geo-location data, receiving, via the system controller, camera data from the sensor array, the camera data indicating camera-generated images captured by the cameras and including a drive wheel of the motor vehicle and a terrain surface of the off-road terrain; determining a region of interest (ROI) embedded and fixed in a predefined location of a camera view of each of the cameras; generating cropped camera images by cropping each of the camera-generated images to remove image data outside the region of interest; receiving, via the system controller, vehicle operating characteristic data and vehicle dynamics data of the motor vehicle from a controller area network (CAN) bus; processing the camera data, the vehicle operating characteristic data, and the vehicle dynamics data via a convolutional neural network (CNN) backbone to predict an occurrence of a driving incident on the off-road terrain within a prediction time horizon, wherein processing the camera data includes the CNN backbone analyzing the cropped camera images to determine wheel characteristics of the drive wheel of the motor vehicle and terrain characteristics of the terrain surface, wherein predicting the occurrence of the driving incident is based on the wheel characteristics of the drive wheel and the terrain characteristics of the terrain surface; and in response to the predicted occurrence of the driving incident, transmitting, via the system controller, a command signal to a resident vehicle system to perform a control operation.

2. The method of claim 1, wherein the wheel characteristics include a loss of ground contact state, a stop in an obstruction state, and / or a loss of normal operating feature state.

3. The method of claim 1, wherein the terrain characteristics include a terrain type and / or a terrain condition, wherein the terrain type includes sand, mud, asphalt, rock, or the like, and the terrain condition includes wet, dry, snowing, icy, or loose / tight.

4. The method of claim 1, wherein processing the camera data includes the CNN backbone analyzing the cropped images to determine obstruction characteristics of an obstruction obstructing driving of the motor vehicle, wherein predicting the occurrence of the driving incident is based on the obstruction characteristics of the obstruction.

5. The method of claim 4, wherein the obstruction characteristics include an obstruction height relative to a wheel height of the drive wheel and / or a body height of a body of the motor vehicle.

6. The method of claim 1, wherein processing camera data includes: for each of the camera-generated images independent of one another, analyzing individual frames of the region of interest to assign characteristics to the drive wheel and / or the terrain surface contained in the camera-generated images; and ​ ​ Based on the region of interest analysis, image analysis via a convolutional neural network backbone using multiple task heads generates indicative characteristics and assessable situational cues via a situational classifier module operable to predict the occurrence of a driving incident.

7. The method of claim 1, wherein processing camera data comprises: generating a continuous frame of a region of interest via a convolutional neural network backbone for all camera generated images to extract features of a drive wheel and / or a terrain surface contained in the camera generated images; linking the camera generated images into a series or chain via a cascading module; and extracting temporal information for each camera generated image via a recurrent neural network (RNN).

8. The method of claim 1, wherein the network of cameras comprises a vehicle body bottom camera mounted to a motor vehicle proximate to its chassis, the vehicle body bottom camera operable to capture an outwardly facing downward view from the vehicle body and generate a signal indicative of the view.

9. The method of claim 1, wherein the vehicle dynamics data comprises vehicle roll data, vehicle pitch data, vehicle yaw data, vehicle lateral / longitudinal speed data, vehicle lateral / longitudinal acceleration data, wheel speed data, and / or steering angle data, and wherein the vehicle operating characteristic data comprises throttle position data, brake force data, park brake status, powertrain mode data, and / or suspension height data.

10. The method of claim 1, wherein the resident vehicle system comprises an autonomous driving control module operable to autonomously drive the motor vehicle, the control operation comprising automating driving maneuvers of the motor vehicle based on the predicted occurrence of the driving incident.

11. The method of claim 1, wherein the resident vehicle system comprises a vehicle navigation system having a display device, the control operation comprising displaying an alert having a remedial driving maneuver via the display device to prevent the occurrence of the driving incident.

12. A non-transitory computer readable medium storing instructions executable by a system controller operable to control operation of a motor vehicle, the motor vehicle comprising a sensor array having a network of cameras mounted at discrete locations on the motor vehicle, the instructions when executed cause the system controller to perform operations comprising: receiving, via a wireless communication device, vehicle geolocation data indicative of the motor vehicle being on or entering an off-road terrain; receiving, from the sensor array, camera data indicative of camera generated images captured by the cameras, and each camera generated image containing a drive wheel of the motor vehicle and / or a terrain surface of the off-road terrain; determining a region of interest (ROI) embedded and fixed at a predefined location of a camera view of each of the cameras; generating cropped camera images by cropping each of the camera generated images to remove image data outside the region of interest; receiving, from a controller area network (CAN) bus of the motor vehicle, vehicle operating characteristic data and vehicle dynamics data of the motor vehicle; processing the camera data, the vehicle operating characteristic data, and the vehicle dynamics data via a convolutional neural network (CNN) backbone to predict an occurrence of a driving incident on the off-road terrain within a prediction time horizon, wherein processing the camera data includes the convolutional neural network backbone analyzing the cropped camera images to determine wheel characteristics of drive wheels of the motor vehicle and terrain characteristics of a terrain surface of the off-road terrain, wherein predicting the occurrence of the driving incident is based on the wheel characteristics of the drive wheels and the terrain characteristics of the terrain surface; and and in response to the predicted occurrence of the driving incident, transmitting a command signal to a resident vehicle system to perform a control operation.

13. A motor vehicle, comprising: a vehicle body; a plurality of drive wheels mounted to the vehicle body; a prime mover mounted to the vehicle body and operable to drive one or more of the drive wheels to propel the motor vehicle; a sensor array including a network of cameras mounted at discrete locations on the vehicle body; and an electronic system controller programmed to: receive, via a wireless communication device, vehicle geolocation data indicative of the motor vehicle being on or entering an off-road terrain; in response to the vehicle geolocation data, receive, from the sensor array, camera data indicative of camera-generated images captured by the cameras, and each camera-generated image containing one or more of a drive wheel and / or a terrain surface of the off-road terrain; define a fixed region of interest (ROI) within a camera view of each of the cameras embedded in the cameras; and generate cropped camera images by cropping each of the camera-generated images to remove image data outside the region of interest; receive, from a controller area network (CAN) bus, vehicle operating characteristic data and vehicle dynamics data of the motor vehicle; process the camera data, the vehicle operating characteristic data, and the vehicle dynamics data via a convolutional neural network (CNN) backbone to predict an occurrence of a driving incident on the off-road terrain within a prediction time horizon, wherein processing the camera data includes the convolutional neural network backbone analyzing the cropped camera images to determine off-road driving characteristics, including: wheel characteristics of drive wheels of the motor vehicle; terrain characteristics of a terrain surface on which the plurality of drive wheels of the motor vehicle are located; and / or obstacle characteristics of an obstacle impeding driving of the motor vehicle, wherein predicting the occurrence of the driving incident is based on the off-road driving characteristics; and in response to the predicted occurrence of the driving incident, transmit a command signal to a resident vehicle system to perform a control operation.

14. The motor vehicle of claim 13, wherein the electronic system controller is further programmed to: for each of the camera-generated images independently of one another, analyze individual frames of the region of interest to assign respective characteristics to the drive wheels and / or the terrain surface contained in the camera-generated images; and based on the region of interest analysis, generate, via the convolutional neural network backbone using a plurality of task heads, situation cues indicative of the characteristics and assessable via a situation classifier module operable to predict the occurrence of the driving incident.

15. A method of operating a motor vehicle, comprising: receiving, via a wireless communication device, vehicle geolocation data indicative of the motor vehicle being on or entering an off-road terrain; in response to the vehicle geolocation data, receiving, from a sensor array, camera data indicative of camera-generated images captured by a network of cameras mounted at discrete locations on the vehicle body, and each camera-generated image containing one or more of a drive wheel and / or a terrain surface of the off-road terrain; defining a fixed region of interest (ROI) within a camera view of each of the cameras embedded in the cameras; and generating cropped camera images by cropping each of the camera-generated images to remove image data outside the region of interest; receiving, from a controller area network (CAN) bus, vehicle operating characteristic data and vehicle dynamics data of the motor vehicle; processing the camera data, the vehicle operating characteristic data, and the vehicle dynamics data via a convolutional neural network (CNN) backbone to predict an occurrence of a driving incident on the off-road terrain within a prediction time horizon, wherein processing the camera data includes the convolutional neural network backbone analyzing the cropped camera images to determine off-road driving characteristics, including: wheel characteristics of drive wheels of the motor vehicle; terrain characteristics of a terrain surface on which the plurality of drive wheels of the motor vehicle are located; and / or obstacle characteristics of an obstacle impeding driving of the motor vehicle, wherein predicting the occurrence of the driving incident is based on the off-road driving characteristics; and in response to the predicted occurrence of the driving incident, transmitting a command signal to a resident vehicle system to perform a control operation.

15. The motor vehicle of claim 13, wherein the electronic system controller is further programmed to: analyze successive frames of regions of interest via a convolutional neural network backbone for all camera-generated images to extract features of drive wheels and / or terrain surfaces contained in the camera-generated images; link the camera-generated images into series or chains via a cascading module; and extract temporal information for each of the camera-generated images via a recurrent neural network (RNN).

Images