Method and system for improving user alertness in an autonomous vehicle

Abstract

A portable electronic monitoring device that provides an in-vehicle user alerting system regarding how a semi-autonomous vehicle is being automatically driven during a driving period, the device being removably and stably mountable to the vehicle and including: a sensor set including at least one sensor that senses an external environment outside the vehicle and movement of the vehicle within the external environment, an interface that receives user input commands and delivers alert outputs, a processor operably connected to the sensor set and the interface, the interface that receives user input commands and delivers alert outputs, and the processor operably connected to the sensor set and the interface, wherein the sensor set is configured to monitor the automatic operation of the semi-autonomous vehicle during a driving period within the external environment and to generate sensor data indicative of driving events regarding the automated driving behavior of the vehicle relative to the external environment occurring during the driving period. The processor is configured to process sensor data during the driving period, compare the detected autonomous driving behavior of the vehicle in the external environment with a model of expected autonomous vehicle driving behavior in a particular driving event, and if the detected autonomous driving behavior deviates from the expected autonomous vehicle driving behavior by more than a threshold or identify a dangerous driving event, and if a dangerous driving event is detected, issue a warning alert through an interface to alert the driver to the occurrence of the dangerous driving event.

Description

[Technical Field]

[0001] The present invention relates to methods and systems for improving and maintaining user alertness in automated vehicles. More particularly, the present disclosure relates to automated vehicle operation monitoring and alerting users to potential hazards, threats, dangers, or situations that may require manual intervention. The present invention is particularly applicable to Level 2 and Level 3 automated vehicles, but may be used with any manually or automated vehicle. [Background technology]

[0002] The levels of vehicle autonomy are defined by the Society of Automotive Engineers (SAE) International and range from Level 0, where the vehicle is not autonomous and manual control is required at all times, to Level 5, where full autonomy is achieved and no manual assistance is required.

[0003] The transition from manually operated vehicles to autonomous vehicles begins with small, incremental changes that achieve Level 1 autonomy. Cruise control is a form of vehicle automation developed in the 1940s and 1950s that maintains the vehicle's speed at a desired level. Cruise control does not meet the requirements for Level 1 autonomy, but more recently, adaptive cruise control has emerged, where the vehicle automatically adapts its speed to match the rest of the vehicle. Subsequent developments, such as parking assistance and lane assistance, which assist the driver (hereafter referred to as the user) in routine tasks, are 21st-century developments, but Level 1 vehicles still require constant user awareness.

[0004] Level 2 in the SAE classification introduces partial automation in certain conditions. Tesla's Autopilot system is an example of a Level 2 system. The Autopilot system operates the vehicle in certain circumstances by regulating the vehicle's steering and speed. While Autopilot mimics a fully autonomous system, its functionality is limited outside of the specific circumstances for which it was designed. Therefore, the user is still required to pay attention while operating the vehicle.

[0005] Unfortunately, the illusion of full autonomy that such systems provide can be dangerous if the user does not understand the system's limitations. There have been high-profile incidents where vehicles were unable to revert to manual control, continued to drive autonomously, and eventually malfunctioned or operated, causing serious injury to the user, vehicle passengers, or other road users.

[0006] Some vehicles have already achieved Level 3 autonomous driving, and the first Level 4 vehicles are expected to hit the market within the next few years. While these systems are more capable in a wider range of situations and circumstances, manual control is still required at certain times. If the vehicle cannot revert to manual control, the user may be put at risk.

[0007] Furthermore, although semi-autonomous vehicles are expected to significantly reduce road traffic incidents, the variety of potential hazards to which vehicles are exposed on the roads is enormous. Until Level 5 autonomy is achieved, vehicle systems are not expected to be flawless. Situations in which a vehicle is operated comfortably may cause unexpected problems, for example, if a minor malfunction occurs in a sensor.

[0008] More generally, it is important to keep in mind the context of automated vehicles in society. They are primarily a means of transportation and need to be accepted as such. Acceptance depends on all road users, including vehicle users, drivers, passengers, other vehicle users / operators, and pedestrians, trusting that automated vehicles are safe. Therefore, increasing trust in automated vehicles, whatever they may be, is required. Summary of the Invention

[0009] One aspect of the invention provides a portable electronic monitoring device that provides a user alert system within a vehicle regarding how a semi-autonomous vehicle is being automatically driven during a driving period. The device is removably and stably mountable to the vehicle. The device includes a sensor set including at least one sensor that senses an external environment outside the vehicle and the vehicle's movement within the external environment. The device includes an interface that receives user input and delivers output. The device includes a processor operably connected to the sensor set and the interface. The sensor set is configured to monitor the automatic operation of the semi-autonomous vehicle within the external environment during the driving period and generate sensor data indicative of driving events regarding the vehicle's automated driving behavior relative to the external environment occurring during the driving period. The processor is configured to process the sensor data during the driving period to compare the detected automated driving behavior of the vehicle in the external environment with a model of expected automated vehicle driving behavior for a particular driving event, and to identify when the detected automated driving behavior deviates from the expected automated vehicle driving behavior by more than a threshold or identify a dangerous driving event. If a dangerous driving event is detected, the processor is configured to generate a warning alert via the interface to alert the driver of the occurrence of the dangerous driving event.

[0010] Providing the above-described device is beneficial in that it allows the vehicle driver or user to be assured that the vehicle is operating correctly when in autonomous mode. The device can be configured to have a low tolerance level for external vehicle events and threats so that it reacts more critically to the vehicle's operation. By having an additional, independent warning system, users can rest assured knowing that they are safe and that their vehicle operation is being reviewed and checked by a separate system. In this regard, the monitoring device does not play a role in controlling the vehicle itself, but alerts the driver so that they can take control themselves when necessary (i.e., disable the vehicle's autonomous mode and return the vehicle to driver control). The device adds an additional level of security in addition to the driver's supervision of their own vehicle operation. Thus, the user may interact less with the vehicle unless absolutely necessary. In some situations, the additional security provided by the device may lower insurance premiums.

[0011] While the above refers to semi-autonomous and autonomous vehicles, the device may also be used when the vehicle is manually driven and the user is not the driver. For example, in a ride-sharing situation, the device may be used to check that the driver's driving meets required standards.

[0012] The at least one sensor may include a proximity sensor. The proximity sensor may include an infrared sensor, a camera, and / or an ultra-wideband sensor. The sensor set may include at least one external weather monitoring sensor. The at least one external weather monitoring sensor may include a barometric pressure sensor and / or an ambient light sensor. The sensor set may include at least one position sensor. The at least one position sensor may include a gyroscope, a magnetic sensor, an altimeter, a geolocation sensor, and / or an accelerometer. The sensor set may include an audio sensor. The sensor data may include an audio signal. It is understood that some of these sensors may be implemented by a combination of a device's camera and a software algorithm executed by the device's processor that processes the captured images and produces specific measurements. For example, in some embodiments, the proximity sensor may be implemented by a camera that captures images of a vehicle in the external environment and an algorithm that determines the vehicle's proximity from the vehicle's apparent size in the images. Another example includes an ambient light sensor that may use a software program to determine the ambient light of an image captured by the camera as a function of luminance.

[0013] In some embodiments, the portable monitoring device (such as a smartphone) includes a local wireless communications link to a personal communication device that provides a user interface to the monitoring device. This advantageously reduces the size and cost of the portable monitoring device and takes advantage of the fact that most drivers have smartphones. However, in other embodiments, the portable monitoring device can include a user interface, and in some embodiments, can be a smartphone itself programmed with a downloadable app. This option further reduces costs because the device itself does not need to be provided, but rather the driver's general-purpose smartphone is simply configured to function as the monitoring device through downloadable software.

[0014] The interface, in some embodiments, may include a touchscreen and a loudspeaker. The interface, in some embodiments, may include a projector configured to project an image onto a surface of the vehicle (such as the windshield) to create a head-up display.

[0015] Optionally, the portable electronic monitor device includes a wireless communication engine for communication with a remote server, the wireless communication engine configured to receive information about the external environment in which the vehicle is traveling.

[0016] Optionally, the portable electronic monitor device includes an AI engine configured to operate as a neural network for learning and modeling the autonomous driving behavior of the vehicle, and a processor operably connected to the AI ​​engine. The AI ​​engine may include a neural network trained to model expected vehicle driving behavior. The neural network may be trained using sensor data collected from manual and / or automated operation of the vehicle prior to the current driving period. The sensor data collected prior to the current driving period may be confirmed data sensed during one or more driving periods in which no dangerous driving event was recognized. Based on the neural network and the sensor data, the AI ​​engine may be configured to generate a model of expected autonomous vehicle driving behavior during a particular driving event.

[0017] Optionally, the processor is configured to determine a threshold value for the particular driving event, and if a comparison of the detected automated driving behavior with a model of the automated vehicle's expected driving behavior for the particular driving event indicates that a deviation has occurred, compare the deviation with the threshold value and determine whether the deviation exceeds the threshold value.

[0018] The threshold may be determined based on a driving event and at least one other parameter selected from the group consisting of: driver reaction time; vehicle automation level; vehicle condition; road type; weather conditions; and one or more user settings. The at least one other parameter may include driver reaction time, and the sensor set may include at least one sensor that senses the vehicle's interior environment. The processor may be configured to determine the driver reaction time based on current and / or historical sensor data sensed from the sensor that senses the vehicle's interior environment. The driving event may include a vehicle maneuver, and the threshold may be based on one or more of: vehicle speed during the maneuver; vehicle braking during the maneuver; and vehicle steering angle during the maneuver. The driving event may include an interaction with another vehicle, and the threshold may be based on one or more of: speed of one or each vehicle during the interaction period; braking during the interaction period; proximity of the other vehicle; direction of movement of the other vehicle; location of the other vehicle; whether the other vehicle is perceived as being automatically operated or capable of operation; and / or behavior of the other vehicle.

[0019] The processor may be configured to determine a classification framework for the particular driving event; assign a value to a deviation of the detected automated driving behavior from an expected automated driving behavior based on the classification framework; and compare the value to a predetermined threshold, the threshold being a value of the classification framework. The classification framework may include a plurality of discrete category values. The classification framework may include continuous numeric values.

[0020] Multiple thresholds may be provided for identifying unsafe driving events, each of which may correspond to a different warning signal.

[0021] Optionally, the sensor set includes at least one sensor sensing the vehicle's interior environment. The sensor set may be configured to monitor the vehicle's interior environment during a driving period and generate sensor data indicative of the driver's current attention state during the driving period. The processor may be configured to determine a required driver attention state for current driving of the semi-autonomous vehicle in the external environment; compare the driver's current attention state with the required driver attention state; and generate a warning alert signal if the current attention state deviates from the required attention state by more than a threshold value. The required attention state may be determined based on one or more vehicle parameters. The one or more vehicle parameters may include a vehicle automation level, a vehicle speed, a vehicle occupancy level, and / or a quality of automated vehicle driving. The required attention state may be determined based on one or more external environmental parameters.

[0022] The one or more external environmental parameters may include road type, road quality, traffic density, weather type, classification of how urban or rural the environment is, driving behavior of other vehicles in the vicinity, and / or the presence of one or more dangerous driving events and / or other threats.

[0023] The processor may be configured to determine, when an unsafe driving event is detected, a point in time before which manual resumption of control of the vehicle is required, and to generate a warning signal at the latest before that point in time.

[0024] In some embodiments, the device is a smartphone and may be configured with downloadable applications. [Brief explanation of the drawings]

[0025] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1A] FIG. 1A is a schematic diagram of a known semi-autonomous vehicle system. [Figure 1B] FIG. 1B is a schematic diagram of a known semi-autonomous vehicle. [Figure 2A] FIG. 2A is a schematic diagram of a semi-autonomous vehicle system incorporating a mobile communication device according to an embodiment of the present invention. [Figure 2B] FIG. 2B is a schematic diagram of a semi-autonomous vehicle incorporating a mobile communication device according to an embodiment of the present invention. [Figure 3] FIG. 3 is a schematic diagram of the mobile communication device of FIG. 2B, according to one embodiment of the invention. [Figure 3A] FIG. 3A is a schematic diagram of a set of sensors used in the mobile communication device of FIG. [Figure 4] FIG. 4 is a flow chart illustrating a method for setting up the mobile communication device of FIG. [Figure 5] FIG. 5 is a flow chart illustrating a method for analyzing monitoring data collected using the mobile communication device of FIG. 3 in the method of FIG. [Figure 6] FIG. 6 is a flow chart illustrating a method for analyzing monitoring data collected using the mobile communication device of FIG. 3 in the method of FIG. [Figure 7] FIG. 7 is a flow chart illustrating a method for analyzing monitoring data collected using the mobile communication device of FIG. 3 in the method of FIG. [Figure 8] FIG. 8 is a flow chart illustrating a method for analyzing monitoring data collected using the mobile communication device of FIG. 3 in the method of FIG. [Figure 9] FIG. 9 is a flow chart illustrating a method for analyzing monitoring data collected using the mobile communication device of FIG. 3 in the method of FIG. [Figure 10] FIG. 10 is a flow chart illustrating a method for analyzing monitoring data collected using the mobile communication device of FIG. 3 in the method of FIG. [Figure 11]FIG. 11 is a flow chart illustrating an alerting method for alerting a user using the mobile communications device of FIG. 3 according to one of the methods of identifying that an alert is desired of FIGS. 5-9. DETAILED DESCRIPTION OF THE INVENTION

[0026] Description of the Prior Art 1A illustrates a known system 10 for semi-autonomous vehicle information exchange. The system 10 includes a semi-autonomous vehicle 12 having an autonomous system 14 disposed thereon, wirelessly connected to an autonomous system content provider server 16. The communication network 18 is a wide area network, such as the Internet. Communication between the vehicle 12 and the server 16 is permitted over the communication network 18 using a suitable wireless communication protocol, such as 3G / 4G / 5G, or local Wi-Fi areas (Wi-Fi hotspots), forming a wireless mesh network.

[0027] Given the context of FIG. 1A , FIG. 1B depicts a semi-autonomous vehicle 12 in more detail, schematically. The following paragraphs refer to both FIG. 1A and FIG. 1B . The vehicle 12 in FIG. 1B has four wheels 20 (two are shown in FIG. 1B ) and a conventionally arranged interior 22. The interior 22 is depicted as having a set of front seats 24, a set of back seats 26, and a steering wheel 28 mounted on a dashboard 30. It is understood that the interior 22 includes elements other than those depicted, including accelerator and brake pedals used by a user to manually operate the vehicle 12, as well as auxiliary driving switches such as indicators, windscreen wiper controls, or headlight buttons. The driver of the vehicle 12, hereafter referred to as user 32, sits in the front seat 24. References herein to user 32 refer to the operator of the vehicle 12, specifically the operator who manually controls the vehicle 12 when manual control is desired. The interior 22 of the vehicle 12 may have different layouts.

[0028] An autonomous system 14 is provided within vehicle 12. The autonomous system 14 is integrated into vehicle 12 to provide the vehicle 12 with specific automated or semi-automated driving capabilities, such as autonomous driving in specific conditions and situations. The autonomous system 12 receives sensor data from multiple on-board sensors 34 (e.g., cameras and proximity sensors) and / or an engine management system 36 (which generates multiple parameters about the vehicle's state and behavior), references data from internal or remote systems, such as an autonomous content provider server 16, and utilizes the received data to control the driving of the vehicle 12. The vehicle 12 is thus semi-automated and can be classified as at least Level 2 autonomy according to the SAE International Scale. Therefore, under conditions where autonomy is permitted, the user does not need to manually operate the vehicle 12.

[0029] As shown in FIG. 1A , instructions and data from the autonomous system 14 related to the automated operation of the semi-autonomous vehicle 12 are communicated wirelessly through a communication network 18. In this regard, the autonomous system 14 has a built-in wireless transmitter and receiver (not shown) for communicating with the autonomous system content provider server 16 through the communication network 18. The semi-autonomous vehicle 12 can sense its environment and collate sensor data about the environment using the autonomous system 14. The vehicle 12 reacts to a given situation based on the collated sensor data. Responses (e.g., vehicle control responses) are typically determined locally within the autonomous system 14 with minimal reaction time, although certain responses may be determined remotely by a remote processing system if the situation does not require a time-sensitive immediate response. The autonomous system content provider server 16 is a remote processing system that can operate the vehicle through the autonomous system 14 based on instructions from the remote processing system. For example, the remote processing system may access traffic information and issue instructions to adjust vehicle speed in anticipation of potential traffic conditions. Additionally, the autonomous system content provider server 16 may provide information, such as mapping information, to the autonomous system 14 so that decisions can be made locally by the autonomous system 14 within the vehicle 12. Data collected by the autonomous system 14 on the semi-autonomous vehicle 12 may also be uploaded to the autonomous content server provider 16 for analysis or later use.

[0030] Although only one semi-autonomous vehicle 12 is shown, in this embodiment, multiple other semi-autonomous vehicles (not shown) are typically in communication with the first content provider server over a communications network. In another embodiment, multiple fully autonomous vehicles are also provided. Similarly, in another embodiment, multiple autonomous content provider servers are provided, with each autonomous content provider server connected to a different set of semi-autonomous vehicles.

[0031] In all semi-autonomous vehicles (i.e., SAE Levels 1 through 4), the user must be able to take manual control of the vehicle in certain situations (e.g., if the autonomous system makes an error based on its sensor inputs) and must be alert and ready to provide such manual override control. The definition of a semi-autonomous vehicle as a vehicle capable of both manual and automatic operation applies here. At lower levels, the user must also monitor the vehicle's operation to resume control, for example, if it is determined (by the vehicle or the user) that an error has occurred or that manual control is required to prevent a dangerous driving situation. However, the higher the level of automation in a semi-autonomous vehicle, the less attention the user typically pays to the driving situation, as the user places more trust in the autonomous system to control the vehicle. However, this is precisely the problem: there is a greater risk that the user will not notice the semi-autonomous vehicle's error, which could result in the vehicle causing an accident or crash.

[0032] The present embodiment is directed to overcoming this problem in known semi-autonomous vehicles. A separate supervisory monitoring system is provided to provide oversight of the semi-autonomous vehicle's operation and user behavior while the semi-autonomous vehicle is operating autonomously. The monitoring system is implemented in a portable mobile communication device having its own set of sensors and the ability to process data received from the sensor set. An example of this could be, but is not limited to, a smartphone. The portable mobile communication device is removably attached to the vehicle to monitor the vehicle's autonomous system. In use, the portable device is removably attached or secured to the vehicle (e.g., by a holder or mount) and is typically located on the windshield so that the portable mobile communication device can use its sensors to monitor the environment around the vehicle. In some embodiments, the portable mobile communication device includes a smartphone with rear and front cameras running a downloaded application that configures the smartphone to operate as a monitoring system during driving periods when the semi-autonomous vehicle is being driven by the autonomous system. Smartphone embodiments are described in more detail below.

[0033] Preferably, the oversight provided by the supervisory monitoring system can be more rigorous and cautious about hazards than that determined by the operation of the autonomous system, without affecting the automated operation of the vehicle. This embodiment allows for a higher level of vigilance for vehicle hazards and driving events compared to an autonomous system installed in the vehicle. This allows the supervisory monitoring system to maintain a watchful eye on the driving environment and provide an even greater level of safety in serving as a safe companion to the user. Additionally, while there are currently no standards applicable to such semi-autonomous vehicles and autonomous systems, this embodiment provides a way to monitor all the different proprietary systems integrated into different vehicles using an independent, non-proprietary system. In other words, the supervisory monitoring system can operate to monitor the automated operation of the vehicle and ensure that the autonomy does not lead to unsafe driving. If a dangerous driving event is identified, the system operates to alert the user using its integrated sensor set and interface.

[0034] Therefore, this embodiment provides an advantageous way to provide a consistent standard across all autonomous systems in autonomous and semi-autonomous vehicles without requiring all manufacturers to change their independent development of these autonomous systems. An additional benefit is that it can provide an additional layer of safety to the autonomous systems, which can also reduce / decrease vehicle insurance costs. Additionally, users may feel more confident in the autonomy of their vehicles if they can monitor both the data they can independently gather using their own senses and the data collected by an independent surveillance monitoring system.

[0035] FIGS. 2A and 2B illustrate a surveillance monitoring system. FIG. 2A illustrates a system 38 for semi-autonomous vehicle information exchange. In common with the known system of FIG. 1A, the system 38 includes a semi-autonomous vehicle 12 having an autonomous system 14 wirelessly connected to an autonomous system content provider server 16. The communication network 18 is a wide area network, such as the Internet. Communication between the vehicle 12 and the server 16 is permitted over a communications network using, for example, a suitable wireless communications protocol, such as 3G / 4G / 5G, or local Wi-Fi areas (Wi-Fi hotspots), forming a wireless mesh network. FIG. 2A differs from FIG. 1A in that the autonomous vehicle 12 also includes a surveillance monitoring system 40 wirelessly connected to a surveillance system content provider server 42 via the communications network 18. Where appropriate, monitoring data collected by the surveillance monitoring system 40 regarding the operation of the vehicle 12 and user behavior while the vehicle 12 is operating autonomously is communicated over the communications network 18. The surveillance monitoring system 40 analyzes the collected monitoring data locally and / or communicates the collected data to a surveillance system content provider server 42 for remote analysis (especially where higher processing power is required).

[0036] The surveillance monitoring system 40 is shown in FIG. 2B as being removably mounted within the autonomous vehicle 12 in front of the user 32, i.e., on the windshield of the vehicle 12. FIG. 2B shows the autonomous system 16, built-in sensors 34, engine management system 36, and user 34 within the vehicle 12, similar to those shown and described in FIG. 1B. From the mount 44, the surveillance monitoring system 40 monitors one or more of the interior of the vehicle 12, the external environment of the vehicle 12, and the general autonomous operation of the vehicle 12 in the external environment. Generally, monitoring of the external environment includes monitoring the forward direction of travel of the vehicle 12, although in some embodiments, appropriate hardware is used to monitor the external environment from the sides of the vehicle 12, the rear of the vehicle 12, and even under the vehicle. The use of additional hardware and monitoring system hardware is discussed below with reference to FIGS. 3 and 3A.

[0037] In its oversight role, the supervisory monitoring system 40 monitors, independent of the autonomous system 14, at least one of the user 32, the vehicle 12, or the semi-autonomous vehicle 12 for any activity that may pose a risk to the continued automated operation of the vehicle. When referring to the independence of the supervisory monitoring system 40 herein, it is meant that the supervisory monitoring system 40 separately monitors and analyzes data without any input from or output to the autonomous system 16. In addition to its monitoring duties, the supervisory monitoring system 40 also provides alerts to bring identified hazards to the user's attention as needed. The terms "user" and "driver" are used interchangeably herein.

[0038] To provide oversight, the supervisory monitoring system 40 uses sensor data to monitor potential hazards. The supervisory monitoring system 40 may be operable to analyze the autonomous system 16 in one or more of at least two driving modes, in which different sets of sensors provide relevant data used by the supervisory monitoring system 40 to determine hazards. In a supervisory embodiment or configuration (driving mode), the supervisory monitoring system 40 is entirely separate from the autonomous system 16. In this mode, the supervisory monitoring system 40 receives data from a set of sensors that are not part of the vehicle 12's built-in sensors 34 and processes and communicates the data using processing and communication systems that are not part of the autonomous system 14. In other words, in supervisory mode operation, the supervisory monitoring system 40 and the autonomous system 14 do not share any of the same systems or data. Alternatively, as a verification embodiment or verification configuration (driving mode) of the supervisory monitoring embodiment described above, the supervisory monitoring system 40 may act as an observer, monitoring and analyzing data received by the autonomous system to ensure that decisions made by the autonomous system 14 are those expected based on the received data. In this embodiment, the surveillance monitoring system 40 and the autonomous system 14 share data streams but operate to process and communicate data separately. To this end, the surveillance monitoring system 40 is configured to interface with the vehicle 12 in some manner. In some embodiments, the surveillance monitoring system 40 connects to the vehicle 12 via a USB port or using the vehicle's on-board diagnostics (OBD) port. In the following description, the surveillance monitoring system will be assumed to be the former (surveillance monitoring embodiment), completely independent of the autonomous system, although it will be understood that the same concepts may be applied to the latter system (verification embodiment) as well.

[0039] The surveillance monitoring system 40 can be integrated into the vehicle 12 as software running on the vehicle hardware and / or as separate hardware with dedicated modules and / or custom software. In the non-limiting example described above, the surveillance monitoring system includes a portable electronic monitor device such as a smartphone, particularly a portable communication device (also referred to herein as a mobile communication device or mobile device) in this embodiment. The mobile device has dedicated function modules and / or downloaded custom software (applications) to provide relevant supervisory functions. Providing the mobile device as part of the surveillance monitoring system allows for independence from the autonomous system and also vehicle independence: the mobile device can be configured to provide independent vehicle operation supervision for any semi-autonomous vehicle. The mobile device provides sensor and output hardware that enables the supervisory and alert functions required to serve as a surveillance monitoring system. The term mobile means that the device is portable to the user, can be removed from the vehicle in a straightforward manner, and can be removed by any user without the need for a trained technician.

[0040] The use of a mobile device for this form of monitoring assistance ensures that sufficient sensing, processing, and alert hardware is preferably provided. If the mobile device is the user's personal smartphone, the user is likely to have the device with them at all times, and the system can be relatively easily set up by the user in any desired vehicle, for example, by placing it in a smartphone holder mounted on the dashboard of the vehicle 12. The advantages of the system are further explained below with reference to the following figures, the features of which are described in more detail below.

[0041] As shown in Figure 3, the exemplary mobile device 50 includes a main processor 52 connected to a monitoring system 54. The monitoring system 54 includes sub-processors in the form of an external environment monitoring processor 54a, a user monitoring processor 54b, and a vehicle monitoring processor 54c. In use, the monitoring system 54 interacts with the other modules of the mobile device 50 shown in Figure 3 through the main processor 52 such that the mobile device 50 functions as the surveillance monitoring system 40. The monitoring system 54 receives data from sensors 56 and / or other modules of the mobile device 50 and analyzes the received data. Based on the results analyzed and generated by the monitoring system 54 and sent to the main processor 52, the main processor 52 determines actions to take for semi-autonomous operation of the vehicle 12.

[0042] Within the mobile device 50, the main processor 52 is also in communication with an alert system 58, a navigation system 60, a user interface 62, a communications engine 64, and data storage 66. The alert system 58, in this embodiment, includes a signal generator (not shown) that uses its built-in user interface 62 to generate control signals for the mobile device 50 to generate sensory alerts. The sensory alerts may be a vibration of the mobile device 50 via a vibration motor in the mobile device 50, an audible alert generated from a loudspeaker in the mobile device 50, and / or a visual alert created by a particular illumination of the display of the mobile device 50, such as an illumination flash to attract the user's attention. In other embodiments, the mobile device 50 may not have a user interface and may instead use the user interface of a user's personal mobile communications device (such as a smartphone) to interface with the mobile device 50. In this embodiment, the personal user's smartphone may be operably connected to the mobile device wirelessly, for example, via a Bluetooth® connection.

[0043] The mobile device 50 of FIG. 3 has a number of different sensors 56. A set of sensors 68 from which sensors 56 of the mobile device 50 may be selected is shown in FIG. 3A. A core set of sensors 70 is shown in FIG. 3A with dotted lines. The core set of sensors 70 includes one or more cameras 72, one or more microphones 74, one or more accelerometers 76, a gyroscope 78, and sensors of a navigation system 60: a location determination sensor 80, such as a GPS receiver, for determining the current location of the mobile device 50, and a geographic compass 82 or compass function using the mobile device 50's magnetic sensors. Other forms of geographic location determination sensors may also be provided, such as a terrestrial wireless positioning system, a location sensor using the Doppler effect, or a Wi-Fi hotspot location sensor. Non-core sensors of the mobile device 50 outside the dotted lines include an altimeter 84, which determines the device's 50 elevation above sea level. This provides assistance in determining the geographic location of the mobile device 50 and helps to better understand external weather conditions. Another sensor is a barometric pressure sensor 86, which is provided to indicate the current atmospheric pressure to which the mobile device 50 is exposed and aids in determining and corroborating external weather conditions. An ambient light sensor 88 may be configured to determine external lighting conditions and aid in adjusting safety thresholds based on current visibility in available ambient light. An infrared proximity sensor 90 enables detection of the presence of subjects within the vehicle 12 or corroborates the position image of the driver of the vehicle 12 captured by a rear-facing (with respect to the vehicle) camera, for example, in poor ambient lighting conditions. One or more ultra-wideband (UWB) sensors 92 are provided to detect objects within the field of view of the sensors (occupants present in the vehicle 12 and objects present in the vicinity of the vehicle 12, e.g., other vehicles or objects). Obviously, the UWB sensors 92 are directional, so two such UWB sensors 92 facing in opposite directions are required when monitoring both the interior and exterior of the vehicle 12.Unlike visual images captured by a camera, these are not sensitive to ambient light conditions, whereas camera-based techniques cannot perform in dark environments, so it is advantageous to use one or more UWB sensors 92 that use pulsed radar transmissions and reflections.

[0044] The above-described sensor set 68 shown in FIG. 3A can be provided in different combinations in different embodiments of the present invention. While a core set of sensors 70 is shown in dotted lines in FIG. 3A, the use of an altimeter 84 or barometric pressure sensor 86 is not required in some embodiments. Similarly, an ambient light sensor 88 and an IR proximity sensor 90 are optional in some embodiments, as image processing algorithms may be used to detect ambient light levels in captured camera images. Finally, the use of a UWB sensor 92 with the core set of sensors 70 in other embodiments is also optional, but can have significant advantages for determining the location of objects in low light or poor visibility conditions, both inside and outside the vehicle 12.

[0045] In considering the sensors 56 and general monitoring capabilities of the surveillance monitoring system 40, the surveillance monitoring system 40 at a minimum monitors the external environment in the forward direction of travel of the vehicle 12. Where possible, the surveillance monitoring system 40 is also configured to monitor the external environment in the rearward direction of travel of the vehicle 12 and / or on both lateral sides of the vehicle 12. When the surveillance monitoring system 40 is incorporated into a mobile device 50, additional monitoring capabilities, such as the ability to monitor different areas of the vehicle 12, can be achieved by providing additional mobile devices that connect wirelessly or otherwise to the original mobile device 50. For example, in embodiments where the surveillance monitoring system 40 is configured to monitor the external environment on both lateral sides of the vehicle 12, a side-facing camera module is connected to the mobile device 50 to monitor the external environment through the driver and passenger side side windows. In embodiments where the surveillance monitoring system 40 is configured to monitor the external environment behind the vehicle 12, a camera module is mounted inside the vehicle 12 near the rear windscreen to provide supervision of the rear of the vehicle 12. The side-facing and rear-facing camera modules may be incorporated into additional mobile devices. Additionally, if the surveillance monitoring device 40 has multiple cameras with different fields of view facing the same direction, as is common in modern smartphones, the surveillance monitoring system may use images captured by different cameras for different purposes. For example, a camera with a wide-angle lens may have a wide field of view and be able to capture external activity to the side of the vehicle 12 without the need for a side-facing camera. Similarly, a telephoto (zoom) lens may be useful in situations where a surveillance monitoring device has a rear-facing camera (relative to the vehicle) mounted on the windshield or dashboard that can view external activity occurring behind the vehicle 12 through the rear windshield of the vehicle 12.

[0046] Returning to Figure 3, the user interface 62 allows a user to input commands into the mobile device 50 and output information, such as sensory alerts, to the user. The communications engine 64 enables communications between the mobile device surveillance monitoring system 40 and the content provider server 42 over the communications network 18, as shown in Figure 2A.

[0047] In an alternative embodiment or in addition, data storage 66 stores a monitoring software program that, when executed on main processor 52, enables mobile device 50 to function as surveillance monitoring system 40. For example, an application or "app" downloadable from a content provider (such as the App Store®) may be stored as an executable monitoring software program and selectable for execution by a user.

[0048] The embodiment shown in FIG. 3 also includes a dedicated AI (artificial intelligence) processor 94 configured to operate as a neural network. The AI ​​processor 94, also referred to as an AI engine, is used to analyze data generated by the sensors 56 and executed by the autonomous system 14 during autonomous driving. Driving patterns, and in particular how the autonomous system reacts to different driving events (such as another vehicle suddenly changing lanes in front of the vehicle), can be monitored to determine a model (not shown) that describes how the vehicle's autonomous system 14 will function. This model of how the autonomous system operates can be compiled and used to predict how the autonomous system 14 will react, particularly when a critical driving event occurs. Such a model can then be used by the main processor 52 as a predictive model of the semi-autonomous vehicle 12 to determine whether the supervisory monitoring system 40 needs to issue an alert to the user to intervene earlier than would otherwise be possible. The alert is issued early because the supervisory monitoring system 40 acts predictively, using a predictive model of what action the autonomous system 14 will take, rather than on the reaction the autonomous system 14 will take. In addition to semi-autonomous driving, reaction times to user intervention in certain events can be monitored and used to determine the likely reaction time for a given event. Different users have different reaction times, and the timing of alerts can be adjusted accordingly, for example, for slower drivers, to provide proactive alerts. This trained AI model of the vehicle 12 created by the mobile device 50 can be uploaded to the monitoring system content provider server 42 for later use and storage, if desired. The AI ​​engine includes a neural network and operates as a neural network. The neural network is trained to model vehicle behavior using data collected during vehicle operation. The operation during which the data was collected can be automated or manual.In either situation, the user is asked to confirm through the user interface that no dangerous or unexpected driving events occurred during the driving period during which data was collected. That is, during the learning period, the user is present in the vehicle to confirm the start of the learning driving period. The user pays attention to driving the vehicle (either normally if driving manually, or automatically if operating automatically) throughout the learning driving period. At the end of the learning driving period, the device seeks confirmation from the user to certify that the data collected during the learning driving period is suitable for training the neural network. If the user confirms suitability, the AI ​​engine updates the neural network based on the collected data; otherwise, the data is discarded. The user may be the vehicle owner or an engineer. In some situations, the neural network may be trained with data collected from vehicles of the same type and downloaded to the device based on user input about the type of vehicle being operated.

[0049] 3 is provided as an example, and it is understood that in other embodiments, the mobile device 50 incorporates and / or is connected to additional modules not shown to further enhance the operation of the surveillance monitoring system 40. For example, an additional alert module, such as a projector configured to create a head-up display on the vehicle windshield, or two sets of lights, one located on either side of the mobile device 50, paired with the mobile device 50 to provide real-time hazard tracking to the user. Such additional modules may be incorporated into, i.e., integrated into, the mobile device 50, incorporated into a cradle or holder in which the mobile device 50 is mounted, or incorporated elsewhere in the vehicle 12.

[0050] The surveillance monitoring system 40 operates according to one or more surveillance processes, examples of which are set forth in the flowcharts of Figures 4 to 11. Each of the processes of Figures 4 to 11 is described with respect to the surveillance mobile communications device 50 shown in Figure 3, but it will be understood that the processes may be applied to any form of surveillance monitoring system 40, particularly portable systems.

[0051] 4 is a flow diagram illustrating a preliminary method for device setup, starting at A. In the first stage of method 400, in step 402, the mobile device 50 is configured in the vehicle 12. Configuring the mobile device 50 in step 402 includes at least one configuration process. Various configuration processes are described below.

[0052] In one configuration process, the mobile device 50 is configured by positioning it in the vehicle 12 to enable proper and complete monitoring. In this process, if the mobile device 50 is a smartphone, the process includes removably mounting the mobile device 50 to the vehicle 12 using a cradle or holder so that the rear camera of the mobile device 50 faces the exterior of the vehicle 12 (forward-facing) and the front camera of the mobile device 50, together with the device screen, faces the interior of the vehicle 12, specifically the user (facing rearward relative to the vehicle 12). A preferred location for the cradle or holder would be, for example, the vehicle dashboard or front windscreen. FIG. 2B illustrates one possible position, with the mobile device 50 positioned near the windshield to provide a good view of the road ahead and the interior of the vehicle 12.

[0053] Other configuration processes include configuration 400 running a configuration process on the mobile device 50 to calibrate and ensure the sensors 56 are positioned correctly. If the mobile device 50 is partially incorrectly configured, the user is prompted to correct the configuration. The mobile device 50 analyzes images received from the device's rear-facing camera 72, which is positioned to face the vehicle's 12's external environment. Image analysis is performed to determine that the orientation and angle of the mobile device 50 relative to the driving surface and vehicle 12 are correct and that there are no obstructions in the image (i.e., the camera's field of view) that could cause errors in subsequent processing. Similarly, if a UWB sensor 92 is used, it is important to configure it to ensure that there are no objects (e.g., on the dashboard) that obscure the view and alter the results. The mobile device 50 may also analyze images obtained from the mobile device's forward-facing camera, which is configured to face the user of the vehicle 12. The analysis may determine, for example, whether the user's hands on the steering wheel are clearly visible in the image, and whether the user's face is clearly visible in the image and that no obstructions are present. Facial recognition may be employed on images acquired by the rear-facing camera to identify and track the user, using AI processor 94 to determine whether the captured image matches a pre-stored set of images that AI processor 94 has trained on. This is particularly useful because the user may not be perfectly aligned with the camera for facial recognition, making it particularly useful to use AI processor 94 on a partial image to determine the user's identity. It is understood that the user also needs to see the external environment while operating vehicle 12, and therefore mobile device 50 may be configured (positioned) to allow the user to have a significant, unobstructed view through the vehicle windows; for example, mobile device 50 may be positioned in a location within the user's peripheral vision so that the road ahead is visible.

[0054] Further configuration includes a data entry phase during which parameters are set, either by the user or automatically, to configure the surveillance monitoring system for the vehicle and the setup for the mobile device 50 within the vehicle 12. Parameters are set so that the mobile device 50 can apply the configuration to its operation, such as the vehicle type, vehicle autonomy level, the relative location of the mobile device 50 and the user within the vehicle 12, the desired level of monitoring, the user's identity, the identities of any passengers, and the destination and / or projected route of the vehicle 12. In response, the mobile device 50 adapts how it processes received information in accordance with one or more of these parameters. As one example, for a configured vehicle type, the surveillance monitoring system 40 communicates with the surveillance system content provider server 42 to access relevant data, such as the vehicle type, the type of autonomy with which the vehicle 12 is operated, and relevant information about how the vehicle will react to certain situations. In this regard, the type of autonomy under which the vehicle 12 is operated may be available by checking predetermined information provided by the vehicle manufacturer or an AI model previously created by another user and uploaded to the monitoring system content provider server 42. The mobile device 50 accesses data it stores (either on the mobile device 50 itself or from the autonomous system content provider server) to determine whether it has previously monitored the operation of the vehicle 12 and details of previous recorded vehicle 12 operations. The mobile device 50 recalls cases where the vehicle's operation was unexpected or a malfunction occurred. This information may also be available in a trained AI model for the vehicle 12 previously created by the mobile device 50 and uploaded to the monitoring system content provider server 42. Thus, this information may be downloadable from the monitoring system content provider server 42, even if it is not already present in the mobile device 50.

[0055] Yet another configuration process includes disabling one or more features of the mobile device 50 during a driving session. In some embodiments of this process, disabling one or more features includes automatically disabling functions of the mobile device 50, such as receiving notifications to reduce user distraction or data communication to specific locations to ensure the user's sensitive data is properly protected. Alternatively, or in addition, disabling may include identifying features the user will disable and / or enable based on their preferences. Other user preferences, such as alert volume, alert type, or user interface graphic placement, may also be configured at this stage, through adjusting personal settings, activating pre-defined profiles, or other methods.

[0056] The configuration is preferably performed before the vehicle 12 begins driving (before the driving phase begins), but can also be performed while the vehicle 12 is operating automatically. The mobile device 50 may reconfigure while the vehicle 12 is operating automatically. In some embodiments, the mobile device 50 automatically determines that a user is present in the vehicle 12 and prompts the user to confirm whether they wish to perform monitoring functions of the autonomous system 14.

[0057] Once the mobile device 50 is configured within the vehicle 12, a new driving period begins, as determined by the surveillance monitoring system 40 in step 404. Once a new driving period begins and is determined by the surveillance monitoring system 40, the mobile device 50 begins providing its main function of monitoring the operating conditions of the vehicle 12 and analyzing the monitoring data accordingly in order to alert the user.

[0058] At step 404, the start of a new driving period is determined by the user physically indicating to the mobile device 50 through a user interface that a new driving period has begun, or by the mobile device 50 automatically identifying the start of a driving period. The automatic determination by the mobile device 50 is performed based on successful completion of device configuration at the vehicle 12, such as identifying movement indicative of the start of a driving period of the vehicle 12, movement of the vehicle 12 at a predetermined speed, the presence of a user relative to the device at the vehicle 12, and / or other indications that a driving period has begun. For example, for the automatic determination, the mobile device 50 senses forward acceleration of the vehicle 12 via an accelerometer in the mobile device 50. If the device automatically detects the start of a new driving period, this acceleration by the vehicle 12 above a predetermined level is taken to indicate that a new driving period has begun.

[0059] After the start of a new driving period is determined in step 404, the mobile device 50 collects monitoring data in step 406. The monitoring data is data that may be collected by the mobile device 50 using one or more of the sensors 56, the navigation system 60, the user interface 62, the communication engine 64, and other data receiving modules, such as external, connected sensors. The collection of monitoring data begins in step 406, and it is expected that the collection of monitoring data will continue during each subsequent step in the process. In other words, the monitoring data is collected continuously during the driving period.

[0060] The monitoring data may include, by way of example, one or more of the following: still images and / or video acquired by the forward-facing and / or rear-facing cameras 72; radar data from one or more UWB sensors 92; sound data acquired from one or more microphones 74; acceleration data acquired from one or more accelerometers 76 within the mobile device 50; position data including relative and absolute orientation, elevation above sea level, and location gathered from the compass 82, GPS system 80, or gyroscope 78, altimeter 84, or other sensors within the mobile device 50; traffic and / or weather information acquired via the communications network 18; weather data from the barometric pressure sensor 86; road speed limit data; and location-related data such as population or area type around a road. Receiving monitoring data related to population or area type is important for understanding the type of hazard or threat in a situation in which the vehicle 12 is located. The area type is also important for accessing the hazard type. This type of monitoring data is typically received from a source external to the mobile device 50. Other examples of monitoring data received from external sources include data from wearable or other connected devices within the vehicle 12, such as a smartwatch, to determine biometric parameters of the user (driver), such as the user's heart rate.

[0061] Collecting the monitoring data at step 406 enables the mobile device 50 to perform one or more analytical processes at step 408, designated processes B, C, D, E, F, or G, each of which is illustrated in FIGS. 5 through 10. Each analytical process B-G provides a different monitoring function and is performed by the main processor 52 and / or the AI ​​processor 94, which executes a monitoring software program. In the implementation of some processes, some or all of the parts of the device 50, particularly the AI ​​processor 94, main processor 52, and monitoring system 54, may operate in sync to provide method results and perform the appropriate analysis. The AI ​​processor 94 is particularly used for vehicle action analysis and for comparing simulations of actions expected to be taken by the vehicle 12 with the autonomous system. In use, the AI ​​processor 94 is considered equivalent or substantially equivalent to the autonomous system 14 in terms of its processing power, except that it is capable of controlling the vehicle 12.

[0062] Analysis processes B-G are not mutually exclusive and can be executed simultaneously or individually in any combination. When processes are executed simultaneously, a hierarchy of analysis, i.e., which processes take priority, may be implemented to ensure that the mobile device 50 safely prioritizes responses and optimizes data movement and available processing power. It is expected that one or more of optional processes B-G and optional alert process H (FIG. 11) will be executed repeatedly until the driving period ends. The above describes how the start of a driving period can be indicated by a user or automatically. Similarly, the end of a driving period is determined when the user indicates the end via the mobile device 50, or when the mobile device 50 detects through sensor data collection that the vehicle 12 is inactive, i.e., not actively being driven, e.g., stationary and with the ignition off, as determined automatically.

[0063] Analysis processes B and C of FIGS. 5 and 6, which may follow the process of FIG. 4, are methods for monitoring the external environment of vehicle 12 generally, rather than the interior of vehicle 12 or operation in general. When desired, the external environment is monitored while vehicle 12 is being driven to identify threats or hazards and alert the user to these threats. While monitoring during periods when autonomous system 14 is operating is of particular interest, the period during which surveillance monitoring system 40 is operating may cover the entire driving session, even when the user is manually operating semi-autonomous vehicle 12. This has the advantage of alerting the user, for example, if they are distracted while driving and their attention to their driving is suboptimal. In these cases, if surveillance monitoring system 40 identifies a hazard, it can alert the user to take appropriate action.

[0064] 5, which follows the processing of FIG. 4, the mobile device 50 analyzes, at step 502, the monitoring data collected at step 408. The analysis is performed at this step 502 to identify threats to the vehicle 12 in the external environment.

[0065] It is then determined at step 508 whether the identified threat is serious enough to warrant an alert to the user. If yes, alert processing begins at step 510. An example of alert processing 510 is shown as letter H in Figure 11 and described below. If the threat is not serious enough to warrant an alert, processing running on mobile device 50 returns to step 502 to continue analyzing the collected data.

[0066] In step 502, identifying threats preferably includes analyzing monitoring data related to the external environment, identifying objects or features in the external environment and actions the objects are performing, and determining whether one or more objects or features constitute a threat.

[0067] The categorization of threats in step 504 is generally according to how soon action needs to be taken. Example categorizations include three levels of threat: an imminent threat, such as another vehicle making a sharp turn in front of the monitored vehicle 12; a medium-term threat identification, such as another vehicle behaving erratically but not engaging in any dangerous behavior relative to the actively monitored vehicle 12; and a long-term threat identification, such as the actively monitored vehicle 12 noticing traffic congestion up the road. The identification of threats is dependent on a variety of vehicle and environmental parameters, such as the location of the vehicle 12, road type, road surface quality, vehicle type, vehicle autonomy level, current speed, acceleration, direction of travel, occupancy or other driving parameters, and / or other aspects such as time of day, season, traffic data, or road quality.

[0068] In step 506, a threat level is determined based on the categorized individual threats. In an embodiment, threat categorization and determination includes assigning a threat level to each feature or object based on an analysis of the received data regarding the threat, and assigning a value to each individual observation identified in the external environment corresponding to a risk to the monitored vehicle 12. An overall threat level is then a function, such as the sum, average, and / or maximum, of the values ​​assigned to the individual observations, such that an overall threat level can be assigned to the external environment. In an embodiment, the threat level is determined by increasing the threat level each time a new individual threat is identified (and decreasing the threat level when an individual threat is identified as having passed), with the threat level starting from a base reference level at the beginning of a driving period. The value by which the threat level is increased or decreased depends on the categorization of the threat. In other embodiments, the threat level is determined by a predictive mechanism applied to the mobile device 50, which identifies likely outcomes and assigns a threat level based on the likely outcomes. These predicted outcomes may be predetermined or determined through the use and analysis of the AI ​​processor 94. In some embodiments, threat levels are assigned by identification of predetermined scenarios.

[0069] In some examples, the surveillance monitoring system 40 identifies whether nearby vehicles are manually or automatically operated and identifies threats based on this data: manually operated vehicles are more likely to pose a threat than automatically operated vehicles and are identified as a higher level or category of threat than automatically operated vehicles. Vehicle tracking data is used in this determination and in identifying other threats. The surveillance monitoring system 40 can be configured to perform one or more vehicle tracking operations, which may include: determining the number and density of other vehicles and determining a threat if the number or density of other vehicles is higher than average; identifying the speed of surrounding vehicles and determining a threat based on the general speed of vehicles or speed limits for the area - if all vehicles are traveling above the speed limit, the threat level in the environment is higher compared to being assigned otherwise; tracking the behavior of a particular vehicle over time in a highway setting and identifying whether the vehicle is behaving erratically. While vehicles are the most common on roads, other objects may be present; in some embodiments, the surveillance monitoring system 40 is configured to use sensor data to identify and categorize non-vehicle foreign objects in the road, such as animals or debris, as threats. The same is true for weather and its severity, which may also be assigned a threat level.

[0070] Just as threat identification relies on a variety of environmental and vehicle parameters, one or more of the same parameters may also be used to categorize and / or assign a threat level.

[0071] In step 508, it is determined whether the threat warrants an alert to the user. This is done by setting an alert state. If a threat level is used, the alert state includes a predetermined threshold or set of thresholds for the state and / or parameters against which the threat level value is compared. If threats are categorized, the alert state includes identification of a particular category of threat or a predetermined number of threats within a category and / or identification of a temporal threat, i.e., a threat type that has existed for a predetermined period of time. In some embodiments, the alert state includes a weighting value based on the operating parameters of the vehicle 12. High vehicle density often poses a lower risk to stationary vehicles compared to vehicles moving at high speeds, and therefore different combinations of sensor parameters can be used to define the alert state. The alert state may include identifying a predetermined threat or scenario stored in the vehicle's data storage. It is also possible to provide multiple alert states against which the threat level is compared. This is discussed in more detail with reference to FIG. 11.

[0072] In addition to identifying threats in the external environment, threats may also exist inside the vehicle 12. An interior monitoring process 600, shown in Figure 6, uses sensors in the surveillance monitoring system 40 to monitor the user to ensure that the user's behavior is appropriate in response to identified threats. In other words, process 600 is a process that monitors both the external environment and the interior of the vehicle 12 to provide more appropriate alerts.

[0073] Collected monitoring data regarding the external environment and interior (interior environment) of vehicle 12 is analyzed to at least identify threats and user behavior in step 602. Threat level determination is shown here as a separate step as step 604. The threat level determination described above for steps 502, 504, and 506 of Figure 5 also applies to threat level determination in this process 600.

[0074] 6, a threat level is determined in step 604, and a desired user alertness appropriate to the threat level is determined in step 606. In other words, surveillance monitoring system 40 identifies how the user should behave in order to appropriately respond to the identified threat or threat level.

[0075] Simultaneously, in step 608, the surveillance monitoring system 40 determines the current user alertness by analyzing the monitoring data collected about the interior of the vehicle 12. As an example, a rear-facing camera (relative to the vehicle's main direction of movement) is used for user monitoring. Analysis of images or image sequences obtained by the camera is analyzed by a user monitoring processor in the surveillance monitoring system by identifying patterns (signifiers) of data indicative of user alertness, such as: the user's position within the vehicle 12; the user's body position; the direction the user is facing; whether the user's eyes are open or closed; whether the user's eyes (if open) are focused; where the user's hands are and whether they are on the steering wheel; the item the user is interacting with, such as a book, mobile device 50, food, or drink; whether there are other passengers in the vehicle 12 and whether the user is interacting with them; the user's eye blink rate; the user's physical reaction speed to the alert; yawning, and how often the user is yawning; and the user's breathing rate based on an image analysis of the user's chest. User monitoring is also performed by the surveillance monitoring system's microphone. Sound data from the surveillance monitoring system's microphone is analyzed to determine signifiers of user alertness, such as: the user's breathing rate; whether the user is talking to other passengers; and whether the user is vocally responding to an alert. In some cases, sound data is analyzed to recognize speech and determine the topic of conversation to determine whether the user is paying attention to the external environment. The surveillance monitoring system 40 can also analyze data received from other devices within the vehicle 12 to identify signifiers of user alertness. Data from a wearable device (not shown) may be analyzed to identify the user's biometric parameters, such as the user's heart rate, breathing rate, secretions, caffeine level, alcohol level, and / or blood oxygen level.

[0076] Although referred to herein as user alertness, it is contemplated that user alertness encompasses any aspect of user behavior that may affect how quickly a user can perform tasks within vehicle 12, particularly how quickly the user can return to manual control of vehicle 12 in response to the occurrence of an alert condition. For example, if the user is in a normal driving position, such as with feet on the pedals of vehicle 12, hands on the steering wheel, and attention focused on the road, the user is likely to be able to immediately and quickly return to manual control. The surveillance monitoring system 40 may therefore also use, e.g., an AI processor, stored examples of how the user has previously responded to alerts to determine overall alertness, since some users may not immediately respond to an alert in the correct manner, even when determined to be relatively alert. Furthermore, because different users have different reaction times to alerts, having some understanding of past response times helps the surveillance monitoring system 40 determine how quickly to issue an alert so that the user can react in time, e.g., to avoid a driving hazard.

[0077] Thus, user alertness can be categorized in terms of alertness level or value, or in terms of expected reaction time to return to manual control or to perform other tasks. The techniques described above also apply to determining the desired user alertness, as determined in step 606.

[0078] Once both the desired alertness of the user and the actual alertness of the user relative to the threat level have been determined in steps 606 and 608, the surveillance monitoring system 40 performs a comparison in step 610 to determine whether the current alertness meets the desired alertness.

[0079] If the user's alertness does not meet the desired level, then internal monitoring process 600 proceeds at step 510 to an alert procedure, such as procedure H of Figure 11. If the user's alertness meets the desired level, then process 600 returns to the analysis step at step 602 to continue monitoring threats and the user's alertness to ensure the desired level is met.

[0080] In some embodiments, instead of using sensor data from the surveillance monitoring system's sensors to passively monitor the user's behavior and alertness, the surveillance monitoring system 40 solicits input from the user through a user interface to ensure the user is at the correct alertness level. In such devices, the required alertness for a threat level is determined and based on the user's response to user input requests from the surveillance monitoring system 40. For example, the user may be required to periodically "check in" with the surveillance monitoring system 40 by interacting through the user interface or by speaking specific phrases to indicate alertness and caution. If the user does not interact with the surveillance monitoring system 40 as required by a predetermined deadline, the user's current alertness is determined to not meet the required user alertness for the threat level.

[0081] When considering user behavior, in some situations, a user may behave unsafely, creating a threat internally and not safely reverting to manual control when a different threat occurs, posing a threat to their own safety or to vehicle 12. Even if no threats or hazards are identified in the external environment, it is important that the user remain alert or at least not behave internally in a way that could pose a risk to the continued operation of vehicle 12. Thus, process D700 shown in FIG. 7 monitors user behavior and determines whether the user behavior is safe.

[0082] Process D700 begins in step 702 with monitoring and analyzing collected data to determine user behavior. User behavior is typically identified similarly to how user alertness is determined in process C600 of FIG. 6, although it is understood that any user behavior categorization may be used. In particular, data collected from sensors in surveillance monitoring system 40 monitoring the interior of vehicle 12 is used to identify how a user is behaving. User behavior may be assigned a category or behavior level, with unsafe behavior resulting in a level of alertness that may be categorized in FIG. 6 or FIG. 7.

[0083] Process 700 continues with surveillance monitoring system 40 determining whether the user's behavior is safe at step 704. If the user's behavior is determined to be safe, process 700 returns to step 702 and continues monitoring and analyzing the user's behavior. If the user's behavior is determined to be unsafe at step 704, the alert process of H510 of FIG. 11 is initiated, or other alert processing is initiated.

[0084] The categorization of unsafe behavior, in this non-limiting embodiment, depends on the level of automation of the vehicle 12. For example, in a level 4 vehicle, if a user reads a book while driving, this behavior would not fall into the category of unsafe behavior, but would not fall into the category of acceptable behavior in a level 2 vehicle. Some actions, such as sleeping, not wearing a seat belt, or drinking alcohol, are typically prohibited in vehicles of all levels.

[0085] The process 700 shown in FIG. 7 is assumed to be performed before any other processes to identify whether the user is safe to operate the vehicle 12, but may be performed at any appropriate time.

[0086] Unsafe behavior can occur not only in the user, but also in the vehicle 12. Erratic or unsafe vehicle control by the autonomous system can constitute unsafe behavior. The monitoring system monitors the vehicle's behavior and the driving events to which the vehicle is exposed, with the authority to determine whether an unsafe driving event has occurred if the actual driving outcome deviates from the expected driving outcome. Based on this data, the system can attribute the behavior to a vehicle malfunction. Process E800, shown in FIG. 8, is provided for monitoring the behavior of the vehicle 12 under the control of the autonomous system to identify deviations from what is expected in automated vehicle operation, to identify a malfunction, and to alert the user accordingly.

[0087] In Figure 8, monitoring data is received by a processor at step 802 and an analysis is performed on the collected monitoring data. The analysis identifies the driving behavior of the vehicle 12 for at least one driving event. The analysis is then used to compare with a model of expected vehicle behavior. Based on the vehicle's driving behavior and any deviations from the expected behavior, step 804 identifies whether there has been a risky driving event. If the vehicle 12 is behaving in an unexpected, unknown, or erratic manner, the vehicle may be malfunctioning.

[0088] A dangerous driving event is identified by comparing the detected automated driving behavior to a model of the automated vehicle's expected driving behavior for a particular driving event. If the detected behavior deviates from the model, it is likely that the vehicle has experienced a dangerous driving event. The deviation is compared to a threshold for the driving event to determine whether the driving event should be classified as dangerous.

[0089] Generally, a model of expected driving behavior is a rule or set of rules for responding to a driving event. For a driving event, such as when a vehicle is driving in an area with a set speed limit, the model may be that the vehicle should travel at or below the set speed limit. For these types of events, where the model is a separate rule, a deviation can be determined as exceeding the speed limit. A threshold can then be set as a dangerous speed level depending on the speed limit, road type, and other factors. The threshold can be a percentage above the permitted speed limit. In other embodiments, a driving event includes complex situations that require modeling of vehicle behavior using simulation techniques and an AI engine. For example, to navigate a complex intermittent traffic system with multiple vehicles moving and temporary instructions to avoid portions of the road, the modeling capabilities of the AI ​​engine can be used to map expected vehicle responses. If the vehicle does something different and a deviation occurs, the processor may determine that a dangerous driving event has occurred depending on the threshold value.

[0090] Deviations may be quantified by comparison to a classification framework. In other words, the processor determines a scale for deviations that can be compared in a particular driving event. By creating a classification framework, deviations can be assigned: a normalized value that can be compared to a single unified threshold, a value that can be compared to predetermined or variable thresholds specific to the event, and / or a category that can be compared to a categorical threshold. Thus, a classification quantifies deviations and also includes thresholds. In some embodiments, multiple thresholds are provided, each corresponding to a different alert that the system will generate and provide to the user (discussed below).

[0091] While deviations are based on the vehicle's actions in a particular situation, the thresholds to which they are compared can be based on other criteria. In particular, thresholds can be viewed as a measure of how dangerous vehicle operation is before the monitoring system determines that the user needs to be alerted and resume manual control. Thus, the system must consider not only the type of driving event occurring, external factors, vehicle operation, and so forth, but also other factors that affect how quickly manual control can be regained and how quickly the user will be required to react to a threat if manual control is regained. Thus, thresholds are also based on one or more driving parameters, including the driver's reaction time; the vehicle's level of automation; vehicle conditions; road type; weather conditions; and one or more user settings. When thresholds are based on reaction time, the system uses internal monitoring sensors to determine how quickly the user can regain manual control based on current behavior. The user settings can refer to settings that indicate how critical the system should be to vehicle behavior.

[0092] If the vehicle 12 is behaving normally and expectedly when compared to the model, or is at least considered to be within the bounds of normal driving, i.e., no unsafe driving events or deviations are identified, processing returns to receiving and analyzing monitoring data at step 802. If the vehicle 12 exhibits erratic behavior that is categorized as an unsafe driving event, method 800 transitions to alert processing at step 510, such as alert process H of FIG.

[0093] As noted above, some unsafe driving events may be isolated, occasional occurrences, while others may indicate or be caused by vehicle malfunction. A non-limiting example of such a malfunction is an autonomous system operating vehicle 12 at a speed above the road speed limit for the current location. When process 800 of FIG. 8 is performed using surveillance monitoring system 40, analysis of the monitoring data in step 802 determines the speed limit for the current location based on the navigation metadata and the current vehicle speed 12 based on the received navigation and / or image data. A comparison reveals that the vehicle's current speed exceeds the speed limit for the current location. If this speed limit is consistently exceeded in a pedestrian area where the speed limit is 30 miles per hour, surveillance monitoring system 40 identifies such exceedance as inappropriate and concludes that there may have been a malfunction in vehicle 12. Surveillance monitoring system 40 then alerts the user using one or more alert modules according to an alert process. However, in a highway setting where the speed limit is 70 miles per hour, the surveillance monitoring system 40 may identify that the vehicle 12 is exceeding this speed limit, but may also identify that there is a heavily loaded vehicle in the inside lane to the vehicle 12, or that there is a vehicle attempting to change lanes. In these situations, a short-term or temporary exceedance of the speed limit is generally acceptable and would not be classified as a malfunction by the surveillance monitoring system 40. The surveillance monitoring system 40 may continue to monitor the current speed of the vehicle 12 at step 802 to identify if the exceedance continues, and if the exceedance of the speed limit is deemed unfounded, initiate alert processing at step 510.

[0094] Other examples of malfunctions include jerky movement of the vehicle 12 identified by the accelerometer, such as not coming to a complete stop at a stop sign, missing a roadside warning sign or alert, not stopping for crossing pedestrians, missing a turn, or drifting out of the correct lane on a highway. These are malfunctions in the control operation of the autonomous system. Malfunctions in the general operation of the vehicle 12 can also be identified by the surveillance monitoring system 40. For example, noises from the vehicle 12 can be recorded and analyzed to identify possible engine and / or tire and / or exhaust problems in the vehicle 12's main hardware. Similarly, accelerometers can be used to detect problems with the vehicle's suspension, such as frequent vertical acceleration caused by an uneven road surface indicating a lack of motion absorption.

[0095] A vehicle malfunction can be a systematic malfunction or a random malfunction. For example, if a sensor is not working or is returning faulty measurements to the autonomous system, the vehicle 12 may be functioning systematically in an unexpected manner for an extended period of time. A malfunction may be random and short-lived if a sensor is temporarily obscured or if a minor fault is quickly corrected. In analyzing its data, the surveillance monitoring system 40 can compare two or more incidents of vehicle behavior in response to comparable threats or events in the external environment to identify whether the malfunction is recurring or whether the error has been corrected. Thus, as new monitoring data is received, the analysis in step 802, which is ongoing processing, should determine a result based on the monitoring data instantaneously, but also over a period of time to better distinguish between random and systematic malfunction issues. The surveillance monitoring system 40 then acts on the identified fault.

[0096] If a systematic fault occurs, the alert given to the user in step 510 may be more urgent or of a higher level than if the fault were due to a random malfunction issue. For apparently random errors, the surveillance monitoring system 40 may continue to monitor the operation of the autonomous system in step 802, paying particular attention to possible malfunctions if a malfunction is again identified and escalates into a more serious, systematic malfunction.

[0097] It is important to note that a malfunction does not necessarily mean a change from conventional driving. A malfunction can also be caused by incorrect or missing information programmed into the vehicle's processing capabilities or autonomous system. An example of this type of malfunction is the vehicle not recognizing a particular type of road surface that requires the vehicle to reduce its speed, such as an uneven, temporary road surface or a cobblestone road surface, leading to an inappropriate reduction in vehicle speed when approaching these types of road surfaces. Another example of this type of malfunction is when road markings are not clear, such as in rural areas with weathered road surfaces. These occurrences are considered malfunctions even if the vehicle's operation under its own autonomous system is not impaired.

[0098] During the process of FIG. 8 , the surveillance monitoring system 40 determines that the vehicle 12 malfunctions due to a particular condition, such as an occurrence in the external environment or preceded by a sign from the vehicle 12 itself. For example, a common occurrence in the external environment is precipitation, such as snow. The vehicle may also produce a malfunction sign in the form of a repetitive noise that can be detected by the surveillance monitoring system 40. FIG. 9 provides a method F900 for monitoring these precursor conditions that could lead to a vehicle malfunction. The conditions are not limited to the current vehicle 12 but also include conditions affecting vehicles of the same type and / or semi-autonomous vehicles in general. That is, the surveillance monitoring system 40 may know the condition based on previous malfunctions of the current vehicle 12, malfunctions in other similar vehicles of the same type or level of automation, and / or known defects in semi-autonomous vehicles in general. By monitoring such conditions, the method F900 may also prevent the vehicle from malfunctioning under such conditions by alerting the user before the situation becomes too serious to take action.

[0099] Method 900 begins at step 902 by analyzing the monitoring data, looking for predetermined conditions in which the vehicle previously malfunctioned. The analysis at step 902 is performed to identify one or more predetermined conditions. The identification may include one or more of the following: accessing locally stored predetermined conditions; accessing an updated list of predetermined conditions from the monitoring system content provider server 42 periodically (at a predetermined interval, such as each time a new driving season begins or daily); and / or receiving a push data package containing an updated list of predetermined conditions from the monitoring system content provider server 42 to the surveillance monitoring system 40 when a connection to the content server is established over a communications network.

[0100] If a predetermined condition is identified, the surveillance monitoring system 40 establishes a severity of the condition and categorizes it accordingly, at step 904. The categorization at step 904 may include assigning a threat level the same or substantially similar to the assigned threat level, assigning a distinct category; categorizing based on an updated list of predetermined conditions; and / or categorizing based on data in a data package accompanying the updated list.

[0101] Simultaneously, a severity action level for the condition is also identified in step 906. In an embodiment, the severity action level is delivered to the surveillance monitoring system 40 from the surveillance system content provider server 42 along with receiving an updated list of predetermined conditions so that it is accessible and stored with the predetermined conditions. Alternatively, or additionally, the surveillance monitoring system 40 may access the action level from the surveillance system content provider server 42 whenever a condition is identified or an action level is calculated, and the action level may be variable depending on other parameters such as the location of the vehicle 12, the road surface, vehicle damage, or other parameters affecting operation of the vehicle 12 in autonomous mode.

[0102] Once the condition severity has been categorized and the action level accessed, the severity and action level are compared in step 908. If the condition severity is greater than or equal to the action level, the supervisory monitoring system 40 enters alert processing in step 910 and also monitors the condition and the vehicle's response to the condition in more detail, particularly in step 910. If the condition severity is not greater than or equal to the action level, the alert processing is not initiated and the condition is monitored in more detail in step 910. After these steps, the supervisory monitoring system 40 continues analyzing the monitored data, particularly the condition and other data collected by the supervisory monitoring system 40, by returning to step 902.

[0103] In some embodiments, the surveillance monitoring system 40 alerts the user that a predetermined condition has been identified and is being monitored at the start of the method regardless of the severity of the condition.

[0104] In addition to monitoring malfunctions of the current vehicle 12, this method may also be used to identify other vehicles that pose a risk to the current vehicle 12 in certain conditions. The predetermined conditions may comprise conditions that pose a higher risk to manually operated vehicles, such as icy conditions, or conditions that pose a significant risk to automatically operated vehicles operating with a particular operating system because they have previously malfunctioned under the same conditions.

[0105] Autonomy in road vehicles is expected to be effective in preventing many road traffic incidents. However, it is still possible that an autonomously acting vehicle may be the sole part of an incident or may involve other semi-autonomous or autonomous vehicles, or pedestrians, or manually operated vehicles. The surveillance monitoring system 40 provides important oversight when an incident occurs.

[0106] Process G408 of Figure 10 is performed to ensure that events are properly recorded, particularly for insurance purposes. In Figure 10, data is analyzed to identify events in step 1002. The term "event" is intended to include at least collisions, and optionally near misses and sudden malfunctions. Collisions and near misses can be identified based on accelerometer data from surveillance monitoring system 40 combined with image and noise data from surveillance monitoring system 40.

[0107] If an event is identified, the surveillance monitoring system 40 accesses all data deemed relevant to the event in step 1004. In particular, the device accesses historical data leading up to the event by utilizing a buffer in which data is collected and then stored for a predetermined period of time. The surveillance monitoring system 40 then analyzes the event and its cause in step 1006. In some embodiments, step 1006 is optional, and the analysis is performed at a later stage or not at all.

[0108] The data and any analysis performed are sent to a more secure, permanent location in step 1008. This may include storing the data in a memory location within the surveillance monitoring system 40 and / or communicating it from the surveillance monitoring system 40 to a remote server and database over a communications network for later access. Once the data is secured, the surveillance monitoring system 40 alerts the user in step 1010 of the identity of the event and that it has been securely stored.

[0109] In some embodiments, the method also includes performing a check to identify injuries to the user. If an event is identified in step 1002, and if one or more predetermined emergency conditions are met that indicate the event is serious, the surveillance monitoring system 40 may assess the user's health and automatically alert emergency services that an event occurred, what the event was, how many users were in the vehicle, where the event occurred, and other pertinent information to assist the emergency services. An automatic alert of emergency services may also be provided if the event is a collision with a pedestrian.

[0110] While each of the above-described methods of FIGS. 5 through 10 is independent of the others, it is understood that different elements of each process can be combined or processed simultaneously. In some cases, processes are performed hierarchically based on available data, or alerts are prioritized in response to current concurrent processes. For example, if a vehicle malfunction likelihood parameter is more urgent for the user than an upcoming intersection, a process that provides an alert to the user based on a vehicle malfunction can be prioritized. In some embodiments, if multiple threats or potential malfunctions are identified, a threat or malfunction likelihood parameter can be assigned to each, and alerts can be prioritized according to the likelihood parameter. Instead of a likelihood parameter, a severity parameter of the potential consequences of ignoring the threat or malfunction can be assigned. In some embodiments, the likelihood parameter and the severity parameter can be combined.

[0111] Alerts to the user may be delivered in any of several different ways. In surveillance monitoring system 40, the alert system includes an alert module that determines which hardware to alert the user with. In particular, if the device is a mobile device 50, it is envisioned that the alert module is connected to: one or more loudspeakers of surveillance monitoring system 40, a display or interface of surveillance monitoring system 40, a vibration generator of surveillance monitoring system 40, one or more device camera flashes, and / or a communications engine for communicating with external devices, such as a wearable, configured to alert the user using an external warning device, such as a vibration generator, loudspeaker, or display, or a light that flashes an alert.

[0112] When alerting the user through a display or loudspeaker, surveillance monitoring system 40 provides a warning message indicating what the user is expected to do, clearly identifying the threat, and / or providing a warning noise to return the user's attention to the road. Speech synthesis may be used to provide such warnings through the loudspeaker, and the user may vocally respond as they interact with surveillance monitoring system 40.

[0113] 11 shows an example alert process, H1100. The process dictates how the surveillance monitoring system 40 should alert the user and escalate the alert if the user does not take action.

[0114] The alert process 1100 begins with an appropriate alert level for the identified threat, event, behavior, or determined condition, and an initial alert level n is set in step 1102. The alert level may be set to 0 at the start of an operating period.

[0115] Then, in step 1104, a maximum alert level appropriate for the threat, n max The maximum alert level is the alert level the system can escalate to if the user does not take action on the previous alert.

[0116] n and n in steps 1100 and 1102 maxOnce n is determined, an alert according to alert level n is sent to the user via surveillance monitoring system 40 or a connected device in step 1104. The alert may be specific to the corresponding threat type. For example, alert level n may be used to determine parameters of the alert, such as its duration, loudness or brightness, number of repetitions, or how many different types of alerts to use at one time. If a low-level threat, such as a traffic jam, was previously identified, i.e., if the n value is low, the alert may be a low-level alert, such as simply notifying the user at a standard volume through a loudspeaker that a threat has been identified. If a high-level threat, such as irregular behavior of a nearby vehicle, i.e., if the n value is high, surveillance monitoring system 40 may immediately alert the user of the identified threat repeatedly through both the screen and the loudspeaker at a high volume and brightness.

[0117] Once the alert is sent in step 1104, the surveillance monitoring system 40 analyzes data received since the alert in step 1106 to identify new user behavior, where the surveillance monitoring system 40 collects data to identify whether the user responded to the alert.

[0118] The surveillance monitoring system 40 determines, in step 1108, whether user behavior identifies a response to the alert. The response to the alert may depend on the threat type or the alert level. In some embodiments, if the threat is particularly severe, the user may be asked to immediately return to manual control of the vehicle 12. In other situations, the user's attention to the road may be a sufficient response for the surveillance monitoring system 40. In some embodiments, the required response may be a more positive user acknowledging the threat or alert, either verbally or by input into the surveillance monitoring system 40 interface.

[0119] If a response, or at least an appropriate response, is detected in step 1108, the surveillance monitoring system 40 records the alert and the alert level reached in step 1110 and continues analyzing the environment using analysis processes B through G, etc. in step 408. The surveillance monitoring system 40 will not send further alerts as long as the user continues to act as desired.

[0120] If no response or an inadequate response is detected in step 1108, the device checks whether the threat still exists in step 1112. If the threat has ended and is no longer present, the reason for the alert and the alert level are recorded in step 1110. If the threat still exists and has not been overcome or avoided, the surveillance monitoring system 40 determines in step 1114 whether n is greater than n max If n is not the maximum alert level, step 1116 increments n by 1, and returns to step 1104 to send a new alert to the user according to the incremented n value in step 1104. max If so, the alert is repeated as determined in step 1114, and if the alert is sufficiently serious, an escalation procedure is entered in step 1118. Escalation may involve an alert to an emergency system or remotely accessing the vehicle control system to command a pullover.

[0121] While the surveillance monitoring system is expected to be immediately usable by the user, in some embodiments the user may learn the system. Using machine learning algorithms, the surveillance monitoring system 40 may monitor the operation of the vehicle 12 in manual and automated driving modes and monitor examples of user actions. Using the learning data obtained from the user and vehicle 12 in these situations, the surveillance monitoring system 40 may subsequently adjust its monitoring and alert the user and vehicle 12.

[0122] In particular, the system may monitor threats and hazards and associate identified threats and hazards with a user's response reactions. The system may then monitor the user to extract an identifier indicating the user's awareness of a particular hazard. When functioning as a surveillance monitoring system, the system may monitor for hazards, access from memory a user identifier corresponding to the user's response to the hazard, analyze the user's monitoring data to determine whether the user identifier is present, and if not, analyze the data to alert the user to the hazard. This is particularly useful because different users are likely to have different habits and different body language. Such a system may also enable better detection of poor user behavior, such as when the user is intoxicated or fatigued and unable to continue driving.

[0123] Additionally, the system can also gain knowledge of threats based on the user's reaction to the situation and the associated avoidance actions to take. This can be applied to autonomous situations. By parsing data from many different drivers, a large amount of learning data can be collected to further refine the machine learning algorithms used in the surveillance monitoring system.

[0124] In some situations, using machine learning in combination with training data from users in manual control of a low-level semi-autonomous vehicle may allow for sufficient data collection for system training in a high-level autonomous vehicle, and vice versa.

[0125] While the above describes a user of the monitoring system 40 for an autonomous vehicle, the monitoring system 40 can also be used in manual control situations. The monitoring system 40 can also be used by the user to improve their driving by providing post-driving analysis so that the user is not distracted by alerts. If the user of the monitoring system 40 is a passenger rather than the driver of the vehicle, the monitoring system 40 can alert the user if the driver is driving dangerously. For example, in a taxi / ride-sharing service, the user can begin a new driving session and be privately warned of possible threats the driver has not considered. Thus, the feedback can be used to improve the quality of driving for the taxi / ride-sharing service. The resulting data can be converted into a driver rating system and loaded into a central database. Such a system can be linked to a website to provide independent driver evaluations, with ratings viewable by potential customers and employees alike. This has the advantage of being a "cross-platform rating" system that monitors Uber®, Lyft®, and other drivers in the exact same way, without the intervention of commercial ride-sharing companies.

[0126] Such a system could therefore also provide independent vehicle driving behavior monitoring for insurance purposes: while insurance premiums would naturally be lower for all drivers of automatically operated vehicles, some drivers may be safer and their premiums may be even lower.

[0127] The use of mobile devices 50, such as smartphones, also opens up more possibilities for vehicle-to-vehicle communication. A threat identified by one device at a particular location can be communicated to devices in other vehicles passing in the opposite direction, allowing for further threat preparedness. Such communication can be achieved over a communications network or with a more local communications protocol, such as Bluetooth®.

[0128] It is understood that the term "interface" as used herein is a broad term that encompasses several different possible embodiments. For example, in one embodiment, the interface may be a user interface, such as a touchscreen of a mobile communication device that functions as a monitoring device. In another embodiment, the interface may be a screen and keyboard or alternatively a display and interaction actuators, such as buttons, through which a user can enter commands. In a further embodiment, the user interaction interface may be provided on a device that is physically separate from the mobile monitoring device but functionally and operationally linkable to the monitoring device through an interface.

Claims

1. a portable electronic surveillance monitor device that provides an in-vehicle user alert system as to how a semi-autonomous vehicle is being automatically driven by an autonomous system during a driving period, the portable electronic surveillance monitor device being removably and stably mountable to the vehicle and independent of the autonomous system; a sensor set including at least one sensor that senses an external environment outside the vehicle and movement of the vehicle within the external environment; a user interface for receiving user input commands and delivering warning alerts; a processor operably connected to the sensor set and the user interface; the set of sensors is configured to monitor automated operation of the semi-autonomous vehicle within the external environment during the driving period and generate sensor data indicative of driving events occurring during the driving period regarding automated driving behavior of the vehicle relative to the external environment during the automated operation; the processor: processing the sensor data during the driving period to compare the sensor data indicative of a driving event with a model of expected autonomous vehicle driving behavior during a particular driving event; Identifying whether the sensor data indicative of a driving event deviates from the expected driving behavior of the automated vehicle by more than a threshold, the automated driving behavior deviating by more than the expected automated driving behavior including a dangerous driving event; a portable electronic surveillance monitor device configured to, if a risky driving event is identified, issue the warning alert through the user interface to alert a driver to the occurrence of the risky driving event.

2. The portable electronic surveillance monitoring device of claim 1 , wherein the at least one sensor includes a proximity sensor, the proximity sensor including at least one of an infrared sensor, a camera, and / or an ultra-wideband sensor.

3. 3. The portable electronic surveillance monitoring device of claim 1 or 2, wherein the set of sensors includes at least one external weather monitoring sensor.

4. A portable electronic surveillance monitor device as described in any one of claims 1 to 3, further comprising a local wireless communication link to a personal communication device providing the user interface to the portable electronic surveillance monitor device.

5. 5. The portable electronic surveillance monitor device of claim 1, wherein the set of sensors includes at least one position sensor, the at least one position sensor including a gyroscope, a magnetic sensor, an altimeter, a geolocation sensor, or an accelerometer.

6. The portable electronic surveillance monitoring device of claim 1 , wherein the set of sensors includes an audio sensor and the sensor data includes an audio signal.

7. 7. The portable electronic surveillance monitor device of claim 1, wherein the user interface includes a touchscreen and a loudspeaker.

8. 8. The portable electronic surveillance monitor device of claim 1, wherein the user interface includes a projector configured to project an image onto a surface of the vehicle to create a head-up display.

9. 9. The portable electronic surveillance monitor device of claim 1, wherein the portable electronic surveillance monitor device is a communications device including a wireless communications engine for communication with a remote server, the wireless communications engine configured to receive information about the external environment in which the vehicle is traveling.

10. 10. The portable electronic surveillance monitor device of claim 1, further comprising an artificial intelligence (AI) engine configured to operate as a neural network to learn and model autonomous driving behavior of the vehicle, the processor operably connected to the AI ​​engine.

11. 11. The portable electronic surveillance monitor device of claim 10, wherein the AI ​​engine includes a neural network trained to model expected vehicle driving behavior.

12. 12. The portable electronic surveillance monitor device of claim 11, wherein the neural network was trained using sensor data collected from manual and / or automated operation of the vehicle prior to a current driving period.

13. 13. The portable electronic surveillance monitor device of claim 12, wherein the sensor data collected prior to the current driving episode is data that has been identified as having been sensed during one or more driving episodes in which the number of recognized unsafe driving events is zero.

14. 14. The portable electronic surveillance monitor device of claim 11, wherein the AI ​​engine is configured to generate the model of expected autonomous vehicle driving behavior during the particular driving event based on the neural network and sensor data.

15. the processor determines the threshold for the particular driving event; if the comparison between the detected automated driving behavior and the model of expected automated vehicle driving behavior for the particular driving event indicates that a deviation has occurred; 15. A portable electronic surveillance monitor device according to any preceding claim, configured to compare the deviation with a threshold value and determine whether the deviation exceeds the threshold value.

16. 16. The portable electronic surveillance monitor device of claim 15, wherein the threshold is determined based on the driving event and at least one other parameter selected from the group consisting of: the driver's reaction time; the vehicle's level of automation; the vehicle's condition; road type; weather conditions; and one or more user settings.

17. 17. The portable electronic surveillance monitor device of claim 16, wherein the at least one other parameter includes the driver's reaction time, the set of sensors includes at least one sensor that senses an interior environment of the vehicle, and the processor is configured to determine the driver's reaction time based on current and / or historical sensor data sensed from the sensor that senses the interior environment of the vehicle.

18. 18. The portable electronic surveillance monitor device of claim 16 or 17, wherein the driving event includes a vehicle maneuver and the threshold is based on one or more of vehicle speed during the maneuver, vehicle braking during the maneuver, and vehicle steering angle during the maneuver.

19. A portable electronic surveillance monitor device as described in any one of claims 16 to 18, wherein the semi-autonomous vehicle includes a first vehicle, the driving event includes an interaction with a second vehicle, and the threshold is based on one or more of: the speed of the first or second vehicle during the interaction period, braking during the interaction period, the proximity of the second vehicle, the direction of movement of the second vehicle, the location of the second vehicle, whether the second vehicle is perceived to be operating or capable of operating automatically, and / or the behavior of the second vehicle.

20. the processor: determining a classification framework for the particular driving event, the classification framework including one or more classification values; assigning a normalized value to the deviation of the detected automated driving behavior from the expected automated driving behavior based on the classification framework; 20. A portable electronic surveillance monitor device as claimed in any one of claims 15 to 19, configured to compare the normalised value to a threshold value, the threshold value being one of the one or more classification values.

21. 21. The portable electronic surveillance monitor device of claim 20, wherein the classification framework includes a plurality of discrete category classification values.

22. 21. The portable electronic surveillance monitor device of claim 20, wherein the classification framework comprises continuous classification values.

23. 23. The portable electronic surveillance monitor device of claim 1, wherein multiple thresholds are provided to identify whether the sensor data indicative of a driving event for the automated driving behavior deviates beyond the expected automated vehicle driving behavior, each threshold corresponding to a different warning alert.

24. 24. The portable electronic surveillance monitoring device of claim 1, wherein the set of sensors includes at least one sensor for sensing an interior environment of the vehicle.

25. 25. The portable electronic surveillance monitor device of claim 24, wherein the set of sensors is further configured to monitor an interior environment of the vehicle during the driving period and generate sensor data indicative of a current state of attention of the driver during the driving period.

26. the processor: determining a required state of attention of the driver for current operation of the semi-automated vehicle within the external environment; comparing the current attention state of the driver with a desired attention state of the driver; 26. The portable electronic surveillance monitoring device of claim 25, configured to generate the warning alert if the current attention state deviates from the desired attention state by more than a threshold value.

27. 27. The portable electronic surveillance monitor device of claim 26, wherein the required attention state is determined based on one or more vehicle parameters.

28. 28. The portable electronic surveillance monitor device of claim 27, wherein the one or more vehicle parameters include a level of automation of the vehicle, a vehicle speed, a vehicle occupancy level, and / or a quality of automated vehicle operation.

29. 27. The portable electronic surveillance monitoring device of claim 26, wherein the required state of attention is determined based on one or more external environmental parameters.

30. 30. The portable electronic surveillance monitor device of claim 29, wherein the one or more external environmental parameters include road type, road quality, traffic density, weather type, a classification of how urban or rural the external environment is, the driving behavior of other vehicles in the vicinity, and / or the presence of the one or more dangerous driving events and / or other threats.

31. 31. The portable electronic surveillance monitor device of claim 1, wherein the processor is configured to determine a time point until manual resumption of control of the vehicle is required when a deviation beyond the expected driving behavior of the autonomous vehicle is identified, and to generate the warning alert no later than said time point.

Images