Psychomotor vigilance test for people tasked with monitoring autonomous vehicles

Abstract

When a person is tasked with monitoring a vehicle 200 operating in an autonomous driving mode, assessing the likelihood that the person experiences a fatigue event can include receiving a set of response times for a psychomotor vigilance test administered to the person. The test can include a plurality of trials involving the person lifting a finger from a user input device. It can be determined whether the person passes or fails each of the plurality of trials. A model 710 trained using data from previous psychomotor vigilance tests administered to the person can be identified for the person. The determination of whether the person passes or fails each of the plurality of trials can be input into the model to determine a value representative of the likelihood of a fatigue event. An intervention response can be initiated based on the value.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 598,752, filed October 10, 2019, and is a continuation of that U.S. Patent Application, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] This application relates to a psychomotor alertness test for a person tasked with monitoring autonomous vehicles. Background Technology

[0004] Autonomous vehicles, such as those that do not require a human driver when operating in autonomous driving mode, can be used to assist in transporting passengers or goods from one location to another. Testing of these vehicles typically involves a "test driver" tasked with monitoring the autonomous vehicle to ensure its safe operation. For example, it can be expected that a person will monitor the vehicle and its environment while it is operating in autonomous driving mode, and take control of the vehicle in case of an emergency or other such situation. Monitoring such vehicles is known to increase human susceptibility to fatigue when it is caused by sleep deprivation, poor sleep quality, fatigue resulting from monitoring itself, or the interaction of these contributing factors.

[0005] The psychomotor alertness test (PVT) can be used to assess a person's current state of consciousness by observing patterns in their reaction times. For example, once a person recognizes a displayed image or message or the playback of specific audio, the PVT might require them to perform a specific action, such as pressing a button. Current PVTs have demonstrated that single-group PVT assessment criteria can be used to predict the future task performance (such as driving performance) of a group of fatigued individuals. However, individuals vary greatly in their baseline PVT performance, which may make existing PVTs less sensitive to fatigue responses in some individuals and oversensitive to fatigue responses in others. This could, in turn, lead to group-level PVT pass / fail criteria that produce false negatives for some and false positives for others. Summary of the Invention

[0006] One aspect of this disclosure provides a method for assessing the likelihood of a person experiencing a fatigue event, the person being tasked with monitoring a vehicle operating in an autonomous driving mode. The method includes: receiving, by one or more server computing devices, a set of response times for a psychomotor alertness test administered to the person, the test comprising a set of trials; determining, by the one or more server computing devices, whether the person passed or failed for each trial in the set of trials; identifying, by the one or more server computing devices, a model associated with the person, the model being trained using data from previous psychomotor alertness tests administered to the person; inputting the determination of whether the person passed or failed for each trial in the set of trials into the model by the one or more server computing devices to determine a value representing the likelihood of a fatigue event; and initiating an intervention response by the one or more server computing devices based on the value.

[0007] In one example, the method further includes determining the intervention response by comparing the value to one or more thresholds. In another example, the method further includes determining a test score based on whether the person passed or failed for each trial in the set of trials, and the result includes the score. In this example, the score represents the person's pass rate or failure rate. In another example, the method further includes inputting the date and time information of the set of trials into the model to determine the value. In another example, the method further includes inputting shift information about the test person's shifts for monitoring vehicles at relative points in time into the model to determine the value. In another example, the method further includes inputting the amount of time since the test person's last rest into the model to determine the value. In another example, the method further includes inputting information about the test person's circadian rhythm into the model to determine the value. In another example, the method further includes determining a threshold time amount based on whether the person is left-handed or right-handed, and determining whether the person passed or failed for each trial in the set of trials is based on said threshold time amount. In another example, the method further includes using a model of how response time changes over time due to muscle fatigue to determine a threshold time amount, and determining whether the person passes or fails for each trial in the set of trials based on the threshold time amount. In another example, the method further includes using a model of how the person's response time changes over time to determine a threshold time amount, and determining whether the person passes or fails for each trial in the set of trials based on the threshold time amount. In another example, the method further includes using a model of how the response time of people in similar situations changes over time to determine a threshold time amount, and determining whether the person passes or fails for each trial in the set of trials based on the threshold time amount. In another example, the method further includes determining the threshold time amount based on the orientation of the user input device used to administer the set of trials, and determining whether the person passes or fails for each trial in the set of trials based on the threshold time amount. In another example, the method further includes determining the threshold time amount based on the orientation of the user input device used to administer the set of trials, and determining whether the person passes or fails for each trial in the set of trials based on the threshold time amount. In another example, the method further includes, once an intervention response is identified, sending one or more signals to one or more of the vehicle's computing device, the PVT system applying the test, and the computing device of the remote assistance operator to initiate the identified intervention response. In this example, the signals include instructions for the testing system to apply a second test. Additionally, the signals include instructions for the vehicle to stop operating in autonomous driving mode. Furthermore, the signals include instructions for the vehicle to prevent a person from operating the vehicle in manual driving mode. Additionally or alternatively, the signals enable the person to communicate with the remote assistance operator.

[0008] Another aspect of this disclosure provides a method for training a model to estimate the probability of a fatigue event in a person tasked with monitoring a vehicle operating in an autonomous driving mode. The method includes: one or more server computing devices accessing results from multiple sets of psychomotor alertness tests administered to the person at different time points, the results including the person's response time; one or more server computing devices determining a score for each set of tests based on the results; one or more server computing devices accessing information from a remote monitoring system identifying estimates of the person's fatigue levels at different time points; one or more server computing devices determining whether the information indicates the person has experienced one or more fatigue events; and one or more server computing devices using the determined scores and the determination of whether the information indicates the person has experienced one or more fatigue events to train a personalized model for that person, such that when data from a set of tests is input into the model, the model outputs a value indicating the probability that the person has experienced a fatigue event.

[0009] Another aspect of this disclosure provides a method for applying a psychomotor alertness test to a person tasked with monitoring a vehicle operating in an autonomous driving mode. The method includes: providing visual stimuli to the person on a touch-sensitive display by one or more processors while the person rests their finger on the display; determining, based on feedback from the touch-sensitive display, a time point at which the finger is removed from the display by one or more processors; measuring, by one or more processors, an amount of time between the time of providing the visual stimulus and said time point for the person to remove their finger from the display; and transmitting this time amount to a remote computing system by one or more processors to determine the likelihood of a fatigue event while the person is tasked with monitoring the vehicle.

[0010] In one example, each stimulus and duration corresponds to a single trial, and the test comprises multiple trials. In another example, the method further includes, prior to providing visual stimuli, instructing the person via a display to place their finger on the display by one or more processors. Attached Figure Description

[0011] Figure 1 This is an example of a PVT system.

[0012] Figure 2 This is a functional diagram of an example vehicle according to an exemplary embodiment.

[0013] Figure 3 These are example exterior views of a vehicle based on various aspects of this disclosure.

[0014] Figure 4 These are example schematic diagrams of systems based on various aspects of this disclosure.

[0015] Figure 5 yes Figure 4 Example block diagram of the system.

[0016] Figure 6 These are example flowcharts based on various aspects of this disclosure.

[0017] Figure 7 This is an example block diagram representing the training of a model according to various aspects of this disclosure.

[0018] Figure 8 These are example flowcharts based on various aspects of this disclosure.

[0019] Figure 9 These are example flowcharts based on various aspects of this disclosure. Detailed Implementation

[0020] Overview

[0021] This technology relates to psychomotor alertness testing (PVT) for individuals tasked with monitoring the driving of a vehicle operating in autonomous driving mode. For example, it can be expected that a person will monitor the vehicle and its surroundings while the vehicle is operating in autonomous driving mode, and take control of the vehicle in case of an emergency or other such situation. Monitoring such a vehicle is known to increase a person's susceptibility to fatigue when fatigue is caused by sleep deprivation, poor sleep quality, fatigue induced by the monitoring itself, or the interaction of these contributing factors. To test a person's current state of consciousness through patterns in their reaction time, the PVT may require the person to perform specific actions, such as pressing a button or screen, or lifting a finger away from a button or screen once the person recognizes a displayed image or message or plays specific audio. Current PVTs have demonstrated that single-set PVT evaluation criteria can be used to predict future task performance (such as driving performance). However, individuals vary greatly in their baseline PVT performance, which may make existing PVTs less sensitive to responses from some individuals and oversensitive to responses from others. This, in turn, may lead to group-level PVT pass / fail criteria that produce false negatives for some and false positives for others.

[0022] The testing system may include output devices, such as touchscreens and / or speakers, and user input devices. The user input device may connect to a computing device that measures the time between the time it takes for information to be displayed on the output device and the time it takes for a person to complete the test (e.g., by tapping on the touchscreen with a finger or alternatively by lifting a finger from the touchscreen). For example, the computing device may provide a stimulus, such as displaying information on the screen, and start a timer or counter. When a person's finger is removed from the touchscreen, the counter can be used to determine how long the user input device takes to register the person's response for the test. This can correspond to a response time, which essentially takes into account both reaction time (the time it takes for a person to recognize a stimulus) and physical movement time (the time required for a person to move his or her finger). The response time can then be compared to some pass / fail criterion (such as a threshold time measure) to determine whether the person passes or fails the test.

[0023] As described above, the testing system can be used in a vehicle with an autonomous driving mode. Therefore, the testing system may be able to communicate directly or indirectly with various remote computing devices and with the vehicle's computing devices.

[0024] To address the aforementioned drawbacks associated with false positives (or, more precisely, failures that shouldn't actually be considered failures), the features described herein can reduce errors caused by using movement time counts as response time and by creating personalized pass / fail criteria (which take into account errors caused by differentiating baseline performance between individuals). For example, instead of simply monitoring button or screen presses, each trial of the test can be configured to involve finger lifts. In this respect, each trial of the test might begin with a person's finger resting on a touchscreen or other touch-sensitive device.

[0025] People can perform multiple tests at different times of the day during different shifts. The testing system can send reaction times to a remote server computing device. In some cases, multiple trials from a single test can be scored. Scoring can be based on response time.

[0026] This information can be combined with other signals. For example, in-vehicle or remote monitoring systems can be used to determine a person's fatigue. As another example, the testing system may include options for a person to self-report his or her fatigue level on the same or different assessment scales.

[0027] Results of individual PVTs, along with other signals, can be used to train a model. This model can be a regression or statistical model trained by one or more server computing devices. PVT data can include the time points and dates relative to the shifts performed by individuals conducting a series of PVTs, as well as both response times and / or scores. This PVT data can be used as training input. Data from in-vehicle or remote monitoring systems and / or self-reported data can be used to determine whether a set of PVTs corresponds to a fatigue event (e.g., loss of concentration or even falling asleep). These determinations (fatigue events or non-fatigue events) can be used as training output.

[0028] A model can be trained such that new PVT data and points about his or her shift will provide an estimate of a person's current and / or future fatigue state. The more PVT data and other signals used to train the model, the more accurate the estimate of the probability that a person will experience a fatigue event. The model can then be associated with the person, for example, using an identification code or other information, and saved for later retrieval.

[0029] The timing and frequency of tests applied by the testing system can vary depending on the circumstances. To apply a test, when such a situation occurs, the vehicle's computing device can send a signal to the testing system. The testing system can test the person by applying a set of tests and evaluating their response times to those tests as described above. The response times of the tests, along with date and time information, shift information regarding the person's relative shift points, and any additional information discussed above (if available), can then be sent to and received by a remote server computing device.

[0030] The remote server computing device can then use the reaction time to determine whether the person passed or failed for each trial in the set of tests. This may also include determining the scores for the trials in the set. A model associated with the person can be identified, as described above, which is trained using data from previous tests administered to the person, as well as other signals as described above. The determination of whether the person passed or failed for each PVT in the set, or alternatively, the response time, can be input into the model to determine a value representing the probability of a fatigue event.

[0031] To identify and initiate an intervention response, the value can then be compared to a threshold or multiple thresholds to determine whether and what type of intervention response to take. Once an intervention response is identified, the server computing device can send a signal to the vehicle's computing device, the PVT system, and / or the remote assistance operator to initiate the identified intervention response. Various different types of intervention responses can be employed.

[0032] The features described in this paper can provide a reliable and effective system for identifying potential fatigue events in individuals tasked with monitoring the driving of vehicles operating in autonomous driving mode. For example, because the model is trained specifically for individuals or similar situations, it is more likely to identify anomalous situations. Furthermore, this improves the system's effectiveness and reduces false positives because the threshold time used to determine whether a person passes or fails for a given PVT can be adjusted to suit the specific individual or similar situation. In other words, the threshold time can be set in a way that considers individual variation (e.g., due to muscle fatigue).

[0033] Example System

[0034] The test system may include one or more computing devices 110 having one or more processors 120 and a memory 130 storing instructions 132 and data 134. The memory 130 stores information accessible by the one or more processors 120, including instructions 132 and data 134 that can be executed or otherwise used by the processors 120. The memory 130 may be any type of memory capable of storing processor-accessible information, including computing device-readable media or other media that store data readable by electronic devices, such as hard disk drives, memory cards, ROM, RAM, DVDs or other optical discs, and other writable and read-only memories. Systems and methods may include different combinations of the foregoing, whereby different portions of instructions and data are stored on different types of media.

[0035] Instruction 132 can be any set of instructions to be executed directly (such as machine code) or indirectly (such as scripts) by a processor. For example, instructions can be stored as computing device code on a computing device-readable medium. In this regard, the terms "instruction" and "program" may be used interchangeably herein. Instructions can be stored in object code format for direct processing by a processor, or in any other computing device language, including collections or scripts of independent source code modules that are interpreted on demand or pre-compiled. The function, methods, and routines of instructions are explained in more detail below.

[0036] Processor 120 can retrieve, store, or modify data 134 according to instruction 132. For example, although the claimed subject matter is not limited to any particular data structure, the data can be stored in a computing device register, stored as a table with multiple different fields and records in a relational database, stored as an XML document, or a flat file. The data can also be formatted in any computing device-readable format.

[0037] One or more processors 120 can be any conventional processor, such as a commercially available CPU or GPU. Alternatively, one or more processors can be a dedicated device such as an ASIC or other hardware-based processor. Although Figure 1 While the processor, memory, and other components of computing device 110 are shown functionally as being housed in the same block, those skilled in the art will understand that a processor, computing device, or memory may actually include multiple processors, computing devices, or memories that may or may not be housed in the same physical housing. For example, memory may be a hard disk drive or other storage medium located in a housing different from that of computing device 110. Therefore, references to processors or computing devices will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0038] The computing device 110 may also include an output device 160 (such as a touchscreen and / or a speaker) and a user input device 150 (such as a touchscreen, button, or other touch-sensitive device). In this respect, the output device and the user input device may be the same device (e.g., a touchscreen). The user input device 150 may be incorporated into or connected to a computing device such as the computing device 110, which can measure the time between the time it takes for information to be displayed on the output device and the time it takes for a person to complete a test (e.g., by tapping on the touchscreen with a finger or alternatively by lifting a finger off the touchscreen). For example, the computing device 110 may provide a stimulus (such as displaying information on a screen) and start a timer or counter. When a person's finger is removed from the touchscreen, the counter may be used to determine how long the user input device 150 takes to register the person's response to the test. This may correspond to a response time, which actually takes into account both reaction time (the time it takes for a person to recognize a stimulus) and physical response time (the time it takes for a person to move his or her finger). The response time can then be compared to some pass / fail criterion (such as a threshold time amount) to determine whether the person passes or fails the test.

[0039] As an example, a single test can contain up to 60-70 trials, which together can take approximately 3 minutes to complete. For instance, the test can utilize an interstimulation interval (ISA) of 1-4 seconds, varying randomly between trials / stimuli. If the ISA is set to 1-3 seconds, there will be more trials within the same test (currently 3 minutes). Additionally, the test length can be increased or decreased, which will also increase or decrease the total number of trials in the test.

[0040] Test system 100 may also include communication system 140, which enables test system 100 to communicate with other computing devices. For example, the communication system may include wired and / or wireless connections (such as transmitters and receivers) that enable the test system to communicate with other computing devices. As an example, the communication system may enable the test system to use various protocols, including short-range communication protocols such as Bluetooth, Bluetooth LE, the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local area networks, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi, and HTTP, as well as various combinations thereof. Such communication can be facilitated by any device capable of transmitting data to and from other computing devices, such as modems and wireless interfaces.

[0041] As mentioned above, the testing system can be used in vehicles with autonomous driving modes. Figure 2 This is an example block diagram of vehicle 200, and Figure 3 This is an example view of vehicle 200. In this example, vehicle 200 is a vehicle with an autonomous driving mode and one or more other driving modes (such as semi-autonomous or manual driving modes). While certain aspects of this disclosure are particularly useful in combination with specific types of vehicles, the vehicle can be any type of vehicle, including but not limited to cars, trucks, motorcycles, buses, recreational vehicles, etc.

[0042] Go to Figure 2 The computing device 110 of the test system can communicate with one or more computing devices 210 in the vehicle. The one or more computing devices 210 may include one or more processors 220, a memory 230 storing instructions 232 and data 234, and other components typically found in general-purpose computing devices. These processors, memory, instructions, and data may be configured identically or similarly to those of processor 120, memory 130, instructions 132, and data 134.

[0043] In one aspect, computing device 210 may be part of an autonomous control system capable of communicating with various components of the vehicle to control the vehicle in an autonomous driving mode. For example, returning to Figure 2 The computing device 210 can communicate with various systems 250, 260, and 270 via wired or wireless connections. For example, these systems may correspond to deceleration systems, acceleration systems, steering systems, routing systems for determining the route the vehicle follows between two or more locations, planning systems for planning trajectories, positioning systems, perception systems for detecting objects in the vehicle environment, etc. The computing device can use these systems to control the vehicle 200 in autonomous driving and semi-autonomous driving modes.

[0044] Vehicle 200 may also include a remote monitoring system 280. This system may include hardware features such as a video camera, microphone, and speaker, as well as one or more computing devices and communication systems, which may be configured identically or similarly to computing device 110 and communication system 140. The hardware features may be used to enable a remote operator to “inspect” the test driver and to enable two-way communication between the remote operator and the test driver.

[0045] Turning Figure 3 As an example, vehicle 200 includes a roof-top housing 310 and a dome-shaped housing 312, which may include LIDAR sensors as well as various camera and radar units. Additionally, housing 320 located at the front of vehicle 200 and housings 330 and 332 located on the driver's and passenger's sides of the vehicle may each house LIDAR sensors. For example, housing 330 is located in front of the driver's door 360. Vehicle 100 also includes housings 340 and 342 for cameras and / or radar units also located on the top of vehicle 200. Additional radar units and cameras (not shown) may be located at the front and rear ends of vehicle 200 and / or at other locations along the top or roof housing 310.

[0046] The computing device 210 may include a communication system 240 that is the same as or similar to the communication system 140. The communication system enables the computing device 210 to communicate with other devices located remotely from the vehicle. In this way, information from the test system 100 can be transmitted to remote devices. Thus, the test system 100 can communicate directly or indirectly with the vehicle's computing device 210 and various remote computing devices (such as those used as part of autonomous vehicle services and other computing devices) via the vehicle's computing device.

[0047] Figure 4 and Figure 5 These are schematic and functional diagrams of example system 400, which includes multiple computing devices 410, 420, 430, 440 and a storage system 450 connected via network 460. System 400 also includes vehicles 200A, 200B, 200C, and 200D that can be configured identically or similarly to vehicle 200. Although only a few vehicles and computing devices are depicted for simplicity, a typical system could include significantly more.

[0048] like Figure 4 As shown, each of computing devices 410, 420, 430, and 440 may include one or more processors, memory, data, and instructions. Such processors, memory, data, and instructions may be configured similarly to one or more processors 120, memory 130, data 134, and instructions 132 of computing device 110.

[0049] Network 460 and intermediate nodes can include a variety of configurations and protocols, including short-range communication protocols such as Bluetooth, Bluetooth LE, the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local area networks, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi, and HTTP, as well as various combinations thereof. Similarly, communication can be facilitated by any device capable of transmitting data to and from other computing devices, such as modems and wireless interfaces.

[0050] In one example, one or more computing devices 410 may include one or more server computing devices, such as a load-balanced server farm, that exchange information with different nodes in the network for the purposes of receiving data from other computing devices, processing data, and sending data to other computing devices. For example, one or more computing devices 410 may include one or more server computing devices capable of communicating via network 460 with computing device 210 of vehicle 200 or similar computing devices of other vehicles, as well as computing devices 420, 430, and 440. For example, each of vehicles 200A, 200B, 200C, and 200D may correspond to vehicle 200 and may be part of a fleet of autonomous vehicle services that can be dispatched to various locations by server computing device 410. In this respect, server computing device 410 (in conjunction with storage system 450) can be used as a dispatch system for autonomous vehicle services, which can be used to dispatch vehicles such as vehicle 200 and vehicle 200A to different locations for picking up and dropping off passengers. Additionally, server computing device 410 can use network 460 to send information to people such as 422, 432, and 442 and present the information to people such as 422, 432, and 442 on displays (such as displays 424, 434, and 444 of computing devices 420, 430, and 440). In this respect, computing devices 420, 430, and 440 can be considered as client computing devices.

[0051] like Figure 4As shown, each client computing device 420, 430, 440 may be a personal computing device intended for use by a person 422, 432, 442, and has all the components typically used in conjunction with a personal computing device, including one or more processors (e.g., a central processing unit (CPU)), memory for storing data and instructions (e.g., RAM and internal hard disk), displays such as displays 424, 434, 444 (e.g., a monitor with a screen, touchscreen, projector, television, or other device operable to display information), and user input devices 426, 436, 446 (e.g., a mouse, keyboard, touchscreen, or microphone). The client computing device may also include a camera for recording video streams, speakers, network interface devices, and all components for connecting these elements to each other.

[0052] While client computing devices 420, 430, and 440 may each comprise a full-size personal computing device, they may alternatively comprise mobile computing devices capable of wirelessly exchanging data with a server via a network such as the Internet. By way of example only, client computing devices may include mobile phones, or devices such as wirelessly enabled PDAs, tablet PCs, wearable computing devices or systems, or netbooks capable of accessing information via the Internet or other networks.

[0053] Each of the client computing devices can be a remote monitoring workstation used by an administrator or remote assistance operator (e.g., persons 422, 432, 444) to provide concierge or remote assistance services to the drivers of test vehicles 200A, 200B, 200C, 200D. For example, representative 442 can use remote monitoring workstation 440 to communicate via telephone calls or audio connections with persons through their respective client computing devices or vehicles 200A, 200B, 200C, 200D to ensure the safe operation of vehicles 100 and 100A and the safety of the test drivers, as described in further detail below. Although in Figure 4 and Figure 5 Only a few remote monitoring workstations 440 are shown in the diagram, but any number of such workstations can be included in a typical system.

[0054] Similar to memory 130, storage system 450 can be any type of computerized storage device capable of storing information accessible by server computing device 410, such as hard disk drives, memory cards, ROM, RAM, DVDs, CD-ROMs, writable and read-only memory. Additionally, storage system 450 can include a distributed storage system where data is stored on multiple different storage devices that may be physically located in the same or different geographical locations. Storage system 450 can be as follows: Figure 3 and Figure 4 As shown, it is connected to a computing device via network 460, and / or can be directly connected to or incorporated into any computing device 410, 420, 430, 440, etc.

[0055] Storage system 450 can be configured to store various types of information. This information may include the status of the fleet's vehicles (service journeys, passengers, etc.), current location, and expected future location. The information may also include data associated with specific test drivers. For example, each test driver identified in the storage system may be associated with: one or more models (or model parameter values ​​used with a common model for all test drivers), reaction time, score, date and time of PVT, shift information, fatigue information from a remote monitoring system and / or self-reported data, and additional information, as discussed further below.

[0056] Example Method

[0057] In addition to the operations described above and shown in the diagrams, various other operations will now be described. It should be understood that the following operations need not be performed in the exact order described below. Instead, various steps can be processed in different orders or simultaneously, and steps can be added or omitted.

[0058] As described above, to address the aforementioned drawbacks associated with false positives (or, more precisely, failures that should not actually be considered failures), the features described herein can reduce errors caused by counting movement time as response time and by creating personalized pass / fail criteria (which take into account errors caused by differentiating baseline performance between individuals). For example, instead of simply monitoring button or screen presses, each trial of the test can be configured to involve finger lift-off. For instance, if the person being tested keeps their finger off the screen sometimes by 1 inch and sometimes by 3 inches, the variation in movement time will be counted as a variation in reaction time, even though in both cases the person may have begun to react just as quickly. In this respect, each trial of the test can begin with the person's finger placed, rested, or otherwise positioned on a touchscreen or other touch-sensitive device. Thus, the initial conditions for each trial may be the same or nearly the same, since the movement distance remains constant across all tests and all the people being tested. In other words, there is virtually no variation between each trial or across many tests, except for the characteristics of the person being tested. Additionally, this arrangement may also be less muscle-intensive for some individuals, as lifting the finger while resting is easier than keeping it on the input device for testing. In other words, a person can focus on waiting for a stimulus, rather than keeping his or her finger in a specific position on the touchscreen.

[0059] A person, or more precisely, a test driver, can conduct multiple tests (PVTs) at different times of the day during different shifts (e.g., periods during which the test driver is expected to monitor the vehicle in autonomous driving mode, which may also include one or more rest periods). Test system 100 can send reaction times to a remote server computing device, such as server computing device 410. This data can be stored, for example, in storage system 450.

[0060] Figure 6 This is an example flowchart 600 for applying a psychomotor alertness test to a person tasked with monitoring a vehicle operating in autonomous driving mode, which can be executed by one or more processors of one or more computing devices (such as processor 120 of computing device 110 of test system 100). In box 610, a visual stimulus is provided on the touch-sensitive display for display while the person rests their finger on it. In box 620, based on feedback from the touch-sensitive display, the point in time at which the finger is removed from the display is determined. In box 630, the amount of time between the time the visual stimulus was provided and that point in time is measured for the person to remove their finger from the display. In box 640, the amount of time is provided to a remote computing system to determine the likelihood of fatigue events while the person is tasked with monitoring the vehicle. Each stimulus and amount of time corresponds to a single trial, and the test includes multiple trials. Additionally, the test may include providing the person with instructions to place their finger on the display before providing the visual stimulus.

[0061] In some cases, multiple tests from a single PVT can be scored by server computing device 410 and / or computing device 110, and then sent to server computing device 410. The scores can also be stored in storage system 450.

[0062] Scoring can be determined based on response time. In one example, if the threshold time is 300 milliseconds and the reaction time is less than 355 milliseconds, the test driver can be considered to have passed. If the response time is greater than 355 milliseconds, the test driver can be considered to have failed or his / her attention has lapsed. As another example, if the test driver's response time is measured within 100 milliseconds of displaying or playing the test stimulus or before displaying or playing the test stimulus, the test driver can also be considered to have failed or been distracted. In this respect, 100 milliseconds can also identify spurious response times, as 100 ms corresponds to the transmission limit of the human nervous system. In cases where spurious response times or "false starts" are prevalent, the test can be ignored and reapplied, or the test can simply be identified as a failure. The ratio of the number of passed or failed tests to the total number of tests in the test can be determined, representing the passenger's pass rate or failure rate. Therefore, this ratio can provide a score on a scale of 0 to 1. Other scoring methods can also be used.

[0063] This information can be combined with other signals. For example, remote monitoring system 280 can be used to determine the fatigue of a test driver. For example, as described above, remote monitoring system 280 may include one or more video or still cameras that provide remote assistance operators (such as people 422, 432, 442) to “check on” the test driver while he or she is monitoring a vehicle such as vehicle 200. For example, the remote assistance operator can identify signals such as the test driver “dozing off,” slowly closing his or her eyes, or not monitoring the road with his or her eyes. The same video or camera data can also be used by automated systems (such as in-vehicle monitoring systems) to look for similar signals. As another example, the testing system may include an option for the test driver to self-report his or her fatigue status on the same or different evaluation scales, such as entering values ​​on scales of 0 to 1, 0 to 10, 0 to 100, etc. This fatigue information may also be stored in storage system 450. As yet another example of other signals, a pattern of time-to-reset (TTR) may also indicate fatigue. In other words, the time a driver spends placing his or her finger back on the display before each task may increase over time. Therefore, fatigue can produce a longer TTR due to distraction caused by alertness.

[0064] Results from PVT and other signals from a specific test driver can be used to train a model to determine parameter values ​​for that particular test driver. Alternatively, the model can be trained for test drivers in similar situations, rather than for a specific test driver. For example, drivers whose baseline PVT performance (i.e., scores) are very similar to other drivers can be grouped together, and results from PVT and other signals from these drivers can be used to train a model. In this respect, the model can generate a set of parameters that is effective specifically for detecting fatigue in that group. The model can be a regression model, a neural network, a random forest, or any other predictive model that can produce numerical predictions on a scale. The model can be trained by one or more server computing devices, such as server computing device 410.

[0065] Figure 7 This is an example representation of training 710 for a specific test driver, but a similar approach can be used to train a model for test drivers in similar situations. PVT data for a specific test driver can be used as training input 720. PVT data may include the time and date of the shift for a specific test driver performing a series of PVTs, as well as both response time and / or score.

[0066] Data from a remote monitoring system and / or self-reported data for that particular test driver can be used to determine whether a set of PVTs corresponds to a fatigue event (e.g., loss of concentration or even falling asleep). For example, the PVT data can be correlated (e.g., manually) with other signals from two groups: those signals corresponding to subsequent fatigue events during the same shift, and those signals not corresponding to subsequent fatigue events during the same shift. These determinations (fatigue events or non-fatigue events) can be used as training output 730. Training can produce parameter values ​​for a model useful for predicting fatigue events for a particular test driver.

[0067] Alternatively, the model can be a statistical model that correlates PVT data with the probability of fatigue events for a specific test driver. This can be most useful when there is not a large amount of training data for a particular test driver.

[0068] A model can be trained such that new PVT data and time points related to his or her shift will provide an estimate of the test driver's current and / or future fatigue state. For example, by examining reaction time immediately before or just after the start of a shift, the model may be better able to predict the test driver's future fatigue (e.g., later during the shift). In some cases, the training data may also include additional information, such as the test driver's circadian rhythm, the time since the test driver's last rest (or "task time" corresponding to the duration the test driver spends continuously monitoring the vehicle), so that this information can also be used as input to the model to provide an estimate of the test driver's current and / or future fatigue state. This estimate can be a value, for example, on a scale of 0 to 1, representing the probability that the test driver will experience a fatigue event (e.g., loss of concentration or even falling asleep). The more PVT data and other signals used to train the model, the more accurate the estimate of the probability that the test driver will experience a fatigue event. The model can then be associated with the test driver, for example, using an identification code or other information, and stored in storage system 450 for later retrieval.

[0069] Figure 8 This is an example flowchart 800 for training a model to estimate the probability of fatigue events for a person tasked with monitoring a vehicle operating in autonomous driving mode, which can be performed by one or more server computing devices (such as one or more server computing devices 410). In box 810, results from multiple sets of psychomotor alertness tests administered to the person at different time points are accessed. The results include the person's response time. In box 820, a score for each set of tests is determined based on the results. In box 830, information from a remote monitoring system (or an onboard monitoring system) that identifies estimates of the person's fatigue levels at different time points is accessed. In box 840, it is determined whether this information indicates that the person has experienced one or more fatigue events. In box 850, a personalized model for the person is trained using the determined scores and the determination that the information indicates the person has experienced one or more fatigue events, such that when data from a set of tests is input into the model, the model outputs a value indicating the probability that the person has experienced a fatigue event. As described above, training can provide parameter values ​​for the model specific to the person. When the parameter values ​​are used, the model can output a value indicating the probability that the person has experienced a fatigue event.

[0070] The timing and frequency of tests applied by the testing system can vary depending on the circumstances. For example, tests can be initiated when the vehicle is stopped, such as at the beginning or end of a test driver's shift, or immediately before or after a test driver's rest period. Additionally, tests can be performed more frequently during night shifts than day shifts. To apply a test, the vehicle's computing equipment can send a signal to the testing system when this occurs. The rate of test application can be higher at the beginning or end of a shift. In this regard, the server computing equipment can access information about the vehicles monitored by each test driver, as well as information about the start and end of each test driver's shift. In other cases, other signals, such as those from remote assistance operators or automated monitoring systems, can determine that the test driver is experiencing or may experience fatigue events. Remote assistance operators or remote monitoring systems may then send signals to the testing system to apply the test.

[0071] The testing system can test a test driver by applying a set of tests and evaluating the test driver's response time to those tests as described above. The response time of the tests, along with date and time information, shift information regarding the relative points of the test driver's shift, and any additional information discussed above (if available), can then be sent to and received by a remote server computing device.

[0072] The remote server computing device can then use the reaction time to determine whether each test driver in the test group passes or fails. In this case, a failure (as described above) could indicate that the test driver is not sufficiently alert to monitor the autonomous vehicle. The server computing device can also determine the score for the test group. To do this, as described above, the remote server computing device can compare the reaction time to a threshold time and determine the ratio.

[0073] A model can be identified that is associated with a specific test driver, as described above, and is trained using data from previous tests administered to that test driver, as well as other signals as described above. For each PVT test driver in the group, a definitive outcome of pass or fail, or alternatively, response time, can be input into the model to determine a value representing the probability of a fatigue event. In other words, ratios, along with date and time information, shift information, and any additional information, can be input into the model for that specific test driver to assess the likelihood of a fatigue event by providing a value, as indicated above. Analyzing data at server computing device 410 provides the flexibility to quickly change logic, thresholds, etc., and also allows remote assistance operators, server computing devices, or others to request a given test driver to perform another test as soon as possible (e.g., at the next break). Additionally, the model can be sensitive to recent data aggregated at server computing device 410 (or storage system 450). For example, if the most recent data from remote monitoring system 280 also suggests fatigue, the same test performance level (e.g., score) may result in a higher probability of a fatigue event.

[0074] To identify and initiate an intervention response, the server computing device 410 can then compare the value with one or more thresholds to determine whether and what type of intervention response to take. The threshold or thresholds can be hand-tuned or selected depending on the desired accuracy and recall values ​​of the PVT system. In other words, the threshold can be lower (e.g., a lower value) to increase the amount or size of the intervention response, or higher (e.g., a higher value) to decrease the amount or size of the intervention response. Once an intervention response is identified, the server computing device can signal the vehicle's computing device, the PVT system, and / or the remote assistance operator to initiate the identified intervention response.

[0075] Various types of intervention responses can be employed. For example, intervention responses may include providing supportive options and, where applicable, task reassignment. For instance, the test driver may be offered another test or another set of trials, and may be able to contact one of the remote assistance operators who can interact with the test driver as described above, or even be relieved of the current shift's monitoring vehicle duties. In this regard, the server computing device 410 can use tests and models to automatically determine whether the test driver should be reassigned tasks before or even during their shift. In some cases, depending on the test driver's overall testing performance, the test driver may be automatically or arbitrarily assigned less safety-critical tasks, or the vehicle may even be prevented from controlling the vehicle in manual driving mode based on signals. For example, the test driver may also be assigned tasks associated with improved alertness, such as tasks with higher engagement or rest periods. If the test driver has some fatigue indicators but still below the lower threshold values ​​described above, additional onboard or remote monitoring can be added to improve fatigue detection should a fatigue event occur.

[0076] The ratings for specific test drivers can also be adjusted based on past performance and / or physical fatigue (which can be used as input to the model). For example, even when some test drivers are tired, they may have faster reflexes, while others may experience an increase in response time due to physical fatigue. In this respect, these test drivers may have the same reaction time, but different response times. Therefore, the threshold time used to determine whether a test driver passes or fails a particular test can be adjusted. To do this, a model can be generated to show how the reaction time of that test driver or test drivers in similar situations (same time of day, same shift, etc.) changes over time. This model could be a statistical regression model, which provides another way to separate performance variability due to fatigue from variability due to other factors. In another example, although the duration of each test in a PVT is very short (e.g., just lifting a finger), it does require physical movement, which will change over time due to cumulative fatigue (i.e., the test driver's finger muscles will become fatigued even if the test driver hasn't yet). Therefore, the threshold time used to determine whether a test driver passes or fails the test may increase over time depending on model fluctuations.

[0077] Additionally, depending on the touchscreen's orientation, the test may be more or less prone to physical fatigue. For example, if the touchscreen is oriented in an upright position, the test driver may be more susceptible to muscle fatigue because he or she must maintain an elevated position. As another example, a driver with a longer reach may experience physical fatigue faster than one with a shorter reach, which the driver may compensate for by tilting towards the touch-sensitive display or other user input device used to administer the test. In this way, data can also be included in the model to determine threshold time amounts.

[0078] Similarly, because the physical posture differs for left-handed and right-handed individuals, the variability may differ depending on the dominant hand of the test driver. In this regard, different models may be available for determining the threshold time amounts for left-handed and right-handed individuals. Likewise, the model for left-handed or right-handed individuals may be specific to that test driver, test drivers in similar situations (same time of day, same shift, etc.), or all test drivers.

[0079] Figure 9 This is an example flowchart 900 for assessing the likelihood of a person experiencing a fatigue event when tasked with monitoring a vehicle operating in autonomous driving mode. It can be executed by one or more server computing devices (such as one or more server computing devices 410). For example, in box 910, a set of response times from a psychomotor alertness test administered to the person is received. This test comprises multiple trials. In box 920, it is determined whether the person passed or failed for each trial in the set of trials. In box 930, a model associated with the person is identified. This model is trained using data from previous psychomotor alertness tests administered to the person. In box 940, the determination of whether the person passed or failed for each trial in the set of trials is input into the model to determine a value representing the likelihood of a fatigue event. In box 950, an intervention response is initiated based on this value.

[0080] The features described in this paper can provide a reliable and effective system for identifying potential fatigue events in individuals tasked with monitoring the driving of vehicles operating in autonomous driving mode. For example, because the model is trained specific to a particular person or situation-like individual, it is more likely to identify anomalous situations. For instance, if a particular person has an average score of 0.8 at a specific time on a day during the last three months, but once scores 0.7 (which could be one or more standard deviations outside that person's normal performance), the model can still indicate that the person may have experienced a fatigue event, even if a score of 0.7 is considered acceptable under broader criteria. Furthermore, this can further improve the system's effectiveness and reduce false positives because the threshold time amount used to determine whether the person passed or failed for a particular PVT can be adjusted to suit specific individuals or situation-like individuals. In other words, the threshold time amount can be set in a way that allows for consideration of individual variability (e.g., due to muscle fatigue).

[0081] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the above features can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be understood as illustrative rather than restrictive of the subject matter defined by the claims. Furthermore, the examples provided herein and clauses phrased with "such as," "comprising," etc., should not be construed as limiting the subject matter of the claims to the specific examples; rather, these examples are intended to illustrate only one of many possible embodiments. In addition, the same reference numerals in different figures can identify the same or similar elements.

Claims

1. A method for assessing the likelihood of a person experiencing a fatigue event, the person having the task of monitoring a vehicle operating in an autonomous driving mode, the method comprising: A set of response times is received by one or more server computing devices for a set of tests in the psychomotor alertness test (PVT) applied to the person, each of the set of response times having a reaction time and a physical movement time; One or more server computing devices determine whether a person passes or fails for each trial in the group of trials based on a threshold time amount, wherein the threshold time amount is determined to reduce the impact of physical travel time on determining whether a person passes or fails for each trial in the group of trials; One or more server computing devices identify a PVT model associated with a person from multiple PVT models associated with multiple corresponding persons, wherein each of the multiple PVT models is trained using data from previous PVT applied to that person; One or more server computing devices input the definitive result of whether the person passed or failed for each trial in this set of trials into the identified PVT model to determine a value representing the probability of a fatigue event; and An intervention response is initiated by one or more server computing devices based on the value.

2. The method of claim 1, further comprising determining an intervention response by comparing the value with one or more thresholds.

3. The method of claim 1, further comprising determining a PVT score based on whether the person passed or failed for each trial in the set of trials, wherein, The results include the scores.

4. The method of claim 3, wherein, The score represents the person's pass rate or failure rate.

5. The method of claim 1, further comprising inputting the date and time information of the group of trials into the identified PVT model to determine the value.

6. The method of claim 1, further comprising inputting shift information about the relative time points of the shifts of the PVT personnel for monitoring vehicles into the identified PVT model to determine the value.

7. The method of claim 1, further comprising: The amount of time since the last rest of the person in the PVT is input into the identified PVT model to determine the value.

8. The method of claim 1, further comprising inputting information about the PVT and the relationship between the person and his or her circadian rhythm into the identified PVT model to determine the value.

9. The method of claim 1, further comprising determining a threshold time based on whether the person is left-handed or right-handed.

10. The method of claim 1, further comprising: The threshold time amount is determined using a model of how the response time changes over time due to muscle fatigue.

11. The method of claim 1, further comprising: The threshold time amount is determined using a model of how the individual's response time changes over time.

12. The method of claim 1, further comprising: The threshold time amount is determined using a model of how response times change over time for people in similar situations.

13. The method of claim 1, further comprising: The threshold time was determined based on the orientation of the user input device used to apply the set of tests.

14. The method of claim 1, wherein, Initiating an intervention response includes sending one or more signals to the vehicle's computing devices, the PVT system applying the PVT, and one or more computing devices of a remotely assisting operator.

15. The method of claim 14, wherein, The one or more signals include instructions for applying a second PVT to the PVT system.

16. The method of claim 15, wherein, The one or more signals include instructions for the vehicle to stop operating in autonomous driving mode.

17. The method of claim 16, wherein, The one or more signals include instructions for preventing a person from operating the vehicle in manual driving mode.

18. The method of claim 15, wherein, The one or more signals enable the person to communicate with a remote assistance operator.

19. The method of claim 1, wherein, The one or more server computing devices are further configured to determine a threshold time amount using a model of how the person's response time changes over time, and wherein the one or more server computing devices are further configured to determine, based on the threshold time amount, whether the person passes or fails for each trial in the set of trials.

20. A system for assessing the likelihood of a person experiencing a fatigue event, the person having the task of monitoring a vehicle operating in an autonomous driving mode, the system comprising one or more server computing devices configured to perform the method as described in any one of claims 1-19.

21. The system of claim 20, further comprising a vehicle.

Images