USE OF STEER-BY-WIRE STEERING SYSTEMS TO APPRECIATE HUMAN VISUAL-MOTOR ABILITIES

Abstract

A method includes, while a vehicle is restricted to a parked state, presenting a visual tracking application on a display of the vehicle, wherein the visual tracking application includes a moving target user interface element (target UL element) and a tracking UI element, detecting, using a steer-by-wire (SbW) steering system of the vehicle, movements of a steering wheel by an operator of the vehicle, moving based on the detected movements of the tracking UL element in the visual tracking application, and determining a corresponding tracking error between a location of the moving target UL element in the visual tracking application and a location of the tracking UL element in the visual tracking application at multiple time points.The procedure also includes processing the corresponding tracking errors to estimate the operator's visuomotor skills.

Description

INTRODUCTION

[0001] The information provided in this section serves the purpose of providing a general overview of the context of the disclosure. Neither the work of the inventors currently named, to the extent described in this section, nor those aspects of the description that could not otherwise qualify as prior art at the time of filing, are expressly or implicitly recognized as prior art in relation to the present disclosure.

[0002] A steer-by-wire (SbW) steering system for a vehicle is a technology that replaces the mechanical links between a steering wheel and the front wheels of a vehicle with electronic signals. An operator inputs the desired steering angle via a sensor-equipped steering wheel, which transmits the information to a control unit. The control unit then commands the electric actuators to adjust the wheel angles according to the operator's input, the vehicle's speed, its stability, and road conditions. An SbW steering system can improve a vehicle's steering performance, fuel economy, and safety, as well as enable new features such as autonomous driving and variable steering ratios.

[0003] The present disclosure relates generally to the use of SbW steering systems for estimating human visuomotor skills. SUMMARY

[0004] One aspect of the disclosure provides a vehicle that includes a steer-by-wire (SbW) steering system, a steering wheel configured to control the SbW steering system in response to operator input, a display, data processing hardware, and memory hardware. The memory hardware communicates with the data processing hardware and stores instructions which, when executed by the data processing hardware, cause the data processing hardware to perform operations. The operations include restricting the vehicle to a parked state and, while the vehicle is restricted to the parked state, presenting a visual tracking application on the display, wherein the visual tracking application includes a moving target user interface element (target UL element) and a tracking UL element.Detecting, using the SbW steering system, the movements of the steering wheel by a vehicle operator; moving the tracking UL element in the visual tracking application based on the detected steering wheel movements; determining a corresponding tracking error between a location of the moving target UI element in the visual tracking application and a location of the tracking UL element in the visual tracking application at multiple time points; and processing the corresponding tracking errors to estimate a visuomotor skill of the operator.

[0005] The implementations of the disclosure may include one or more of the following optional features. According to some implementations, moving the tracking UL element in the visual tracking application based on the detected steering wheel movements includes moving the tracking UL element back and forth in the visual tracking application as the operator turns the steering wheel. Moving the tracking UL element back and forth as the operator turns the steering wheel may include moving the tracking UL element by a predetermined amount in the visual tracking application in response to the steering wheel being turned by a predetermined amount. According to some examples, detecting steering wheel movements by the operator using the SbW steering system includes providing haptic feedback as the steering wheel is turned.

[0006] According to some examples, processing the corresponding tracking errors to estimate the operator's visuomotor skill includes at least one of the following: determining a root mean square of the corresponding tracking errors, determining a frequency above which the magnitudes of the corresponding tracking errors correspond to a first threshold, determining a phase delay of the corresponding tracking errors, or using a McRuer transition model. According to some implementations, the operations also include determining whether the estimated operator's visuomotor skill meets a criterion and, if the estimated operator's visuomotor skill meets the criterion, restricting the operator's ability to operate the vehicle.Restricting the operator's control of the vehicle may include at least one of preventing the operator from operating the vehicle at all, limiting one operation of the vehicle by the operator, or notifying an emergency contact about the operator's restricted control of the vehicle.

[0007] According to some implementations, the operations also include identifying the vehicle operator and storing the operator's estimated visuomotor ability in an operator profile associated with the identified operator. The operations may further include retrieving a previously estimated visuomotor ability from the operator profile and determining, based on this estimated ability and the previously estimated visuomotor ability, whether the operator's vehicle operation should be restricted. Identifying the vehicle operator may involve using biometric data or user authentication information to identify the operator.

[0008] Another aspect of the disclosure provides a computer-implemented method, executed by a vehicle's data processing hardware, that causes the data processing hardware to perform operations. These operations include restricting the vehicle to a parked state and, while the vehicle is restricted to the parked state: presenting a visual tracking application on the display, wherein the visual tracking application includes a movable target user interface element (target UL element) and a tracking UL element; detecting, using the SbW steering system, the movements of the steering wheel by a vehicle operator; and moving the tracking UL element in the visual tracking application based on the detected movements of the steering wheel.and determining a corresponding tracking error between a location of the moving target UI element in the visual tracking application and a location of the tracking UI element in the visual tracking application at multiple time points; and processing the corresponding tracking errors to estimate a visuomotor skill of the operator.

[0009] The implementations of the disclosure may include one or more of the following optional features. According to some implementations, moving the tracking UL element in the visual tracking application based on the detected steering wheel movements involves moving the tracking UL element back and forth by a predetermined amount when the operator turns the steering wheel back and forth by a predetermined amount.According to some examples, processing the corresponding tracking errors to estimate the operator's visuomotor ability involves at least one of determining a root mean square of the corresponding tracking errors, determining a frequency above which the magnitudes of the corresponding tracking errors correspond to a first threshold, determining a phase delay of the corresponding tracking errors, or using a McRuer transition model.

[0010] According to some implementations, the operations also include determining whether the operator's estimated visuomotor ability meets a criterion, and, if the operator's estimated visuomotor ability meets the criterion, restricting the operator's ability to operate the vehicle. Restricting the operator's ability to operate the vehicle may include at least one of preventing the operator from driving the vehicle at all, limiting the operator's ability to drive the vehicle, or notifying an emergency contact of the operator's restricted ability to operate the vehicle.

[0011] According to some implementations, the operations also include identifying the vehicle operator and storing the operator's estimated visuomotor ability in an operator profile assigned to the identified operator. The operations may further include retrieving a previously estimated visuomotor ability from the operator profile and determining, based on this estimated ability, whether the operator's vehicle operation should be restricted.

[0012] A further aspect of the disclosure provides a system that includes data processing hardware and storage hardware. The storage hardware communicates with the data processing hardware and stores instructions which, when executed by the data processing hardware, cause the data processing hardware to perform operations. These operations include restricting the vehicle to a parked state and, while the vehicle is restricted to the parked state: presenting a visual tracking application on the display, wherein the visual tracking application includes a moving target user interface element (target UL element) and a tracking UL element; and detecting, using the SbW steering system, the movements of the steering wheel by an operator of the vehicle.Moving the tracking ul element in the visual tracking application based on the detected movements of the steering wheel; and determining a corresponding tracking error between a location of the moving target ul element in the visual tracking application and a location of the tracking ul element in the visual tracking application at multiple time points; and processing the corresponding tracking errors to estimate a visuomotor skill of the operator.

[0013] The implementations of the disclosure may include one or more of the following optional features. According to some examples, processing the corresponding tracking errors to estimate the operator's visuomotor skill includes at least one of determining a root mean square of the corresponding tracking errors, determining a frequency above which the magnitudes of the corresponding tracking errors correspond to a first threshold, determining a phase delay of the corresponding tracking errors, or using a McRuer transition model.According to some implementations, the operations also include determining whether the operator's estimated visuomotor ability meets a criterion, and, if the operator's estimated visuomotor ability meets the criterion, restricting the operator's ability to operate the vehicle, where restricting the operator's ability to operate the vehicle includes at least one of preventing the operator from driving the vehicle at all, limiting the operator's ability to drive the vehicle, or notifying an emergency contact of the operator's restricted ability to operate the vehicle.

[0014] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the following description. Further aspects, features, and advantages will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described here serve only to illustrate selected configurations and are not intended to limit the scope of protection of the present disclosure; they show: Fig. 1 a view of an exemplary vehicle comprising a steer-by-wire (SbW) steering system and a system for estimating human visuomotor skills according to the principles of the present disclosure; Fig. 2 A schematic view of the system for estimating human visuomotor skills according to Fig. 1; Fig. 3 a view of the vehicle's instrument panel according to Fig. 1, which includes a display for presenting a visual tracking application; Fig. 4 a schematic view of the SbW steering system according to Fig. 1; Fig. 5. A view of an exemplary visual tracking application; Fig. 6 a schematic view of an exemplary frequency domain analysis model; Fig. 7. A flowchart of an exemplary sequence of operations for a procedure for using SbW steering systems to estimate human visuomotor skills; Fig. 8 a flow chart of a further exemplary arrangement of operations for a procedure for using SbW steering systems to estimate human visuomotor skills; Fig. 9 a flow chart of a further exemplary arrangement of operations for a procedure for using SbW steering systems to estimate human visuomotor skills; Fig. 10 a flow chart of a further exemplary arrangement of operations for a procedure for using SbW steering systems to estimate human visuomotor skills; Fig. 11 a flow chart of a further exemplary arrangement of operations for a procedure for using SbW steering systems to estimate human visuomotor skills.

[0016] The corresponding reference symbols in all drawings indicate the corresponding parts. DETAILED DESCRIPTION

[0017] Exemplary configurations relating to the accompanying drawings are now described in more detail. Exemplary configurations are provided so that this disclosure is comprehensive and fully conveys the scope of protection of the disclosure to those skilled in the art. Specific details, such as examples of specific components, devices, and methods, are set forth to provide a comprehensive understanding of the configurations of this disclosure. It is evident to those skilled in the art that specific details need not be used, that the exemplary embodiments of the configurations can be embodied in many different forms, and that the specific details and the exemplary configurations are not intended to be interpreted in a way that limits the scope of protection of the disclosure.

[0018] The terminology used here serves only to describe specific exemplary configurations and is not intended to be restrictive. The singular articles "a," "an," and "the," as used here, are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises," "comprehensive," "including," and "exhibiting" are inclusive and therefore specify the presence of features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, one or more other steps, one or more other operations, one or more other elements, one or more other components, and / or groups thereof.The procedural steps, processes, and operations described here are not intended to necessarily require their execution in the specific order discussed or illustrated, unless they are specifically identified as such. Additional or alternative steps may be used.

[0019] When an element or layer is described as "at," "interacting with," "connected with," "attached to," or "coupled to" another element or layer, it may be directly at, interacting with, connected with, attached to, or coupled to the other element or layer, or there may be intermediate elements or layers. Conversely, when an element is described as "directly at," "directly interacting with," "directly connected with," "directly attached to," or "directly coupled to" another element or layer, there may be no intermediate elements or layers. Other words used to describe the relationship between elements should be interpreted in the same way (e.g., "between" as opposed to "directly between," "adjacent" as opposed to "directly adjacent," etc.).The term “and / or”, as used here, includes all combinations of one or more of the associated listed elements.

[0020] The terms "first," "second," "third," etc., can be used here to describe different elements, components, areas, layers, and / or sections. These elements, components, areas, layers, and / or sections should not be restricted by these terms. These terms can only be used to distinguish one element, component, area, layer, or section from another area, layer, or section. Terms such as "first," "second," and other numerical terms do not imply sequence or order unless clearly indicated by the context. Consequently, a first element, component, area, layer, or section could be...which will be discussed below, can be referred to as a second element, a second component, a second area, a second layer or a second section, without deviating from the lessons of the exemplary configurations.

[0021] In this application, including the definitions below, the term "module" may be replaced by the term "circuit". The term "module" may refer to, be part of, or include an application-specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field-programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; a memory (shared, dedicated, or group) that stores the code executed by a processor; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-a-chip.

[0022] The term "code," as used above, can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared processor" includes a single processor that executes some or all of the code from multiple modules. The term "group processor" includes a processor that, in combination with additional processors, executes some or all of the code from one or more modules. The term "shared memory" includes a single memory that stores some or all of the code from multiple modules. The term "group memory" includes memory that, in combination with additional memory, stores some or all of the code from one or more modules. The term "memory" can be a subset of the term "computer-readable medium."The term "computer-readable medium" does not include the transitory electrical and electromagnetic signals that propagate through a medium and can therefore be considered a tangible, non-transient storage medium. Non-restrictive examples of non-transient storage include tangible, computer-readable media, including non-volatile memory, magnetic memory, and optical memory.

[0023] The devices and methods described in this application can be implemented in whole or in part by one or more computer programs executed by one or more processors. The computer programs contain processor-executable instructions stored on at least one non-transient tangible, computer-readable medium. The computer programs may also contain and / or rely on stored data.

[0024] A software application (i.e., a software resource) can refer to computer software that causes a computing device to perform a task. Depending on the context, a software application may be called an "application," an "app," or a "program." Examples of applications include, but are not limited to, system diagnostics applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0025] Non-transient memory can include physical devices used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transient memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (which is typically used, for example, for firmware such as boot programs).The examples of volatile memory include, but are not limited to, read / write memory (RAM), dynamic read / write memory (DRAM), static read / write memory (SRAM), phase change memory (PCM), as well as disks or tapes.

[0026] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a procedural and / or object-oriented high-level programming language and / or in assembly / machine language. The terms "machine-readable medium" and "computer-readable medium," as used herein, refer to any computer program product, any non-transient computer-readable medium, any device, and / or apparatus (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including any machine-readable medium that receives machine instructions as a machine-readable signal.The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0027] Various implementations of the systems and techniques described herein may be realized in a digital electronic and / or optical circuit arrangement, an integrated circuit arrangement, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include the implementation in one or more computer programs that are executable and / or interpretable on a programmable system containing at least one programmable processor, which may be a special-purpose or general-purpose processor, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to a storage system, at least one input device, and at least one output device.

[0028] The processes and logical sequences described in this application can be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by acting on input data and producing an output. The processes and logical sequences can also be executed by a specialized logic circuit arrangement, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Examples of processors suitable for executing a computer program include general-purpose and specialized microprocessors, and any or more processors of any type of digital computer. Generally, a processor receives instructions and data from read-only memory, read / write memory, or both.The essential elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or is operationally connected to them to receive data from them, send data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing instructions and data of a computer program include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.The processor and memory can be supplemented by or incorporated into a special logic circuit arrangement.

[0029] To provide interaction with a user, one or more aspects of the disclosure may be implemented in a computer that has a display device, such as a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touchscreen, for displaying information to the user, and optionally a keyboard and a pointing device, such as a mouse or a trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; feedback provided to the user may be, for example, any form of sensory feedback, such as visual, auditory, or tactile feedback; while input may be received from the user in any form, including auditory, speech, or tactile input.Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0030] Unless expressly stated otherwise, the phrase "at least one of A, B or C" shall refer to any combination or subset of A, B, C, such as: (1) at least one A alone; (2) at least one B alone; (3) at least one C alone; (4) at least one A with at least one B; (5) at least one A with at least one C; (6) at least one B with at least one C; and (7) at least one A with at least one B and at least one C. Unless expressly stated otherwise, the phrase "at least one of A, B and C" shall refer to any combination or subset of A, B, C, such as: (1) at least one A alone; (2) at least one B alone; (3) at least one C alone; (4) at least one A with at least one B; (5) at least one A with at least one C; (6) at least one B with at least one C; and (7) at least one A with at least one B and at least one C.Furthermore, unless expressly stated otherwise, “A or B” shall refer to any combination of A and B, such as: (1) A alone; (2) B alone; and (3) A and B.

[0031] The configurations shown and described here can be used in conjunction with any type of locally or remotely controlled vehicle (e.g., a passenger car, truck, aircraft, train, motorcycle, bicycle, etc.) that incorporates an SbW steering system.

[0032] In particular, in the Fig. 1, Fig. 2, Fig. 3 and Fig. Figure 4 shows a vehicle 10 incorporating a SbW steering system 11, in conjunction with a human visuomotor skill estimation system 12, which uses the SbW steering system 11 to estimate the visuomotor skill of a vehicle operator. The SbW steering system 11 replaces mechanical links between the steering wheel 14 and the front wheels 16 of the vehicle 10 with electronic signals. An operator 100 inputs a desired steering angle via a sensor-equipped steering wheel 14 (e.g., using a steering feel motor 42), which sends the information to a steering module 25 of a control module 22. The steering module 25 then commands the electric actuators (e.g., one or more motors 44) of a steering rod 26 to adjust the wheel angles according to the operator's input and the vehicle's speed, stability, and road conditions.The present disclosure relates generally to the use of the SbW steering system 11 for estimating human visuomotor skills, which is a novel use of SbW steering systems that has not been investigated before.

[0033] The estimation system 12 of human visuomotor skills includes an estimation module 20 of human visuomotor skills, which can be stored and executed by any control module 22 of the vehicle 10. Specifically, the control module 22 can contain machine-readable instructions to execute the estimation in any of the Fig. The operations shown in Figures 7-11 are stored, for example, in a memory hardware 23, which can be executed by a data processing hardware (e.g., a processor 24) of the control module 22 to carry out the operations. According to the illustrated example, the human visuomotor skill estimation module 20 is connected to the SbW steering system 11 for receiving steering inputs from the user via the steering wheel 14 of the vehicle 10, and, based on the user's steering inputs, controls a visual tracking application 52 for estimating visuomotor skills.

[0034] The estimation module 20 of human visuomotor skills restricts the vehicle 10 to a parked state and uses the SbW steering system 11 to estimate the visuomotor skill of the operator 100 while the vehicle 10 is restricted to the parked state. Here, the vehicle 10 is restricted to the parked state, so the SbW steering system 11 can be safely used to estimate the visuomotor skill of the operator 100. In particular, the estimation module 20 of human visuomotor skills presents a visual tracking application 52 (see Fig. 5) on a display 18 of the vehicle 10 while the vehicle 10 is restricted to a parked state. The visual tracking application 52 includes a moving target user interface element (target UL element) 54 and a cursor or tracking UL element 56. The display 18 can, for example, and without restriction, be a display used to show an instrument panel on an instrument panel 15 of the vehicle 10 (e.g., directly in front of the steering wheel 14), a head-up display, or an infotainment system display. According to the illustrated example after Fig. In step 5, operator 100 moves the steering wheel 14 back and forth to move the tracking UL element 56 in the visual tracking application 52 back and forth, while the estimation module 20 of human visuomotor skills moves the movable target UI element 54 in the visual tracking application 52 back and forth. For the visual tracking application 52, operator 100 is instructed or guided to attempt to move the tracking UL element 56 so that it moves with and remains above the movable target UI element 54. Here, the extent to which the tracking UL element 56 moves with and remains above the movable target UI element 54 represents a visuomotor skill of operator 100.

[0035] The human visuomotor skills estimation module 20 detects the movements of the steering wheel 14 by the operator 100 of the vehicle 10 using the SbW steering system 11, wherein the visual tracking application 52 moves the tracking ul element 56 in the visual tracking application 52 based on the detected movements of the steering wheel 14. According to some implementations, the SbW steering system 11 provides haptic feedback (e.g., vibrations or resistance) to the operator 100 via the steering wheel 14 when the operator 100 interacts with the visual tracking application 52. According to some examples, the human visuomotor skills estimation module 20 detects a steering angle of the steering wheel 14 (e.g., in degrees), wherein the visual tracking application 52 uses an integrator with a gain of, for example, 1 / 2. B. forty (40) determines how much the tracking ul element 56 is to be moved in pixels.When the operator 100 rotates the steering wheel 14 by a predetermined amount (e.g., a predetermined number of degrees), the visual tracking application 52 can move the tracking UI element 56 by a predetermined amount (e.g., a predetermined number of pixels). Here, the gain can be chosen, for example, to mimic a natural relationship between the steering wheel angle and the yaw rate, or chosen so that the operator 100 can comfortably rotate the steering wheel 14 within a comfortable range while maintaining fine control of the tracking UI element 56. According to some implementations, the movable target UI element 54 is moved back and forth at varying rates and / or over varying distances.

[0036] The estimation module 20 of human visuomotor skills determines a corresponding tracking error e at several time points between a location of the moving target UI element 54 in the visual tracking application 52 and a location of the tracking UI element 56 in the visual tracking application 52. The moving target UI element 54 may, for example, be located at a distance r from a left side 58 of the visual tracking application 52, while the tracking UI element 56 is located at a distance y from the left side 58 of the visual tracking application 52. Here, the tracking error e at a specific time point is a difference between y and r. For example, e = r - y, or e = abs(r - y).

[0037] Once the visual tracking application 52 is finished or stopped, the Human Visual-Motor Skills Estimator 20 processes the corresponding tracking errors e to estimate the operator's 100 visual-motor skill. Here, the visual-motor skill represents the operator's 100 ability to integrate or coordinate visual information with a motor action. That is, how well the operator can translate a visual image / plan into an accurate motor action. According to some examples, the Human Visual-Motor Skills Estimator 20 determines a root mean square (RMS) value of the corresponding tracking errors e as an estimate of the visual-motor skill.Additionally or alternatively, the human visuomotor skills estimation module 20 can determine a frequency above which the magnitudes of the corresponding tracking errors correspond to a first threshold as an estimate of visuomotor ability. Additionally or alternatively, the human visuomotor skills estimation module 20 can determine a phase delay of the corresponding tracking errors as an estimate of visuomotor ability. According to some examples, the human visuomotor skills estimation module 20 uses a frequency domain analysis (e.g., using a McRuer transition model) to estimate visuomotor ability.

[0038] Fig. Figure 6 illustrates an exemplary frequency domain analysis model 600. Here, the frequency domain analysis model 600 is based on a McRuer transition model. According to the illustrated example, r is the position of the moving target ul element 54 and y is the position of the tracking ul element 56 at a specific time. Here, e=r−y Hue(s)P(s)=ωce−τss r(t)=A3 sin(ω3rt+φ3) d(t)=∑i=110Bi sin(ωidt+φi) P(s)=40s where ω c where is the transition frequency and τ is the transport delay time.

[0039] According to some examples, the assessment module 20 of human visuomotor skills determines whether the estimated visuomotor skill of operator 100 meets a criterion. If the operator's estimated visuomotor skill meets the criterion, it restricts operator 100's operation of vehicle 10. For example, the assessment module 20 of human visuomotor skills can prevent operator 100 from operating the vehicle at all, restrict operator 100's operation of the vehicle (e.g., limiting driving below a specific speed, limiting driving to specific environmental conditions, etc.), and / or notify an emergency contact of operator 100's restricted operation of vehicle 10.

[0040] According to some implementations, the human visuomotor skill estimator 20 identifies operator 100 of vehicle 10, storing operator 100's estimated visuomotor skill in an operator profile 21 associated with the identified operator 100. Here, the human visuomotor skill estimator 20 can retrieve a previously estimated visuomotor skill from operator profile 21 and, based on this estimated skill and the previously estimated visuomotor skill, determine whether operator 100's operation of vehicle 10 should be restricted. According to some examples, the stored visuomotor skill estimates can be tracked over a longer period or used to identify trends in visuomotor skills, e.g.,to be assessed due to an illness or disease.

[0041] According to some examples, identifying the operator 100 involves using biometric data acquired by one or more biometric sensors 27 for the operator 100. Additionally or alternatively, identifying the operator 100 can be based on comparing user authentication data obtained from the operator 100 or a device associated with the operator 100 with previously stored user authentication data 28.

[0042] Fig. Figure 7 is a flowchart of an exemplary sequence of operations for a computer-implemented procedure 700 for estimating human visuomotor skills. The operations can be performed by data processing hardware (e.g., the processor 24 according to Fig. 1) are executed based on the execution of instructions stored in memory hardware (e.g., memory hardware 23 after Fig. 1) are stored. Many other ways of implementing Procedure 700 can be used. For example, the order in which the operations are executed can be changed, and / or one or more of the operations and / or interactions can be modified, eliminated, subdivided, or combined. Additionally, the operations can be sorted according to Fig. 7. For example, by separate processing threads, processors, devices, discrete logic, circuits, etc., sequentially and / or in parallel.

[0043] In operation 702, procedure 700 includes restricting vehicle 10 to a parked state. While vehicle 10 is restricted to the parked state, in operation 704, procedure 700 includes presenting the visual tracking application 52 on the display 18. The visual tracking application 52 includes a moving target user interface element (target UL element) 54 and a tracking UL element 56.

[0044] While the vehicle 10 is restricted to the parked state, procedure 700, in operation 706, includes detecting the movements of the steering wheel 14 by an operator 100 of the vehicle 10 using the SbW steering system 11. While the vehicle 10 is restricted to the parked state, procedure 700, in operation 708, includes moving the tracking UI element 56 in the visual tracking application 52 based on the detected movements of the steering wheel 14. While the vehicle 10 is restricted to the parked state, procedure 700, in operation 710, includes determining a corresponding tracking error e between a location r of the moving target UI element 54 in the visual tracking application 52 and a location y of the tracking UI element 56 in the visual tracking application 52 at several time points.The procedure 700 in operation 712 then includes processing the corresponding tracking errors e to estimate the visuomotor ability of operator 100.

[0045] Fig. Figure 8 is a flowchart of an exemplary sequence of operations for a computer-implemented procedure 800 for estimating human visuomotor skills. The operations can be performed by data processing hardware (e.g., the processor 24 according to Fig. 1) are executed based on the execution of instructions stored in memory hardware (e.g., memory hardware 23 after Fig. 1) are stored. Many other ways of implementing Procedure 800 can be used. For example, the order in which the operations are executed can be changed, and / or one or more of the operations and / or interactions can be modified, eliminated, subdivided, or combined. Additionally, the operations can be further subdivided according to Fig. 8 e.g., by separate processing threads, processors, devices, discrete logic, circuits, etc., executed sequentially and / or in parallel.

[0046] In operation 802, procedure 800 includes presenting the visual tracking application 52 in response to user input. In operation 804, procedure 800 includes determining a corresponding tracking error e between a location r of the moving target UI element 54 in the visual tracking application 52 and a location y of the tracking UL element 56 in the visual tracking application 52 at several time points. Subsequently, in operation 804, procedure 800 includes processing the corresponding tracking errors e to estimate a visuomotor ability 805 of the operator 100.

[0047] In operation 806, procedure 800 includes assessing whether the operator's estimated visuomotor ability 805 meets a criterion. If the operator's estimated visuomotor ability 805 meets the criterion, it restricts the operator's operation of the vehicle 10. The human visuomotor ability assessment module 20 can, for example, prevent the operator from operating the vehicle 10 at all, restrict the operator's operation of the vehicle (e.g., limiting driving below a specific speed, limiting driving under specific environmental conditions, etc.), and / or notify an emergency contact of the operator's restricted operation of the vehicle 10.According to some examples, a previously estimated visuomotor ability is retrieved from operator profile 21 and used to determine whether operator 100's operation of vehicle 10 should be restricted. During operation 812, the estimated visuomotor ability is stored in operator profile 21.

[0048] In Operation 814, Procedure 800 includes determining whether to release the visuomotor skills test in Operations 802 and 804. This determination is based, for example, on data acquired by the vehicle's sensors and / or cameras 816, or on biometric authentication in Operation 818. Biometric authentication can be based, for example, on comparing biometric data 820, such as a facial image or fingerprint, with data stored in the operator profile 21. Exemplary sensors 816 are included in the vehicle but are not limited to alcohol or drug sensors.Additionally or alternatively, Operation 814 can automatically clear the visuomotor skills test for a teenage operator, a disabled operator, an elderly operator, an operator driving to a party, an operator previously charged or convicted of driving under the influence of alcohol or drugs, etc.

[0049] Fig. Figure 9 is a flowchart of an exemplary sequence of operations for a computer-implemented procedure 900 for estimating human visuomotor skills. The operations can be performed by data processing hardware (e.g., the processor 24 according to Fig. 1) are executed based on the execution of instructions stored in memory hardware (e.g., memory hardware 23 after Fig. 1) are stored. Many other ways of implementing Procedure 900 can be used. For example, the order in which the operations are executed can be changed, and / or one or more of the operations and / or interactions can be modified, eliminated, subdivided, or combined. Additionally, the operations can be further subdivided according to Fig. 9 e.g., by separate processing threads, processors, devices, discrete logic, circuits, etc., executed sequentially and / or in parallel.

[0050] In Operation 902, Procedure 900 involves detecting that vehicle 10 is being started. In Operation 904, Procedure 900 involves determining whether the visuomotor skills test is enabled. Here, Operation 904 can enable the visuomotor skills test if the sensors detect that the operator 100 may be under the influence of drugs or alcohol, for a teenage operator, a disabled operator, an elderly operator, an operator driving to a party, an operator previously charged or convicted of driving under the influence of alcohol or drugs, etc. If the visuomotor skills test is not enabled, the control system exits Procedure 900.

[0051] If the visual-motor skills test is authorized, in operation 906, procedure 900 includes restricting the vehicle 10 to a parked state and displaying the visual tracking application 52 on the display 18 of the vehicle 10. In operation 908, procedure 900 includes requesting, guiding, or instructing the operator 100 to perform the visual tracking test.

[0052] In Operation 910, Procedure 900 includes determining whether a visual-motor skill estimated using the visual tracking test meets a criterion, i.e., whether the operator 100 has passed the visual tracking test. If the operator 100 has passed the visual tracking test, Procedure 900 in Operation 912 includes displaying on Display 18 that the operator 100 has passed the test and removing the restriction that the vehicle 10 remains in a parked state. In Operation 914, Procedure 900 includes storing the estimated visual-motor skill in the operator profile 21 along with, for example, a date, a time, an operator identifier, etc.

[0053] If operator 100 fails the test at operation 910, procedure 900 at operation 916 includes requesting, guiding, or instructing operator 100 to repeat the visual tracking test. At operation 918, procedure 900 includes determining whether a visuomotor skill estimated using the repeated visual tracking test meets a criterion, i.e., whether operator 100 has passed the visual tracking test. If operator 100 has passed the visual tracking test, the control proceeds to operation 912.Otherwise, if operator 100 fails the visual tracking test, procedure 900, during operation 920, includes displaying on display 18 that operator 100 has failed the test and continuing the restriction that vehicle 10 remains in a parked state. Procedure 900 can, for example, prevent all driving operations of the vehicle by operator 100, restrict driving operations of the vehicle by operator 100 (e.g., restrict driving below a specific speed, restrict driving to specific environmental conditions, etc.), and / or notify an emergency contact of operator 100's restricted operation of vehicle 10.

[0054] Fig. Figure 10 is a flowchart of an exemplary sequence of operations for a computer-implemented procedure 1000 for estimating human visuomotor skills. The operations can be performed by data processing hardware (e.g., the processor 24 according to Fig. 1) are executed based on the execution of instructions stored in memory hardware (e.g., memory hardware 23 after Fig. 1) are stored. Many other ways of implementing Procedure 1000 can be used. For example, the order in which the operations are executed can be changed, and / or one or more of the operations and / or interactions can be modified, eliminated, subdivided, or combined. Additionally, the operations can be sorted according to Fig. 10. e.g., by separate processing threads, processors, devices, discrete logic, circuits, etc., are executed sequentially and / or in parallel.

[0055] In Operation 1002, Procedure 1000 includes assessing a visuomotor ability 1004 for an operator 100. In Operation 1006, Procedure 1000 includes arbitrating whether operator 100's operation of the vehicle 10 should be restricted. Operation 1006 may, for example, remove any restrictions on operator 10's operation of the vehicle 10; prevent operator 100 from driving the vehicle; limit operator 100's driving (e.g., restrict driving to below a specific speed, restrict driving to specific environmental conditions, etc.); and / or notify an emergency contact of operator 100's restricted operation of the vehicle 10.As shown, any applied restrictions can be stored in operator profile 21 for later retrieval and use when performing future arbitrations.

[0056] The arbitration in operation 1006 can take place based on the information 1008 obtained from an operator profile 21 (e.g., preferences, limitations, history, previous estimates of visuomotor skills, etc.) for the operator 100, an operator identity 1010, and / or operator status information 1012. In operation 1014, the procedure 1000 involves authenticating the operator's identity 1010 based on the user authentication information 1016 obtained from the operator 100 or from a user device assigned to the operator 100, and / or the biometric user data 1020 obtained using one or more sensors in the vehicle and / or the cameras 1022. Here, the operator identification 1010 can, for example, be...based on matching the user authentication information 1016 and / or the biometric data 1020, such as a facial image or a fingerprint, with the data stored in the operator profile 21.

[0057] Examples of sensors included in vehicle 816 are not limited to alcohol or drug sensors. Operator condition information 1012 can reflect whether the operator 100 is under the influence of alcohol or drugs, or is drowsy.

[0058] Fig. Figure 11 is a flowchart of an exemplary sequence of operations for a computer-implemented procedure 1100 for estimating human visuomotor skills. The operations can be performed by data processing hardware (e.g., the processor 24 according to Fig. 1) are executed based on the execution of instructions stored in memory hardware (e.g., memory hardware 23 after Fig. 1) are stored. Many other ways of implementing procedure 1100 can be used. For example, the order in which the operations are executed can be changed, and / or one or more of the operations and / or interactions can be modified, eliminated, subdivided, or combined. Additionally, the operations can be further subdivided according to Fig. 11 e.g. by separate processing threads, processors, devices, discrete logic, circuits, etc., executed sequentially and / or in parallel.

[0059] In Operation 1102, Procedure 1100 includes performing initial interactions with an operator 100. In Operation 1104, Procedure 1100 includes performing a visuomotor skills test to estimate a visuomotor skill 1106 for an operator 100. In Operation 1108, Procedure 1000 includes arbitrating whether to restrict the operator 100's operation of the vehicle 10. Operation 1108 may, for example, remove any restrictions on the operator 10's operation of the vehicle 10; prevent the operator 100 from driving the vehicle; limit the operator 100's driving of the vehicle (e.g., restrict driving below a specific speed, restrict driving to specific environmental conditions, etc.); and / or notify an emergency contact of the operator 100's restricted operation of the vehicle 10.Any restrictions applied can be stored in operator profile 21 for later retrieval and use when performing future arbitrations.

[0060] The arbitration in Operation 1108 can take place based on the information 1109 obtained from an operator profile 21 (e.g., preferences, limitations, history, previous estimates of visuomotor skills, etc.) for the operator 100 and / or the operator status information 1111. In Operation 1100, the procedure 1000 involves determining the operator's identity 1112 based on the user authentication information obtained from the operator 100 or from a user device assigned to the operator 100, and / or the user biometric data 1113 obtained using one or more sensors in the vehicle and / or the cameras. Here, the operator identification 1112 can be determined, for example, based on matching the user information and / or the biometric data 1113, such as...A facial image or fingerprint, using the data stored in the operator profile 21, can be used. Examples of sensors included in the vehicle are not limited to alcohol or drug sensors. Here, the operator status information 1111 can reflect whether the operator 100 is under the influence of alcohol or drugs, or is drowsy.

[0061] Several implementations have been described. However, it is recognized that various modifications can be made without deviating from the inventive concept and scope of protection of the disclosure. Accordingly, other implementations fall within the scope of protection of the following claims.

[0062] The preceding description has been provided for illustrative and descriptive purposes. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular configuration are generally not restricted to that particular configuration but are interchangeable and may be used in a selected configuration, even if not specifically shown or described. Furthermore, they may be varied in various ways. Such variations are not to be considered a deviation from the disclosure, and all such modifications are to be contained within the scope of protection of the disclosure.

Claims

[1] Vehicle comprising: a steer-by-wire steering system (SbW steering system); a steering wheel configured to control the SbW steering system in response to input from an operator; an advertisement; Data processing hardware; and Storage hardware that communicates with the data processing hardware and stores instructions which, when executed by the data processing hardware, cause the data processing hardware to perform operations that include: Restricting the vehicle to a parked state; while the vehicle is restricted to the parked state: Presenting a visual tracking application on the display, wherein the visual tracking application includes a movable target user interface element (target UL element) and a tracking UI element; Detecting steering wheel movements by a vehicle operator using the SbW steering system; Moving the tracking UL element in the visual tracking application based on the detected movements of the steering wheel; and Determining a corresponding tracking error between a location of the moving target UL element in the visual tracking application and a location of the tracking UL element in the visual tracking application at multiple time points; and Processing the corresponding tracking errors to estimate the operator's visuomotor skills. [2] Vehicle according to claim 1, wherein moving the tracking UI element in the visual tracking application based on the detected movements of the steering wheel comprises moving the tracking UL element back and forth in the visual tracking application when the operator turns the steering wheel back and forth. [3] Vehicle according to claim 2, wherein the back-and-forth movement of the tracking ul element when the operator turns the steering wheel back and forth comprises moving the tracking ul element by a predetermined amount in the visual tracking application in response to the steering wheel being turned by a predetermined amount. [4] Vehicle according to claim 1, wherein the detection of movements of the steering wheel by the operator using the SbW steering system comprises providing haptic feedback when the steering wheel is turned. [5] Vehicle according to claim 1, wherein the processing of the corresponding tracking errors to estimate the visuomotor ability of the operator comprises at least one of the following: Determining a root mean square of the corresponding tracking errors; Determining a frequency above which the magnitudes of the corresponding tracking errors correspond to a first threshold value; Determining a phase delay of the corresponding tracking errors; or Using a McRuer transition model. [6] Vehicle according to claim 1, wherein the operations further comprise: Determine whether the operator's estimated visuomotor ability meets a criterion; and If the estimated visual-motor ability of the operator meets the criterion, restricting the operator's ability to operate the vehicle. [7] Vehicle according to claim 6, wherein restricting the operation of the vehicle by the operator comprises at least one of the following: Preventing the operator from making any driving operations on the vehicle; Limiting the operator's ability to drive the vehicle; or Notify an emergency contact about the operator's limited ability to operate the vehicle. [8] Vehicle according to claim 1, wherein the operations further comprise: Identifying the vehicle operator; and Storing the estimated visuomotor ability of the operator in an operator profile assigned to the identified operator. [9] Vehicle according to claim 8, wherein the operations further comprise: Retaining a previously estimated visuomotor skill from the operator profile; and Determine, based on the estimated visuomotor ability and the previously estimated visuomotor ability, whether the operator's operation of the vehicle should be restricted. [10] Vehicle according to claim 8, wherein identifying the operator of the vehicle includes using biometric data or user authentication information to identify the operator.

Images