Method and apparatus for determining operational intent

Abstract

To improve the accuracy of determining whether an operation performed by an operator is in line with the operator's intent. [Solution] In the operation intent determination method, the operation target operated by the operator is determined (S5), a sensory stimulus is sent from the determined operation target to the operator (S6), and based on the operator's biological information when the sensory stimulus is sent from the operation target, it is determined whether or not the operator has an intention to operate the operation target (S8), and if it is determined that there is an intention to operate, the operation target is made to move (S10).

Description

Technical Field

[0001] The present invention relates to an operation intention determination method and an operation intention determination device.

Background Art

[0002] The following Patent Document 1 describes a control device that controls a device based on the biological information of a user. The control device includes a biological information acquisition unit that acquires the biological information of the user, a will detection unit that detects the presence or absence of a first will by the user based on the information from the biological information acquisition unit, and a determination unit that determines whether to validate or invalidate the operation content by the user based on the detection result of the will detection unit and the operation of the device by the user.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the control device of Patent Document 1 above, even if the intention of the operation by the user is detected, when there are a plurality of candidates for operations that can be performed by the user, it is impossible to determine which operation the user is trying to perform. For this reason, it may not be possible to prevent an incorrect operation that goes against the intention of the user. An object of the present invention is to improve the determination accuracy of whether an operation performed by an operator is an operation that matches the operator's intention.

Means for Solving the Problems

[0005] In one aspect of the present invention, a method for determining an operator's intent to operate involves identifying an object being operated by the operator, providing a sensory stimulus from the identified object to the operator, determining whether the operator has an intent to operate the object based on the operator's biological information at the time the sensory stimulus is provided by the object, and operating the object if it is determined that there is an intent to operate. [Effects of the Invention]

[0006] According to the present invention, it is possible to improve the accuracy of determining whether an operation performed by an operator is in line with the operator's intentions. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows an example of the hardware configuration of the operation support device according to the embodiment. [Figure 2] Figure 1 is a block diagram showing an example of the controller's functional configuration. [Figure 3] This is a flowchart of an example of a method for determining the operational intent of an embodiment. [Figure 4] This is an example of a scatter plot showing the P3 component of event-related potentials for driving operations and the stimulus-preceding negative potential (SPN) for the operation of the target object. [Figure 5] This is an explanatory diagram of an example of an attention allocation decision map. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will be described below with reference to the drawings. Note that the drawings are schematic and may differ from actual ones. Furthermore, the embodiments of the present invention described below are illustrative examples of devices and methods for realizing the technical concept of the present invention, and the technical concept of the present invention is not limited to the structure, arrangement, etc., of the components described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims described in the patent claims.

[0009] (First Embodiment) (composition) Figure 1 shows an example of the hardware configuration of the operation support device according to the embodiment. The operation support device 1 is a device that suppresses the occurrence of erroneous operations by measuring the operator's brain activity when an operator performs an operation on a certain object, determining whether or not there is an intention to operate on the object, and operating the object based on the determination result.

[0010] In the following description, we will explain an example where the operator performing the operation on the target is a vehicle occupant (e.g., the driver), and the target is an in-vehicle device 2 installed in the vehicle. However, the present invention is not limited to determining the operating intent of a vehicle occupant, but can be broadly applied to determining the operating intent of an operator performing an operation on various targets.

[0011] The operation support device 1 comprises a behavior monitoring sensor 4, a brain activity sensor 5, a sensory stimulus generator 6, and a controller 7. For example, the operation support device 1 may be installed in a vehicle on which the occupants are riding. The behavior monitoring sensor 4, the brain activity sensor 5, the sensory stimulus generator 6, and the controller 7 form an operation intent determination device 3 that determines the occupant's intention to operate the in-vehicle equipment 2.

[0012] Here, the in-vehicle device 2 to be operated may be, for example, the vehicle's in-vehicle infotainment (IVI) system, navigation system, audio-visual equipment, or air conditioning system. Furthermore, in the first embodiment, the in-vehicle device 2 to be operated may be a driving control element used for driving operations (for example, a steering wheel, accelerator pedal, brake pedal, shift lever, turn signal, parking brake).

[0013] The behavior monitoring sensor 4 may be a sensor for monitoring the behavior of the occupants. For example, the behavior monitoring sensor 4 may include a camera installed inside the vehicle to photograph the occupants. The behavior monitoring sensor 4 may also include an RGB-D camera capable of acquiring the distance to the subject. For example, the behavior monitoring sensor 4 may be a sensor that detects the operation input of the in-vehicle device 2, or a contact sensor or pressure sensor that detects whether or not an occupant is in contact with the in-vehicle device 2. The behavior monitoring sensor 4 generates occupant monitoring data and outputs it to the controller 7.

[0014] Brain activity sensor 5 is a sensor that detects the brain activity of the occupant. For example, brain activity sensor 5 may be an electroencephalogram (EEG) sensor that detects the occupant's brain waves by having multiple electrodes attached to the occupant's head. The multiple electrodes of the EEG sensor may be arranged, for example, in the parietal region of the occupant's head, which is related to cognitive function, in accordance with the International 10-20 method. The number and placement of the electrodes are not particularly limited. Furthermore, the method of attaching the multiple electrodes of the brain activity sensor 5 to the head is not particularly limited; for example, it may consist of a wearable electrode cap or band equipped with multiple electrodes. The brain activity sensor 5 detects data on the occupant's brain activity (e.g., electroencephalography) and outputs the detected brain activity data to the controller 7.

[0015] The sensory stimulus generator 6 is installed in the in-vehicle device 2 and generates sensory stimuli to be delivered to the occupant. In other words, the sensory stimulus generator 6 generates sensory stimuli delivered to the occupant from the in-vehicle device 2, which is operated by the occupant. For example, the sensory stimulus generator 6 may be a display device provided in an in-vehicle device 2, such as an in-vehicle infotainment system, navigation system, or audiovisual equipment, that presents visual stimuli to the occupants.

[0016] Furthermore, the sensory stimulus generator 6 may be a speaker or buzzer provided in the in-vehicle equipment 2 that presents auditory stimuli to the occupants. If the in-vehicle equipment 2 is a device capable of outputting sound (for example, an in-vehicle infotainment system, a navigation system, audiovisual equipment, etc.), the in-vehicle equipment 2 itself may be used as the sensory stimulus generator 6 that presents auditory stimuli. The sensory stimulus generator 6 may also be a vibration device or a tactile feedback device provided in the vehicle-mounted device 2 that presents a tactile stimulus to the occupant by vibrating the vehicle-mounted device 2 or outputting ultrasonic waves.

[0017] The controller 7 is an electronic control unit (ECU: Electronic Control Unit) that determines the operation intention of the vehicle-mounted device 2 by the occupant based on the monitoring data output from the behavior monitoring sensor 4 and the data of the brain activity output from the brain activity sensor 5. The controller 7 includes a processor 7a and peripheral components such as a storage device 7b. The processor 7a may be, for example, a CPU (Central Processing Unit) or a MPU (Micro-Processing Unit). The storage device 7b may include a semiconductor storage device, a magnetic storage device, an optical storage device, etc. For example, the storage device 7b may include memories such as registers, cache memories, a ROM (Read Only Memory) and a RAM (Random Access Memory) used as a main storage device.

[0018] The functions of the controller 7 described below are realized, for example, when the processor 7a executes a computer program stored in the storage device 7b. Note that the controller 7 may be formed by dedicated hardware for executing each information process described below. For example, the controller 7 may include a functional logic circuit set in a general-purpose semiconductor integrated circuit. The controller 7 may have a programmable logic device (PLD) such as a field-programmable gate array (FPGA).

[0019] Figure 2 is a block diagram of an example of the functional configuration of the controller 7 in Figure 1. The controller 7 comprises a first operation intent determination unit 10, an occupant operation estimation unit 11, a sensory stimulus generation unit 12, an event-related potential calculation unit 13, a second operation intent determination unit 14, and an in-vehicle equipment control unit 15. The first operation intent determination unit 10 determines whether or not the occupant intends to perform some operation based on the brain activity data output by the brain activity sensor 5.

[0020] Here, when the first operation intent determination unit 10 determines whether or not there is an "intention to perform some operation," the determination is not whether or not there was an intention to operate a specific in-vehicle device 2, but rather whether or not an intention to perform some operation was detected, regardless of what the intention was (whether the intention was to operate the in-vehicle device 2 or any other operation). In the following explanation, the crew member's intention that is the subject of determination by the first operation intention determination unit 10 may be referred to as the "first operation intention."

[0021] For example, the first operation intent determination unit 10 may determine whether or not the occupant has reached out based on a signal generated by the behavior monitoring sensor 4 monitoring the occupant's behavior. For example, the first operation intent determination unit 10 may determine whether or not the occupant has reached out based on the occupant's behavior captured by the camera of the behavior monitoring sensor 4. For example, the first operation intent determination unit 10 may extract partial images of the occupant's body parts from the captured images generated by the RGB-D camera of the behavior monitoring sensor 4. Known methods, such as the known background subtraction method, may be used to extract partial images of the occupant's body parts.

[0022] Next, the first operation intent determination unit 10 may estimate the coordinates of the crew member's joints using partial images of the crew member's body parts. For example, for each pixel in the partial image of the body part, a feature may be calculated using the depth distance around the pixel. This feature may then be applied to a cluster learning machine to calculate the body parts, the pixels of each body part may be clustered, and representative points of the clusters may be extracted to estimate the coordinates of the crew member's joints. The first operation intent determination unit 10 may determine whether or not the occupant has reached out their hand based on the estimated joint position coordinates of each part of the occupant's body.

[0023] The first operation intent determination unit 10 may determine whether the occupant attempted to perform an operation based on the brain activity data detected by the brain activity sensor 5 when it is determined that the occupant has reached out their hand. For example, the brainwave waveform obtained when the occupant is attempting to perform an operation may be stored in the memory device 7b in advance, and if a brainwave waveform identical or similar to the stored brainwave waveform is detected, it may be determined that there is a first operation intent (i.e., it may be determined that the occupant is attempting to perform an operation). On the other hand, if the brainwave waveform detected by the brain activity sensor 5 when it is determined that the occupant has reached out their hand is not identical or similar to the stored brainwave waveform, it may be determined that there is no first operation intent.

[0024] The occupant operation estimation unit 11 estimates the in-vehicle device 2 operated by the occupant based on signals generated by the behavior monitoring sensor 4, which monitors the occupant's behavior. In the following description, the in-vehicle device 2 that the occupant operation estimation unit 11 estimates to have been operated by the occupant may be referred to as the "operated device". For example, the occupant operation estimation unit 11 estimates the target of operation based on the occupant's behavior captured by the camera of the behavior monitoring sensor 4. For example, the occupant operation estimation unit 11 may estimate the joint position coordinates of the occupant from the captured image generated by the RGB-D camera of the behavior monitoring sensor 4, similar to the first operation intent determination unit 10 described above.

[0025] The occupant operation estimation unit 11 may detect occupant gestures from estimated joint position coordinates and estimate the object to be operated based on at least one of the shape, orientation, or position of the detected gesture. For example, the occupant operation estimation unit 11 may detect the occupant's hand gestures and estimate the object to be operated based on at least one of the shape, orientation, or position of the detected gesture. For example, the occupant operation estimation unit 11 may estimate the object to be operated using a database (e.g., a classifier) ​​that has learned the features of gestures used to operate the in-vehicle equipment 2.

[0026] The sensory stimulus generation unit 12 generates sensory stimuli to be transmitted from the target to the occupant by operating the sensory stimulus generator 6 provided on the target when the occupant operation estimation unit 11 estimates the target. For example, if the object being operated is an in-vehicle infotainment system, navigation system, or audiovisual equipment, the sensory stimulus generation unit 12 may present visual stimuli to the occupant from a sensory stimulus generator 6, which is a display device provided in these devices.

[0027] For example, the in-vehicle device 2 to be operated may be equipped with a contactless graphical user interface (GUI). The contactless GUI may accept contactless operations on GUI elements, which are operation elements (e.g., buttons, etc.) placed on the GUI screen, based on the occupant's hand gestures detected by the occupant operation estimation unit 11. The sensory stimulus generator 6 may present a CG image of a hand (a so-called "ghost hand") that virtually represents the occupant's hand gestures operating the GUI elements as a visual stimulus to the display device. For example, the sensory stimulus generator 6 may present the occupants with illuminated or flashing displays on the display devices of these devices as visual stimuli, or it may present the occupants with interactive agents displayed on these devices as visual stimuli.

[0028] Alternatively, the sensory stimulus generation unit 12 may present auditory stimuli to the occupant from a sensory stimulus generator 6, such as a speaker or buzzer, provided on the object being operated. For example, if the object being operated is a device capable of voice output (e.g., an in-vehicle infotainment system, navigation system, audiovisual equipment, etc.), the in-vehicle device 2 itself may be used as the sensory stimulus generator 6. For example, the voice of an interactive agent assisting in the operation of these in-vehicle devices 2 may be presented to the occupant as an auditory stimulus.

[0029] Alternatively, the sensory stimulation generation unit 12 may, for example, activate a vibration device or tactile feedback device provided on the object being operated to vibrate the object or emit ultrasonic waves, thereby presenting tactile stimuli to the occupant. The event-related potential calculation unit 13 extracts event-related potentials generated by sensory stimuli presented by the sensory stimulus generation unit 12 from the brain activity data (e.g., electroencephalogram) output by the brain activity sensor 5.

[0030] For example, the event-related potential calculation unit 13 may extract the P3 component (P300 component) of the event-related potential. In the following description, the P3 component of the event-related potential may be simply referred to as the "P3 component". The P3 component is an event-related potential (stimulus-response positive potential) that reflects the allocation of processing resources and information updates at the central level, which respond approximately 300 milliseconds after the stimulus event is presented to the subject.

[0031] The amplitude of the P3 component depends on the level of attention the occupant is paying to the stimulus; the higher the level of attention, the larger the amplitude of the P3 component. Therefore, when the in-vehicle device 2 is operated, a sensory stimulus is presented from the in-vehicle device 2, and the amplitude of the P3 component generated by this sensory stimulus is calculated. The higher the level of attention the occupant is paying to the in-vehicle device 2, the larger the calculated amplitude of the P3 component. Thus, the level of attention the occupant is paying to the operated in-vehicle device 2 can be estimated based on the amplitude of the P3 component generated by the sensory stimulus presented from the operated in-vehicle device 2.

[0032] For example, the event-related potential calculation unit 13 may extract the electroencephalogram potential that occurs approximately 300 milliseconds after the sensory stimulus generation unit 12 provides a sensory stimulus from the object being operated to the occupant (i.e., the occupant operation estimation unit 11 estimates the operation of the in-vehicle equipment 2 by the occupant) as the P3 component generated by the sensory stimulus.

[0033] Alternatively, the event-related potential calculation unit 13 may extract a stimulus-preceding negativity (SPN), which is an event-related potential that is less affected by the frequency of stimulus occurrence. The SPN is a potential that increases when information about the consequences of a subject's actions is provided as a stimulus, and it reflects the degree of the subject's involvement in the actions they performed. Therefore, when an occupant operates the in-vehicle device 2, if the SPN generated by the sensory stimulus provided by the operated device 2 is extracted, the magnitude of the occupant's attention to the operation of the operated device (i.e., the occupant's attention to the operated device) can be estimated from the amplitude of the SPN.

[0034] For example, the event-related potential calculation unit 13 may, when the occupant operation estimation unit 11 estimates that the occupant operated the in-vehicle device 2, extract the electroencephalogram potential that occurs from the time the occupant operates the in-vehicle device 2 until the time the sensory stimulus is delivered as the SPN. For example, the electroencephalogram potential that occurs immediately before the time the sensory stimulus is delivered may be extracted as the SPN.

[0035] The second operation intent determination unit 14 determines whether the occupant had an intention to operate the target of the operation, based on the amplitude of the event-related potential calculated by the event-related potential calculation unit 13. In other words, it determines whether the occupant operated the in-vehicle device 2 as intended or unintentionally. In other words, the operational intent that the second operational intent determination unit 14 is to determine is whether or not the occupant who operated a specific in-vehicle device 2 had the intention to operate that in-vehicle device 2. In the following explanation, the operational intent of the occupant that the second operational intent determination unit 14 is to determine may be referred to as the "second operational intent."

[0036] For example, the second operation intent determination unit 14 may determine whether the occupant was paying attention to the object being operated (i.e., whether the occupant who operated the in-vehicle device 2 was paying attention to the in-vehicle device 2) based on whether the amplitude of the event-related potential calculated by the event-related potential calculation unit 13 is above a predetermined threshold. For example, it may be determined that the occupant was paying attention to the object being operated if the amplitude of the event-related potential is above a predetermined threshold, and that the occupant was not paying attention to the object being operated if the amplitude of the event-related potential is below a predetermined threshold. The second operation intent determination unit 14 may determine that the occupant had the intention to operate the object (i.e., the second operation intent) if the occupant was paying attention to the object. Conversely, if the occupant was not paying attention to the object, the unit may determine that the occupant did not have the second operation intent.

[0037] The in-vehicle equipment control unit 15 controls the operation of the in-vehicle equipment 2 operated by the occupant based on the determination result of the first operation intent determination unit 10 and the determination result of the second operation intent determination unit 14. If the first operation intent determination unit 10 determines that there is no first operation intent of the occupant, the in-vehicle equipment control unit 15 will prohibit the operation of the in-vehicle equipment 2 even if the in-vehicle equipment 2 is operated.

[0038] Furthermore, even if the first operation intent determination unit 10 determines that the occupant has a first operation intent, if the second operation intent determination unit 14 determines that the occupant does not have a second operation intent, the in-vehicle equipment control unit 15 will prohibit the operation of the target device. If the first operation intent determination unit 10 determines that the occupant has a first operation intent, and the second operation intent determination unit 14 determines that the occupant has a second operation intent, the in-vehicle equipment control unit 15 permits the operation of the target device.

[0039] (operation) Figure 3 is a flowchart of an example of a method for determining the intent of an operation according to the embodiment. In step S1, the first operation intent determination unit 10 determines whether or not the occupant has reached out their hand. If the occupant has not reached out their hand (step S1:N), the process returns to step S1. If the occupant has reached out their hand (step S1:Y), the process proceeds to step S2.

[0040] In step S2, the first operation intent determination unit 10 determines whether the crew member has a first operation intent. If the crew member has a first operation intent (step S2:Y), the process proceeds to step S5. If the crew member does not have a first operation intent (step S2:N), it is determined that the crew member has no intention to perform any operation (step S3), and the process proceeds to step S4. In step S4, the in-vehicle equipment control unit 15 prohibits the operation of the in-vehicle equipment 2. After that, the process ends.

[0041] In step S5, the occupant operation estimation unit 11 identifies the in-vehicle device 2 that was operated by the occupant. In step S6, the sensory stimulus generation unit 12 generates sensory stimuli to be transmitted from the target to the occupant by operating the sensory stimulus generator 6 provided on the target.

[0042] In step S7, the event-related potential calculation unit 13 extracts event-related potentials generated by sensory stimuli from the brain activity data output by the brain activity sensor 5. The second operation intention determination unit 14 calculates the occupant's level of attention to the target of operation based on the event-related potentials. In step S8, the second operation intent determination unit 14 determines whether the occupant is paying attention to the object being operated based on the calculated level of attention.

[0043] If the crew is not paying attention to the object being operated on (step S8:N), the process proceeds to step S11. If the crew is paying attention to the object being operated on (step S8:Y), the process proceeds to step S9. In step S9, the second operation intent determination unit 14 determines that the occupant has an operation intent (second operation intent) for the target of the operation. In step S10, the in-vehicle equipment control unit 15 authorizes the operation of the target device. The process then terminates.

[0044] In step S11, the second operation intent determination unit 14 determines that there is no operation intent (second operation intent) of the occupant toward the object to be operated. In step S12, the in-vehicle equipment control unit 15 prohibits the operation of the target device. The process then terminates.

[0045] (Second Embodiment) The amplitude of event-related potentials varies from person to person. Therefore, if the level of attention of the crew member towards the target is estimated based on the amplitude of the event-related potential, as in the example above, and the presence or absence of a second intention to operate is determined based on a comparison of the level of attention with the threshold, the quality of the threshold setting may affect the determination result.

[0046] Therefore, in the second embodiment, in addition to the crew's attention level to the object being operated, the operator's attention level to the primary task, which is a task that occurs more frequently than the operation on the object being operated, is calculated. It is known that the total amount of attention a person can dedicate is roughly constant. Therefore, the sum of the crew's attention to the main task and their attention to the object being manipulated is roughly constant, and a negative correlation exists where an increase in one leads to a decrease in the other.

[0047] Based on this relationship, the degree to which a crew member paid attention to the object being manipulated can be estimated based on attention allocation, which is the ratio between attention to the primary task and attention to the object being manipulated. This attention allocation can be determined by the ratio between the amplitude of the event-related potential detected by the crew member when performing the primary task and the amplitude of the event-related potential generated by the sensory stimulus provided by the object being manipulated.

[0048] Therefore, by estimating the extent to which the crew paid attention to the object being operated based on attention allocation, which is the ratio between attention to the primary task and attention to the object being operated, it is possible to more accurately estimate whether or not the crew had a second intention to operate. The following description will illustrate an example where the occupant is the driver of the vehicle and the primary task is the operation of the vehicle. However, the present invention is not limited to such an example and can be broadly applied to various cases where there are tasks other than the operation being operated.

[0049] For example, driving operations may include steering the steering wheel, pressing the accelerator and brake pedals, using the turn signals, shift lever, and parking brake. Therefore, in the second embodiment, driving controls used for driving operations (e.g., steering wheel, accelerator pedal, brake pedal, turn signal, shift lever, parking brake, etc.) are excluded from the in-vehicle equipment 2 that is subject to the determination of the occupant's second operating intention.

[0050] The occupant operation estimation unit 11 estimates whether or not an occupant performed a driving operation based on signals generated by the behavior monitoring sensor 4 monitoring the occupant's behavior. For example, the occupant operation estimation unit 11 may extract feature quantities of the estimated occupant's joint position coordinates and use a database (e.g., a classifier) ​​that has learned the feature quantities of driving operations to estimate whether or not an occupant performed a driving operation. Alternatively, the occupant operation estimation unit 11 may estimate whether or not an occupant has performed a driving operation based on detection signals from sensors that detect operation inputs to the driving controls, or from contact sensors and pressure sensors that detect whether or not an occupant is in contact with the driving controls.

[0051] The event-related potential calculation unit 13 extracts event-related potentials generated by driving operations from brain activity data (e.g., electroencephalogram) output by the brain activity sensor 5. For example, the event-related potential calculation unit 13 may extract the P3 component generated in response to the driving operation, which is the main task, and its sensory stimuli (such as auditory or visual stimuli). For example, the event-related potential calculation unit 13 may extract the electroencephalogram potential generated approximately 300 milliseconds after the driving operation as the P3 component generated by the driving operation.

[0052] For example, the event-related potential calculation unit 13 may extract SPNs that occur in response to the driving operation and its sensory stimuli, which are the main tasks. For example, the event-related potential calculation unit 13 may predict the feedback time at which result information (feedback stimulus) for the driving operation is given to the occupant, and extract the electroencephalogram potentials that occur from the time of the driving operation to the feedback time as SPNs. For example, the electroencephalogram potential that occurs immediately before the feedback time may be extracted as an SPN.

[0053] For example, feedback stimuli may include the generation of yaw motion due to steering wheel operation, acceleration or deceleration due to operation of the accelerator or brake pedal, or changes in the meter display due to operation of the turn signal, shift lever, or parking brake. To predict the timing of feedback, the length of time elapsed from the time the driving operation is performed to the time of feedback may be set individually according to the type of driving operation. For example, a known fixed length may be set in advance according to the type of driving operation, or the length of time elapsed from the time the driving operation is performed to the time of feedback may be recorded in a database by detecting the occurrence of the driving operation and the feedback stimulus.

[0054] The second operation intent determination unit 14 calculates the amplitude α of the event potential generated by the sensory stimulus provided to the occupant from the in-vehicle device 2 as the occupant's level of attention to the target of operation (hereinafter referred to as "target of operation attention level Ao"). The second operation intent determination unit 14 also calculates the amplitude β of the event potential generated by the driving operation as the occupant's level of attention to the driving operation (hereinafter referred to as "driving attention level Ad"). The second operation intent determination unit 14 may also calculate the driving attention level Ad as the average value of the amplitude β of the event potential generated for multiple driving operations occurring within a predetermined period of time.

[0055] The second operation intent determination unit 14 estimates the occupant's attention allocation to the target of operation based on the attention level Ao of the target of operation and the attention level Ad of the driver. For example, the second operation intent determination unit 14 may estimate the occupant's attention allocation to the target of operation based on the attention ratio (Ad / Ao), which is the ratio of the attention level Ad of the driver to the attention level Ao of the target of operation. For example, if the attention ratio (Ad / Ao) is below a threshold, it can be determined that the attention level Ao of the target is sufficiently large and that the crew has a secondary intention to operate on the target. Conversely, if the attention ratio (Ad / Ao) is above a threshold, it can be determined that the attention level Ao of the target is small and that the crew does not have a secondary intention to operate on the target.

[0056] For example, the second operation intent determination unit 14 may calculate the amplitude of the P3 component generated by the sensory stimulus given from the operation target to the occupant as the operation target attention level Ao, and the amplitude β of the P3 component generated by the driving operation as the driving attention level Ad. However, it is known that the amplitude of the P3 component is greatly influenced by the frequency of the stimulus given to the subject; the higher the frequency, the smaller the amplitude of the P3 component generated in response to the same intensity stimulus.

[0057] Here, there is a difference in the frequency of occurrence between the driver's actions while the vehicle is in operation and the operation of the non-driver control device 2. The frequency of operation of the non-driver control device 2 is lower than the frequency of the driver's actions. Therefore, even if attention is focused on driving operations and less on the target of operation, if the frequency of operation on the target of operation is low, the attention level Ao on the target of operation will be high, leading to an overestimation of attention to the target of operation. Consequently, simply comparing the amplitude of the P3 component for driving operations with the amplitude of the P3 component for the target of operation to estimate attention allocation may result in estimation errors due to the frequency of stimuli.

[0058] For example, the second operation intent determination unit 14 calculates the amplitude β of the P3 component generated by the driving operation as the driver attention level Ad, and calculates the amplitude α of the SPN generated by the sensory stimulus given to the occupant in response to the operation of the target as the target attention level Ao. Because SPN has the characteristic of being less affected by the frequency of occurrence, even if the frequency of operations on the target is lower than the frequency of the main task, there is little risk of the target attention level Ao being overestimated.

[0059] Figure 4 shows an example of a scatter plot of the amplitude α of the P3 component generated by the driving operation and the amplitude β of the SPN in relation to the operation being controlled. The horizontal axis shows the amplitude of the P3 component in relation to the driving operation, and the vertical axis shows the amplitude of the SPN in relation to the operation being controlled. Figure 4 shows that a negative correlation exists between the amplitude β of the P3 component in relation to the driving operation and the amplitude α of the SPN in relation to the operation of the target being operated.

[0060] For example, the second operation intent determination unit 14 may estimate the occupant's attention allocation to the target of operation based on the total attention level (Ad+Ao), which is the sum of the driver attention level Ad and the target of operation attention level Ao, and the attention level ratio (Ad / Ao), which is the ratio of the driver attention level Ad to the target of operation attention level Ao. For example, the second operation intent determination unit 14 may determine whether the total attention level (Ad+Ao) is less than the first threshold Th1.

[0061] Figure 5 shows an example of an attention allocation judgment map used to estimate the attention allocation of the crew. The horizontal axis represents the attention level (Ao) of the target of operation, and the vertical axis represents the attention level (Ad) of driving. The discriminant line L1 is the straight line formed by attention levels (Ao, Ad) that satisfy the total attention level (Ad + Ao) = first threshold Th1. The discriminant line L2 is the straight line formed by attention levels (Ao, Ad) that satisfy the attention level ratio (Ad / Ao) = second threshold Th2.

[0062] If the total attention level (Ad+Ao) is less than the first threshold Th1, that is, if the attention level (Ao,Ad) is within the region R1 closer to the origin than the discrimination line L1, the second operation intent discrimination unit 14 may estimate that the occupant's level of alertness is reduced. This allows the system to determine whether the crew's level of attention to the object being operated is insufficient, or whether the crew's level of alertness is reduced in the first place (i.e., the total amount of attention the crew can afford to pay is insufficient). In this case, the second operation intent determination unit 14 may determine that the crew does not have a second operation intent towards the object being operated.

[0063] For example, the second operation intent determination unit 14 may determine whether the attention ratio (Ad / Ao) is equal to or greater than the second threshold Th2. If the total attention level (Ad+Ao) is less than the first threshold Th1 and the attention level ratio (Ad / Ao) is less than the second threshold Th2, that is, if the attention level (Ao,Ad) is farther from the origin than the discrimination line L1 and the driving attention level Ad is within the region R2 smaller than the discrimination line L2, the second operation intention determination unit 14 may estimate that the occupant's level of alertness has not decreased and that the occupant's attention level to the target of operation is sufficiently high. In this case, the second operation intention determination unit 14 may determine that the occupant has a second operation intention to the target of operation.

[0064] Furthermore, if the total attention level (Ad+Ao) is less than the first threshold Th1 and the attention level ratio (Ad / Ao) is greater than or equal to the second threshold Th2, that is, if the attention level (Ao,Ad) is farther from the origin than the discrimination line L1 and the driving attention level Ad is greater than the discrimination line L2, the second operation intention determination unit 14 may estimate that the occupant's level of alertness has not decreased, but the occupant's attention to the target of operation is low (i.e., they are concentrating on the driving operation rather than the target of operation). It may also determine that the occupant does not have a second operation intention towards the target of operation.

[0065] The second operation intent determination unit 14 may calculate the amplitude β of the SPN generated by the driving operation as the driver attention level Ad, and the amplitude α of the SPN generated by the sensory stimulus given to the occupant in response to the operation of the target as the target attention level Ao.

[0066] (Effects of the embodiment) (1) The controller 7 identifies the object being manipulated by the operator, provides sensory stimulation from the identified object to the operator, determines whether the operator has an intention to manipulate the object based on the operator's biological information when the sensory stimulation is provided from the object, and operates the object if it is determined that there is an intention to manipulate it. This improves the accuracy of determining whether an operation performed by an operator aligns with the operator's intent. As a result, it becomes possible to determine which target the operator intended to operate.

[0067] (2) The controller 7 may provide the operator with at least one of the following as a sensory stimulus from the object being operated: a visual stimulus, an auditory stimulus, or a tactile stimulus. This makes it possible to generate brain activity related to the level of attention given to the operation of the object being manipulated. (3) The controller 7 may calculate the operator's level of attention to the sensory stimuli given by the object being manipulated based on the operator's biological information, and may determine whether or not there is an intention to manipulate based on the level of attention to the sensory stimuli given by the object being manipulated. This makes it possible to determine whether or not the operator was paying attention to the object being operated on.

[0068] (4) The controller 7 may determine whether or not there is an intention to operate based on the allocation of attention between the operator's level of attention to the sensory stimuli given by the object being operated on and the operator's level of attention to the main task, which is a task that occurs more frequently than the operation on the object being operated on. This improves the accuracy of determining whether or not the operator was paying attention to the object being operated on.

[0069] (5) The brain activity sensor 5 may detect the operator's brain activity as biological information. The controller 7 may calculate event-related potentials for the sensory stimuli and operations of the main task and for the sensory stimuli given by the operation target, based on the brain activity detected by the brain activity sensor 5, and calculate attention allocation based on the calculated event-related potentials. This improves the accuracy of determining whether or not the operator was paying attention to the object being operated on.

[0070] (6) The controller 7 may calculate attention allocation based on the first event-related potential for the sensory stimuli and manipulation of the main task, and the second event-related potential for the sensory stimuli given by the object being manipulated. The second event-related potential may be an event-related potential that is correlated with the first event-related potential and is less affected by the frequency of occurrence than the first event-related potential. This improves the accuracy of determining whether or not the operator was paying attention to the object being operated on.

[0071] (7) The controller 7 may calculate a stimulus-response positive potential as the first event-related potential and a stimulus-preceding negative potential as the second event-related potential. This improves the accuracy of determining whether or not the operator was paying attention to the object being operated on. (8) The controller 7 may determine whether or not there is an intention of the operator to perform some operation, and if it determines that there is no intention of the operator to perform any operation, it may determine that there is no intention to perform an operation on the target of the operation. This improves the accuracy of determining the intent of the operator controlling the object being manipulated. [Explanation of Symbols]

[0072] 1...Operation support device, 2...In-vehicle equipment, 3...Operation intent determination device, 4...Behavior monitoring sensor, 5...Brain activity sensor, 6...Sensory stimulus generator, 7...Controller, 7a...Processor, 7b...Storage device, 10...First operation intent determination unit, 11...Occupant operation estimation unit, 12...Sensory stimulus generation unit, 13...Event-related potential calculation unit, 14...Second operation intent determination unit, 15...In-vehicle equipment control unit

Claims

1. Identify the object being manipulated by the operator, The identified target of the operation provides sensory stimulation to the operator. Based on the operator's biological information when sensory stimulation is applied from the object being manipulated, it is determined whether or not the operator has an intention to manipulate the object being manipulated. If it is determined that the aforementioned operation intention exists, the target of the operation is operated. A method for determining the intent of an operator, characterized by the features described above.

2. The method for determining the intent of an operation according to claim 1, characterized in that at least one of visual stimuli, auditory stimuli, or tactile stimuli is provided to the operator as a sensory stimulus from the object being operated.

3. Based on the operator's biological information, the operator's level of attention to the sensory stimuli given by the object being operated is calculated. The method for determining the intent to operate according to claim 1, characterized in that it determines whether or not there is an intent to operate based on the degree of attention to the sensory stimulus given from the object to be operated.

4. The method for determining the presence or absence of the intention to perform an operation according to claim 3, characterized in that it determines the presence or absence of the intention to perform the operation based on the allocation of attention between the operator's level of attention to sensory stimuli given from the object to be operated and the operator's level of attention to the main task, which is a task that occurs more frequently than the operation on the object to be operated.

5. The brain activity of the operator is detected as the aforementioned biological information. Based on the detected brain activity, event-related potentials for the sensory stimuli and manipulation of the main task, and event-related potentials for the sensory stimuli given by the object being manipulated are calculated. The attention allocation is calculated based on the calculated event-related potential. The method for determining the intent of an operation as described in feature 4.

6. The attention allocation is calculated based on the first event-related potential for the sensory stimulus and manipulation of the main task, and the second event-related potential for the sensory stimulus given by the object being manipulated. The second event-related potential is an event-related potential that is correlated with the first event-related potential and is less affected by the frequency of occurrence than the first event-related potential. The method for determining the intent of an operation according to feature 5.

7. The method for determining the intent of an operation according to claim 6, characterized in that a stimulus-response positive potential is calculated as the first event-related potential, and a stimulus-preceding negative potential is calculated as the second event-related potential.

8. Determine whether the operator intends to perform some kind of operation, If it is determined that the operator has no intention to perform any operation, then it is determined that there is no intention to operate on the target of the operation. The method for determining the intent of an operation as described in claim 1.

9. A first sensor detects the operator's behavior, A second sensor that detects the operator's biological information, A sensory stimulus generator provided on an object that can be operated by the operator and outputs sensory stimuli to be given to the operator, A controller that determines the object being operated by the operator based on the detection result of the first sensor, provides sensory stimulation to the operator from the sensory stimulation generator of the determined object, determines whether the operator has an intention to operate the object based on the operator's biological information detected by the second sensor when the sensory stimulation is provided from the sensory stimulation generator, and operates the object if it is determined that there is an intention to operate it. An operation intent determination device characterized by comprising the following:

Images