Abnormality detection device
The abnormality determination device identifies passengers in the driver's seat to detect abnormal conditions not visible in facial information, preventing accidents by allowing passenger intervention and automatic vehicle stop.
Patent Information
- Application Number
- DE112019003500
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-07-09
- Filing Date
- 2019-07-02
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2039-07-02
AI Technical Summary
Existing vehicle control devices fail to detect abnormal conditions of drivers that do not appear in facial information, such as when a passenger intervenes in driving due to the driver's abnormal state.
An abnormality determination device that identifies a person in the driver's seat using cameras and determines if a passenger is present in the operable area, triggering an abnormality process if a passenger is detected, including vehicle control to stop the vehicle if necessary.
Detects abnormal conditions of drivers that cannot be identified through facial information alone, preventing potential accidents by allowing passengers to intervene in driving and automatically stopping the vehicle when necessary.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Reference to related application
[0001] This international application is based on and claims priority from Japanese patent application No. JP 2020-9179 A, filed with the Japan Patent Office on July 9, 2018. The entire disclosure of Japanese patent application No. JP 2020-9179 A is incorporated herein by reference. Technical area
[0002] The present disclosure relates to an abnormality determination apparatus. State of the art
[0003] DE 11 2016 005 314 T5 discloses an automated driving assistance device that supports automated driving control of a passenger car, and includes a driver monitoring unit and a manual driving resumption level setting unit. The driver monitoring unit monitors the state of the driver driving the passenger car. The manual driving resumption level setting unit sets a level indicating whether or not a switch from automatic driving to manual driving can be performed in a predetermined switching zone, step by step, based on the driver's state detected by the driver monitoring unit.
[0004] DE 11 2018 006 579 B4 discloses an occupant detection device comprising: a face region detection unit for executing a process for detecting a plurality of face regions corresponding to a plurality of faces in an image captured by a camera for capturing an image of a vehicle interior; a first determination unit for determining whether or not there is a seat region including two or more face regions among a plurality of seat regions corresponding to a plurality of seats in the captured image; a second determination unit for determining whether or not a time duration of a state in which there is a seat region including two or more face regions has exceeded a reference time; and a warning output control unit for executing control for outputting a warning when the time duration exceeds the reference time.
[0005] JP 2016-85563 A discloses a vehicle control device that performs appropriate processing for safety when the vehicle driver is in an abnormal state that impedes driving. The vehicle control device determines whether the driver is in a drowsy state or an unresponsive state based on the driver state monitoring information and the driver's facial information captured in the image captured by the camera. If it is determined that the driver is in a drowsy state or an unresponsive state, vehicle control such as deceleration and stopping is executed. Summary of the invention
[0006] There are various possible abnormal states of the driver. A detailed study by the inventors has identified the following problem. That is, the vehicle control device that determines the driver's abnormal state based on information only from the driver, as described above, fails to detect the abnormal state that does not appear in the driver's facial information. Therefore, the determination of the driver's abnormal state is insufficient.
[0007] An object of the present disclosure is to make it possible to detect an abnormal condition of a driver that cannot be detected using information from the driver alone. This object is achieved by the abnormality determination device according to claims 1 and 5. Further developments according to the invention are the subject of the dependent claims.
[0008] According to one aspect of the present disclosure, a vehicle-mounted abnormality determination device is provided, including an identification unit, a person determination unit, and an abnormality processing unit. The identification unit is configured to identify a person in a captured image captured by a camera (1) that captures an area including a driver's seat of the vehicle. The person determination unit is configured to determine whether a person other than the driver of the vehicle is present in an operable area, which is set as an area where a steering wheel of the vehicle can be operated, based on an identification result from the identification unit.The abnormality processing unit executes an abnormality process assuming that the driver is in an abnormal state in response to the person determination unit determining that the person other than the driver is present in the operable area.
[0009] With such a configuration, it is possible to detect an abnormal state of the driver that cannot be detected using information from the driver alone. For example, if a passenger senses an abnormal state of the driver in a bus or the like, the passenger can intervene in the steering operation. In such a case, a passenger other than the driver is within the operable range. Therefore, even if the abnormal state of the driver cannot be detected using information from the driver alone, it is determined that someone other than the driver is within the operable range, and therefore the abnormal process is executed. Brief description of the drawings Fig. 1 is a block diagram showing a configuration of an abnormality determination system; Fig. 2 is a flowchart of an abnormality determination process according to a first embodiment; Fig. 3 is a diagram illustrating a setting area defined around a steering wheel in a two-dimensionally captured image; Fig. 4 is a timing chart showing an example that determines that the driver is in an abnormal state; and Fig. 5 is a flowchart of an abnormality determination process according to a second embodiment. Embodiments for carrying out the invention
[0010] Exemplary embodiments of the present disclosure are described below with reference to the drawings. [1. First Example][1-1. Configuration]
[0011] One in Fig. The abnormality determination system 100 shown in Fig. 1 is a system installed in a vehicle such as a bus that carries a large number of passengers; the abnormality determination system 100 includes cameras 1, an abnormality determination device 2, an HMI (Human Machine Interface) 3, and a vehicle control device 4.
[0012] Each camera 1 is an imager capable of capturing an image of an area including a driver's seat of a vehicle; for example, the cameras 1 are arranged at a plurality of respective positions in front of a driver of the vehicle. The camera 1 outputs a signal representing a captured image to the abnormality determination device 2. Based on the captured image acquired by the camera 1, the abnormality determination device 2 can specify the three-dimensional position of a person or the like included in the captured image.
[0013] The abnormality determination device 2 mainly comprises a known microcomputer including a CPU 21, a ROM 22, a RAM 23, a flash memory 24, and the like. The CPU 21 executes the program stored in the ROM 22, which is a non-transitory tangible storage medium. By executing this program, a process corresponding to the program is executed. Specifically, the abnormality determination device 2 executes a Fig. 2, which will be described later according to the program. The abnormality determination device 2 outputs a signal representing the result of the abnormality determination process to the HMI 3 and the vehicle control device 4. The abnormality determination device 2 may include one microcomputer or a plurality of microcomputers. In addition, the method for executing each function of the abnormality determination device 2 is not limited to software, and part or all of its functions may be performed using one or more hardware circuits. For example, while the above-described functions can be implemented by an electronic circuit that is hardware, the electronic circuit may be implemented by a digital circuit, an analog circuit, or a combination of these.
[0014] The HMI 3 has a display 31, a loudspeaker 32 and an enable switch 33.
[0015] The display 31 is a vehicle display that displays an image and is provided at a position where a driver of the vehicle can visually recognize it.
[0016] The speaker 32 is a vehicle speaker used together with a car navigation device, an audio device, and the like.
[0017] The release switch 33 is a switch provided in an area where the driver of the vehicle can operate or control, and can stop the notification of the driver's abnormal state. That is, as described later in the present embodiment, the driver's abnormal state is notified when it is determined that the driver is in an abnormal state. However, even if the driver is in a normal state, it may be the case that the driver's posture is determined to be abnormal because the driver's posture has slumped when attempting to pick up an object near the driver's seat. In such a case, the driver's abnormal state is notified. Therefore, the abnormality determination system 100 is configured to stop the notification by pressing the release switch 33.
[0018] The vehicle control device 4 includes a motor 41, a brake actuator 42, and a steering actuator 43. The vehicle control device 4 safely and automatically stops the vehicle when it is determined through the abnormality determination process described later that the driver is in an abnormal state. Specifically, the vehicle control device 4 brakes or stops the vehicle by, for example, controlling the motor 41 or the brake actuator 42 in the vehicle. Furthermore, the vehicle control device 4 returns the vehicle to the shoulder by, for example, also performing steering angle control of the steering through the steering actuator 43. [1-2. Process]
[0019] Next, the abnormality determination process executed by the abnormality determination device 2 will be described with reference to the flowchart of Fig. 2. This abnormality determination process is executed periodically while the ignition switch is on.
[0020] First, in S101, the abnormality determination device 2 identifies a person by analyzing a captured image captured by the camera 1. In this embodiment, a person is identified in the captured image using a known pattern matching method. In addition, in S101, the abnormality determination device 2 further recognizes a human face of the identified person by analyzing the captured image. In this embodiment, a human face is recognized based on the relative positions of facial feature points such as eyes and nose specified in the captured image.
[0021] Then, in S102, the abnormality determination device 2 determines whether the driver is present in the captured image based on the identification result in S101. Specifically, the abnormality determination device 2 determines whether the driver is present in the captured image based on the recognition result. In the present embodiment, the facial feature point information of the driver registered in advance in the flash memory 24 is compared with the facial feature point information of the person recognized in the captured image to identify the driver. That is, the facial feature point information of the driver registered in advance and the facial feature point information of the recognized person are compared to determine whether both correspond to the same person.If it is determined that both correspond to the same person, it is determined that the driver is present in the captured image. If the abnormality determination device 2 determines in S102 that the driver is not present in the captured image, the process proceeds to S107, which will be described later. That is, if the driver is not present in the captured image, it means that the driver is not present in the driver's seat. Therefore, it is determined that the driver is in an abnormal state.
[0022] On the other hand, when the abnormality determination device 2 determines in S102 that the driver is present in the captured image, the process proceeds to S103.
[0023] In S103, the abnormality determination device 2 determines the driver's posture in the captured image. The driver's posture is determined by detecting the inclination of the head relative to the body based on the identification result in S101.
[0024] Subsequently, in S104, the abnormality determination device 2 determines whether the driver's posture is normal. In the present embodiment, if an inclination of the driver's head with respect to the body is equal to or greater than a predetermined threshold and such an inclination continues for a predetermined period of time, it is determined that the driver's posture is abnormal. If the abnormality determination device 2 determines that the driver's posture is abnormal in S104, the abnormality determination device 2 proceeds the process to S107, which will be described later. In other words, if the driver's posture is abnormal, for example, if the driver is weak and the head is tilted, it is determined that the driver is in an abnormal state.
[0025] On the other hand, when the abnormality determination device 2 determines in S104 that the driver's posture is normal, the process proceeds to S105.
[0026] In S105, the abnormality determination device 2 determines whether a person other than the driver is present in the captured image based on the recognition result in S101. In the present embodiment, if a person other than the person identified as the driver in the captured image is identified, it is determined that a person other than the driver is present. If the abnormality determination device 2 determines in S105 that no person other than the driver is present in the captured image, the process returns to S101.
[0027] On the other hand, if the abnormality determination device 2 determines in S105 that a person other than the driver is present in the captured image, the process proceeds to S106.
[0028] In S106, the abnormality determination device 2 determines whether a person other than the driver is present in an operable area, which is predefined as an area in which the steering wheel of the vehicle can be operated. The operable area is, for example, an area in which the driver can operate the steering wheel to drive the vehicle. In the present embodiment, as shown in Fig. 3, the setting area A is set around the steering wheel in the two-dimensional captured image B. In the present embodiment, if a person other than the driver, who was determined to be present in S105, is not present in the setting area A in the captured image B, it is determined that no person other than the driver is present in the operable area. If it is determined in S106 that no person other than the driver is present in the operable area, the abnormality determination device 2 returns the process to S101. That is, if it is determined that the driver is in a normal state and a passenger is not near the steering wheel, it is determined that the driver is not in an abnormal state.
[0029] On the other hand, if the abnormality determination device 2 determines in S106 that a person other than the driver is present in the operable area, the abnormality determination device 2 proceeds the process to S107. That is, even in cases where it is determined that the driver is in a normal state, if a person other than the driver is present in the setting area A, it is highly likely that a passenger present near the steering wheel will interfere with the driving of the vehicle. Therefore, it is determined that the driver is in an abnormal state.
[0030] In S107, the abnormality determination device 2 determines that the driver is in an abnormal state and therefore notifies the driver of the vehicle of the abnormal state. In the present embodiment, the display 31 is used to display an image to notify the driver of the abnormal state, and the speaker 32 is used to notify the driver of the abnormal state through sounds or the like. Note that the abnormality determination device 2 executes S107 as the abnormality process.
[0031] Subsequently, in S108, the abnormality determination device 2 determines whether the release operation has been executed. In the present embodiment, whether the release operation has been executed is determined by the driver pressing the release switch 33. If the release switch 33 is pressed, it is determined that the release operation has been executed. If the abnormality determination device 2 determines in S108 that the release operation has not been executed, the process proceeds to S109.
[0032] In S109, the abnormality determination device 2 determines whether the notification state for notifying that the driver is in an abnormal state has continued for a predetermined period of time or longer. If the abnormality determination device 2 determines in S109 that the notification state has not continued for a predetermined period of time or longer, the process returns to S108.
[0033] If it is determined in S108 that the release operation has been performed, the abnormality determination device 2 returns the process to S101. That is, even in cases where the driver is notified of an abnormal state, it is determined that the driver is not in an abnormal state if the driver performs the release operation within a predetermined period of time.
[0034] On the other hand, when the abnormality determination device 2 determines in S109 that the notification state has continued for the predetermined period of time or longer, the process proceeds to S110.
[0035] In S110, the abnormality determination device 2 executes the control for automatically stopping the vehicle by the previously described vehicle control device 4. The abnormality determination device 2 executes S110 as the abnormality process. Thereafter, the abnormality determination device 2 ends the abnormality determination process of Fig. 2. [1-3. Effects]
[0036] According to the first embodiment described above, the following effects are achieved.
[0037] (1a) According to this embodiment, it is possible to detect an abnormal state of the driver which cannot be detected by the information of the driver alone.
[0038] Here, an example of a specific situation in which it is determined that the driver is in an abnormal state in the present embodiment will be explained by using the timing chart of Fig. 4 described.
[0039] At time T1, the driver becomes abnormal without losing his posture. In this case, it is conceivable that the driver's posture is determined to be normal and the driver is not determined to be abnormal. However, at time T2, when a passenger who has sensed an abnormal state of the driver intervenes in driving, a person other than the driver is present in the setting area A. Therefore, it is determined that a person other than the driver is present in the operable area, and it is determined that the driver is in an abnormal state. Then, at time T3, the driver's abnormal state is notified. Thereafter, if the notification state continues for a predetermined period or longer without the driver executing the release operation, control for automatically stopping the vehicle is executed at time T4.
[0040] As described above, if it is determined whether the driver is in the abnormal state based on information only about the driver, it may be erroneously determined that the driver is in the normal state. In contrast, the configuration of the present embodiment determines whether the driver is in an abnormal state by considering the information of the person other than the driver. Accordingly, it is possible to automatically stop the vehicle by detecting an abnormal state of the driver that cannot be detected from information only about the driver.
[0041] (1b) In the present embodiment, it is determined whether the notification state has continued for a predetermined period or longer. Then, even in cases where the driver is notified of the abnormal state, the notification is stopped if the driver performs the release operation within a predetermined period. Therefore, it is possible to prevent the automatic vehicle stop control from being executed even if the driver is notified of the abnormal state even though the driver is normal.
[0042] In the first embodiment, S101 corresponds to the process as an identification unit; S102 and S106 correspond to the process as a personal determination unit; and S107 and S110 correspond to the process as an abnormality processing unit. [2. Second Embodiment][2-1. Configuration]
[0043] Since the abnormality determination system of a second embodiment has the same basic configuration as that of the first embodiment, the same components as those of the first embodiment are denoted by the same reference numerals, and such description will be omitted. [2-2. Process]
[0044] Next, the abnormality determination process executed by the abnormality determination device 2 will be described with reference to the flowchart of Fig. 5. It should be noted that the processing of S209 to S212 in Fig. 5 is the same as the processing of S107 to S110 in Fig. 2.
[0045] In S201, the abnormality determination device 2 identifies a person by analyzing a captured image captured by the camera 1. In the present embodiment, unlike the first embodiment, recognition of the face of the identified person is not required.
[0046] In S202, the abnormality determination device 2 determines whether a person is present in the operable area based on the identification result in S201. If, in S202, the person identified in S201 is present in the setting area A, it is determined that a person is present in the operable area, as in S106 of the first embodiment. If the abnormality determination device 2 determines in S202 that a person is present in the operable area, the process proceeds to S203.
[0047] In S203, the abnormality determination device 2 determines whether two or more people are present in the operable area. If it is determined in S203 that two or more people are present in the operable area, the abnormality determination device 2 proceeds the process to S209. That is, if two or more people are present in the setting area A, the people presumed to be the driver and a passenger, where the passenger is a person other than the driver near the steering wheel, are likely to interfere with the driving of the vehicle. Therefore, it is determined that the driver is in an abnormal state.
[0048] On the other hand, if the abnormality determination device 2 determines in S203 that no more than two people are present in the operable area, that is, if there is a person who is assumed to be the driver in the operable area, the abnormality determination device 2 advances the process to S204.
[0049] In S204, the abnormality determination device 2 determines the posture of the person present in the operable area. The determination process in S204 is the same as the processing in S103 of the first embodiment.
[0050] Subsequently, in S205, the abnormality determination device 2 determines whether the posture of the person present in the operable area is normal. The determination process in S205 is the same as the processing in S104 of the first embodiment. If the abnormality determination device 2 determines in S205 that the posture of the person present in the operable area is normal, the abnormality determination device 2 returns the process to S201. That is, if it is determined that the person presumed to be the driver is in a normal state and the passenger is not near the steering wheel, it is determined that the driver is not in an abnormal state.
[0051] On the other hand, if the abnormality determination device 2 determines in S205 that the posture of the person present in the operable area is abnormal, the abnormality determination device 2 proceeds the process to S209. In other words, if the posture of the person presumed to be the driver is abnormal, for example, if the person presumed to be the driver loses his head and leans, it is determined that the driver is in an abnormal state.
[0052] On the other hand, if the abnormality determination device 2 determines in S202 that no person is present in the operable area, the process proceeds to S206. In other words, if no person is present in the operable area, the following cases are possible. That is, there is a possibility that the driver has lost his mind and disappeared from the operable area, or there is a possibility that a driver who intentionally disappears from the operable area because of picking up an object near the driver's seat will return to the detectable area. Therefore, in the subsequent processing from S206 to S208, it is determined whether the driver is in an abnormal state.
[0053] In S206, the abnormality determination device 2 determines whether a person has appeared in the operable area. In the present embodiment, if a person is present in the setting area A, it is determined that a person has appeared in the operable area. If the abnormality determination device 2 determines in S206 that no person has appeared in the operable area, the process proceeds to S207.
[0054] In S207, the abnormality determination device 2 determines whether the absence state in which no person appears in the operable area has continued for a predetermined period of time or longer. In the present embodiment, it is determined whether the absence state has continued for three (3) seconds or longer. If the abnormality determination device 2 determines in S207 that the absence state has continued for a predetermined period of time or longer, the process proceeds to S209. That is, there is a possibility that the driver's disappearance from the operable area is not due to an intentional action, and therefore, it is determined that the driver is in an abnormal state.
[0055] On the other hand, if the abnormality determination device 2 determines in S207 that the absence state has not continued for the predetermined period or longer, the process returns to S206. That is, if the absence state in the driver's operable range is less than 3 seconds, it is too early to determine that the driver is in an abnormal state, so it is not determined that the driver is in an abnormal state.
[0056] On the other hand, when the abnormality determination device 2 determines in S206 that a person appears in the operable area, the process proceeds to S208.
[0057] In S208, the abnormality determination device 2 determines whether (i) an appearing position, which is a position through which a person appears in the operable area, and (ii) a disappearing position, which is a position through which a person disappears from the operable area, are the same. Specifically, the abnormality determination device 2 determines whether an appearing position through which a person appears in the operable area in the current cycle is the same as a disappearing position through which a person disappears from the operable area in the previous cycle, which is stored in the flash memory 24 before the current cycle. Here, when there are a plurality of disappearing positions, the disappearing position used for the previous determination is the last disappearing position, that is, immediately before the appearing person appears at the appearing position in the current cycle.Then, the abnormality determination device 2 determines whether the appearing person and the disappearing person are the same person by determining whether the appearing position and the disappearing position are the same.
[0058] In the present embodiment, each of the appearing position and disappearing position in the operable area is classified into one of two positions, left and right, with respect to the driver's seat. As described below, it is determined whether the appearing person and the disappearing person are the same person.
[0059] The following first describes an example case where the appearing and disappearing positions are different. For example, the appearing position is on the left side of the driver's seat, and the disappearing position is on the right side of the driver's seat. In such a case, the driver may lose consciousness and disappear on the right side of the driver's seat; then, the passenger who intervenes in driving after the driver's abnormal state is determined may appear from the left side of the driver's seat. In this case, the driver is the disappearing person, and the passenger is the appearing person. Furthermore, it is unlikely that the driver disappears on the right side of the driver's seat and appears from the left side of the driver's seat.Therefore, if the appearing position and the disappearing position are different, it is assumed that the disappearing person is the driver and the appearing person is the passenger. It is thus determined that the appearing person and the disappearing person are not the same person.
[0060] The following describes an example case where the appearing position and the disappearing position are the same; for example, the appearing position is on the left side of the driver's seat, and the disappearing position is on the left side of the driver's seat. In such a case, the following situation is conceivable. That is, a person assumed to be a driver temporarily disappears to the right side of the driver's seat, for example, by intentionally bending down, but returns to the original posture to appear from the right side of the driver's seat after completing the intended action. For example, it is conceivable that the driver bent down to pick up an object near the driver's seat and disappeared from the operable range, but picked up the object and returned.Therefore, if the appearing position and the disappearing position are the same, it is very likely that the disappearing person and the appearing person are the driver, and thus the appearing person and the disappearing person are determined to be the same person.
[0061] If the abnormality determination device 2 determines in S208 that the appearing position and the disappearing position are not the same, the process proceeds to S209. That is, if the appearing person and the disappearing person are not the same person, it is determined that the driver is in an abnormal state.
[0062] On the other hand, if the abnormality determination device 2 determines in S208 that the appearing position and the disappearing position are the same, the process returns to S201. That is, since it is highly possible that the appearing person and the disappearing person are the same person, it is determined that the driver is not in an abnormal state. [2-3. Effects]
[0063] According to the second embodiment described in detail above, the following effects can be achieved in addition to the effects (1a) and (1b) of the first embodiment described above.
[0064] (2a) In this embodiment, when it is determined that the driver is in an abnormal state based on the appearing position and the disappearing position, the driver's abnormal state is notified. That is, based on the appearing position and the disappearing position, it is determined whether the appearing person and the disappearing person are the same person. Thus, it is possible to determine whether the driver is in an abnormal state by determining whether the appearing person and the disappearing person are the same person without identifying the driver by using face recognition.
[0065] (2b) In this embodiment, if a person appears in the operable area within a predetermined period of time after the person disappears from the operable area, it is not determined that the driver is in an abnormal state. For example, consider a case where a driver intentionally disappears from the operable area because he or she picks up an object near the driver's seat and then returns to the detectable area within a predetermined period of time. Even in such a case, the configuration according to the second embodiment can prevent the driver from being erroneously determined to be in an abnormal state.Additionally, in the present embodiment, if the person does not appear in the operable area even after a predetermined period of time has elapsed after the person has disappeared from the operable area, it is determined that the driver is in an abnormal state. Therefore, for example, if (i) the driver loses his mind and disappears from the operable area, and (ii) no passenger intervenes in driving, the driver can be determined to be in an abnormal state by detecting the driver's abnormal state.
[0066] In the second embodiment, S201 corresponds to the process as an identification unit, S202 corresponds to the process as a presence determination unit, S203 corresponds to the process as a person determination unit, and S209 and S212 correspond to the process as an abnormality processing unit. [3. Further examples]
[0067] While the embodiment of the present disclosure has been described above, the present disclosure is not limited to the foregoing embodiment and may be modified in various ways.
[0068] (3a) In the first embodiment, an exemplary configuration is shown in which the driver is identified by comparing the facial feature point information of the pre-registered driver, and the facial feature point information of the recognized person has been exemplified. The face recognition method for identifying the driver is not limited to this. For example, when the ignition switch is turned on, information on the facial feature points of the person sitting in the driver's seat may be stored as driver information for comparison. In addition, even if the driver is identified by additionally comparing the human facial feature point information acquired in the previous cycle before the current cycle and the human facial feature point information acquired in the current cycle.
[0069] (3b) In the second embodiment described above, an exemplary configuration is shown in which each of the positions, appearing position and disappearing position, in the operable range is classified into one of two positions, left and right positions, with respect to the driver's seat. The classification method is not limited to this. For example, it may be classified into one of four positions: left, right, upper, and lower. In such a case, for example, when the appearing position is the upper left side of the driver's seat and the disappearing position is one of the lower right side, upper right side, and lower left side of the driver's seat, it can be determined that the appearing position and the disappearing position are not the same.That is to say, not only when the left and right positions of the appearing position and the disappearing position are different, but even when the left and right positions are the same but the upper and lower positions are different, it can be determined that the appearing position and the disappearing position are not the same.
[0070] (3c) The setting area may be a setting area defined in three-dimensional space. That is, in each of the above embodiments, the rectangular area in the captured image is defined as the setting area A. However, it is not necessary to be limited to this. For example, a rectangular parallelepiped area in which the setting area A has a certain depth in each of the above-described embodiments may be defined as a setting area.
[0071] (3d) In the second embodiment, an example case will be described in which, when the appearing position and the disappearing position are the same, it is determined that the appearing person and the disappearing person are the same person, and it is determined that the driver is not in an abnormal state. However, the method for determining whether the driver is in an abnormal state is not limited to this. For example, when the appearing person appears in the operable area from outside the setting area, that is, outside the operable area, it can be determined that the driver is in an abnormal state regardless of the direction in which the person appears.
[0072] (3e) In each of the foregoing embodiments, the abnormality determination system 100 may be mounted on a vehicle other than a bus.
[0073] (3f) A function of one configuration element in the above-described embodiments may be implemented by multiple configuration elements. Functions of multiple configuration elements may be implemented by one configuration element. Part of the configuration according to the above-described embodiments may be omitted. At least part of the configuration of the above embodiments may be supplemented or replaced by another configuration of the above embodiment, or the like.
[0074] (3g) The present disclosure can be implemented in various forms in addition to the above-described abnormality determination device 2, such as a system including the abnormality determination device 2 as a component, a program that causes a computer to function as the abnormality determination device 2, a medium in which the program is stored, and an abnormality determination method.
Claims
[1] An abnormality determination device (2) mounted on a vehicle, comprising: an identification unit (S101, S201) configured to identify a person in a captured image captured by a camera (1) that captures an image of an area including a driver's seat of the vehicle; a person determination unit (S106, S203) configured to determine whether two or more people are present in an operable area defined as an area in which a steering wheel of the vehicle can be operated based on an identification result from the identification unit (S101, S201); and an abnormality processing unit (S107, S110, S209, S212) configured to execute an abnormality process assuming that the driver is in an abnormal state in response to the person determination unit (S106, S203) determining that two or more people are present in the operable area, wherein: the abnormality processing unit (S107, S110, S209, S212) is configured to execute the abnormality process based on (i) a disappearing position at which a person disappears from the operable area and (ii) an appearing position at which a person appears in the operable area. [2] An abnormality determination device (2) according to claim 1, wherein: the identification unit (S101, S201) is further configured to recognize a human face in the captured image; the person determination unit (S106, S203) is configured to determine whether a person other than the driver is present in the operable area based on the recognition result from the identification unit (S101, S201); and the abnormality processing unit (S107, S110, S209, S212) is configured to execute the abnormality process in response to the person determination unit (S106, S203) determining that the person other than the driver is present in the operable area. [3] An abnormality determination device (2) according to claim 1, further comprising: a presence determination unit (S202) configured to determine whether a person is present in the operable area based on the identification result from the identification unit (S101, S201), where: each of the positions, disappearing position and appearing position, is classified into one of two positions, a left position and a right position with respect to the driver's seat; and in cases where a person appears in the operable area after the presence determination unit (S202) has determined that no person is present in the operable area, the abnormality processing unit (S107, S110, S209, S212) is configured to carry out the abnormality process in response to the appearing position and the immediately preceding disappearing position being different from each other, and not to execute the abnormality process in response to the appearing position and the immediately preceding disappearing position being the same. [4] An abnormality determination device (2) according to claim 3, wherein: in response to a state in which no person appears in the operable area, which lasts for a predetermined period of time or longer after the presence determination unit (S202) has determined that no person is present in the operable area, the abnormality process is executed by the abnormality processing unit (S107, S110, S209, S212). [5] An abnormality determination device (2) mounted on a vehicle, comprising: an identification unit (S101, S201) configured to identify a person in a captured image captured by a camera (1) that captures an image of an area including a driver's seat of the vehicle; a person determination unit (S106, S203) configured to determine whether a passenger other than a driver of the vehicle is present in an operable area set as an area in which a steering wheel of the vehicle can be operated based on an identification result from the identification unit (S101, S201); and an abnormality processing unit (S107, S110, S209, S212) configured to execute an abnormality process assuming that the driver is in an abnormal state in response to the person determination unit (S106, S203) determining that the passenger is present in the operable area, wherein: the person determination unit (S106, S203) determines whether two or more persons are present in the operable area based on the identification result from the identification unit (S101, S201); and the person determination unit (S106, S203) is configured to determine that the passenger is present in the operable area in response to determining that two or more people are present, wherein: the abnormality processing unit (S107, S110, S209, S212) is configured to execute the abnormality process based on (i) a disappearing position at which a person disappears from the operable area and (ii) an appearing position at which a person appears in the operable area. [6] An abnormality determination device (2) according to claim 5, wherein: the identification unit (S101, S201) is further configured to recognize a human face in the captured image; and the person determination unit (S106, S203) is configured to determine whether the passenger is present in the operable area based on the recognition result from the identification unit (S101, S201). [7] An abnormality determination device (2) according to claim 5, further comprising: a presence determination unit (S202) configured to determine whether a person is present in the operable area based on the identification result from the identification unit (S101, S201), where: each of the positions, disappearing position and appearing position, is classified into one of two positions, a left position and a right position with respect to the driver's seat; and in cases where a person appears in the operable area after the presence determination unit (S202) has determined that no person is present in the operable area, the abnormality processing unit (S107, S110, S209, S212) is configured to carry out the abnormality process in response to the appearing position and the immediately preceding disappearing position being different from each other, and not to execute the abnormality process in response to the appearing position and the immediately preceding disappearing position being the same. [8] An abnormality determination device (2) according to claim 7, wherein: in response to a state in which no person appears in the operable area, which lasts for a predetermined period of time or longer after the presence determination unit (S202) has determined that no person is present in the operable area, the abnormality process is executed by the abnormality processing unit (S107, S110, S209, S212).
Citation Information
Patent Citations
automated driving assistance device, automated driving assistance system, automated driving assistance method and automated driving assistance program
DE112016005314T5
Occupant detection device and occupant detection method
DE112018006579B4
Transmitted wavefront interferometer
JP2008209179A
On-vehicle control device
JP2016085563A
JP002016085563A
