Abnormal attitude determination device, abnormal attitude determination method, and vehicle control system

Abstract

The present invention comprises: a feature detecting unit (13) for posture determination that, on the basis of the face of an occupant in a captured image, detects the face orientation of the occupant, the head position of the occupant, and the positions of parts of the face of the occupant; a posture type determining unit (151) that performs posture collapse type determination by comparing the face orientation and head position of the occupant with the face orientation and head position in a reference posture to determine whether the posture of the occupant corresponds to any of a plurality of posture collapse types, and when the face of the occupant could not be detected, determines that the posture of the occupant corresponds to lying down ; a forward leaning determining unit (152) that performs forward leaning determination to determine whether the posture of the occupant corresponds to forward leaning on the basis of the positions of parts of the face of the occupant based on a detection captured image when the posture type determining unit (151) has determined that the posture of the occupant corresponds to lying down; and an abnormal posture determining unit (153) that determines whether the posture of the occupant is an abnormal posture on the basis of the posture collapse type determination result and the forward leaning determination result.

Description

[Technical field]

[0001] The present disclosure relates to an abnormal attitude determination device, an abnormal attitude determination method, and a vehicle control system. [Background technology]

[0002] Conventionally, there is known a technology for detecting an abnormal state of an occupant by judging the occupant's posture collapse based on an image of the occupant in a vehicle. One of the posture collapse patterns is "head down" in which the occupant falls forward and continues to maintain a posture with the face close to the steering wheel. For example, Patent Document 1 discloses an image processing device that identifies a nose feature point and a left shoulder feature point of a driver captured in an image, and estimates that the driver is head down if the amount of decrease in the distance between the nose feature point and the left shoulder feature point in the vertical direction of the moving body exceeds a threshold value. Incidentally, in recent years, imaging devices that are mounted on vehicles and capture images of the interior of the vehicle have become increasingly smaller, which may result in a narrower imaging range of the imaging devices. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-149442 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, when checking a sign ahead, a vehicle occupant may lean forward, or assume a so-called "leaning forward" posture. This "leaning forward" is not caused by the occupant being in an abnormal state. In other words, it does not fall under the category of posture loss that is used to detect an abnormal state of the occupant. Here, whether the occupant is "leaning forward", which is a posture error, or "leaning forward", which is not a posture error, the occupant leans forward, so the occupant's face may not be captured in the captured image. In particular, in images captured by imaging devices with narrower imaging ranges in recent years, there is a high possibility that the occupant's face will not be captured in the captured image if the occupant is "leaning forward" or "leaning forward". With conventional technology, when the occupant's face is no longer captured in the captured image, it is not possible to distinguish whether the occupant is in a "prone" or "leaning forward" position, which could result in the occupant being mistakenly detected as being in a "prone" position when in fact the occupant is in a "leaning forward" position. Incidentally, the technology of the image processing device disclosed in Patent Document 1 does not take into consideration the case where the face of the occupant is no longer captured in the captured image, and therefore the above problem remains unresolved.

[0005] The present disclosure has been made to solve such problems, and aims to provide an abnormal posture determination device that makes it possible to prevent an occupant's "leaning forward" posture from being mistakenly detected as a "prone" posture. [Means for solving the problem]

[0006] The abnormal posture determination device according to the present disclosure includes a face detection unit that detects the face of a vehicle occupant in an image captured in an area in which the face of the occupant should be, a posture determination feature detection unit that detects the facial direction of the occupant, the position of the occupant's head, and the positions of the occupant's facial features based on the face of the occupant in the image detected by the face detection unit, and a posture determination feature detection unit that performs a posture deviation type determination as to whether the occupant's posture falls into one of a plurality of posture deviation types by comparing the facial direction and head position of the occupant detected by the posture determination feature detection unit with the facial direction and head position of the occupant in a reference posture in which it is assumed that the occupant is not experiencing posture deviation, and if the face detection unit cannot detect the occupant's face, the occupant's posture is determined to fall into one of a plurality of posture deviation types. The vehicle is equipped with a posture type determination unit which determines that the occupant's posture corresponds to prone position among the posture collapse types, a forward lean determination unit which, if the posture type determination unit determines that the occupant's posture corresponds to prone position, makes a forward lean determination as to whether the occupant's posture corresponds to forward lean, which does not correspond to any of the multiple posture collapse types, based on the positions of the occupant's facial features based on the detected captured image, which is an captured image traced back from when the occupant's posture was determined to correspond to prone position and when the face detection unit was able to detect the occupant's face, and an abnormal posture determination unit which determines whether the occupant's posture is an abnormal posture based on the posture collapse type determination result by the posture type determination unit and the forward lean determination result by the forward lean determination unit. Effect of the Invention

[0007] According to the abnormal posture determination device of the present disclosure, it is possible to prevent an occupant's "leaning forward" posture from being mistakenly detected as a "prone" posture. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of an abnormal posture determination device according to a first embodiment. [Diagram 2] 11A and 11B are diagrams for explaining a plurality of types of poor posture. [Diagram 3]Figures 3A and 3B are diagrams illustrating an example of the transition in the position of the driver's face on the captured image when the driver assumes a "prone" position and the driver's face is no longer captured in the captured image, and when the driver assumes a "leaning forward" position and the driver's face is no longer captured in the captured image. [Figure 4] 4 is a diagram for explaining a specific method of forward lean determination performed by a forward lean determination unit in the first embodiment. FIG. [Diagram 5] 1 is a diagram showing a configuration example of an operation control system according to a first embodiment. [Figure 6] 4 is a flowchart for explaining the operation of the abnormal posture determination device according to the first embodiment. [Figure 7] 4 is a flowchart showing an example of the operation of a driving assistance function by the driving control system according to the first embodiment. [Figure 8] 8A and 8B are diagrams illustrating an example of a hardware configuration of the abnormal posture determination device according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1 The abnormal posture determination device according to the first embodiment determines whether or not the posture of a vehicle occupant is an abnormal posture, based on a captured image in which at least the face of the vehicle occupant is captured. In the following first embodiment, as an example, the occupant of the vehicle, who is the subject of the abnormal attitude determination device for determining whether or not the vehicle is in an abnormal attitude, is the driver of the vehicle.

[0010] FIG. 1 is a diagram showing an example of the configuration of an abnormal posture determination device 1 according to the first embodiment. The abnormal posture determination device 1 according to the first embodiment is assumed to be mounted on a vehicle. The abnormal posture determination device 1 is connected to an image capture device 2 and an output device 3 , and the abnormal posture determination device 1 , the image capture device 2 and the output device 3 constitute an abnormal posture determination system 100 . The abnormal attitude determination system 100, together with the driving control system 400, the steering mechanism 5, and the braking / driving mechanism 6, constitutes a vehicle control system SYS.A.

[0011] The imaging device 2 is mounted on a vehicle and installed so as to be able to capture at least an area in which the driver's face should be present. In the first embodiment, it is assumed that the imaging device 2 is installed, for example, near the center of the instrument panel in the vehicle width direction or in the center console. For example, the imaging device 2 may be shared with a so-called DMS (Driver Monitoring System) that is installed for the purpose of monitoring the interior of the vehicle. The imaging device 2 is a visible light camera or an infrared camera. The imaging device 2 outputs the captured image to the abnormal posture determination device 1. The abnormal posture judging device 1 judges whether or not the posture of the driver is abnormal based on the captured image captured by the imaging device 2. The abnormal posture judging device 1 will be described in detail later. When the abnormal posture determination device 1 determines that the posture of the driver is abnormal, it causes the output device 3 to output a warning that the posture of the driver is abnormal.

[0012] The output device 3 is mounted on the vehicle and outputs a warning that the driver's posture is abnormal. The output device 3 is, for example, an audio output device such as a speaker. The output device 3 may be, for example, provided in an audio device provided in a vehicle. When the abnormal posture determination device 1 outputs information for outputting a warning (hereinafter referred to as "warning information"), the output device 3 outputs a warning sound or a voice message notifying that the driver's posture is abnormal. The output device 3 may be, for example, a display device such as a display. For example, when warning information is output from the abnormal posture determination device 1, the output device 3 displays a message notifying that the driver's posture is abnormal. The output device 3 may be, for example, a turn signal, a hazard lamp, or a headlight provided on the exterior of the vehicle so as to be visible from other vehicles. For example, the abnormal attitude determination device 1 may output warning information to the output device 3 to notify a person outside the vehicle, such as a passenger of the other vehicle, that the driver of the vehicle is in an abnormal state. The output device 3 may be mounted on, for example, another vehicle. The abnormal posture determination device 1 can transmit warning information to the output device 3 mounted on the other vehicle, and can output a voice or display a message to inform the passengers of the other vehicle that the driver of the vehicle is in an abnormal state.

[0013] Moreover, the abnormal posture determination device 1 outputs the determination result as to whether or not the driver's posture is an abnormal posture to a driving control system 400 connected to the abnormal posture determination system 100. 1, the abnormal attitude determination system 100 is connected to a driving control system 400. The driving control system 400 cooperates with the steering mechanism 5 or the braking / driving mechanism 6, and performs driving assistance using the determination result of the abnormal attitude determination system 100. Details of the driving control system 400 will be described later.

[0014] As shown in FIG. 1, abnormal posture determination device 1 includes image acquisition unit 11, face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16. The posture determining feature detection unit 13 includes a face direction detection unit 131 , a head position detection unit 132 , and a features position detection unit 133 . The determination unit 15 includes a posture type determination unit 151 , a forward lean determination unit 152 , and an abnormal posture determination unit 153 .

[0015] The image acquisition unit 11 acquires a captured image from the imaging device 2 . The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12.

[0016] The face detection unit 12 detects the face of the driver in the captured image acquired by the image acquisition unit 11 . In detail, the face detection unit 12 detects the driver's facial feature points indicating the driver's facial parts by using a known image recognition technique such as edge detection for the captured image. The facial parts are the corners and corners of the eyes, the nose, the mouth, the eyebrows, the chin, and the like. The driver's facial feature points are represented, for example, by coordinates on the captured image. The face detection unit 12 may also detect the driver's facial area. The driver's facial area is, for example, the smallest rectangle that surrounds the outline of the driver's face. The driver's facial area is, for example, represented by the coordinates of the four corners of the smallest rectangle on the captured image. Since the installation position and the angle of view of the imaging device 2 are known in advance, even if a plurality of passengers are captured in the captured image, the face detection unit 12 can determine which area of ​​the captured image is the face of the driver. For example, an area in the captured image where the driver's face may exist (hereinafter referred to as the "driver detection area") is set in advance, and the face detection unit 12 detects the facial feature points and face area of ​​the driver in the driver detection area using a known image recognition technique. The face detection unit 12 outputs information relating to the detected face of the driver (hereinafter referred to as “face information”) to the posture determination feature detection unit 13. The face information is, for example, a captured image to which information capable of identifying the driver's facial feature points and facial area has been added.

[0017] The posture determination feature detection unit 13 detects features for determining the posture of an occupant, in this case, the driver. The features for determining the posture of the driver are the driver's face direction, the driver's head position, and the positions of the driver's facial features. In the first embodiment, the driver's face direction and head position detected by the posture determination feature detection unit 13 are, for example, the face direction and head position in real space. In the first embodiment, the positions of the driver's facial features detected by the posture determination feature detection unit 13 are, for example, the positions of the facial features on a captured image. In the first embodiment, as an example, the driver's facial features for determining the driver's posture are the driver's eyes. In the following description, it is assumed that the driver's facial features are the driver's eyes.

[0018] The facial direction detection unit 131 of the posture determination feature detection unit 13 detects the facial direction of the driver in real space based on the facial information output from the face detection unit 12, specifically, based on the driver's face in the captured image detected by the face detection unit 12. The face direction detection unit 131 may detect the face direction of the driver using, for example, a known face direction detection technique for detecting the face direction from a captured image. The face direction of the driver is represented, for example, by an angle (yaw angle, pitch angle, or roll angle) relative to a predetermined reference axis (hereinafter referred to as the "reference axis") in the front-rear direction of the vehicle. The face direction detection unit 131 outputs information relating to the detected face direction of the driver (hereinafter referred to as “face direction information”) to the reference posture estimation unit 14 and the determination unit 15. The face direction information includes information on the yaw angle, pitch angle, or roll angle with respect to a reference axis.

[0019] The head position detection unit 132 of the posture determination feature detection unit 13 detects the head position of the driver in real space based on the face information output from the face detection unit 12, more specifically, based on the face of the driver in the captured image detected by the face detection unit 12. In the first embodiment, the head position of the driver in the captured image is indicated, for example, by the center of the driver's eyebrows. The head position detection unit 132 detects, for example, a point in real space corresponding to the center of the driver's eyebrows in the captured image as the driver's head position. Note that this is only an example, and the position of the driver's head in the captured image may be indicated, for example, by the center of the driver's face area or the center of a straight line connecting the driver's eyes. In this case, the head position detection unit 132 detects, for example, a point in real space corresponding to the center of the driver's face area in the captured image or the center of a straight line connecting the driver's eyes as the driver's head position in real space. The head position detection unit 132 may detect the head position of the driver by using, for example, a known coordinate conversion technique for converting points on a captured image into points in real space. The driver's head position is represented, for example, by coordinates in real space. Head position detection section 132 outputs information relating to the detected head position of the driver (hereinafter referred to as “head position information”) to reference posture estimation section 14 and determination section 15. The head position information includes coordinate information of the driver's head position.

[0020] The part position detection unit 133 of the posture determination feature detection unit 13 detects the position of the driver's eyes on the captured image based on the face information output from the face detection unit 12, specifically, based on the face of the driver in the captured image detected by the face detection unit 12. The face detection unit 12 detects the positions of the driver's right eye and left eye on the captured image. In the first embodiment, the position of the driver's eyes is the center of a straight line connecting the inner corner and the outer corner of the driver's eye on the captured image. The part position detection unit 133 may detect the position of the driver's eyes by using, for example, a known image recognition technology. The position of the driver's eyes is represented, for example, by coordinates on the captured image. Parts position detection section 133 outputs information relating to the detected position of the driver's eyes (hereinafter referred to as “parts position information”) to reference posture estimation section 14 and determination section 15. The part position information includes coordinate information of the position of the driver's right eye and coordinate information of the position of the driver's left eye.

[0021] The reference posture estimation unit 14 estimates the driver's reference posture based on the driver's facial direction, head position, and eye position detected by the posture determination feature detection unit 13 during a predetermined period (hereinafter referred to as the "reference setting period"). The reference set period is, for example, a period from when the vehicle ignition is turned on until a preset time (for example, 10 seconds) has elapsed. Note that this is merely an example, and the reference set period may be, for example, a predetermined time (for example, 3 seconds) when the vehicle speed is equal to or greater than a preset speed (for example, 25 [km / h]) and the steering angle is within a preset range (for example, ±20 degrees). The reference posture estimation unit 14 estimates, for example, the average values ​​of the driver's facial direction, head position, and eye position detected by the posture determination feature detection unit 13 during the reference setting period as the driver's facial direction, head position, and eye position in the driver's reference posture. The reference posture estimation unit 14 outputs to the determination unit 15 information relating to the estimated reference posture of the driver (hereinafter referred to as “reference posture information”). The reference posture information includes information on the driver's facial orientation, head position, and eye position in the reference posture. The driver's facial orientation in the reference posture is represented by a yaw angle, pitch angle, or roll angle with respect to a reference axis. The driver's head position in the reference posture is represented, for example, by coordinates in real space. The driver's eye position in the reference posture is represented, for example, by coordinates on a captured image. The reference posture information includes information on the driver's facial orientation, such as a yaw angle, pitch angle, or roll angle with respect to a reference axis. The reference posture information also includes information on the driver's head position, such as coordinate information on the driver's head position. The reference posture information also includes information on the driver's eye position, such as coordinate information on the driver's right eye position and coordinate information on the driver's left eye position.

[0022] The determination unit 15 determines whether or not the driver's posture is abnormal based on the face direction, head position, and eye position of the driver detected by the posture determination feature detection unit 13.

[0023] The posture type determination unit 151 of the determination unit 15 determines whether or not the driver's posture corresponds to any one of a plurality of predetermined posture error types (hereinafter referred to as "posture error types") based on the driver's face direction, head position, and eye position detected by the posture determination feature detection unit 13. In the first embodiment, the determination by the posture type determination unit 151 of whether the driver's posture corresponds to any one of a plurality of posture error types is also referred to as "posture error type determination." The plurality of posture deviation types are set in advance as types of abnormal postures that are caused by the occupant being in an abnormal state.

[0024] Here, Figure 2 (partially modified from the "Basic Design Document for Driver Abnormality Automatic Detection System" of the Advanced Safety Vehicle Promotion Study Group, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, March 2018) is a diagram to explain the various types of posture problems. As shown in Fig. 2, there are multiple modes of posture failure. Fig. 2 shows "face down" in which the driver falls forward and continues to face near the steering wheel, "head down" in which the driver continues to face downward, "backward lean" in which the driver's upper body is tilted backward and the face continues to face upward, "backward lean" in which the driver's upper body is arched up and the face continues to face upward, "neck only sideways" in which the driver's face continues to be tilted to the left or right, "sideways" in which the driver's upper body is tilted to the left or right and the face continues to be tilted in the same direction, and "leaning to the side" in which the driver's upper body is tilted to the left or right. In the present disclosure, the multiple pre-defined types of posture problems are "face down", "head down", "backward arch", "arched back", "head only sideways", "sideways", and "leaning" shown in Fig. 2. Information on the multiple types of posture problems is stored in a location that can be referenced by posture type determination unit 151.

[0025] The posture type determination unit 151 performs posture correction type determination as to whether the driver falls into any of a plurality of posture correction types by comparing the driver's facial direction detected by the facial direction detection unit 131 and the driver's head position detected by the head position detection unit 132 with the driver's facial direction in a reference posture (hereinafter referred to as the "reference facial direction") and the driver's head position (hereinafter referred to as the "reference head position"). In detail, the posture type determination unit 151 determines which of a plurality of posture failure types the driver belongs to, depending on how much the driver's face direction has changed compared to a reference face direction, or how much the driver's head position has changed compared to a reference head position. Note that the conditions for determining which posture failure type the driver belongs to when the driver's face direction has changed compared to the reference face direction, or when the driver's head position has changed compared to the reference head position, are set in advance and stored in the posture type determination unit 151. The posture type determining section 151 may determine the posture defect type by using a learning device such as a Support Vector Machine (SVM).

[0026] Here, if the face detection unit 12 fails to detect the driver's face, the posture type determination unit 151 determines that the driver's posture corresponds to "prone" among the multiple posture disorders. The case where the face detection unit 12 fails to detect the driver's face means that the posture determination feature detection unit 13 fails to detect the driver's facial direction, head position, and part positions. The posture type determination unit 151 can grasp, via the posture determination feature detection unit 13, that the face detection unit 12 fails to detect the driver's face. When the driver falls forward, it becomes increasingly difficult to capture the driver's face in the captured image, and eventually the entire driver's face may not be captured. In particular, if the imaging device 2 is a narrow-angle camera with a narrow imaging range, there is a high possibility that the driver's face will not be captured if the driver falls forward. Therefore, when the face detection unit 12 fails to detect the face of the driver, the posture type determination unit 151 assumes that this is due to the driver having fallen forward, and determines that the driver's posture corresponds to "prone". Note that the determination of the driver's posture as "prone" by the posture type determination unit 151 due to the driver's face not being detected is a provisional, temporary determination, and the final determination of whether the driver's posture can be "prone" is made by the abnormal posture determination unit 153. Details of the abnormal posture determination unit 153 will be described later.

[0027] The posture type determination unit 151 outputs the posture error type determination result to the lean forward determination unit 152. The posture error type determination result includes information indicating whether the driver's posture falls into any one of a plurality of posture error types, and, when it is determined that the driver's posture falls into any one of the plurality of posture error types, information capable of identifying the posture error type.

[0028] When the posture type determination unit 151 determines that the driver's posture corresponds to "prone", the leaning forward determination unit 152 of the determination unit 15 determines whether the driver's posture corresponds to "leaning forward". In the first embodiment, the determination by the leaning forward determination unit 152 of whether the driver's posture corresponds to "leaning forward" is also referred to as "leaning forward determination". The "leaning forward" posture is a posture in which the body is inclined as if it is about to fall forward. For example, the driver may assume the "leaning forward" posture when checking a sign in front of the vehicle. As shown in Figure 2, "leaning forward" is a type of posture that does not fall into any of the multiple posture collapse types. In other words, "leaning forward" cannot be considered an abnormal posture caused by the occupant being in an abnormal state.

[0029] As described above, when the driver's face is no longer captured in the captured image and the face detection unit 12 cannot detect the driver's face, the posture type determination unit 151 provisionally determines the posture to be "prone". However, even when the driver assumes a "leaning forward" posture, the driver leans his / her body forward in the same manner as when the driver assumes a "prone" posture. Therefore, if the abnormal posture determination device 1 uniformly determines that the driver's posture is "prone" because the driver's face is no longer captured in the captured image, there is a possibility that the abnormal posture of the driver will be erroneously detected. Therefore, when the posture type determination unit 151 determines that the driver's posture corresponds to "prone", the leaning forward determination unit 152 determines whether the determination is incorrect and the driver's posture is not "leaning forward".

[0030] Here, Figures 3A and 3B are figures for explaining an example of the transition in the position of the driver's face on the captured image when the driver assumes a "prone" position and the driver's face is no longer captured on the captured image, and when the driver assumes a "leaning forward" position and the driver's face is no longer captured on the captured image. Fig. 3A shows an example of the transition of the position of the driver's face on the captured image when the driver's face is no longer captured on the captured image because the driver has taken a "prone" posture, and Fig. 3B shows an example of the transition of the position of the driver's face on the captured image when the driver has taken a "leaning forward" posture. In Fig. 3A and Fig. 3B, the captured image is indicated by Im, the driver is indicated by Dr, and the seat is indicated by S. It is assumed that the imaging device 2 that captured the images shown in FIGS. 3A and 3B is a narrow-angle camera. In the first embodiment, the upper left corner of the captured image is the origin, the rightward direction of the captured image is the positive direction of the x-axis, and the downward direction of the captured image is the direction of the y-axis.

[0031] As shown in FIG. 3A, when the driver assumes a "prone" position, the driver's face moves downward and to the left in the captured image (see the middle image in FIG. 3A), then goes out of frame and is no longer captured in the captured image (see the right image in FIG. 3A). On the other hand, as shown in Figure 3B, when the driver assumes a "leaning forward" posture, the driver's face moves to the upper left in the captured image (see the middle image in Figure 3B), then goes out of frame and is no longer captured in the captured image (see the right image in Figure 3B). In this way, the driver's face is not captured in the captured image in either case, but the movement of the position of the driver's face in the captured image before the driver's face is not captured in the captured image differs between the case where the driver is in a "prone" posture and the case where the driver is in a "leaning forward" posture. The leaning forward determination unit 152 uses this to determine whether the driver's posture corresponds to "leaning forward" or not.

[0032] When the posture type determination unit 151 determines that the driver's posture corresponds to "prone", the leaning forward determination unit 152 performs a leaning forward determination as to whether the driver's posture corresponds to "leaning forward" or not, based on the position of the driver's eyes based on the captured image (hereinafter referred to as the "detected captured image") at the time when the face detection unit 12 was able to detect the driver's face, going back from the time when the posture type determination unit 151 determined that the occupant's posture corresponds to "prone".

[0033] For example, posture determination feature detection unit 13 associates face information, face direction information, head position information, and part position information, assigns detection times, and stores them in a chronological order in a storage unit (not shown). The storage unit is provided in a location that can be referenced by abnormal posture determination device 1. When the posture type determination unit 151 determines that the driver's posture corresponds to "prone", the lean-forward determination unit 152 refers to the storage unit and detects a detected captured image acquired prior to the determination. The lean-forward determination unit 152 can specify the detected captured image, for example, from face information stored in the storage unit. For example, if the face detection unit 12 can detect the driver's face based on the captured image acquired by the image acquisition unit 11 immediately before the captured image when the posture type determination unit 151 determines that the driver's posture corresponds to "prone", the lean-forward determination unit 152 detects the captured image acquired by the image acquisition unit 11 immediately before as the detected captured image. Then, the lean-forward determination unit 152 compares the eye height of the driver on the detected captured image with the eye height of the driver on the captured image in the reference posture (hereinafter referred to as "reference eye height"), and determines that the driver's posture is "leaning forward" if the driver's eyes on the detected captured image are higher than a preset threshold (hereinafter referred to as "height determination threshold"). The lean-forward determination unit 152 can specify the reference eye height based on the reference posture information output from the reference posture estimation unit 14. The eye height of the driver on the captured image is represented by the y coordinate of the position of the driver's eyes on the captured image.

[0034] The lean-forward determination unit 152 determines whether or not the posture of the occupant corresponds to "leaning forward" based on, for example, the following formula (1). (Reference eye height) - (Eye height on detected captured image) > (Height determination threshold) (1) If formula (1) is satisfied, the lean-forward position determination unit 152 determines that the occupant's posture is "leaning forward." When formula (1) is not satisfied, the lean-forward determination unit 152 determines that the occupant's posture is not "leaning forward," in other words, that the occupant is "leaning forward."

[0035] Note that a plurality of patterns of height determination threshold values ​​are set in advance in accordance with combinations of the installation position of the imaging device 2 and the position of the vehicle handlebar, and are stored in a location that the forward lean determination unit 152 can refer to. For example, before a product is shipped, an administrator or the like experimentally sits a subject in the driver's seat with the steering wheel in various positions and verifies what the eye height is in the captured image when the subject is in the reference posture and just before the subject's face is no longer captured when "leaning forward." Then, the administrator or the like sets a height determination threshold for each handlebar position from the eye height in the captured image in the reference posture and the eye height in the captured image immediately before the subject's face is no longer captured when "leaning forward". The administrator or the like sets the height determination threshold, for example, to the difference between the eye height in the reference posture and the eye height immediately before the subject's face is no longer captured when "leaning forward".

[0036] FIG. 4 is a diagram for explaining a specific method of forward lean determination by forward lean determination unit 152 in the first embodiment. In FIG. 4, the left side shows an example of a captured image of the driver in the reference posture, and the right side shows an example of a captured image of the driver just before the driver's face is no longer captured in the image when he or she "leans forward." As described above, when the driver takes a "leaning forward" posture, the driver's face moves to the upper left on the captured image. The eye height of the driver in the captured image taken just before the driver's face is no longer captured while "leaning forward" (shown as 402 in FIG. 4) is higher than the eye height on the captured image in the reference posture (shown as 401 in FIG. 4). In other words, when the driver takes a "leaning forward" posture, the above formula (1) is satisfied.

[0037] The leaning forward determination unit 152 determines whether or not the driver's posture corresponds to "leaning forward" for each of the driver's right eye and left eye, based on the above formula (1). When it is determined that the driver's posture corresponds to "leaning forward" for at least one of the driver's right eye and left eye, the leaning forward determination unit 152 determines that the driver's posture corresponds to "leaning forward".

[0038] It should be noted that the above-described method of forward lean determination by the forward lean determination unit 152 is merely one example, and the forward lean determination unit 152 may perform the forward lean determination using other methods. For example, in the case where the facial feature is the eye, the leaning forward judgment unit 152 may judge that the driver's posture is "leaning forward" when the height of the driver's eyes on the detected captured image detected by the posture judgment feature detection unit 13 is higher than the top end of the steering wheel on the detected captured image, i.e., when the driver's eyes are at a higher position than the top end of the steering wheel on the detected captured image. When leaning forward, the eye position is assumed to be lower than the handlebar position, and when leaning forward, the eye position is assumed to be higher than the handlebar position.

[0039] Also, for example, the forward lean determination unit 152 may compare the driver's eye height with the height of the top end of the steering wheel in real space, in other words, in three-dimensional space, and determine that the driver's posture is leaning forward when the driver's eye height in the three-dimensional space is higher than the top end of the steering wheel in the three-dimensional space, that is, when the driver's eyes are located higher than the top end of the steering wheel in the three-dimensional space.

[0040] The lean-forward determination unit 152 can calculate the height of the driver's eyes in the three-dimensional space by converting the position of the driver's eyes on the detected captured image from a two-dimensional position to a three-dimensional position. Further, the forward lean determination unit 152 can calculate the three-dimensional position of the steering wheel and the height of the upper end of the steering wheel in the three-dimensional space, based on operation information of the tilt steering wheel and the telescopic steering wheel mounted on the vehicle. The coordinate system of the three-dimensional position of the driver's eyes and the three-dimensional position of the top end of the steering wheel is defined as a vehicle coordinate system in which the position of the imaging device 2 is the origin, the right direction with respect to the vehicle's traveling direction is the X-axis, the upward direction of the vehicle is the Y-axis, and the direction opposite to the vehicle's traveling direction is the Z-axis.

[0041] An example of a method for calculating the eye height of the driver in the three-dimensional space and the height of the top end of the steering wheel in the three-dimensional space by the lean-forward determination unit 152 will be described.

[0042] First, an example of a method for calculating the eye height of the driver in three-dimensional space by the forward lean determination unit 152 will be described. The lean-forward determination unit 152 calculates the distance from the position of the imaging device 2 to the driver's eyes (defined as a first distance). When the driver approaches the imaging device 2, the driver's face is captured larger in the detected captured image. Also, when the driver approaches the imaging device 2, the distance between the driver's eyes increases in the detected captured image. The leaning forward determination unit 152 calculates the first distance, for example, using statistical information on the size of the driver's face in the detected captured image or the distance between the driver's eyes in the detected captured image. Furthermore, the lean-forward determination unit 152 calculates an angle (defined as a first angle) of a line connecting the position of the imaging device 2 and the driver's eyes with respect to the Z-axis direction, based on the distance on the image from the center of the detected captured image to the driver's eyes. At this time, the lean-forward determination unit 152 takes into consideration the mounting angle of the imaging device 2. The mounting angle of the imaging device 2 is known in advance. Based on the calculated first distance and first angle, the forward lean determination unit 152 can calculate the height of the driver's eyes in a three-dimensional space using trigonometric functions. Also, the forward lean determination unit 152 can calculate the three-dimensional position of the driver's eyes.

[0043] Next, an example of a method for calculating the height of the upper end of the handlebar in three-dimensional space by the forward lean determination unit 152 will be described. The forward lean determination unit 152 calculates the angle of the steering wheel (defined as a second angle) based on the tilt steering operation information. The forward lean determination unit 152 also calculates the depth of the steering wheel position (defined as a second distance) based on the telescopic steering operation information. Since the size of the handlebar and its mounting position on the vehicle are known in advance, lean-forward determination unit 152 can calculate the three-dimensional position of the top end of the handlebar based on the second angle and the second distance. Also, lean-forward determination unit 152 can calculate the height of the top end of the handlebar in three-dimensional space.

[0044] The forward lean determination unit 152 outputs the forward lean determination result to the abnormal posture determination unit 153. The forward lean determination result includes information indicating whether or not the posture of the driver corresponds to "leaning forward". The forward lean determination section 152 outputs the posture error type determination result outputted from the posture type determination section 151 together with the forward lean determination result to the abnormal posture determination section 153 .

[0045] The abnormal posture determination unit 153 determines whether or not the driver's posture is abnormal, based on the posture failure type determination result by the posture type determination unit 151 and the forward lean determination result by the forward lean determination unit 152. In detail, even if the posture type determination unit 151 determines that the driver's posture is “prone”, if the forward lean determination unit 152 determines that the driver's posture is “leaning forward”, the abnormal posture determination unit 153 determines that the driver's posture is “leaning forward”, i.e., the driver is not in an abnormal posture. When the posture type determination unit 151 determines that the driver's posture is "head down" and the leaning forward determination unit 152 determines that the driver's posture is not "head down", the abnormal posture determination unit 153 determines the duration of the state in which the driver's posture is "head down". For example, the abnormal posture determination unit 153 stores the acquired posture failure type determination result in a storage unit (not shown) in association with information on the date and time when the posture failure type determination result was acquired. The abnormal posture determination unit 153 may determine the duration of the state in which the driver's posture is "head down" from the stored posture failure type determination result. When the duration of the state in which the driver's posture is "head down" reaches a preset threshold (hereinafter referred to as "abnormal posture determination threshold"), the abnormal posture determination unit 153 determines that the driver's posture is "head down", that is, that the driver has an abnormal posture. Furthermore, when the posture type determination unit 151 determines that the driver's posture corresponds to a posture error type other than "head down", the abnormal posture determination unit 153 determines the duration of the state in which the driver's posture corresponds to a posture error type other than "head down". The abnormal posture determination unit 153 may determine the duration of the state in which the driver's posture corresponds to a posture error type other than "head down" from the posture error type determination result stored. When the duration of the state corresponding to a posture error type other than "head down" reaches the abnormal posture determination threshold, the abnormal posture determination unit 153 determines that the driver's posture is of the abnormal posture type determined by the posture type determination unit 151, that is, that the driver has an abnormal posture. When the posture type determination unit 151 determines that the posture of the driver does not correspond to any of the plurality of posture abnormality types, the abnormal posture determination unit 153 determines that the driver is not in an abnormal posture. The abnormal posture determination unit 153 outputs to the output control unit 16 the determination result as to whether or not the driver is in an abnormal posture (hereinafter referred to as the “abnormal posture determination result”).

[0046] When the abnormal posture determination unit 153 determines that the driver is in an abnormal posture, the output control unit 16 outputs, to the output device 3, warning information for warning the driver. In addition, the output control unit 16 outputs the abnormal attitude determination result output from the abnormal attitude determination unit 153 to the operation control system 400.

[0047] FIG. 5 is a diagram showing an example of the configuration of an operation control system 400 according to the first embodiment. In the following description, a vehicle equipped with the abnormal attitude determination system 100 is referred to as a first vehicle, and a vehicle other than the first vehicle is referred to as a second vehicle. Note that there is a case where the first vehicle and the second vehicle are not distinguished from each other, and the first vehicle and the second vehicle are each referred to as a vehicle. In the following, an example in which the abnormal attitude determination system 100 is used as a part of the driving control system 400 will be described, but the abnormal attitude determination system 100 may be configured to be able to communicate with the driving control system 400 wirelessly or wired, as shown in FIG. 1.

[0048] The driving control system 400 includes a driving control device C3 that controls the driving of the first vehicle by outputting a control signal based on the abnormal attitude determination result by the abnormal attitude determination system 100 to control the steering mechanism 5 and the braking / driving mechanism 6 of the first vehicle. The driving control system 400 also includes a map information storage device C4 that stores map information used for automatic driving of the vehicle, a surrounding situation monitoring device C5 that monitors the surrounding situation of the vehicle, and a vehicle state acquisition device C6 that acquires information indicating the state of the first vehicle.

[0049] In addition, the abnormal posture determination system 100, the driving control device C3, the map information storage device C4, the surrounding condition monitoring device C5, and the vehicle state acquisition device C6, which are provided in the driving control system 400, are each connected to a communication bus C7, and data can be sent and received via the communication bus C7.

[0050] The steering mechanism 5 is a mechanism provided on the first vehicle for determining the traveling direction of the first vehicle, and includes, for example, a steering column, a steering shaft, a rack, a pinion, and a steering actuator 51. The braking / driving mechanism 6 is a mechanism for controlling the traveling speed of the first vehicle and switching between forward and reverse, and includes, for example, an accelerator, a brake, a shift, and a braking / driving actuator 61. The steering actuator 51 that controls the steering mechanism 5 is composed of, for example, an EPS (Electric Power Steering) motor, and the braking / driving actuator 61 that controls the braking / driving mechanism 6 is composed of, for example, an electronically controlled throttle, a brake actuator, and the like.

[0051] The map information storage device C4 is a storage medium in which map information including road connection relationships, the number of lanes, intersection position information, and railroad crossing position information is stored. Here, the map information storage device C4 is assumed to be mounted on the first vehicle, but the map information storage device C4 may be configured as, for example, a server that transmits map data to the driving control device C3 via communication. When the map information storage device C4 is configured as the above-mentioned server, for example, a storage medium may be provided in the driving control system 400, and map information acquired from the map information storage device C4 via the vehicle-mounted communication device C52 described later may be stored in the storage medium.

[0052] The surrounding condition monitoring device C5 monitors the surroundings of the first vehicle, and includes a GPS (Global Positioning System) receiver C51, an in-vehicle communication device C52, an external sensor C53, and a navigation system C54.

[0053] The GPS receiver C51 receives signals transmitted from GPS positioning satellites and detects the current position of the first vehicle.

[0054] The outside vehicle sensor C53 is composed of, for example, at least one of a camera that captures images outside the vehicle, a millimeter wave radar, a LiDAR, and an ultrasonic sensor, and detects the position of a second vehicle, pedestrian, obstacle, etc. that is present in the vicinity of the first vehicle or the distance from the first vehicle.

[0055] The navigation system C54 calculates a route from the current position of the first vehicle to a destination and provides guidance along the calculated route. The navigation system C54 has a display unit, for example, a liquid crystal display, and is housed in an instrument panel. The navigation system C54 also has an operation unit, such as a touch panel or physical buttons, and is configured to be able to accept operations by the occupant. The operation unit of the navigation system C54 may be configured with a microphone or the like to be able to accept operations by voice uttered by the occupant.

[0056] The in-vehicle communication device C52 will be described. The in-vehicle communication device C52 is, for example, a wireless communication device connected to an antenna for wireless communication. The in-vehicle communication device C52 acquires information on the position of the second vehicle or pedestrians, traffic information, etc. by communicating with a communication device of a second vehicle or a communication device installed on a road. Here, the traffic information is, for example, congestion information, traffic regulation information, construction section information, etc.

[0057] Moreover, the in-vehicle communication device C52 can perform inter-vehicle communication by wireless communication with the in-vehicle communication device C52 of the second vehicle existing around the first vehicle. In addition, the in-vehicle communication device C52 may perform mobile communication with a base station outside the first vehicle. Furthermore, the in-vehicle communication device C52 is configured to be able to transmit the information of the first vehicle output onto the communication bus C7 to the second vehicle and a call center, etc. Note that the in-vehicle communication device C52 can output the information received from the second vehicle and the information received from the call center, etc., to the communication bus C7. The in-vehicle communication device C52 may also be configured to be able to acquire information from a server, etc., existing outside the vehicle, such as the map information storage device C4 configured as a server that transmits map data to the driving control device C3.

[0058] The vehicle state acquisition device C6 acquires information indicating the state of the first vehicle, and includes a steering angle sensor C61, a vehicle speed sensor C62, a steering torque sensor C63, an accelerator position sensor C64, and a brake position sensor C65.

[0059] The steering angle sensor C61 is provided, for example, on an EPS motor or a steering wheel, and detects the steering angle of the first vehicle. The vehicle speed sensor C62 is provided, for example, on a wheel, and detects the traveling speed of the first vehicle.

[0060] The steering torque sensor C63 is provided on, for example, a steering wheel and detects the magnitude of the steering wheel operation force applied by the driver. The accelerator position sensor C64 detects the amount of depression of the accelerator pedal by the driver. The brake position sensor C65 detects the amount of depression of the brake pedal by the driver.

[0061] The driving control device C3 of the driving control system 400 controls the steering actuator 51 or the braking / driving actuator 61 mounted on the first vehicle to assist the driver in driving. The driving control device C3 performs automatic driving control of the first vehicle, for example, by outputting a signal to the steering actuator 51 or the braking / driving actuator 61. The driving control device C3 performs braking / driving control of the first vehicle, for example, by controlling the braking / driving actuator 61 composed of an electronically controlled throttle and a brake actuator to operate the brakes to decelerate or stop the first vehicle. The driving control device C3 performs steering control of the first vehicle, for example, by controlling the steering actuator 51 composed of an EPS motor to maintain the lane in which the first vehicle is traveling.

[0062] Furthermore, the driving control device C3 has a plurality of driving assistance functions that assist or substitute for the driver's driving operation by controlling the driving force, braking force, steering force, etc. of the first vehicle. For example, the driving assistance functions include a cruise control function and a lane departure prevention function. In the following description, the cruise control function is referred to as ACC (Adaptive Cruise Control), and the lane departure prevention function is referred to as LKA (Lane Keeping Assist).

[0063] When the driving control device C3 executes the ACC, it controls the traveling speed of the first vehicle by adjusting the driving force and the braking force based on the monitoring information of the vehicle ahead obtained from the surrounding condition monitoring device C5. When the vehicle ahead is not detected, the ACC makes the first vehicle travel at a constant speed at a target speed preset by the driver or the like. On the other hand, when the vehicle ahead is detected, the ACC makes the first vehicle travel following the vehicle ahead while maintaining the distance to the vehicle ahead.

[0064] Furthermore, when the driving control device C3 executes LKA, it controls the steering force and the steering force based on the shape information of the lane markings in the traveling direction acquired from the surrounding condition monitoring device C5. LKA makes the first vehicle travel along the lane by applying a steering force to the steering wheel in a direction that prevents the first vehicle from approaching the lane markings. Note that the road information output by the surrounding condition monitoring device C5 may be used for vehicle control by ACC and LKA.

[0065] Furthermore, when the driving control device C3 executes the emergency evacuation function, it can perform automatic evacuation control (evacuation processing) to automatically stop the first vehicle. When the automatic evacuation control is started, the driving control device C3 causes the surrounding situation monitoring device C5 to search for an evacuation location to stop the first vehicle. Then, the driving control device C3 moves the first vehicle to the evacuation location set by the search by the surrounding situation monitoring device C5, and stops the first vehicle at this evacuation location. Note that the above-mentioned evacuation location may be a shoulder outside the lane on an expressway, and may be a location that avoids a position where a moving body such as a second vehicle, a train, or a pedestrian is likely to be present, such as an intersection, a railroad crossing, or a sidewalk, in addition to a shoulder outside the lane on an ordinary road.

[0066] Here, if the driver is in an abnormal state, there is a high possibility that the operation by the driver is an erroneous operation. For example, if the driver's physical condition suddenly changes and the driver leans forward on the steering wheel, the steering wheel will be operated, but this operation is likely to be an erroneous operation. Also, if the driver's physical condition suddenly changes and the driver leans back, the accelerator pedal may be pressed by mistake.

[0067] Therefore, when the driving control device C3 executes the driving assistance function due to the driver being in an abnormal state, driving operations (hereinafter referred to as override operations) aimed at preventing accidents, such as the driver's intentional driving operations or driving operations by a passenger taking the place of the driver, may be enabled while invalidating erroneous operations by the driver. The vehicle state acquisition device C6 may be provided with an override operation detection unit C66 that detects the override operation.

[0068] The override operation will be described below. Regarding vehicle acceleration, if the driver is in an abnormal state, the driver may lose his / her posture and may erroneously depress the accelerator pedal. Therefore, when the driving support function by the driving control device C3 is being executed, the accelerator operation may be invalidated. In other words, the accelerator operation does not have to be included in the override operation.

[0069] On the other hand, even if the driver is in an abnormal state regarding deceleration of the vehicle, the driver who is in a daze may stop the first vehicle in an attempt to avoid a collision with an obstacle. Therefore, the override operation detection unit C66 acquires the amount of depression of the brake pedal from the brake position sensor C65, and detects the operation of the brake pedal as an override operation if the braking force obtained by depressing the brake pedal is greater than the braking force in the driving support function by the driving control device C3.

[0070] Furthermore, when changing the vehicle's course, even if the driver is in an abnormal state, the passenger may operate the steering wheel instead of the driver. However, when the driver is in an abnormal state, the driver's posture may become unstable and the driver may fall onto the steering wheel, so the override operation detection unit C66 does not need to detect this erroneous operation as an override operation. Therefore, the override operation detection unit C66 detects, as an override operation, the operation of the steering wheel performed when a detection result indicating that the driver's posture is not unstable is obtained from the abnormal posture determination system 100.

[0071] When the override operation detection unit C66 detects an override operation while the driving support function is being executed, the driving control system 400 may substitute the control of the steering actuator 51 or the braking / driving actuator 61 for the driving operation by the passenger or the driver. In other words, while the driving support function is being executed, the driving control of the vehicle may be performed by the override operation detected by the override operation detection unit C66.

[0072] The operation of the abnormal posture determination device 1 according to the first embodiment will be described. FIG. 6 is a flowchart for explaining the operation of the abnormal posture determination device 1 according to the first embodiment. The operation of the abnormal posture determination device 1 starts after the ignition of the vehicle is turned on, for example.

[0073] The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST1). The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12.

[0074] The face detection unit 12 detects the face of the driver in the captured image acquired by the image acquisition unit 11 in step ST1 (step ST2). The face detection unit 12 outputs the face information to the posture determination feature detection unit 13 .

[0075] The facial direction detection unit 131 of the posture determination feature detection unit 13 detects the facial direction of the driver in real space based on the facial information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The face direction detection unit 131 outputs the face direction information to the reference posture estimation unit 14 and the determination unit 15.

[0076] The head position detection unit 132 of the posture determination feature detection unit 13 detects the head position of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST4). Head position detection section 132 outputs the head position information to reference posture estimation section 14 and determination section 15.

[0077] The part position detection unit 133 of the posture determination feature detection unit 13 detects the position of the driver's eyes in the captured image based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST5). The part position detection unit 133 outputs the part position information to the reference posture estimation unit 14 and the determination unit 15.

[0078] If the reference posture estimation unit 14 has not completed estimation of the driver's reference posture (if "NO" in step ST6), the reference posture of the driver is estimated based on the driver's facial direction, head position, and eye position detected by the posture determination feature detection unit 13 during the reference setting period (step ST7). The reference posture estimation unit 14 may determine whether or not estimation of the driver's reference posture has been completed based on, for example, a reference posture estimation completion flag. For example, when the reference posture estimation unit 14 estimates the driver's reference posture, it sets the reference posture estimation completion flag to "1". The initial value of the reference posture estimation completion flag is "0", and the reference posture estimation completion flag is initialized, for example, when the ignition of the abnormal posture determination device 1 is turned off. The reference posture estimation completion flag is stored in a location that can be referenced by the abnormal posture determination device 1.

[0079] If the reference posture estimation unit 14 has completed estimation of the driver's reference posture (if "YES" in step ST6), the posture type determination unit 151 of the determination unit 15 performs a posture correction type determination as to whether the driver's posture corresponds to any of a plurality of predetermined posture correction types based on the driver's facial direction, head position, and eye position detected by the posture determination feature detection unit 13 (the driver's facial direction detected by the facial direction detection unit 131 in step ST3, the driver's head position detected by the head position detection unit 132 in step ST4, and the driver's eye position detected by the parts position detection unit 133 in step ST5) (step ST8). The posture type determination unit 151 outputs the posture error type determination result to the leaning forward determination unit 152 .

[0080] If the posture type determination unit 151 determines in step ST8 that the driver's posture corresponds to "prone", the lean-forward determination unit 152 of the determination unit 15 performs a lean-forward determination as to whether the driver's posture corresponds to "leaning forward" (step ST9). The lean-forward determination unit 152 outputs the lean-forward determination result to the abnormal posture determination unit 153. The forward lean determination unit 152 outputs the posture type determination result output from the posture type determination unit 151 in step ST8 to the abnormal posture determination unit 153 together with the forward lean determination result.

[0081] The abnormal posture determination unit 153 determines whether or not the driver's posture is abnormal based on the posture problem type determination result by the posture type determination unit 151 in step ST8 and the forward lean determination result by the forward lean determination unit 152 in step ST9 (step ST10).

[0082] When the abnormal posture determination unit 153 determines in step ST10 that the driver is in an abnormal posture, the output control unit 16 outputs warning information for warning the driver to the output device 3 (step ST11). In addition, the output control unit 16 outputs the abnormal attitude determination result output from the abnormal attitude determination unit 153 to the operation control system 400.

[0083] FIG. 7 is a flowchart showing an example of the operation of the driving assistance function by the driving control system 400 according to the first embodiment. In the following description, an example will be given in which the driving control system 400 performs an emergency evacuation function for the vehicle as a driving assistance function when the abnormal posture determination system 100 determines that the driver is in a bad posture, in other words, that the driver is in an abnormal state. Note that the operation of the driving control system 400 is started, for example, after the ignition of the vehicle is turned on.

[0084] The driving control device C3 of the driving control system 400 acquires the abnormal posture determination result of the driver of the first vehicle from the abnormal posture determination system 100 (ST201).

[0085] Then, the driving control device C3 starts the emergency evacuation function. First, the driving control device C3 acquires the vehicle state from the vehicle state acquisition device C6 (ST202). The vehicle state is information indicating the state of the first vehicle, and is used for vehicle control in the driving support function such as the emergency evacuation function.

[0086] Next, the driving control device C3 acquires the surrounding conditions of the first vehicle from the surrounding conditions monitoring device C5 (ST203), and sets a position to which the first vehicle is to be evacuated (ST204). Hereinafter, the position to which the first vehicle is to be evacuated is referred to as the evacuation position. The evacuation position may be within a range of 150 m from the current position of the first vehicle, or may be a position that takes the first vehicle 60 seconds or less to move from the current position. In this way, it is possible to prevent the number of times the first vehicle enters the intersection from increasing more than necessary as the first vehicle moves to the evacuation position.

[0087] For example, the driving control device C3 searches for a position where the first vehicle can be stopped safely from the surrounding situation monitoring device C5, and sets the safest position from among the candidates of the positions where the first vehicle can be stopped safely as the evacuation position. Here, the evacuation position is a position where there is no possibility of collision with a moving body such as a second vehicle or a pedestrian, and an obstacle, and where there is no possibility of collision with the moving body and obstacle on the route from the current position of the first vehicle to the evacuation position. Examples of the evacuation position include the edge of a road such as a road shoulder, and inside a lane other than an intersection or a railroad crossing. In addition, if the evacuation position is the edge of a road such as a road shoulder, a space for a passenger to escape from the first vehicle may be secured.

[0088] After setting the evacuation position, the driving control device C3 performs vehicle control to stop the first vehicle at the evacuation position. After setting the evacuation position, the driving control device C3 may display a route from the current position of the first vehicle to the set evacuation position on the display unit of the navigation system C54.

[0089] The driving control device C3 judges whether or not a change of course is required before moving the first vehicle to the evacuation position (ST205). If a change of course is not required before moving the first vehicle to the evacuation position (if "NO" in ST205), that is, if the evacuation position is set within the lane in which the first vehicle is currently traveling, the driving control device C3 controls the steering actuator 51 and the braking / driving actuator 61 to make the first vehicle travel within the lane in which the first vehicle is currently traveling (ST206). Then, the driving control device C3 controls the steering actuator 51 and the braking / driving actuator 61 to decelerate the first vehicle and then stop it at the evacuation position (ST207).

[0090] In the process of ST207, the driving control device C3 may drive the first vehicle at a speed at which an emergency stop can be made, such as 10 km / h. In addition, if it is not necessary to continue driving the first vehicle, such as when the evacuation position is set near the first vehicle, the process of ST206 can be omitted. Furthermore, in the processes of ST206 and ST207, when the driving control device C3 decelerates the first vehicle, the deceleration may be set to, for example, 3 m / s in order to prevent the passenger from falling over. 2 It may be the following:

[0091] Then, after the driving control device C3 stops the first vehicle at the evacuation position, a report is sent to a call center or the like outside the vehicle via the in-vehicle communication device C52 (ST210). The report sent via the in-vehicle communication device C52 includes, for example, a message indicating that the driver of the first vehicle is in an abnormal state, the current position of the first vehicle acquired by the surrounding condition monitoring device C5, and the evacuation position set by the driving control device C3.

[0092] A case where the driving control device C3 determines in the process of ST205 that a lane change is required before the first vehicle is moved to the evacuation position (the case of "YES" in ST205) will be described. Note that lane change includes a case where the first vehicle is moved to a lane adjacent to the lane in which the first vehicle is traveling, such as a lane change, and a case where the first vehicle is deviated from the lane in which the first vehicle is traveling so as to pull the first vehicle over to the side of the road.

[0093] When the driving control device C3 determines that a lane change is required, it controls the steering actuator 51 and the braking / driving actuator 61 to change the lane of the first vehicle (ST208). Here, the lateral movement speed of the vehicle during lane change may be about 0.3 m / s so that a second vehicle or a pedestrian or the like existing at the lane change destination can recognize the lane change of the first vehicle and avoid a collision.

[0094] Then, similar to the process in ST207, the driving control device C3 controls the steering actuator 51 and the braking / driving actuator 61 to decelerate the first vehicle and then stop it at the evacuation position (ST209). After the driving control device C3 stops the first vehicle at the evacuation position, the operation of the driving control system 400 proceeds to the process in ST210, and a report is sent to a call center or the like outside the vehicle via the in-vehicle communication device C52.

[0095] Furthermore, when an override operation is performed while the driving support function is being executed, the driving control system 400 may substitute the control of the steering actuator 51 or the braking / driving actuator 61 for a driving operation by a passenger or a driver. That is, for example, when the override operation detection unit C66 detects an override operation during the processing of ST202 to ST210 in Fig. 7, the vehicle may be driven according to the detected override operation.

[0096] In the above example, when the abnormal posture determination system 100 determines that the driver is in a bad posture, in other words, that the driver is in an abnormal state, the driving control system 400 performs an emergency evacuation function for the vehicle as a driving support function. However, the driving control system 400 may execute ACC or LKA as a driving support function. In addition, the driving support functions may be combined, such as executing the emergency evacuation function when the safety of the surroundings is ensured after executing ACC and LKA. In this way, when the driver is in a bad posture, in other words, when it is determined that the driver is in an abnormal state, the driving support functions can be appropriately combined.

[0097] In the above embodiment 1, the driver's facial features are described as the driver's eyes, but this is merely an example. For example, the driver's facial features may be the nose or eyebrows. For example, when the driver's facial features are the nose, the lean-forward determination unit 152 compares the height of the nose on the detected captured image of the driver with the height of the nose on the captured image of the driver in the reference posture, and if the nose on the detected captured image of the driver is at a position higher than a preset threshold value or more, determines that the driver's posture is leaning forward.

[0098] In the above-described first embodiment, the abnormal posture determination device 1 assumes that, once the reference posture estimation unit 14 estimates the reference posture of the driver, the reference posture is not reviewed until, for example, the ignition of the vehicle is turned off. However, this is merely an example, and, for example, the reference posture estimation unit 14 may constantly monitor the reference posture, and if the reference posture has changed within a range where it can be said that the driver's posture has not collapsed, the reference posture may be updated.

[0099] In the above first embodiment, the reference posture estimating unit 14 estimates the reference posture of the driver, but this is merely an example. For example, an administrator or the like may set a reference posture in advance based on the facial direction, head position, and positions of facial features such as the eyes of a typical driver, and store the reference posture in a location that can be referenced by the determination unit 15. In this case, the abnormal posture determination device 1 can be configured not to include the reference posture estimation unit 14. However, the abnormal posture determination device 1 can set a reference posture more suited to an individual by having the reference posture estimation unit 14 estimate the reference posture, rather than by not having the reference posture estimation unit 14. As a result, the abnormal posture determination device 1 can more accurately determine whether or not the driver is in an abnormal posture.

[0100] In the above embodiment 1, the vehicle occupant for which the abnormal posture determination device 1 determines whether the vehicle is in an abnormal posture is the driver of the vehicle, but this is merely an example. The abnormal posture determination device 1 can determine whether the vehicle occupant other than the driver is in an abnormal posture.

[0101] In addition, in the above embodiment 1, the abnormal posture judgment device 1 is an in-vehicle device mounted on a vehicle, and the image acquisition unit 11, face detection unit 12, posture judgment feature detection unit 13, reference posture estimation unit 14, judgment unit 15, and output control unit 16 are provided in the in-vehicle device. Without being limited to this, some of the image acquisition unit 11, face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16 may be mounted on the in-vehicle device of the vehicle, and the others may be provided in a server connected to the in-vehicle device via a network, so that a system is formed by the in-vehicle device and the server. Moreover, the image acquisition unit 11, face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16 may all be provided in the server.

[0102] 8A and 8B are diagrams illustrating an example of a hardware configuration of the abnormal posture determination device 1 according to the first embodiment. In the first embodiment, the functions of the image acquisition unit 11, face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16 are realized by a processing circuit 1001. That is, the abnormal posture determination device 1 includes the processing circuit 1001 for performing control to determine whether the posture of a vehicle occupant is an abnormal posture by distinguishing between "prone" and "leaning forward" based on a captured image. The processing circuit 1001 may be dedicated hardware as shown in FIG. 8A, or may be a processor 1004 executing a program stored in a memory 1005 as shown in FIG. 8B.

[0103] When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0104] When the processing circuit is the processor 1004, the functions of the image acquisition unit 11, the face detection unit 12, the posture determination feature detection unit 13, the reference posture estimation unit 14, the determination unit 15, and the output control unit 16 are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 1005. The processor 1004 executes the functions of the image acquisition unit 11, the face detection unit 12, the posture determination feature detection unit 13, the reference posture estimation unit 14, the determination unit 15, and the output control unit 16 by reading and executing the program stored in the memory 1005. That is, the abnormal posture determination device 1 includes a memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST11 in FIG. 6 described above. Also, it can be said that the programs stored in the memory 1005 cause the computer to execute the procedures or methods of the processes of the image acquisition unit 11, face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16. Here, the memory 1005 corresponds to, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), or the like.

[0105] The functions of image acquisition unit 11, face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the functions of image acquisition unit 11 and output control unit 16 can be realized by processing circuit 1001 as dedicated hardware, and the functions of face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, and determination unit 15 can be realized by processor 1004 reading and executing a program stored in memory 1005. The storage unit (not shown) is configured, for example, by the memory 1005 . The abnormal posture determination device 1 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as an image capture device 2 or an output device 3.

[0106] As described above, according to the first embodiment, the abnormal posture determination device 1 includes a face detection unit 12 that detects the face of a vehicle occupant in an image captured in which the area in which the face of the occupant should be located is captured, a posture determination feature detection unit 13 that detects the facial direction of the occupant, the position of the occupant's head, and the positions of the occupant's facial features in the captured image based on the face of the occupant in the captured image detected by the face detection unit 12, and a posture determination feature detection unit 13 that compares the facial direction and head position of the occupant detected by the posture determination feature detection unit 13 with the facial direction and head position of the occupant in a reference posture in which it is assumed that the occupant is not suffering from posture deterioration to determine whether the posture of the occupant falls into one of a plurality of posture deterioration types. If the face detection unit 12 cannot detect the face of the occupant, the posture of the occupant is determined to be one of a plurality of posture deterioration types. The system is configured to include a posture type determination unit 151 that determines that the occupant's posture corresponds to the prone posture type, a forward lean determination unit 152 that, when the posture type determination unit 151 determines that the occupant's posture corresponds to the prone posture, performs a forward lean determination as to whether or not the occupant's posture corresponds to a forward lean that does not correspond to any of the multiple posture deviation types based on the positions of the occupant's facial features on the detected captured image, which is an image captured when the face detection unit 12 was able to detect the occupant's face going back from when the posture of the occupant was determined to correspond to the prone posture, and an abnormal posture determination unit 153 that determines whether or not the occupant's posture is an abnormal posture based on the posture deviation type determination result by the posture type determination unit 151 and the forward lean determination result by the forward lean determination unit 152. Therefore, the abnormal posture determination device 1 can prevent the occupant's "forward lean" posture from being erroneously detected as the "prone" posture.

[0107] Various aspects of the present disclosure are summarized below as appendices.

[0108] (Appendix 1) a face detection unit that detects a face of a vehicle occupant in a captured image in which a range in which the face of the vehicle occupant should be present is captured; a posture determination feature detection unit that detects a facial orientation of the occupant, a head position of the occupant, and positions of facial features of the occupant based on the face of the occupant in the captured image detected by the face detection unit; a posture type determination unit which performs a posture error type determination as to whether the posture of the occupant corresponds to one of a plurality of posture error types by comparing the facial direction and head position of the occupant detected by the posture determination feature detection unit with the facial direction and head position of the occupant in a reference posture in which it is assumed that the occupant is not experiencing posture error, and if the face detection unit cannot detect the face of the occupant, determines that the posture of the occupant corresponds to the prone position among the plurality of posture error types; a forward lean determination unit that, when the posture type determination unit determines that the posture of the occupant corresponds to prone posture, performs a forward lean determination as to whether or not the posture of the occupant corresponds to forward leaning that does not correspond to any of the plurality of posture collapse types, based on positions of facial features of the occupant based on a detected captured image, which is an captured image retroactive to when the posture of the occupant was determined to correspond to prone posture and when the face detection unit was able to detect the face of the occupant; an abnormal posture determination unit that determines whether the posture of the occupant is an abnormal posture based on a posture deviation type determination result by the posture type determination unit and a forward lean determination result by the forward lean determination unit. An abnormal posture determination device comprising: (Appendix 2) The abnormal posture determination unit determines that the posture of the occupant is not the abnormal posture when the posture type determination unit determines that the posture of the occupant is prone, but the forward lean determination unit determines that the posture of the occupant is forward leaning. 2. An abnormal posture determination device according to claim 1. (Appendix 3) a reference posture estimation unit that estimates the reference posture of the occupant based on the facial direction, head position, and positions of facial features of the occupant detected by the posture determination feature detection unit during a reference setting period. 3. The abnormal posture determination device according to claim 1 or 2, comprising: (Appendix 4) The forward lean determination unit compares a height of a facial feature of the occupant on the detected captured image with a height of a facial feature of the occupant on the captured image in the reference posture, and determines that the posture of the occupant is forward leaning if the facial feature of the occupant on the detected captured image is located at a position higher than or equal to a height determination threshold value. 4. The abnormal posture determination device according to claim 1, (Appendix 5) the facial features of the occupant are the eyes of the occupant, The forward lean determination unit determines that the posture of the occupant is leaning forward when the height of the occupant's eyes is higher than the top end of the handlebar. 4. The abnormal posture determination device according to claim 1, (Appendix 6) The forward lean determination unit compares the height of the occupant's eyes with the height of the top end of the steering wheel on the detection captured image, and determines that the posture of the occupant is forward leaning when the eyes of the occupant are at a higher position than the top end of the steering wheel on the detection captured image. 6. The abnormal posture determination device according to claim 5, (Appendix 7) The forward lean determination unit compares the height of the occupant's eyes with the height of the top end of the steering wheel in a three-dimensional space, and determines that the posture of the occupant is leaning forward when the occupant's eyes are at a higher position than the top end of the steering wheel in the three-dimensional space. 6. The abnormal posture determination device according to claim 5, (Appendix 8) The forward lean determination unit calculates the height of the occupant's eyes in the three-dimensional space based on the detected captured image and a position of an imaging device that captures the captured image, and calculates the height of the upper end of the steering wheel in the three-dimensional space based on operation information of tilt steering and telescopic steering. 8. The abnormal state determination device according to claim 7, (Appendix 9) an output control unit that outputs warning information for warning the occupant when the abnormal posture determination unit determines that the occupant is in the abnormal posture; 9. The abnormal posture determination device according to claim 1, comprising: (Appendix 10) The facial features of the occupant are the eyes of the occupant. 10. The abnormal posture determination device according to claim 1, (Appendix 11) A face detection unit detects a face of a vehicle occupant in a captured image in which a range in which the face of the vehicle occupant should be present is captured; a posture determination feature detection unit detecting a facial orientation of the occupant, a head position of the occupant, and positions of facial features of the occupant based on the face of the occupant in the captured image detected by the face detection unit; a posture type determination unit performs a posture error type determination as to whether the posture of the occupant corresponds to any one of a plurality of posture error types by comparing the facial direction and head position of the occupant detected by the posture determination feature detection unit with the facial direction and head position of the occupant in a reference posture in which it is assumed that the occupant is not experiencing posture error, and if the face detection unit cannot detect the face of the occupant, determines that the posture of the occupant corresponds to the prone position among the plurality of posture error types; a forward lean determination unit, when the posture type determination unit determines that the posture of the occupant corresponds to prone posture, performs a forward lean determination as to whether or not the posture of the occupant corresponds to a forward lean that does not correspond to any of the plurality of posture collapse types, based on positions of facial features of the occupant based on a detected captured image, which is an captured image retroactive to the time when the posture of the occupant was determined to correspond to prone posture and when the face detection unit was able to detect the face of the occupant; an abnormal posture determination unit determining whether or not the posture of the occupant is an abnormal posture based on a posture imbalance type determination result by the posture type determination unit and a forward lean determination result by the forward lean determination unit; An abnormal posture determination method comprising: (Appendix 12) A vehicle control system including the abnormal attitude determination device according to any one of Supplementary Note 1 to Supplementary Note 10 and a driving control device that controls driving of the vehicle, the abnormal posture determination device outputs a determination result as to whether or not the occupant is in the abnormal posture to the operation control device; The driving control device outputs a control signal for outputting a warning to the occupant or for performing an evacuation process of the vehicle based on the determination result to an alarm control device, a steering mechanism, or a braking / driving mechanism mounted on the vehicle. A vehicle control system comprising:

[0109] In addition, in the present disclosure, any of the components of the embodiments may be modified or any of the components of the embodiments may be omitted. [Industrial Applicability]

[0110] The occupant status detection device of the present disclosure can be mounted in a vehicle and used as a device for detecting whether or not an occupant of the vehicle is capable of driving the vehicle. [Explanation of symbols]

[0111] 1 Abnormal posture determination device, 2 Imaging device, 3 Output device, 5 Steering mechanism, 6 Braking / driving mechanism, 11 Image acquisition unit, 12 Face detection unit, 13 Posture determination feature detection unit, 131 Face direction detection unit, 132 Head position detection unit, 133 Part position detection unit, 14 Reference posture estimation unit, 15 Determination unit, 151 Posture type determination unit, 152 Forward lean determination unit, 153 Abnormal posture determination unit, 16 Output control unit, 100 Abnormal posture determination system, 400 Driving control system, 51 Steering actuator, 61 Braking / driving actuator, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory, C3 Driving control device, C4 Map information storage device, C5 Surrounding situation monitoring device, C51 GPS receiver, C52 In-vehicle communication device, C53 Outside vehicle sensor, C54 Navigation system, C6 vehicle status acquisition device, C61 steering angle sensor, C62 vehicle speed sensor, C63 steering torque sensor, C64 accelerator position sensor, C65 brake position sensor, C66 override operation detection unit.

Claims

1. a face detection unit that detects a face of a vehicle occupant in a captured image in which a range in which the face of the vehicle occupant should be present is captured; a posture determination feature detection unit that detects a facial orientation of the occupant, a head position of the occupant, and positions of facial features of the occupant based on the face of the occupant in the captured image detected by the face detection unit; a posture type determination unit which performs a posture error type determination as to whether the posture of the occupant corresponds to one of a plurality of posture error types by comparing the facial direction and head position of the occupant detected by the posture determination feature detection unit with the facial direction and head position of the occupant in a reference posture in which it is assumed that the occupant is not experiencing posture error, and if the face detection unit cannot detect the face of the occupant, determines that the posture of the occupant corresponds to the prone position among the plurality of posture error types; a forward lean determination unit that, when the posture type determination unit determines that the posture of the occupant corresponds to prone posture, performs a forward lean determination as to whether or not the posture of the occupant corresponds to forward leaning that does not correspond to any of the plurality of posture collapse types, based on positions of facial features of the occupant based on a detected captured image, which is an captured image retroactive to when the posture of the occupant was determined to correspond to prone posture and when the face detection unit was able to detect the face of the occupant; an abnormal posture determination unit that determines whether the posture of the occupant is an abnormal posture based on a posture deviation type determination result by the posture type determination unit and a forward lean determination result by the forward lean determination unit. An abnormal posture determination device comprising:

2. The abnormal posture determination unit determines that the posture of the occupant is not the abnormal posture when the posture type determination unit determines that the posture of the occupant is prone, but the forward lean determination unit determines that the posture of the occupant is forward leaning.

2. The abnormal posture determination device according to claim 1.

3. a reference posture estimation unit that estimates the reference posture of the occupant based on the facial direction, head position, and positions of facial features of the occupant detected by the posture determination feature detection unit during a reference setting period. The abnormal posture determination device according to claim 1 , further comprising:

4. The forward lean determination unit compares a height of a facial feature of the occupant on the detected captured image with a height of a facial feature of the occupant on the captured image in the reference posture, and determines that the posture of the occupant is forward leaning if the facial feature of the occupant on the detected captured image is located at a position higher than or equal to a height determination threshold value.

4. The abnormal posture determination device according to claim 1, wherein the abnormal posture determination device detects an abnormal posture of the object.

5. the facial features of the occupant are the eyes of the occupant, The forward lean determination unit determines that the posture of the occupant is leaning forward when the height of the occupant's eyes is higher than the top end of the handlebar.

4. The abnormal posture determination device according to claim 1, wherein the abnormal posture determination device detects an abnormal posture of the object.

6. The forward lean determination unit compares the height of the occupant's eyes with the height of the top end of the steering wheel on the detection captured image, and determines that the posture of the occupant is forward leaning when the eyes of the occupant are at a higher position than the top end of the steering wheel on the detection captured image.

6. The abnormal posture determination device according to claim 5.

7. The forward lean determination unit compares the height of the occupant's eyes with the height of the top end of the steering wheel in a three-dimensional space, and determines that the posture of the occupant is forward leaning when the occupant's eyes are at a higher position than the top end of the steering wheel in the three-dimensional space.

6. The abnormal posture determination device according to claim 5.

8. The forward lean determination unit calculates the height of the occupant's eyes in the three-dimensional space based on the detected captured image and a position of an imaging device that captures the captured image, and calculates the height of the upper end of the steering wheel in the three-dimensional space based on operation information of tilt steering and telescopic steering.

8. The abnormal posture determination device according to claim 7.

9. an output control unit that outputs warning information for warning the occupant when the abnormal posture determination unit determines that the occupant is in the abnormal posture; The abnormal posture determination device according to claim 1 , further comprising:

10. The facial features of the occupant are the eyes of the occupant.

2. The abnormal posture determination device according to claim 1.

11. A face detection unit detects a face of a vehicle occupant in a captured image in which a range in which the face of the vehicle occupant should be present is captured; a posture determination feature detection unit detecting a facial orientation of the occupant, a head position of the occupant, and positions of facial features of the occupant based on the face of the occupant in the captured image detected by the face detection unit; a posture type determination unit performs a posture error type determination as to whether the posture of the occupant corresponds to any one of a plurality of posture error types by comparing the facial direction and head position of the occupant detected by the posture determination feature detection unit with the facial direction and head position of the occupant in a reference posture in which it is assumed that the occupant is not experiencing posture error, and if the face detection unit cannot detect the face of the occupant, determines that the posture of the occupant corresponds to the prone position among the plurality of posture error types; a forward lean determination unit, when the posture type determination unit determines that the posture of the occupant corresponds to prone posture, performs a forward lean determination as to whether or not the posture of the occupant corresponds to a forward lean that does not correspond to any of the plurality of posture collapse types, based on positions of facial features of the occupant based on a detected captured image, which is an captured image retroactive to the time when the posture of the occupant was determined to correspond to prone posture and when the face detection unit was able to detect the face of the occupant; an abnormal posture determination unit determining whether or not the posture of the occupant is an abnormal posture based on a posture imbalance type determination result by the posture type determination unit and a forward lean determination result by the forward lean determination unit; An abnormal posture determination method comprising:

12. A vehicle control system comprising the abnormal attitude determination device according to claim 1 and a driving control device that controls driving of the vehicle, the abnormal posture determination device outputs a determination result as to whether or not the occupant is in the abnormal posture to the operation control device; The driving control device outputs a control signal for outputting a warning to the occupant or for performing an evacuation process of the vehicle based on the determination result to an alarm control device, a steering mechanism, or a braking / driving mechanism mounted on the vehicle. A vehicle control system comprising:

Images