Abnormal attitude detection device, abnormal attitude detection method, and vehicle control system

Abstract

The present invention comprises: an image acquisition unit (11) that acquires a photographic image capturing a range in which the face of a vehicle passenger should exist; a feature amount extraction unit (12) that extracts a feature amount on the basis of the photographic image; a change amount calculation unit (14) that calculates a change amount of the feature amount from a feature amount of a reference posture of the passenger; a first determination unit (15) that determines whether the posture of the passenger is a normal posture, by comparing the feature amount and the change amount to a normal posture determination condition, comparing the feature amount to a normal posture determination condition, or comparing the change amount to a normal posture determination condition; a second determination unit (16) that, on the basis of the change amount and a machine learning model for which the change amount is an input and information indicating the presence / absence of a collapse of the posture of the passenger is an output, determines whether the posture of the passenger has collapsed, when the first determination unit (15) has determined that the posture of the passenger is not a normal posture; and a detection unit (17) that detects abnormal posture of a passenger when a state in which the second determination unit (16) has determined that the posture of the passenger has collapsed has continued for a period of time used for abnormality detection.

Description

[Technical Field]

[0001] The present disclosure relates to an abnormal attitude detection device, an abnormal attitude detection method, and a vehicle control system. [Background technology]

[0002] Conventionally, a technology has been known that detects abnormal posture of an occupant by determining whether the occupant is misaligned based on features such as facial orientation extracted from an image of the occupant inside a vehicle and a trained model in machine learning (hereinafter referred to as a "machine learning model") (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 208529 Summary of the Invention [Problem to be solved by the invention]

[0004] By using a machine learning model to determine whether an occupant's posture is out of alignment, it is possible to detect abnormal occupant posture with high accuracy. On the other hand, in a scene where feature values ​​such as facial orientation cannot be stably extracted from a captured image, the presence or absence of an occupant's posture is determined based on low-accuracy feature values ​​and the machine learning model. In this case, when determining whether an occupant's posture is out of alignment using the machine learning model, there is a possibility that the occupant may be determined to be in an out-of-alignment state even though they are not. In conventional technology, in scenes where features such as facial orientation cannot be reliably extracted from captured images, the presence or absence of an occupant's posture is determined based on low-precision features and a machine learning model, which can result in the risk of falsely detecting an abnormal occupant's posture. However, the technology disclosed in Patent Document 1 still cannot solve the above problem because it does not take into account the possibility that when determining whether or not an occupant's posture is unstable using a machine learning model in a scene where features cannot be stably extracted from the captured image, the occupant's posture may be erroneously determined to be unstable.

[0005] The present disclosure has been made to solve such problems, and aims to provide an abnormal posture detection device that can detect an abnormal posture of an occupant even when a scene occurs in which features cannot be stably extracted from an captured image. [Means for solving the problem]

[0006] The abnormal posture detection device according to the present disclosure includes an image acquisition unit that acquires an image of an area where the face of a vehicle occupant should be present, a feature extraction unit that extracts feature amounts used to determine the posture of the occupant based on the image acquired by the image acquisition unit, a change amount calculation unit that calculates a change amount of the feature amount extracted by the feature amount extraction unit from a feature amount in a reference posture of the occupant, and a normal posture determination unit that compares the feature amount extracted by the feature amount extraction unit with the change amount calculated by the change amount calculation unit. ,or The feature quantity extracted by the feature quantity extraction unit is compared with the normal posture determination condition. So, ride The vehicle occupant posture detection system includes a first determination unit that determines whether the occupant's posture is normal or not, a second determination unit that, if the first determination unit determines that the occupant's posture is not normal, determines whether the occupant's posture is out of alignment based on the amount of change calculated by the change amount calculation unit and a machine learning model that inputs the amount of change and outputs information indicating whether the occupant's posture is out of alignment, and a detection unit that detects an abnormal occupant's posture when the state in which the second determination unit determines that the occupant's posture is out of alignment continues for an abnormality detection time. [Effects of the Invention]

[0007] According to the abnormal posture detection device according to the present disclosure, even if a scene occurs in which feature amounts cannot be stably extracted from a captured image, an abnormal posture of an occupant can be detected. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of an abnormal posture detection device according to a first embodiment. [Figure 2] FIG. 10 is a diagram for explaining the significance of the first determination unit determining whether or not the driver's posture is normal based on a feature amount based on a current captured image and a relative change in the feature amount in the first embodiment. [Figure 3] 10A and 10B are diagrams for explaining a plurality of types of poor posture. [Figure 4] 1 is a diagram illustrating a configuration example of an operation control system according to a first embodiment. [Figure 5] 10 is a flowchart for explaining an operation in a reference posture feature amount calculation process performed by the abnormal posture detection device according to the first embodiment. [Figure 6] 10 is a flowchart for explaining operations in a change amount calculation process, a first attitude determination process, a second attitude determination process, and an abnormal attitude detection process performed by the abnormal attitude detection device according to the first embodiment. [Figure 7] 7 is a flowchart for explaining detailed operations of a first attitude determination process by a first determination unit, which is performed in step ST14 of FIG. 6. [Figure 8] 7 is a flowchart for explaining detailed operations of the second attitude determination process by the second determination unit, which is performed in step ST15 of FIG. 6. [Figure 9] 7 is a flowchart for explaining detailed operations of an abnormal posture detection process by a detection unit, which is performed in step ST16 of FIG. 6. [Figure 10] 10A and 10B are diagrams for explaining an example of a scene in which feature amounts such as facial orientation cannot be stably extracted from a captured image. [Figure 11] 4 is a flowchart for explaining an example of an operation of a driving assistance function by the driving control system according to the first embodiment. [Figure 12] 12A and 12B are diagrams illustrating an example of a hardware configuration of the abnormal posture detection device according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1 The abnormal posture detection device according to the first embodiment detects whether the posture of a vehicle occupant is abnormal based on a captured image in which at least the face of the vehicle occupant is captured. In the following first embodiment, as an example, the occupant of the vehicle, for whom the abnormal attitude detection device detects whether or not the vehicle is in an abnormal attitude, is the driver of the vehicle.

[0010] FIG. 1 is a diagram illustrating an example of the configuration of an abnormal posture detection device 1 according to the first embodiment. The abnormal attitude detection device 1 according to the first embodiment is assumed to be mounted on a vehicle. The abnormal posture detection device 1 is connected to an imaging device 2, and the abnormal posture detection device 1 and the imaging device 2 together constitute an abnormal posture detection system 100. The abnormal attitude detection system 100, together with the driving control system 300, the steering mechanism 3, and the braking / driving mechanism 4, constitutes a vehicle control system SYS.A.

[0011] The imaging device 2 is mounted on a vehicle and is installed so as to be able to capture an image of at least the area where the driver's face is expected to be present. In the first embodiment, it is assumed that the imaging device 2 is installed, for example, near the center of the instrument panel in the vehicle width direction or on the center console. For example, the imaging device 2 may be shared with a so-called DMS (Driver Monitoring System) that is installed for the purpose of monitoring the interior of the vehicle. The imaging device 2 is a visible light camera or an infrared camera. The imaging device 2 outputs the captured image to the abnormal posture detection device 1. The abnormal posture detection device 1 detects whether or not the posture of the driver is abnormal based on the captured image captured by the imaging device 2. Details of the abnormal posture detection device 1 will be described later.

[0012] The abnormal posture detection device 1 outputs the detection result as to whether or not the driver's posture is abnormal to the driving control system 300 connected to the abnormal posture detection system 100. 1, the abnormal attitude detection system 100 is connected to a driving control system 300. The driving control system 300 is linked to the steering mechanism 3 or the braking / driving mechanism 4, and provides driving assistance using the detection results of the abnormal attitude detection system 100. Details of the driving control system 300 will be described later.

[0013] As shown in FIG. 1, the abnormal posture detection device 1 includes an image acquisition unit 11, a feature extraction unit 12, a reference posture feature calculation unit 13, a change amount calculation unit 14, a first judgment unit 15, a second judgment unit 16, a detection unit 17, and a memory unit 18.

[0014] The image acquisition unit 11 acquires a captured image from the imaging device 2 . The image acquisition unit 11 outputs the acquired captured image to the feature extraction unit 12.

[0015] The feature extraction unit 12 extracts feature amounts used to determine the driver's posture based on the captured image acquired by the image acquisition unit 11. In the first embodiment, the feature amounts used to determine the driver's posture are the driver's facial direction, the driver's head position, and the face detection reliability.

[0016] The feature extraction unit 12 first detects the driver's face in the captured image acquired by the image acquisition unit 11. In detail, the feature extraction unit 12 detects the driver's facial feature points, which indicate the driver's facial features, using a known image recognition technique such as edge detection on the captured image. The facial features include the corners and corners of the eyes, the nose, the mouth, the eyebrows, and the chin. The driver's facial feature points are represented, for example, by coordinates on the captured image. The feature extraction unit 12 may also detect the driver's facial region. The driver's facial region is, for example, the smallest rectangle that encloses the outline of the driver's face. The driver's facial region is represented, for example, by the coordinates of the four corners of the smallest rectangle on the captured image. Since the installation position and angle of view of the image capturing device 2 are known in advance, even if a captured image contains multiple occupants, the feature extraction unit 12 can determine which area of ​​the captured image contains the driver's face. For example, an area in the captured image where the driver's face may be present (hereinafter referred to as the "driver detection area") is set in advance, and the feature extraction unit 12 detects the feature points and face area of ​​the driver's face in the driver detection area using a known image recognition technique.

[0017] The feature extraction unit 12 detects the driver's facial orientation and head position based on the detected information about the driver's face (hereinafter referred to as "face information") and extracts it as a feature. The face information is, for example, a captured image to which information that can identify the driver's facial feature points and facial area is added.

[0018] In the first embodiment, the facial direction and head position of the driver detected by the feature amount extracting unit 12 are, for example, the facial direction and head position in real space. The feature extraction unit 12 detects the direction of the driver's face in real space based on the face information, more specifically, based on the driver's face in the captured image. The feature extraction unit 12 may detect the driver's facial direction using, for example, a known facial direction detection technique for detecting the facial direction from a captured image. The driver's facial direction is expressed, for example, by an angle (yaw angle, pitch angle, or roll angle) relative to the reference axis, with the optical axis of the imaging device 2 as the reference axis. Note that this is merely an example, and the driver's facial direction may also be expressed, for example, by an angle (yaw angle, pitch angle, or roll angle) relative to the reference axis, with a predetermined reference axis in the longitudinal direction of the vehicle as the reference axis.

[0019] Furthermore, the feature extraction unit 12 detects the driver's head position in real space based on the facial information, more specifically, based on the driver's face in the captured image. In the first embodiment, the driver's head position in the captured image is indicated, for example, by the center of the driver's eyebrows. The feature extraction unit 12 detects, for example, a point in real space corresponding to the center of the driver's eyebrows in the captured image as the driver's head position. Note that this is merely an example, and the driver's head position in the captured image may be indicated, for example, by the center of the driver's facial area or the center of a line connecting the inner corners of the driver's eyes. In this case, the feature extraction unit 12 detects, for example, a point in real space corresponding to the center of the driver's facial area in the captured image or the center of a line connecting the inner corners of the driver's eyes as the driver's head position in real space. The feature extraction unit 12 may detect the driver's head position by using, for example, a known coordinate transformation technique for transforming points on the captured image into points in real space. The driver's head position is represented by, for example, coordinates in real space.

[0020] Furthermore, the feature extraction unit 12 calculates the face detection reliability based on the face information and sets this as the extracted feature. In embodiment 1, face detection reliability indicates how reliable a face detected from a captured image is, i.e., how plausible it is as a face, or more specifically, how many facial features were detected from the captured image. The feature extraction unit 12 calculates the face detection reliability in accordance with predetermined conditions that are set in advance. For example, the feature extraction unit 12 calculates the percentage of facial features detected based on the captured image acquired by the image acquisition unit 11 out of all the features defined as facial features, such as the corners of the eyes, the nose, the mouth, the eyebrows, or the chin, as the face detection reliability.

[0021] The feature extraction unit 12 outputs information regarding the driver's facial orientation, head position, and face detection reliability extracted as features (hereinafter referred to as "feature information") to the reference posture feature calculation unit 13 and the change amount calculation unit 14, and also stores the information in chronological order in the memory unit 18. The feature information is information that associates information on the driver's face orientation relative to a reference axis (yaw angle, pitch angle, and roll angle), coordinate information on the driver's head position (X coordinate, Y coordinate, Z coordinate), information indicating the face detection reliability, and a captured image.

[0022] The reference posture feature amount calculation unit 13 estimates the reference posture of the driver and calculates the feature amount in the reference posture of the driver (hereinafter referred to as the "reference posture feature amount"). In the first embodiment, the reference posture refers to the posture of the driver in a state where the driver is considered to be facing forward with respect to the traveling direction of the vehicle. Note that in the first embodiment, the driver facing forward with respect to the traveling direction of the vehicle is assumed to mean that the driver's face is facing forward with respect to the traveling direction of the vehicle. In the first embodiment, "forward" is not limited to strictly being forward, but includes approximately forward. The process of estimating the reference posture of the driver and calculating the feature amounts in the reference posture, which is performed by the reference posture feature amount calculation unit 13 in the abnormal posture detection device 1 according to the first embodiment, is referred to as "reference posture feature amount calculation process."

[0023] The reference posture feature calculation unit 13 determines whether the driver is considered to be facing forward in the vehicle's traveling direction, for example, by determining whether the vehicle speed, steering wheel steering angle, and shift position satisfy predetermined conditions (hereinafter referred to as "vehicle conditions"), and estimates the driver's reference posture. The vehicle conditions are preset, for example, such that "the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km / h), the steering wheel angle is within a predetermined angle range (e.g., ±20 degrees), and the shift position is in "D." The vehicle conditions set include the vehicle speed, steering wheel angle, and shift position assumed in a situation where the driver is assumed to be facing forward in the direction of travel of the vehicle.

[0024] When the vehicle speed, steering wheel angle, and shift position satisfy the vehicle conditions, the reference posture feature amount calculation unit 13 determines that the driver is facing forward in the vehicle's traveling direction. In other words, when the vehicle speed, steering wheel angle, and shift position satisfy the vehicle conditions, the reference posture feature amount calculation unit 13 estimates that the driver's posture is the reference posture. The reference posture feature amount calculation unit 13 may acquire information on the vehicle speed from a vehicle speed sensor C62 (see FIG. 4, which will be described later) mounted on the vehicle. The reference posture feature amount calculation unit 13 may acquire information on the steering wheel steering angle from a steering torque sensor C63 (see FIG. 4, which will be described later) mounted on the vehicle. The reference posture feature amount calculation unit 13 may acquire information on the shift position from a shift position sensor (not shown) mounted on the vehicle.

[0025] The above-described method of determining whether or not the driver is considered to be facing forward with respect to the traveling direction of the vehicle by the reference posture feature amount calculation unit 13 is merely an example. The reference posture feature amount calculation unit 13 may use other methods to determine whether or not the driver is considered to be facing forward with respect to the traveling direction of the vehicle. For example, the reference posture feature amount calculation unit 13 may determine, based on the feature amount information output from the feature amount extraction unit 12, whether or not the driver is considered to be facing forward with respect to the traveling direction of the vehicle. For example, if the reference axis of the driver's facial direction and head position is the optical axis of the imaging device 2, the installation position and angle of view of the imaging device 2 in the driver's vehicle are known in advance, and therefore the reference posture feature amount calculation unit 13 can determine the ranges of the driver's facial direction and head position within which the driver is considered to be facing forward in the traveling direction of the vehicle. Note that when the driver is facing forward in the traveling direction of the vehicle, the ranges of the driver's facial direction and head position detected by the feature amount extraction unit 12 are assumed to be set in advance.

[0026] When the reference posture feature calculation unit 13 estimates that the driver's posture is the reference posture, it stores the feature information output from the feature extraction unit 12 in the memory unit 18 or in a memory area inside the reference posture feature calculation unit 13 as candidate feature information that is a candidate for calculating the reference posture feature.

[0027] When a preset number of pieces of candidate feature information are stored, the reference posture feature calculation unit 13 calculates the driver's reference posture feature, specifically, the face direction (yaw angle, pitch angle, and roll angle), head position (X coordinate, Y coordinate, Z coordinate), and face detection reliability, from the stored preset number of pieces of candidate feature information.

[0028] For example, the reference posture feature calculation unit 13 calculates the most frequent values ​​of the driver's face direction, the driver's head position, and the face detection reliability included in a predetermined number of candidate feature information as the driver's reference posture feature. For example, the reference posture feature calculation unit 13 may calculate the average value of the driver's facial direction, the driver's head position, and the face detection reliability included in a predetermined number of candidate feature information as the driver's reference posture feature.

[0029] The reference posture feature amount calculation unit 13 performs a reference posture feature amount calculation process, calculates the reference posture feature amount of the driver, and then stores information indicating the calculated reference posture feature amount of the driver (hereinafter referred to as "reference posture feature amount information") in the memory unit 18. Then, the reference posture feature value calculation unit 13 sets "1" to a reference posture feature value calculation completion flag, which is provided in a location that can be referenced by the abnormal posture detection device 1 and indicates that the reference posture feature value has been calculated. The initial value of the reference posture feature calculation completion flag is set to "0." The reference posture feature calculation completion flag is initialized when the power supply of the vehicle is turned on, when the detection unit 17 detects an abnormal posture of the driver, etc. At this time, reference posture feature amount calculation section 13 may delete the candidate feature amount information stored in storage section 18 or the internal storage area of ​​reference posture feature amount calculation section 13 .

[0030] The change amount calculation unit 14 calculates the amount of change of the feature amount extracted by the feature amount extraction unit 12 from the reference posture feature amount. In the abnormal posture detection device 1 according to embodiment 1, the process performed by the change amount calculation unit 14 to calculate the change amount from the reference posture feature amount of the feature amount extracted by the feature amount extraction unit 12 is referred to as the "change amount calculation process." The "change amount calculation process", the "first posture determination process" by first determination unit 15 (described later), the "second posture determination process" by second determination unit 16 (described later), and the "abnormal posture detection process" by detection unit 17 (described later) are performed after the "reference posture feature amount calculation process" by reference posture feature amount calculation unit 13 is completed. Details of the "first posture determination process", "second posture determination process", and "abnormal posture detection process" will be described later.

[0031] The change amount calculation unit 14 compares the feature amounts extracted by the feature amount extraction unit 12, specifically the driver's facial direction, head position, and face detection reliability, with the feature amounts included in the reference posture feature amount information stored in the storage unit 18, specifically the driver's facial direction, head position, and face detection reliability, and calculates the differences as change amounts. That is, the change amount calculation unit 14 calculates the amount of change in the driver's facial direction extracted by the feature amount extraction unit 12 from the facial direction in the driver's reference posture, the amount of change in the driver's head position extracted by the feature amount extraction unit 12 from the head position in the driver's reference posture, and the amount of change in the face detection reliability extracted by the feature amount extraction unit 12 from the face detection reliability in the driver's reference posture. The change amount calculation unit 14 calculates the amount of change for each of the yaw angle, pitch angle, and roll angle of the driver's facial direction. The change amount calculation unit 14 also calculates the amount of change for each of the X coordinate, Y coordinate, and Z coordinate of the driver's head position.

[0032] The change amount calculation unit 14 outputs information indicating the calculated change amount (hereinafter referred to as “change amount information”) to the first determination unit 15 together with the feature amount information output from the feature amount extraction unit 12. The change amount information includes information on the change amount of the driver's facial orientation (change amount of yaw angle, change amount of pitch angle, and change amount of roll angle), information on the change amount of the driver's head position (change amount of X coordinate, change amount of Y coordinate, and change amount of Z coordinate), and information indicating the change amount of face detection reliability.

[0033] The first judgment unit 15 judges whether the driver's posture is normal or not by comparing the feature extracted by the feature extraction unit 12 and the change calculated by the change calculation unit 14 with the conditions for determining normal posture, by comparing the feature extracted by the feature extraction unit 12 with the conditions for determining normal posture, or by comparing the change calculated by the change calculation unit 14 with the conditions for determining normal posture. The process of determining whether or not the driver's posture is normal, which is performed by the first determination unit 15 in the abnormal posture detection device 1 according to the first embodiment, is referred to as a "first posture determination process."

[0034] The normal posture determination condition is a condition for determining whether the driver's posture is normal, and is set in advance by an administrator or the like and stored in a location that the first determination unit 15 can refer to. The normal posture determination conditions include, for example, the following conditions:

[0035] "Of the following formulas (1-1), (1-2), (2-1), (2-2), (3-1), (3-2), (4-1), (4-2), (5-1), (5-2), (6-1), and (6-2), all of formulas (1-1) or (1-2), (2-1) or (2-2), (3-1) or (3-2), (4-1) or (4-2), (5-1) or (5-2), and (6-1) or (6-2) must be satisfied. First yaw angle threshold < Change in face direction yaw angle from the reference posture < Second yaw angle threshold (1-1) Third yaw angle threshold < Face direction yaw angle < Fourth yaw angle threshold (1-2) First pitch angle threshold < Change in face pitch angle from the face pitch angle in the reference posture < Second pitch angle threshold (2-1) Third pitch angle threshold < face direction pitch angle < fourth pitch angle threshold (2-2) First roll angle threshold < Change in face direction roll angle from the face direction roll angle in the reference posture < Second roll angle threshold (3-1) Third roll angle threshold < face direction roll angle < fourth roll angle threshold (3-2) First X-coordinate threshold < Change in the X-coordinate of the head position from the reference posture < Second X-coordinate threshold (4-1) Third X coordinate threshold < X coordinate of head position < Fourth X coordinate threshold (4-2) First Y-coordinate threshold < Change in Y-coordinate of head position from the reference posture < Second Y-coordinate threshold (5-1) Third Y-coordinate threshold < Y-coordinate of head position < Fourth Y-coordinate threshold (5-2) First Z coordinate threshold < Change in Z coordinate of head position from the reference posture < Second Z coordinate threshold (6-1) Third Z coordinate threshold < Z coordinate of head position < Fourth Z coordinate threshold (6-2) In addition, First yaw angle threshold < second yaw angle threshold, 3rd yaw angle threshold < 4th yaw angle threshold, First pitch angle threshold < second pitch angle threshold, 3rd pitch angle threshold < 4th pitch angle threshold, First roll angle threshold < second roll angle threshold, 3rd roll angle threshold < 4th roll angle threshold, 1st X coordinate threshold < 2nd X coordinate threshold, 3rd X coordinate threshold < 4th X coordinate threshold, First Y coordinate threshold < second Y coordinate threshold, 3rd Y coordinate threshold < 4th Y coordinate threshold, 1st Z coordinate threshold < 2nd Z coordinate threshold, 3rd Z coordinate threshold < 4th Z coordinate threshold

[0036] If the feature amount extracted by the feature amount extracting unit 12 or the change amount calculated by the change amount calculating unit 14 satisfies the normal posture determining condition, the first determining unit 15 determines that the driver's posture is normal. If the feature extracted by the feature extractor 12 and the change calculated by the change calculator 14 do not satisfy the normal posture determination condition, the first determination unit 15 determines that the driver's posture is not normal.

[0037] Here, the significance of the first judgment unit 15 judging whether the driver's posture is normal or not based on the feature extracted by the feature extraction unit 12, in other words, the feature based on the current captured image, and the change calculated by the change calculation unit 14, in other words, the relative change in the feature from the reference posture feature will be explained.

[0038] Figure 2 is a diagram for explaining the significance of the first determination unit 15 determining whether the driver's posture is normal or not based on the feature amount based on the current captured image and the relative change in the feature amount in embodiment 1. FIG. 2 is a simplified diagram showing an example of the concept of the determination by the first determination unit 15 as to whether or not the driver's posture is normal.

[0039] As described above, in the first embodiment, the reference posture is a posture in which the driver is considered to be facing forward in the traveling direction of the vehicle. Here, for example, as shown on the left side of FIG. 2, a posture in which the driver is driving with his elbows on the ground and the roll angle of his face is 40 degrees can also be estimated as the reference posture. Now, for example, as shown on the right side of Figure 2, let's assume that the driver stops driving with his elbows on the ground and starts driving with a straight posture, i.e., the roll angle of the face becomes 0 degrees. In this case, if the first judgment unit 15 judges whether the driver's posture is normal or not based only on the change in the feature from the reference posture feature, the first judgment unit 15 may judge that the driver's posture in the state shown on the right side of Figure 2 is not normal. The first judgment unit 15 judges whether the driver's posture is normal or not by taking into consideration not only the change in the feature from the reference posture feature but also the feature based on the current captured image, so that even if the reference posture is estimated to be a posture such as that shown on the left side of Figure 2, the first judgment unit 15 can judge that the driver's posture, which should be considered a normal posture as shown on the right side of Figure 2, is a normal posture.

[0040] In the explanation of the example using Figure 2, the driver's facial direction has been used as an example, but the same can be said for the driver's head position. By determining whether the driver's posture is normal or not by taking into consideration not only the relative change in the feature from the reference posture feature but also the feature based on the current captured image, it is possible to determine that the posture of a driver who is assuming a posture that should be considered normal is the normal posture.

[0041] The first determination unit 15 outputs a determination result as to whether or not the driver's posture is determined to be normal (hereinafter referred to as a "normal posture determination result") to the second determination unit 16. At this time, the first determination unit 15 outputs the change amount information output from the change amount calculation unit 14 to the second determination unit 16 together with the normal posture determination result.

[0042] If the first judgment unit 15 judges that the driver's posture is not normal, the second judgment unit 16 judges whether the driver is experiencing poor posture based on the amount of change calculated by the change amount calculation unit 14 and the machine learning model. The machine learning model is a trained model that receives input of the change in feature amount from a reference posture feature amount and outputs information indicating whether the driver's posture is out of alignment. The change amount is, in detail, the change in the driver's facial orientation from the facial orientation in the reference posture, the change in the driver's head position from the head position in the reference posture, and the change in the face detection reliability from the face detection reliability in the reference posture. In embodiment 1, the information indicating whether or not the driver has poor posture output by the machine learning model includes information indicating whether or not the driver has poor posture, and, if the driver has poor posture, information indicating the type of poor posture (hereinafter referred to as "poor posture type").

[0043] The machine learning model is generated in advance and stored in a location that can be referenced by the second determination unit 16, such as the storage unit 18. For example, an administrator or the like generates a machine learning model by having multiple drivers test drive the vehicle, and stores the model in the storage unit 18 or the like.

[0044] It should be noted that a plurality of posture error types are defined in advance as types of posture error that may occur when a driver is experiencing posture error. Figure 3 (partially modified from the "Basic Design Document for Driver Abnormality Automatic Detection System" of the Advanced Safety Vehicle Promotion Study Group, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, March 2018) is a diagram used to explain the different types of poor posture. There are several types of poor posture, as shown in Figure 3. Figure 3 shows the following types of poor posture: "head down" in which the driver leans forward and continues to have their face close to the steering wheel; "head down" in which the driver continues to have their face facing down; "backward lean" in which the driver's upper body leans backward and continues to have their face facing up; "backward lean" in which the driver's upper body leans up and continues to have their face facing up; "head tilted to the side" in which the driver's face is tilted to the left or right; "sideways lean" in which the driver's upper body leans to the left or right and their face is tilted in the same direction; and "leaning to the side" in which the driver's upper body is tilted to the left or right. In the present disclosure, the multiple pre-set posture types are "head down," "head down," "backward arch," "arched back," "head tilted to the side," "head tilted to the side," and "leaning to the side," as shown in Fig. 3. Information about the multiple posture types is stored in a location that can be referenced by the second determination unit 16.

[0045] The second determination unit 16 inputs the amount of change into the machine learning model to obtain information indicating whether or not the driver has poor posture, thereby determining whether or not the driver has poor posture and, if so, the type of poor posture.

[0046] If the first determination unit 15 determines that the driver's posture is normal, the second determination unit 16 does not perform posture deviation determination using the machine learning model. That is, the second determination unit 16 determines that the driver's posture is not deviation, in other words, that the driver is not experiencing deviation.

[0047] The second determination unit 16 outputs to the detection unit 17 the determination result as to whether or not the driver is experiencing poor posture (hereinafter referred to as the “poor posture determination result”). The posture error determination result is information in which information indicating whether or not the driver has posture error is associated with information indicating the type of posture error if the driver has posture error.

[0048] The process of determining whether or not the driver is experiencing poor posture, which is performed by the second determination unit 16 in the abnormal posture detection device 1 according to the first embodiment, as described above, is referred to as "second posture determination process."

[0049] Based on the posture collapse determination result output from the second determination unit 16, the detection unit 17 detects whether the driver's posture is abnormal or not. In the abnormal posture detection device 1 according to the first embodiment, the process performed by the detection unit 17 to detect whether the posture of the driver is abnormal or not is referred to as "abnormal posture detection process."

[0050] The detection unit 17 detects that the driver's posture is abnormal when the state in which the second judgment unit 16 judges that the driver is suffering from poor posture continues for a preset time (hereinafter referred to as the "abnormality detection time"). The detection unit 17 detects that the driver's posture is not abnormal when the second determination unit 16 determines that the driver is not suffering from poor posture, or when the state in which the second determination unit 16 determines that the driver is suffering from poor posture does not continue for the abnormality detection time.

[0051] In detail, the detection unit 17 first determines whether or not the second determination unit 16 has determined that the driver has poor posture. From the result of the posture deviation determination output from the second determination unit 16, the detection unit 17 can determine whether or not the second determination unit 16 has determined that the driver has poor posture.

[0052] If the second judgment unit 16 judges that the driver has no posture imbalance, that is, if a posture imbalance judgment result indicating that the driver has not experienced posture imbalance is output, the detection unit 17 detects that the driver's posture is not abnormal.

[0053] When the second judgment unit 16 judges that the driver has poor posture, that is, when the second judgment unit 16 outputs a posture error judgment result indicating that the driver is experiencing poor posture, the detection unit 17 counts up a counter for counting the abnormality detection time (hereinafter referred to as the "abnormal posture detection counter"). On the other hand, if the second judgment unit 16 does not judge that the driver has poor posture, that is, if the second judgment unit 16 outputs a posture deviation judgment result indicating that the driver is not experiencing poor posture, the detection unit 17 resets the abnormal posture detection counter.

[0054] Next, the second determination unit 16 determines whether or not the abnormal posture detection counter has reached a preset threshold value (hereinafter referred to as "abnormality detection threshold value"). When the abnormal posture detection counter reaches the abnormality detection threshold, the detection unit 17 determines that the state in which the second determination unit 16 determined that the driver is suffering from poor posture has continued for the abnormality detection time, and detects that the driver's posture is abnormal. The detection unit 17 outputs information indicating that the driver's posture is abnormal (hereinafter referred to as "abnormal posture detection information") to the driving control system 300. The abnormal posture detection information includes information indicating that the driver's posture is abnormal and information indicating the posture deviation type. The detection unit 17 can identify the posture deviation type based on the posture deviation determination result output from the second determination unit 16.

[0055] The detection unit 17 may output the abnormal posture detection information to an output device (not shown) as information for analysis in case some unexpected situation occurs.

[0056] If the abnormal posture detection counter has not reached the abnormality detection threshold, the detection unit 17 determines that the state in which the second determination unit 16 determined that the driver is experiencing poor posture has not continued for the abnormality detection time, and does not detect that the driver's posture is abnormal. In other words, the detection unit 17 detects that the driver's posture is not abnormal. The detection unit 17 may output information indicating that the driver's abnormal posture has not been detected as abnormal posture detection information to the driving control system 300 or to an output device.

[0057] The storage unit 18 stores various information such as feature amount information, reference posture feature amount information, and machine learning models. 1, the storage unit 18 is provided in the abnormal posture detection device 1, but this is merely an example. The storage unit 18 may be provided outside the abnormal posture detection device 1 in a location that can be referenced by the abnormal posture detection device 1.

[0058] FIG. 4 is a diagram showing an example of the configuration of an operation control system 300 according to the first embodiment. In the following description, a vehicle equipped with the abnormal attitude detection system 100 will be referred to as a first vehicle, and a vehicle other than the first vehicle will be referred to as a second vehicle. Note that there may be cases where the first vehicle and the second vehicle are not distinguished from each other and the first vehicle and the second vehicle are each referred to as a vehicle. In the following, an example will be described in which the abnormal attitude detection system 100 is used as part of a driving control system 300, but the abnormal attitude detection system 100 may be configured to be able to communicate with the driving control system 300 wirelessly or via a wire, as shown in FIG. 1.

[0059] The driving control system 300 includes a driving control device C3 that controls the driving of the first vehicle by outputting a control signal based on abnormal attitude detection information from the abnormal attitude detection system 100 to control the steering mechanism 3 and braking / driving mechanism 4 of the first vehicle. The driving control system 300 also includes a map information storage device C4 that stores map information used for automatic driving of the vehicle, a surrounding condition monitoring device C5 that monitors the surrounding conditions of the vehicle, a vehicle condition acquisition device C6 that acquires information indicating the condition of the first vehicle, and an alarm control device C7 that controls warnings or notifications to the driver.

[0060] In addition, the abnormal posture detection system 100, driving control device C3, map information storage device C4, surrounding conditions monitoring device C5, vehicle status acquisition device C6, and alarm control device C7 provided in the driving control system 300 are each connected to a communication bus C8, and data can be sent and received via the communication bus C8.

[0061] The steering mechanism 3 is a mechanism provided on the first vehicle for determining the traveling direction of the first vehicle, and includes, for example, a steering column, a steering shaft, a rack, a pinion, and a steering actuator 31. The braking / driving mechanism 4 is a mechanism for controlling the traveling speed of the first vehicle and switching between forward and reverse driving, and includes, for example, an accelerator, a brake, a shift, and a braking / driving actuator 41. The steering actuator 31 that controls the steering mechanism 3 is made up of, for example, an EPS (Electric Power Steering) motor, and the braking / driving actuator 41 that controls the braking / driving mechanism 4 is made up of, for example, an electronically controlled throttle, a brake actuator, etc.

[0062] The map information storage device C4 is a storage medium that stores map information, including road connectivity, the number of lanes, intersection location information, and railroad crossing location information. Here, the map information storage device C4 is assumed to be mounted on the first vehicle. However, the map information storage device C4 may be configured, for example, as a server that transmits map data to the driving control device C3 via communications. When the map information storage device C4 is configured as the server, for example, a storage medium may be provided in the driving control system 300, and map information acquired from the map information storage device C4 via the on-board communication device C52 (described later) may be stored in the storage medium.

[0063] The surroundings monitoring device C5 monitors the surroundings of the first vehicle, and includes a GPS (Global Positioning System) receiver C51, an on-board communication device C52, an external sensor C53, and a navigation system C54.

[0064] The GPS receiver C51 receives signals transmitted from GPS positioning satellites and detects the current position of the first vehicle.

[0065] The outside vehicle sensor C53 is composed of, for example, at least one of a camera that captures images outside the vehicle, a millimeter-wave radar, a LiDAR, and an ultrasonic sensor, and detects the position of a second vehicle, pedestrian, obstacle, etc. that is present around the first vehicle or the distance from the first vehicle.

[0066] The navigation system C54 calculates a route from the current position of the first vehicle to a destination and provides guidance along the calculated route. The navigation system C54 has a display unit configured, for example, as a liquid crystal display, and is housed in an instrument panel. The navigation system C54 also has an operation unit such as a touch panel or physical buttons, and is configured to be able to accept operations by the occupant. The operation unit of the navigation system C54 may be configured with a microphone or the like so as to be able to accept operations via voice uttered by the occupant.

[0067] The in-vehicle communication device C52 will be described. The in-vehicle communication device C52 is, for example, a wireless communication device connected to an antenna for wireless communication. The in-vehicle communication device C52 acquires information on the position of the second vehicle or pedestrians, traffic information, etc. by communicating with a communication device of a second vehicle or a communication device installed on the road. Here, the traffic information includes, for example, congestion information, traffic regulation information, construction section information, etc.

[0068] Furthermore, the in-vehicle communication device C52 can perform inter-vehicle communication via wireless communication with the in-vehicle communication device C52 of a second vehicle located around the first vehicle. In addition, the in-vehicle communication device C52 may perform mobile communication with a base station outside the first vehicle. Furthermore, the in-vehicle communication device C52 is configured to be able to transmit information about the first vehicle output onto the communication bus C8 to the second vehicle and a call center, etc. Note that the in-vehicle communication device C52 can output information received from the second vehicle and information received from the call center, etc., to the communication bus C8. Furthermore, the in-vehicle communication device C52 may be configured to be able to acquire information from a server, etc. located outside the vehicle, such as a map information storage device C4 configured as a server that transmits map data to the driving control device C3.

[0069] The vehicle state acquisition device C6 acquires information indicating the state of the first vehicle, and includes a steering angle sensor C61, a vehicle speed sensor C62, a steering torque sensor C63, an accelerator position sensor C64, and a brake position sensor C65.

[0070] The steering angle sensor C61 is provided, for example, on an EPS motor or a steering wheel, and detects the steering angle of the first vehicle. The vehicle speed sensor C62 is provided, for example, on a wheel, and detects the traveling speed of the first vehicle.

[0071] The steering torque sensor C63 is provided on the steering wheel, for example, and detects the magnitude of the steering wheel operation force by the driver. The accelerator position sensor C64 detects the amount of depression of the accelerator pedal by the driver. The brake position sensor C65 detects the amount of depression of the brake pedal by the driver.

[0072] The warning control device C7 acquires abnormality detection information from the abnormal posture detection system 100. When the warning control device C7 acquires abnormality detection information indicating that the driver's posture is abnormal, it outputs a warning to the driver via the warning unit C71. The warning unit C71 is, for example, an audio device mounted on the first vehicle, and the warning is a voice or notification sound output from the audio device. The warning unit C71 may also be configured as a display unit of the navigation system C54, and the warning may be a warning message displayed on the display unit of the navigation system C54. As described above, the warning may be any warning that can be recognized by the driver of the first vehicle. The driving control device C3 may also be controlled to output a control signal to cause the warning control device C7 to issue a warning.

[0073] Furthermore, the alarm control device C7 stops issuing the alarm if the driver responds to the alarm after issuing the alarm. Here, the case where the driver responds to the warning means a case where the driver is recognized to have made a conscious action in response to the warning, such as when the abnormal posture detection system 100 outputs information indicating that the driver's posture is not abnormal, or when an operation unit provided in the navigation system C54 and a switch provided on the steering wheel receive an operation by the driver. The alarm control device C7 may also output a signal indicating that the driver has responded to the warning to the abnormal posture detection system 100.

[0074] Furthermore, when abnormal posture detection information indicating that the driver's posture is abnormal is output from the abnormal posture detection system 100, the warning control device C7 activates a notification unit C72 such as a turn signal, a hazard lamp, and a headlight that are provided on the exterior of the vehicle so as to be visible from the second vehicle, to notify people outside the first vehicle, such as occupants of the second vehicle, that the posture of the driver of the first vehicle is abnormal. For example, when there is no reaction from the driver to the warning for a predetermined period after the warning is issued, the warning control device C7 activates the notification unit C72 to notify people outside the first vehicle that the posture of the driver of the first vehicle is abnormal.

[0075] In addition, the warning control device C7 may notify the occupants of the second vehicle that the posture of the driver of the first vehicle is abnormal, for example, by displaying an alert message on the display unit of the navigation system C54 of the second vehicle through vehicle-to-vehicle communication via the above-mentioned in-vehicle communication device C52.

[0076] Furthermore, when driving assistance is performed by the driving control device C3, the notification control unit may use the vehicle state acquired by the vehicle state acquisition unit or the signal output from the driving control device C3 to notify people outside the first vehicle, such as occupants of the second vehicle, of the direction of travel of the first vehicle or whether or not it is changing lanes, using the notification unit C72, which is composed of the above-mentioned lighting fixtures, etc., provided on the vehicle exterior, etc.

[0077] The driving control device C3 of the driving control system 300 controls the steering actuator 31 or braking / driving actuator 41 mounted on the first vehicle to assist the driver in driving. The driving control device C3 performs automatic driving control of the first vehicle, for example, by outputting signals to the steering actuator 31 or braking / driving actuator 41. The driving control device C3 performs braking / driving control of the first vehicle, for example, by controlling the braking / driving actuator 41, which is composed of an electronically controlled throttle and a brake actuator, to operate the brakes to slow down or stop the first vehicle. The driving control device C3 performs steering control of the first vehicle, for example, by controlling the steering actuator 31, which is composed of an EPS motor, to maintain the lane in which the first vehicle is traveling.

[0078] Furthermore, the driving control device C3 has a plurality of driving assistance functions that assist or substitute for the driver's driving operation by controlling the driving force, braking force, steering force, etc. of the first vehicle. For example, the driving assistance functions include a cruise control function and a lane departure prevention function. In the following description, the cruise control function will be referred to as ACC (Adaptive Cruise Control), and the lane departure prevention function will be referred to as LKA (Lane Keeping Assist).

[0079] When the driving control device C3 executes the ACC, it controls the traveling speed of the first vehicle by adjusting the driving force and braking force based on monitoring information of the vehicle ahead obtained from the surrounding conditions monitoring device C5. If a vehicle ahead is not detected, the ACC causes the first vehicle to travel at a constant speed at a target speed preset by the driver or the like. On the other hand, if a vehicle ahead is detected, the ACC causes the first vehicle to follow the vehicle ahead while maintaining a distance from the vehicle ahead.

[0080] Furthermore, when LKA is executed, the driving control device C3 controls the steering force and steering force based on shape information of the lane markings in the direction of travel acquired from the surrounding conditions monitoring device C5. LKA applies a steering force to the steering wheel in a direction that prevents the first vehicle from approaching the lane markings, causing the first vehicle to travel along the lane. Note that road information output by the surrounding conditions monitoring device C5 may be used for vehicle control by ACC and LKA.

[0081] Furthermore, when the driving control device C3 executes the emergency evacuation function, it can perform automatic evacuation control (evacuation processing) to automatically stop the first vehicle. When the automatic evacuation control is started, the driving control device C3 causes the surrounding condition monitoring device C5 to search for an evacuation location where the first vehicle should be stopped. Then, the driving control device C3 moves the first vehicle to the evacuation location set by the surrounding condition monitoring device C5 and stops the first vehicle at this evacuation location. Note that, on expressways, the above-mentioned evacuation location may be a shoulder outside the lane in which the vehicle is traveling, and on ordinary roads, it may be a shoulder outside the lane in which the vehicle is traveling, as well as a location such as an intersection, a railroad crossing, or a sidewalk that avoids locations where there is a high possibility of a second vehicle, a train, or a moving object such as a pedestrian being present.

[0082] Here, if the driver's posture is abnormal, it is estimated that the driver is in an abnormal state. If the driver is in an abnormal state, there is a high possibility that the operation by the driver is an erroneous operation. For example, if the driver's physical condition suddenly changes and the driver leans forward on the steering wheel, there is a high possibility that the steering wheel is operated but this operation is an erroneous operation. Also, if the driver's physical condition suddenly changes and the driver leans back, there is a possibility that the accelerator pedal is accidentally depressed.

[0083] Therefore, when the driving control device C3 executes the driving assistance function due to the driver's abnormal posture, it may be possible to validate driving operations (hereinafter referred to as override operations) aimed at preventing accidents, such as intentional driving operations by the driver or driving operations by a passenger in place of the driver, while invalidating erroneous operations by the driver. The vehicle state acquisition device C6 may be provided with an override operation detection unit C66 that detects override operations.

[0084] The override operation will be described below. Regarding vehicle acceleration, if the driver is in an abnormal state, the driver may lose posture and accidentally press the accelerator pedal. Therefore, when the driving assistance function by the driving control device C3 is being executed, the accelerator operation may be disabled. In other words, the accelerator operation does not need to be included in the override operation.

[0085] On the other hand, even when the driver is in an abnormal state regarding deceleration of the vehicle, the driver may become drowsy and stop the first vehicle in an attempt to avoid a collision with an obstacle. Therefore, the override operation detection unit C66 acquires the amount of depression of the brake pedal from the brake position sensor C65, and if the braking force obtained by depression of the brake pedal is greater than the braking force in the driving assistance function of the driving control device C3, detects the operation of the brake pedal as an override operation.

[0086] Furthermore, when changing the vehicle's course, even if the driver is in an abnormal state, a passenger may operate the steering wheel in place of the driver. However, if the driver is in an abnormal state, the driver's posture may become unstable and the driver may fall onto the steering wheel. Therefore, the override operation detection unit C66 does not need to detect this erroneous operation as an override operation. Therefore, the override operation detection unit C66 detects, as an override operation, a steering wheel operation performed when the abnormal posture detection system 100 provides a detection result indicating that the driver's posture is not unstable.

[0087] When the override operation detection unit C66 detects an override operation while the driving support function is being executed, the driving control system 300 may substitute the control of the steering actuator 31 or the braking / driving actuator 41 with the driving operation by the passenger or the driver. In other words, while the driving support function is being executed, the driving control of the vehicle may be performed based on the override operation detected by the override operation detection unit C66.

[0088] The operation of the abnormal posture detection device 1 according to the first embodiment will be described.

[0089] FIG. 5 is a flowchart for explaining the operation of the reference posture feature amount calculation process performed by the abnormal posture detection device 1 according to the first embodiment. The operation of the reference posture feature calculation process by the abnormal posture detection device 1 shown in the flowchart of Figure 5 is repeatedly performed, for example, in accordance with instructions from a control unit (not shown), while the reference posture feature calculation completion flag is "0" until the reference posture feature calculation completion flag becomes "1". For example, when the ignition of the vehicle is turned on, the control unit refers to the reference posture feature calculation completion flag. When the control unit confirms that the reference posture feature calculation completion flag is "0", it instructs the image acquisition unit 11, feature extraction unit 12, and reference posture feature calculation unit 13 of the abnormal posture detection device 1 to perform a reference posture feature calculation process.

[0090] The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST1). The image acquisition unit 11 outputs the acquired captured image to the feature extraction unit 12.

[0091] The feature extraction unit 12 extracts feature amounts used to determine the driver's posture based on the captured image acquired by the image acquisition unit 11 in step ST1 (step ST2). The feature amount extraction unit 12 outputs the feature amount information to the reference posture feature amount calculation unit 13 and stores it in chronological order in the storage unit 18 .

[0092] The reference posture feature amount calculation unit 13 estimates the reference posture of the driver and performs a reference posture feature amount calculation process to calculate the reference posture feature amount (step ST3). In the reference posture feature amount calculation process in step ST3, the reference posture feature amount calculation unit 13 first estimates whether the posture of the driver is the reference posture. When the reference posture feature calculation unit 13 estimates that the driver's posture is the reference posture, it stores the feature information output from the feature extraction unit 12 in the memory unit 18 or in a memory area inside the reference posture feature calculation unit 13 as candidate feature information that is a candidate for calculating the reference posture feature. While the preset number of candidate feature information pieces have not been stored, reference posture feature calculation unit 13 does not set the reference posture feature calculation completion flag to “1” and ends the processing of step ST3. Then, the operation of abnormal posture detection device 1 returns to the processing of step ST1. In step ST3, when a predetermined number of candidate feature information items are stored, the reference posture feature calculation unit 13 calculates the driver's reference posture feature information and face detection reliability from the stored predetermined number of candidate feature information items. Reference posture feature quantity calculation unit 13 stores the reference posture feature quantity information in storage unit 18 and sets the reference posture feature quantity calculation completion flag to “1.” At this time, reference posture feature quantity calculation unit 13 may delete the reference posture feature quantity information stored in storage unit 18 or the internal storage area of ​​reference posture feature quantity calculation unit 13.

[0093] FIG. 6 is a flowchart for explaining operations in the change amount calculation process, the first posture determination process, the second posture determination process, and the abnormal posture detection process performed by the abnormal posture detection device 1 according to the first embodiment. The operations of the change amount calculation process, first attitude determination process, second attitude determination process, and abnormal attitude detection process by the abnormal attitude detection device 1 shown in the flowchart of Figure 6 are repeatedly performed after the reference attitude feature calculation process described using the flowchart of Figure 5 is completed, for example, until the vehicle ignition is turned off. When the reference posture feature calculation process is completed and the reference posture feature calculation completion flag is set to "1", the control unit confirms this and instructs the image acquisition unit 11, feature extraction unit 12, change amount calculation unit 14, first judgment unit 15, second judgment unit 16, and detection unit 17 of the abnormal posture detection device 1 to perform change amount calculation process, first posture judgment process, second posture judgment process, or abnormal posture detection process.

[0094] The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST11). The image acquisition unit 11 outputs the acquired captured image to the feature extraction unit 12.

[0095] The feature extraction unit 12 extracts feature amounts used to determine the driver's posture based on the captured image acquired by the image acquisition unit 11 in step ST11 (step ST12). The feature extraction unit 12 outputs the feature information to the change amount calculation unit 14 and stores it in chronological order in the storage unit 18 .

[0096] The change amount calculation unit 14 performs a change amount calculation process to calculate the amount of change from the reference posture feature amount of the feature amount extracted by the feature amount extraction unit 12 in step ST12 (step ST13). The change amount calculation unit 14 outputs the change amount information to the first determination unit 15 together with the feature amount information output from the feature amount extraction unit 12.

[0097] The first judgment unit 15 performs a first posture judgment process to judge whether the driver's posture is normal or not by comparing the feature extracted by the feature extraction unit 12 in step ST12 with the change amount calculated by the change amount calculation unit 14 in step ST13 with the normal posture judgment condition, or by comparing the feature extracted by the feature extraction unit 12 with the normal posture judgment condition, or by comparing the change amount calculated by the change amount calculation unit 14 with the normal posture judgment condition (step ST14).

[0098] FIG. 7 is a flowchart for explaining the detailed operation of the first attitude determination process by the first determination unit 15, which is performed in step ST14 of FIG.

[0099] First determination unit 15 compares the feature extracted by feature extraction unit 12 and the change calculated by change calculation unit 14 with the conditions for determining normal posture, compares the feature extracted by feature extraction unit 12 with the conditions for determining normal posture, or compares the change calculated by change calculation unit 14 with the conditions for determining normal posture (step ST110), and determines whether the feature extracted by feature extraction unit 12 and the change calculated by change calculation unit 14 satisfy the conditions for determining normal posture, whether the feature extracted by feature extraction unit 12 satisfies the conditions for determining normal posture, or whether the change calculated by change calculation unit 14 satisfies the conditions for determining normal posture (step ST120).

[0100] If the feature extracted by the feature extracting unit 12 or the change calculated by the change amount calculating unit 14 satisfies the normal posture determination condition, if the feature extracted by the feature extracting unit 12 satisfies the normal posture determination condition, or if the change calculated by the change amount calculating unit 14 satisfies the normal posture determination condition (“YES” in step ST120), the first determination unit 15 determines that the driver's posture is normal (step ST130). The first determination unit 15 outputs a normal posture determination result indicating that the driver's posture is normal to the second determination unit 16. At this time, the first determination unit 15 outputs the change amount information output from the change amount calculating unit 14 together with the normal posture determination result to the second determination unit 16.

[0101] If the feature extracted by the feature extraction unit 12 and the change calculated by the change amount calculation unit 14 do not satisfy the normal posture determination condition ("NO" in step ST120), the first determination unit 15 does not determine that the driver's posture is normal. That is, the first determination unit 15 determines that the driver's posture is not normal. The first determination unit 15 outputs a normal posture determination result indicating that the driver's posture is not normal, in other words, that the first determination unit 15 has determined that the driver's posture is not normal, to the second determination unit 16. At this time, the first determination unit 15 outputs the change amount information output from the change amount calculation unit 14 together with the normal posture determination result to the second determination unit 16.

[0102] Returning to the explanation of the flowchart in FIG. The second determination unit 16 performs a second posture determination process to determine whether or not the driver is experiencing poor posture based on the normal posture determination result output from the first determination unit 15 in step ST14 (step ST15).

[0103] FIG. 8 is a flowchart for explaining the detailed operation of the second attitude determination process by the second determination unit 16, which is performed in step ST15 of FIG.

[0104] In step ST14 of FIG. 6, if the first judgment unit 15 judges that the driver's posture is normal (in the case of "YES" in step ST210), the second judgment unit 16 judges that the driver's posture is not poor, in other words, that the driver is not experiencing poor posture, and outputs a posture poor judgment result indicating that the driver is not experiencing poor posture to the detection unit 17, and terminates the processing shown in the flowchart of FIG. 8.

[0105] 6, if the first determination unit 15 determines that the driver's posture is not normal ("NO" in step ST210), the second determination unit 16 determines whether the driver is experiencing poor posture based on the amount of change calculated by the change amount calculation unit 14 in step ST13 of Fig. 6 and the machine learning model (step ST220). The second determination unit 16 outputs a posture error determination result indicating that the driver is experiencing poor posture to the detection unit 17.

[0106] Returning to the explanation of the flowchart in FIG. The detection unit 17 performs an abnormal posture detection process to detect whether the driver's posture is abnormal or not based on the posture collapse determination result output from the second determination unit 16 in step ST15 (step ST16).

[0107] FIG. 9 is a flowchart for explaining the detailed operation of the abnormal posture detection process by the detection unit 17, which is performed in step ST16 of FIG.

[0108] The detection unit 17 determines whether or not the second determination unit 16 has determined in step ST15 of FIG. 6 that the driver's posture has deteriorated (step ST310).

[0109] If the second judgment unit 16 judges that the driver has poor posture (if "YES" in step ST310), that is, if the second judgment unit 16 outputs a posture error judgment result indicating that the driver is experiencing poor posture, the detection unit 17 counts up the abnormal posture detection counter (step ST320).

[0110] On the other hand, if the second judgment unit 16 does not judge that the driver has poor posture (if "NO" in step ST310), that is, if the second judgment unit 16 outputs a posture deviation judgment result indicating that the driver is not experiencing poor posture, the detection unit 17 resets the abnormal posture detection counter (step ST330).

[0111] The second determination unit 16 determines whether or not the abnormal posture detection counter has reached the abnormality detection threshold value (step ST340).

[0112] If the abnormal posture detection counter reaches the abnormality detection threshold ("YES" in step ST340), the detection unit 17 determines that the state in which the second determination unit 16 determined that the driver is experiencing poor posture has continued for the abnormality detection time, and detects the driver's abnormal posture (step ST350). The detection unit 17 outputs abnormal posture detection information indicating that the driver's abnormal posture has been detected to the driving control system 300. The detection unit 17 may output the abnormal posture detection information to an output device as information for analysis in case some unexpected situation occurs.

[0113] If the abnormal posture detection counter has not reached the abnormality detection threshold ("NO" in step ST340), the detection unit 17 determines that the state in which the second determination unit 16 determined that the driver is experiencing poor posture has not continued for the abnormality detection time, and does not detect an abnormal posture of the driver. In other words, the detection unit 17 detects that the driver's posture is not abnormal. The detection unit 17 may output information that an abnormal posture of the driver has not been detected to the driving control system 300 or to an output device.

[0114] In this way, the abnormal posture detection device 1 according to the first embodiment calculates the amount of change from the reference posture feature of the feature extracted based on the captured image capturing the area where the driver's face should be. The abnormal posture detection device 1 first compares the extracted feature and the calculated amount of change with the normal posture determination conditions, and determines whether the driver's posture is normal by comparing the extracted feature with the normal posture determination conditions or by comparing the calculated amount of change with the normal posture determination conditions. If the abnormal posture detection device 1 determines that the driver's posture is not normal, it then determines whether the driver is experiencing poor posture based on the calculated amount of change and a machine learning model. If the state in which it has been determined that the driver is experiencing poor posture continues for the abnormality detection time, the abnormal posture detection device 1 detects the driver's abnormal posture.

[0115] By using a machine learning model to determine whether an occupant's posture is out of alignment, it is possible to detect abnormal occupant posture with high accuracy. On the other hand, in a scene where feature values ​​such as facial orientation cannot be stably extracted from a captured image, the presence or absence of an occupant's posture is determined based on low-accuracy feature values ​​and the machine learning model. In this case, when determining whether an occupant's posture is out of alignment using the machine learning model, there is a possibility that the occupant may be determined to be in an out-of-alignment state even though they are not. Here, FIGS. 10A and 10B are diagrams for explaining an example of a scene in which feature amounts such as facial orientation cannot be stably extracted from a captured image. FIG. 10A shows an example of a case where the driver's face is no longer captured in the captured image because the driver has assumed a "prone" posture. FIG. 10B shows an example of a case where the driver touches his / her face with his / her hand, causing a part of the driver's face to be obscured in the captured image. 10A and 10B, the captured image is indicated by Im and the driver is indicated by Dr. In the first embodiment, the upper left corner of the captured image is the origin, the rightward direction of the captured image is the positive direction of the x-axis, and the downward direction of the captured image is the positive direction of the y-axis.

[0116] For example, if the driver falls into a "prone" posture, the driver's face will not be captured in the captured image, and the feature amount of the driver's facial direction and the like will not be extracted (see FIG. 10A).

[0117] For example, if the driver is touching his / her face with his / her hand, part of the driver's face is covered in the captured image, and the feature amount such as the driver's facial direction cannot be stably extracted from the captured image (see Fig. 10B). In the example of Fig. 10B, the driver's eyes are covered, and therefore the facial direction cannot be stably extracted.

[0118] It is assumed that the amount of change in the feature from the reference posture feature will be large in both the scene shown in Fig. 10A and the scene shown in Fig. 10B. As a result, for example, if the amount of change is input to a machine learning model that determines whether the driver has poor posture, it may be determined that the driver has poor posture in both scenes. In fact, it should not be determined that the driver has poor posture in the scene shown in Fig. 10B. When determining whether a driver has poor posture using a machine learning model, in scenes where features cannot be extracted stably, an unintended change from the reference posture feature may occur, increasing the possibility of misjudging whether the driver has poor posture.

[0119] In contrast, as described above, the abnormal posture detection device 1 according to the first embodiment first determines whether the driver's posture is normal by comparing the extracted feature amount and the calculated change amount with the normal posture determination conditions, or by comparing the extracted feature amount with the normal posture determination conditions, or by comparing the calculated change amount with the normal posture determination conditions. After that, if the abnormal posture detection device 1 determines that the driver's posture is not normal, it determines whether the driver is suffering from poor posture based on the calculated change amount and the machine learning model, and if the state in which it has been determined that the driver is suffering from poor posture continues for the abnormality detection time, it detects the driver's abnormal posture.

[0120] Therefore, the abnormal posture detection device 1 can detect an abnormal posture of the driver even if a scene occurs in which feature amounts cannot be stably extracted from the captured image.

[0121] FIG. 11 is a flowchart for explaining an example of the operation of the driving support function by the driving control system 300 according to the first embodiment. In the following description, an example will be given in which the driving control system 300 performs an emergency evacuation function for the vehicle as a driving assistance function when the abnormal posture detection system 100 detects that the driver's posture is abnormal. Note that the operation of the driving control system 300 is started, for example, after the ignition of the vehicle is turned on.

[0122] First, the driving control device C3 of the driving control system 300 acquires abnormal posture detection information from the abnormal posture detection system 100 (step ST401) and determines whether or not an abnormal posture of the driver has been detected (step ST402). If the abnormal posture detection information does not indicate that an abnormal posture of the driver has been detected ("NO" in step ST402), the operation of the driving control system 300 proceeds to the processing of ST401.

[0123] On the other hand, when the abnormal posture detection information indicates that an abnormal posture of the driver has been detected, the alarm control device C7 activates the warning unit C71 to start warning the driver (step ST403). Here, the warning to the driver by the warning unit C71 is, for example, displaying a warning message on the display unit of the navigation system C54 mounted on the first vehicle, or outputting sound from an audio device mounted on the first vehicle.

[0124] Next, the warning control device C7 determines whether or not a response has been received from the driver in response to the warning (step ST404). Here, the determination of whether or not a response has been received from the driver in response to the warning is made based on, for example, whether or not abnormal posture detection information indicating that an abnormal posture of the driver has been detected has been received from the abnormal posture detection system 100 before a predetermined time has elapsed since the warning was issued from the warning unit C71.

[0125] That is, if abnormal posture detection information indicating that an abnormal posture of the driver has been detected is not received from the abnormal posture detection system 100 before a predetermined time has elapsed since the warning unit C71 was activated, the warning control device C7 determines that a response has been received from the driver to the warning. On the other hand, if a predetermined time has elapsed since the warning unit C71 was activated while abnormal posture detection information indicating that an abnormal posture of the driver has been detected remains received from the abnormal posture detection system 100, the warning control device C7 determines that a response has not been received from the driver to the warning. Note that the determination of whether a response has been received from the driver to the warning may be made, for example, based on whether or not the driver has operated the operating unit to cancel the warning before a predetermined time has elapsed since the warning unit C71 was activated. The predetermined time is, for example, 3 seconds.

[0126] If a response is obtained from the driver in response to the warning ("YES" in step ST404), the warning control device C7 stops the warning from the warning unit C71 (step ST405), and the operation of the driving control system 300 proceeds to the processing of step ST401. On the other hand, if no response is obtained from the driver in response to the warning ("NO" in step ST404), the warning control device C7 activates the notification unit C72 to start notifying people outside the first vehicle (step ST406). Here, notifying the second vehicle means notifying people outside the first vehicle, such as passengers of the second vehicle, that the driver of the first vehicle is not in a condition to continue driving, or notifying that the first vehicle will start a driving support function, such as an emergency evacuation function.

[0127] Then, the driving control device C3 starts the emergency evacuation function. First, the driving control device C3 acquires information about the vehicle state from the vehicle state acquisition device C6 (step ST407). The vehicle state is information indicating the state of the first vehicle, and is used for vehicle control in driving support functions such as the emergency evacuation function.

[0128] Next, the driving control device C3 acquires information about the surrounding conditions of the first vehicle from the surrounding conditions monitoring device C5 (step ST408) and sets a location to which the first vehicle will be evacuated (step ST409). Hereinafter, the location to which the first vehicle will be evacuated will be referred to as the evacuation location. The evacuation location may be within a range of 150 m from the current location of the first vehicle, or a location that takes the first vehicle 60 seconds to move from the current location. This prevents the first vehicle from entering the intersection more than necessary while moving to the evacuation location.

[0129] For example, the driving control device C3 searches for a position where the first vehicle can be stopped safely from the surrounding situation monitoring device C5, and sets the safest position from among the candidate positions where the first vehicle can be stopped safely as the evacuation position. Here, the evacuation position is a position where there is no possibility of collision with a moving object such as a second vehicle or a pedestrian, or an obstacle, and where there is no possibility of collision with the moving object or obstacle on the route from the current position of the first vehicle to the evacuation position. Examples of evacuation positions include the edge of a road such as a road shoulder, or within a lane other than an intersection or a railroad crossing. Furthermore, if the evacuation position is the edge of a road such as a road shoulder, space may be secured to allow passengers to escape from the first vehicle.

[0130] After setting the evacuation position, the driving control device C3 performs vehicle control to stop the first vehicle at the evacuation position. After setting the evacuation position, the driving control device C3 may display a route from the current position of the first vehicle to the set evacuation position on a display unit of the navigation system C54.

[0131] First, the driving control device C3 determines whether a lane change is required before the first vehicle is moved to the evacuation position (step ST410). If a lane change is not required before the first vehicle is moved to the evacuation position (if "NO" in step ST410), that is, if the evacuation position is set within the lane in which the first vehicle is currently traveling, the driving control device C3 controls the steering actuator 31 and the braking / driving actuator 41 to cause the first vehicle to travel within the lane in which the first vehicle is currently traveling (step ST411). Then, the driving control device C3 controls the steering actuator 31 and the braking / driving actuator 41 to decelerate the first vehicle and then stop it at the evacuation position (step ST412).

[0132] In the process of step ST411, the driving control device C3 may drive the first vehicle at a speed at which an emergency stop can be made, such as 10 km / h. In addition, if it is not necessary to continue driving the first vehicle, such as when the evacuation position is set near the first vehicle, the process of step ST411 can be omitted. Furthermore, in the processes of steps ST411 and ST412, when the driving control device C3 decelerates the first vehicle, the speed may be set to, for example, 3 m / s to prevent the passenger from falling. 2 It may be the following:

[0133] Then, after the driving control device C3 stops the first vehicle at the evacuation position, a report is sent to a call center or the like outside the vehicle via the in-vehicle communication device C52 (step ST413). The report sent via the in-vehicle communication device C52 includes, for example, a message indicating that the driver of the first vehicle is in an abnormal posture and may be unable to continue driving, the current position of the first vehicle acquired by the surrounding situation monitoring device C5, and the evacuation position set by the driving control device C3. Note that even when a report is sent via the in-vehicle communication device C52, the driving control device C3 may continue to keep the first vehicle stopped, and the alarm control device C7 may continue to alert people outside the first vehicle.

[0134] The following describes a case where the driving control device C3 determines in the process of step ST410 that a lane change is required before the first vehicle is moved to the evacuation position (the case of "YES" in step ST410). Note that lane change includes a case where the first vehicle is moved to a lane adjacent to the lane in which the first vehicle is traveling, such as a lane change, and a case where the first vehicle is deviated from the lane in which the first vehicle is traveling so as to pull over to the side of the road.

[0135] When the driving control device C3 determines that a lane change is necessary, it controls the steering actuator 31 and the braking / driving actuator 41 to change the lane of the first vehicle (step ST414). Here, the lateral movement speed of the vehicle during lane change may be approximately 0.3 m / s so that a second vehicle or pedestrian, etc., present at the lane change destination can recognize the lane change of the first vehicle and avoid a collision. Furthermore, when a lane change is to be performed, the warning control device C7 may notify a person outside the first vehicle that the first vehicle will start changing lane via the notification unit C72 at least a predetermined time before the lane change is initiated.

[0136] Next, if no response is obtained from the driver in response to the warning from the alarm unit, a notification is made to the second vehicle via the notification unit C72, indicating that the posture of the driver of the first vehicle is abnormal and that the driver may be in a state where he or she cannot continue driving. Note that the notification to the second vehicle by the notification unit C72 is not limited to a notification indicating that the posture of the driver of the first vehicle is abnormal, but may also be a notification indicating that an emergency evacuation function of the vehicle will be started or a notification indicating that the emergency evacuation function of the vehicle is being executed, which is performed in subsequent processing.

[0137] Then, similar to the processing of step ST412, the driving control device C3 controls the steering actuator 31 and the braking / driving actuator 41 to decelerate the first vehicle and then stop it at the evacuation position (step ST415). After the driving control device C3 stops the first vehicle at the evacuation position, the operation of the driving control system 300 proceeds to the processing of step ST413, and a report is sent to a call center or the like outside the vehicle via the in-vehicle communication device C52.

[0138] Furthermore, if an override operation is performed while the driving support function is being executed, the driving control system 300 may substitute the control of the steering actuator 31 or the braking / driving actuator 41 with a driving operation by the passenger or the driver. That is, for example, if the override operation detection unit C66 detects an override operation during the processing of steps ST406 to ST415 in Fig. 11, the vehicle may be driven based on the detected override operation.

[0139] In the above example, when the abnormal posture detection system 100 detects that the driver's posture is abnormal, the driving control system 300 performs an emergency evacuation function for the vehicle as a driving assistance function. However, the driving control system 300 may also execute ACC or LKA as a driving assistance function. Furthermore, driving assistance functions may be combined, such as by executing ACC and LKA and then executing the emergency evacuation function when safety in the surrounding area is ensured. Driving assistance functions can be combined as appropriate.

[0140] In the above-described first embodiment, the abnormal posture detection device 1 is provided with the reference posture feature amount calculation unit 13 and has the function of estimating the reference posture of the driver and calculating the reference posture feature amount in the estimated reference posture, but this is merely an example. For example, the function of estimating the reference posture of the driver and calculating the reference posture feature amount in the reference posture may be possessed by a device (not shown) connected to the abnormal posture detection device 1 outside the abnormal posture detection device 1. In this case, abnormal posture detection device 1 does not necessarily have to include reference posture feature value calculation unit 13. Furthermore, with regard to the operation of abnormal posture detection device 1, the operation of the reference posture feature value calculation process as described using the flowchart shown in FIG. Alternatively, for example, an administrator or the like may set a reference posture based on the facial orientation, head position, and face detection reliability of a typical driver in advance and store the reference posture in a location accessible to the abnormal posture detection device 1, such as the storage unit 18. In this case, the abnormal posture detection device 1 is not required to include the reference posture feature amount calculation unit 13, and the operation of the reference posture feature amount calculation process as described using the flowchart shown in FIG. 5 can be omitted. However, a configuration having a function for estimating the reference posture in the reference posture feature amount calculation unit 13 or an external device allows the abnormal posture detection device 1 to estimate a reference posture and calculate reference posture feature amounts that are more tailored to the individual than a configuration not having a function for estimating the reference posture in the reference posture feature amount calculation unit 13 or an external device. As a result, the abnormal posture detection device 1 can more accurately detect whether the driver is in an abnormal posture.

[0141] In the first embodiment, the amount of change used as an input to the machine learning model includes the amount of change in face detection reliability, but this is merely an example. The amount of change used as an input to the machine learning model may not include the amount of change in face detection reliability. However, when the face detection reliability is included in the amount of change used as input to the machine learning model, the abnormal posture detection device 1 can more accurately determine whether or not posture has deteriorated, compared to when the face detection reliability is not included. Nowadays, imaging devices 2 mounted on vehicles have become smaller. Miniaturized imaging devices 2 often employ narrow-angle lenses. When the imaging device 2 is a so-called narrow-angle camera, the imaging range is narrow. That is, there is a high possibility that the driver's facial features will be out of frame. In other words, the ease of detecting the driver's facial features decreases. In this case, even if the driver's posture is not abnormal, the plausibility of the detected driver's face decreases. By using the face detection reliability as an input to the machine learning model, it is possible to prevent overdetection of poor posture, and it is expected that information indicating whether the driver's posture is poor or not obtained from the machine learning model will be obtained as an inference result with higher accuracy.

[0142] In the first embodiment, the vehicle occupant whose abnormal posture detection device 1 detects whether the vehicle is in an abnormal posture is the driver of the vehicle, but this is merely an example. The abnormal posture detection device 1 can detect whether any vehicle occupant other than the driver is in an abnormal posture.

[0143] In addition, in the above embodiment 1, the abnormal posture detection device 1 is an on-board device mounted on a vehicle, and the image acquisition unit 11, the feature extraction unit 12, the reference posture feature calculation unit 13, the change amount calculation unit 14, the first judgment unit 15, the second judgment unit 16, and the detection unit 17 are provided in the on-board device. Without being limited to this, some of the image acquisition unit 11, feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first judgment unit 15, second judgment unit 16, and detection unit 17 may be mounted on the vehicle's on-board device, and the rest may be provided on a server connected to the on-board device via a network, so that the system is configured with the on-board device and the server. In addition, the image acquisition unit 11, the feature extraction unit 12, the reference posture feature calculation unit 13, the change amount calculation unit 14, the first determination unit 15, the second determination unit 16, and the detection unit 17 may all be provided in the server.

[0144] 12A and 12B are diagrams illustrating an example of a hardware configuration of the abnormal posture detection device 1 according to the first embodiment. In the first embodiment, the functions of the image acquisition unit 11, the feature extraction unit 12, the reference posture feature calculation unit 13, the change amount calculation unit 14, the first determination unit 15, the second determination unit 16, the detection unit 17, and a control unit (not shown) are realized by the processing circuit 101. That is, the abnormal posture detection device 1 includes the processing circuit 101 for performing control to detect whether the posture of a vehicle occupant is an abnormal posture based on a captured image. The processing circuit 101 may be dedicated hardware as shown in FIG. 12A, or may be a processor 104 that executes a program stored in a memory 105 as shown in FIG. 12B.

[0145] When the processing circuitry 101 is dedicated hardware, the processing circuitry 101 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0146] When the processing circuit is processor 104, the functions of image acquisition unit 11, feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first determination unit 15, second determination unit 16, detection unit 17, and a control unit (not shown) are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 105. Processor 104 reads and executes the program stored in memory 105 to execute the functions of image acquisition unit 11, feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first determination unit 15, second determination unit 16, detection unit 17, and a control unit (not shown). That is, abnormal posture detection device 1 includes memory 105 for storing a program that, when executed by processor 104, results in the execution of steps ST1 to ST3 of FIG. 5 and steps ST11 to ST16 of FIG. 6 described above. Furthermore, it can be said that the program stored in memory 105 causes the computer to execute the procedures or methods of the processes of image acquisition unit 11, feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first determination unit 15, second determination unit 16, detection unit 17, and a control unit (not shown). Here, memory 105 corresponds to, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), or EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, flexible disk, optical disk, compact disk, minidisk, DVD (Digital Versatile Disc), etc.

[0147] Note that the functions of image acquisition unit 11, feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first determination unit 15, second determination unit 16, detection unit 17, and a control unit (not shown) may be partially implemented by dedicated hardware and partially implemented by software or firmware. For example, the functions of image acquisition unit 11 may be implemented by processing circuit 101 as dedicated hardware, and the functions of feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first determination unit 15, second determination unit 16, detection unit 17, and a control unit (not shown) may be implemented by processor 104 reading and executing programs stored in memory 105. The storage unit 18 is configured by, for example, a memory 105. The abnormal posture detection device 1 also includes an input interface device 102 and an output interface device 103 that perform wired or wireless communication with devices such as the imaging device 2 or various devices included in the operation control system 300.

[0148] As described above, according to the first embodiment, the abnormal posture detection device 1 includes an image acquisition unit 11 that acquires a captured image in which an area in which the face of a vehicle occupant should be present is captured, a feature extraction unit 12 that extracts feature amounts used to determine the posture of the occupant based on the captured image acquired by the image acquisition unit 11, a change amount calculation unit 14 that calculates a change amount of the feature amount extracted by the feature extraction unit 12 from a feature amount in a reference posture of the occupant, and a normal posture determination condition by comparing the feature amount extracted by the feature extraction unit 12 and the change amount calculated by the change amount calculation unit 14 with the normal posture determination condition. The abnormal posture detection device 1 is configured to include a first determination unit 15 that determines whether the posture of the occupant is normal by comparing the amount of change calculated by the change amount calculation unit 14 with a normal posture determination condition, a second determination unit 16 that, when the first determination unit 15 determines that the posture of the occupant is not normal, determines whether the occupant has lost their posture based on the amount of change calculated by the change amount calculation unit 14 and a machine learning model that inputs the amount of change and outputs information indicating whether the occupant has lost their posture, and a detection unit 17 that detects an abnormal posture of the occupant when the state in which the second determination unit 16 has determined that the occupant has lost their posture continues for an abnormality detection time. Therefore, the abnormal posture detection device 1 can detect an abnormal posture of the occupant even when a scene occurs in which feature amounts cannot be stably extracted from a captured image.

[0149] In addition, in the present disclosure, any of the components of the embodiments may be modified or omitted. [Industrial Applicability]

[0150] The abnormal posture detection device of the present disclosure can detect an abnormal posture of an occupant even when a scene occurs in which feature amounts cannot be stably extracted from a captured image. [Explanation of symbols]

[0151] 1 Abnormal posture detection device, 11 Image acquisition unit, 12 Feature extraction unit, 13 Reference posture feature calculation unit, 14 Change amount calculation unit, 15 First judgment unit, 16 Second judgment unit, 17 Detection unit, 18 Memory unit, 100 Abnormal posture detection system, 2 Imaging device, 300 Driving control system, 3 Steering mechanism, 31 Steering actuator, 4 Braking / driving mechanism, 41 Braking / driving actuator, 101 Processing circuit, 102 Input interface device, 103 Output interface device, 104 Processor, 105 Memory, C3 Driving control device, C4 Map information storage device, C5 Surrounding situation monitoring device, C51 GPS receiver, C52 In-vehicle communication device, C53 Outside vehicle sensor, C54 Navigation system, C6 Vehicle state acquisition device, C61 Steering angle sensor, C62 Vehicle speed sensor, C63 Steering torque sensor, C64 Accelerator position sensor, C65 Brake position sensor, C66 override operation detection unit, C7 alarm control unit, C71 warning unit, C72 notification unit.

Claims

1. an image acquisition unit that acquires a captured image capturing an area where the face of a vehicle occupant should be present; a feature extraction unit that extracts feature amounts used to determine the posture of the occupant based on the captured image acquired by the image acquisition unit; a change amount calculation unit that calculates a change amount of the feature amount extracted by the feature amount extraction unit from the feature amount in a reference posture of the occupant; a first determination unit that determines whether the posture of the occupant is normal by comparing the feature extracted by the feature extraction unit and the change calculated by the change calculation unit with a normal posture determination condition, or by comparing the feature extracted by the feature extraction unit with the normal posture determination condition; a second determination unit that, when the first determination unit determines that the posture of the occupant is not normal, determines whether the occupant has lost their posture based on the amount of change calculated by the change amount calculation unit and a machine learning model that receives the amount of change as an input and outputs information indicating whether the occupant has lost their posture; a detection unit that detects an abnormal posture of the occupant when the state in which the second determination unit has determined that the occupant has lost their posture continues for an abnormality detection time; An abnormal posture detection device comprising:

2. a reference posture characteristic amount calculation unit that estimates the reference posture of the occupant and calculates the characteristic amount in the reference posture; The abnormal posture detection device according to claim 1, further comprising:

3. The feature amount includes a facial orientation of the occupant and a head position of the occupant.

3. The abnormal posture detection device according to claim 1 or 2.

4. The feature amount includes a facial orientation of the occupant, a head position of the occupant, and a face detection reliability.

3. The abnormal posture detection device according to claim 1 or 2.

5. an image acquisition unit acquiring a captured image capturing an area where the face of a vehicle occupant should be present; a feature extraction unit extracting feature amounts used to determine the posture of the occupant based on the captured image acquired by the image acquisition unit; a change amount calculation unit calculating a change amount of the feature amount extracted by the feature amount extraction unit from the feature amount in a reference posture of the occupant; a step in which a first determination unit determines whether the posture of the occupant is normal by comparing the feature extracted by the feature extraction unit and the change calculated by the change calculation unit with a normal posture determination condition, or by comparing the feature extracted by the feature extraction unit with the normal posture determination condition; a second determination unit, when the first determination unit determines that the posture of the occupant is not normal, determining whether the occupant has lost their posture based on the amount of change calculated by the change amount calculation unit and a machine learning model that receives the amount of change as an input and outputs information indicating whether the occupant has lost their posture; a step in which a detection unit detects an abnormal posture of the occupant when a state in which the second determination unit has determined that the occupant has lost their posture continues for an abnormality detection time; An abnormal posture detection method comprising:

6. A vehicle control system comprising the abnormal attitude detection device according to claim 1 and a driving control device that controls driving of the vehicle, the abnormal posture detection device outputs a detection result of whether the occupant is in the abnormal posture to the operation control device; The driving control device outputs a control signal for issuing a warning to the occupant or for performing evacuation processing of the vehicle to an alarm control device, a steering mechanism, or a braking / driving mechanism mounted on the vehicle based on the detection result. A vehicle control system comprising:

Images