Abnormal posture detection device, abnormal posture detection method, and vehicle control system

Abstract

The present invention comprises: an image acquisition unit (11) that acquires a photographic image capturing a range in which the face of a vehicle passenger should exist; a feature amount extraction unit (12) that extracts a feature amount on the basis of the photographic image; a change amount calculation unit (14) that calculates a change amount of the feature amount from a feature amount of a reference posture of the passenger; a first determination unit (15) that determines whether the posture of the passenger is a normal posture, by comparing the feature amount and the change amount to a normal posture determination condition, comparing the feature amount to a normal posture determination condition, or comparing the change amount to a normal posture determination condition; a second determination unit (16) that, on the basis of the change amount and a machine learning model for which the change amount is an input and information indicating the presence / absence of a collapse of the posture of the passenger is an output, determines whether the posture of the passenger has collapsed, when the first determination unit (15) has determined that the posture of the passenger is not a normal posture; and a detection unit (17) that detects abnormal posture of a passenger when a state in which the second determination unit (16) has determined that the posture of the passenger has collapsed has continued for a period of time used for abnormality detection.

Description

Abnormal attitude detection device, abnormal attitude detection method, and vehicle control system

[0001] The present disclosure relates to an abnormal attitude detection device, an abnormal attitude detection method, and a vehicle control system.

[0002] Conventionally, a technology has been known that detects abnormal posture of an occupant by determining whether the occupant is misaligned based on features such as facial orientation extracted from an image of the occupant inside a vehicle and a trained model in machine learning (hereinafter referred to as a "machine learning model") (for example, Patent Document 1).

[0003] International Publication No. 2017 / 208529

[0004] By using a machine learning model to determine whether an occupant's posture is out of alignment, it is possible to detect an abnormal occupant's posture with high accuracy. On the other hand, in a scene where feature values, such as facial orientation, cannot be stably extracted from a captured image, the presence or absence of an occupant's posture is determined based on low-accuracy feature values ​​and a machine learning model. In this case, when determining whether an occupant's posture is out of alignment using a machine learning model, the occupant may be determined to be in an abnormal posture state even when the occupant is not in an abnormal posture state. In conventional technologies, in a scene where feature values, such as facial orientation, cannot be stably extracted from a captured image, the presence or absence of an occupant's posture is determined based on low-accuracy feature values ​​and a machine learning model, which may result in a false detection of an abnormal occupant's posture. Note that the technology disclosed in Patent Document 1 fails to take into account the possibility of falsely determining whether an occupant's posture is out of alignment when a machine learning model is used to determine whether an occupant's posture is out of alignment in a scene where feature values ​​cannot be stably extracted from a captured image, and therefore remains unable to solve the above problem.

[0005] The present disclosure has been made to solve such problems, and aims to provide an abnormal posture detection device that can detect an abnormal posture of an occupant even when a scene occurs in which features cannot be stably extracted from an captured image.

[0006] The abnormal posture detection device according to the present disclosure includes an image acquisition unit that acquires an image of an area where the face of a vehicle occupant should be present, a feature extraction unit that extracts feature amounts used to determine the posture of the occupant based on the image acquired by the image acquisition unit, a change amount calculation unit that calculates a change amount of the feature amount extracted by the feature amount extraction unit from a feature amount in a reference posture of the occupant, and a normal posture determination condition by comparing the feature amount extracted by the feature amount extraction unit with the normal posture determination condition or by comparing the feature amount extracted by the feature amount extraction unit with the normal posture determination condition. The vehicle is equipped with a first determination unit that determines whether the occupant's posture is normal by comparing the amount of change calculated by the amount calculation unit with conditions for determining normal posture; a second determination unit that, when the first determination unit determines that the occupant's posture is not normal, determines whether the occupant has lost their posture based on the amount of change calculated by the amount of change calculation unit and a machine learning model that inputs the amount of change and outputs information indicating whether the occupant's posture has lost its posture; and a detection unit that detects an abnormal posture of the occupant when the state in which the second determination unit has determined that the occupant has lost their posture continues for an abnormality detection time.

[0007] According to the abnormal posture detection device according to the present disclosure, even if a scene occurs in which feature amounts cannot be stably extracted from a captured image, an abnormal posture of an occupant can be detected.

[0008] 10A is a diagram illustrating an example of the configuration of an abnormal posture detection device according to embodiment 1. FIG. 10B is a diagram illustrating the significance of the first determination unit determining whether the driver's posture is normal or not based on feature amounts based on a current captured image and a relative change in the feature amounts in embodiment 1. FIG. 10C is a diagram illustrating a plurality of posture imbalance types. FIG. 10D is a diagram illustrating an example of the configuration of a driving control system according to embodiment 1. FIG. 10E is a flowchart illustrating the operation of a reference posture feature amount calculation process performed by the abnormal posture detection device according to embodiment 1. FIG. 10F is a flowchart illustrating the operation of a change amount calculation process, a first posture determination process, a second posture determination process, and an abnormal posture detection process performed by the abnormal posture detection device according to embodiment 1. FIG. 10G is a flowchart illustrating the detailed operation of the first posture determination process performed by the first determination unit in step ST14 of FIG. 6. FIG. 10H is a flowchart illustrating the detailed operation of the second posture determination process performed by the second determination unit in step ST15 of FIG. 6. FIG. 10H is a flowchart illustrating the detailed operation of the abnormal posture detection process performed by the detection unit in step ST16 of FIG. 6. 12A and 12B are diagrams illustrating an example of a hardware configuration of an abnormal attitude detection device according to embodiment 1.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1. An abnormal posture detection device according to embodiment 1 detects whether the posture of a vehicle occupant is abnormal based on a captured image in which at least the face of the vehicle occupant is captured. In the following embodiment 1, as an example, the vehicle occupant whose posture is to be detected by the abnormal posture detection device as being abnormal is the vehicle driver.

[0010] 1 is a diagram showing an example of the configuration of an abnormal attitude detection device 1 according to embodiment 1. The abnormal attitude detection device 1 according to embodiment 1 is assumed to be mounted on a vehicle. The abnormal attitude detection device 1 is connected to an imaging device 2, and the abnormal attitude detection device 1 and the imaging device 2 together constitute an abnormal attitude detection system 100. The abnormal attitude detection system 100, together with a driving control system 300, a steering mechanism 3, and a braking / driving mechanism 4, constitutes a vehicle control system SYS.A.

[0011] The imaging device 2 is mounted on a vehicle and is installed so as to be able to capture an image of at least the area where the driver's face should be present. In the first embodiment, it is assumed that the imaging device 2 is installed, for example, near the center of the instrument panel in the vehicle width direction or on the center console. For example, the imaging device 2 may be shared with a so-called DMS (Driver Monitoring System) installed for the purpose of monitoring the interior of the vehicle. The imaging device 2 is a visible light camera or an infrared camera. The imaging device 2 outputs the captured image to the abnormal posture detection device 1. The abnormal posture detection device 1 detects whether the driver's posture is abnormal based on the captured image captured by the imaging device 2. Details of the abnormal posture detection device 1 will be described later.

[0012] The abnormal posture detection device 1 outputs the detection result of whether the driver's posture is abnormal to a driving control system 300 connected to the abnormal posture detection system 100. As shown in Fig. 1, the abnormal posture detection system 100 is connected to the driving control system 300. The driving control system 300 works in conjunction with the steering mechanism 3 or the braking / driving mechanism 4, and provides driving assistance using the detection result of the abnormal posture detection system 100. Details of the driving control system 300 will be described later.

[0013] As shown in FIG. 1, the abnormal posture detection device 1 includes an image acquisition unit 11, a feature extraction unit 12, a reference posture feature calculation unit 13, a change amount calculation unit 14, a first determination unit 15, a second determination unit 16, a detection unit 17, and a memory unit 18.

[0014] The image acquisition unit 11 acquires a captured image from the imaging device 2. The image acquisition unit 11 outputs the acquired captured image to the feature extraction unit 12.

[0015] The feature extraction unit 12 extracts feature amounts used to determine the driver's posture based on the captured image acquired by the image acquisition unit 11. In the first embodiment, the feature amounts used to determine the driver's posture are the driver's facial orientation, the driver's head position, and the face detection reliability.

[0016] The feature extraction unit 12 first detects the driver's face in the captured image acquired by the image acquisition unit 11. Specifically, the feature extraction unit 12 detects facial feature points of the driver, which indicate facial features of the driver, using a known image recognition technique, such as edge detection, on the captured image. The facial feature points include the corners and corners of the eyes, the nose, the mouth, the eyebrows, and the chin. The facial feature points of the driver are represented, for example, by coordinates on the captured image. The feature extraction unit 12 may also detect the driver's facial region. The driver's facial region is, for example, the smallest rectangle that encloses the outline of the driver's face. The driver's facial region is represented, for example, by the coordinates of the four corners of the smallest rectangle on the captured image. Because the installation position and the angle of view of the imaging device 2 are known in advance, the feature extraction unit 12 can determine which region of the captured image is the driver's face, even if multiple occupants are captured in the captured image. For example, an area in the captured image where the driver's face may be present (hereinafter referred to as the "driver detection area") is set in advance, and the feature extraction unit 12 detects the feature points and face area of ​​the driver's face in the driver detection area using known image recognition technology.

[0017] The feature extraction unit 12 detects the driver's facial orientation and head position based on the detected information about the driver's face (hereinafter referred to as "face information") and extracts the detected information as features. The face information is, for example, a captured image to which information that can identify the driver's facial feature points and facial area is added.

[0018] In the first embodiment, the facial orientation and head position of the driver detected by the feature extraction unit 12 are, for example, the facial orientation and head position in real space. The feature extraction unit 12 detects the facial orientation of the driver in real space based on facial information, specifically, based on the driver's face in the captured image. The feature extraction unit 12 may detect the facial orientation of the driver using, for example, a known facial orientation detection technique for detecting the facial orientation from the captured image. The facial orientation of the driver is expressed, for example, by an angle (yaw angle, pitch angle, or roll angle) relative to the optical axis of the image capture device 2 as a reference axis. Note that this is merely an example, and the facial orientation of the driver may also be expressed, for example, by an angle (yaw angle, pitch angle, or roll angle) relative to the reference axis, with a predetermined reference axis in the longitudinal direction of the vehicle as the reference axis.

[0019] Furthermore, the feature extraction unit 12 detects the driver's head position in real space based on the facial information, specifically, based on the driver's face in the captured image. In the first embodiment, the driver's head position in the captured image is indicated, for example, by the center of the driver's eyebrows. The feature extraction unit 12 detects, for example, a point in real space corresponding to the center of the driver's eyebrows in the captured image as the driver's head position. Note that this is merely an example, and the driver's head position in the captured image may be indicated, for example, by the center of the driver's facial area or the center of a line connecting the inner corners of the driver's eyes. In this case, the feature extraction unit 12 detects, for example, a point in real space corresponding to the center of the driver's facial area in the captured image or the center of a line connecting the inner corners of the driver's eyes as the driver's head position in real space. The feature extraction unit 12 may detect the driver's head position using, for example, a known coordinate transformation technique that converts points in the captured image into points in real space. The driver's head position is represented, for example, by coordinates in real space.

[0020] The feature extraction unit 12 also calculates face detection reliability based on the face information and uses this as the extracted feature. In the first embodiment, the face detection reliability indicates how reliable a face detected from a captured image is, i.e., how plausible the face is, or more specifically, how many facial features were successfully detected from the captured image. The feature extraction unit 12 calculates the face detection reliability in accordance with predetermined conditions. For example, the feature extraction unit 12 calculates, as the face detection reliability, the proportion of facial features detected based on the captured image acquired by the image acquisition unit 11 out of all facial features defined as facial features, such as the outer corners of the eyes, the inner corners of the eyes, the nose, the mouth, the eyebrows, or the chin.

[0021] The feature extraction unit 12 outputs information on the driver's facial orientation, head position, and face detection reliability extracted as features (hereinafter referred to as "feature information") to the reference posture feature calculation unit 13 and the change amount calculation unit 14, and also stores the information in chronological order in the storage unit 18. The feature information is information in which information on the driver's facial orientation (yaw angle, pitch angle, and roll angle) relative to a reference axis, coordinate information (X coordinate, Y coordinate, Z coordinate) of the driver's head position, information indicating face detection reliability, and a captured image are associated with each other.

[0022] The reference posture feature amount calculation unit 13 estimates the reference posture of the driver and calculates feature amounts in the reference posture of the driver (hereinafter referred to as "reference posture feature amounts"). In the first embodiment, the reference posture refers to the posture of the driver in a state in which the driver is considered to be facing forward with respect to the traveling direction of the vehicle. Note that in the first embodiment, the driver facing forward with respect to the traveling direction of the vehicle is assumed to mean that the driver's face is facing forward with respect to the traveling direction of the vehicle. In the first embodiment, "forward" is not limited to strictly facing forward, but also includes approximately facing forward. The process of estimating the reference posture of the driver and calculating feature amounts in the reference posture performed by the reference posture feature amount calculation unit 13 in the abnormal posture detection device 1 according to the first embodiment is referred to as "reference posture feature amount calculation process."

[0023] The reference posture feature amount calculation unit 13 determines whether the driver is considered to be facing forward with respect to the traveling direction of the vehicle based on, for example, whether the vehicle speed, the steering wheel angle, and the shift position satisfy preset conditions (hereinafter referred to as "vehicle conditions"), and estimates the reference posture of the driver. The vehicle conditions may be, for example, "the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km / h), the steering wheel angle is within a predetermined angle range (e.g., ±20 degrees), and the shift position is "D." The vehicle conditions include the vehicle speed, the steering wheel angle, and the shift position assumed in a situation where the driver is assumed to be facing forward with respect to the traveling direction of the vehicle.

[0024] The reference posture feature value calculation unit 13 determines that the driver is facing forward with respect to the vehicle's traveling direction when the vehicle speed, steering wheel angle, and shift position satisfy the vehicle conditions. That is, the reference posture feature value calculation unit 13 estimates that the driver's posture is a reference posture when the vehicle speed, steering wheel angle, and shift position satisfy the vehicle conditions. The reference posture feature value calculation unit 13 may acquire vehicle speed information from a vehicle speed sensor C62 (see FIG. 4 , described later) mounted on the vehicle. The reference posture feature value calculation unit 13 may also acquire steering wheel angle information from a steering torque sensor C63 (see FIG. 4 , described later) mounted on the vehicle. The reference posture feature value calculation unit 13 may also acquire shift position information from a shift position sensor (not shown) mounted on the vehicle.

[0025] The above-described method of determining whether the reference posture feature amount calculation unit 13 determines whether the driver is facing forward with respect to the vehicle's traveling direction is merely an example. The reference posture feature amount calculation unit 13 may determine whether the driver is facing forward with respect to the vehicle's traveling direction using other methods. For example, the reference posture feature amount calculation unit 13 may determine whether the driver is facing forward with respect to the vehicle's traveling direction based on the feature amount information output from the feature amount extraction unit 12. For example, if the reference axis of the driver's facial direction and head position is the optical axis of the image capture device 2, the installation position and angle of view of the image capture device 2 in the driver's vehicle are known in advance, and therefore the reference posture feature amount calculation unit 13 can determine the ranges of the driver's facial direction and head position within which the driver is considered to be facing forward with respect to the vehicle's traveling direction. Note that the ranges of the driver's facial direction and head position detected by the feature amount extraction unit 12 when the driver is facing forward with respect to the vehicle's traveling direction are pre-set.

[0026] When the reference posture feature calculation unit 13 estimates that the driver's posture is the reference posture, it stores the feature information output from the feature extraction unit 12 in the memory unit 18 or in a memory area inside the reference posture feature calculation unit 13 as candidate feature information that is a candidate for calculating the reference posture feature.

[0027] When a preset number of pieces of candidate feature information are stored, the reference posture feature calculation unit 13 calculates the driver's reference posture feature, specifically, the face direction (yaw angle, pitch angle, and roll angle), head position (X coordinate, Y coordinate, Z coordinate), and face detection reliability, from the stored preset number of pieces of candidate feature information.

[0028] For example, the reference posture feature calculation unit 13 calculates the most frequent value of the driver's facial orientation, the driver's head position, and the face detection reliability included in a predetermined number of pieces of candidate feature information as the driver's reference posture feature. For example, the reference posture feature calculation unit 13 may calculate the average value of the driver's facial orientation, the driver's head position, and the face detection reliability included in a predetermined number of pieces of candidate feature information as the driver's reference posture feature.

[0029] The reference posture feature quantity calculation unit 13 performs a reference posture feature quantity calculation process and calculates the driver's reference posture feature quantity. After calculating the driver's reference posture feature quantity, the reference posture feature quantity calculation unit 13 stores information indicating the calculated reference posture feature quantity of the driver (hereinafter referred to as "reference posture feature quantity information") in the storage unit 18. The reference posture feature quantity calculation unit 13 then sets to "1" a reference posture feature quantity calculation completion flag, which is provided in a location that can be referenced by the abnormal posture detection device 1 and indicates that the reference posture feature quantity has been calculated. The initial value of the reference posture feature quantity calculation completion flag is "0." The reference posture feature quantity calculation completion flag is initialized when the vehicle is powered on, when the detection unit 17 detects an abnormal posture of the driver, or the like. At this time, the reference posture feature quantity calculation unit 13 may delete the candidate feature quantity information stored in the storage unit 18 or in a storage area inside the reference posture feature quantity calculation unit 13.

[0030] The change amount calculation unit 14 calculates the amount of change of the feature amount extracted by the feature amount extraction unit 12 from the reference posture feature amount. In the abnormal posture detection device 1 according to the first embodiment, the process of calculating the amount of change of the feature amount extracted by the feature amount extraction unit 12 from the reference posture feature amount, which is performed by the change amount calculation unit 14, is referred to as the "change amount calculation process." The "change amount calculation process," the "first posture determination process" by the first determination unit 15 (described later), the "second posture determination process" by the second determination unit 16 (described later), and the "abnormal posture detection process" by the detection unit 17 (described later) are performed after the "reference posture feature amount calculation process" by the reference posture feature amount calculation unit 13 is completed. Details of the "first posture determination process," "second posture determination process," and "abnormal posture detection process" will be described later.

[0031] The change amount calculation unit 14 compares the feature amounts extracted by the feature amount extraction unit 12, specifically the driver's facial direction, head position, and face detection reliability, with the feature amounts included in the reference posture feature amount information stored in the storage unit 18, specifically the driver's facial direction, head position, and face detection reliability, and calculates the differences as change amounts. That is, the change amount calculation unit 14 calculates the amount of change in the driver's facial direction extracted by the feature amount extraction unit 12 from the facial direction in the driver's reference posture, the amount of change in the driver's head position extracted by the feature amount extraction unit 12 from the head position in the driver's reference posture, and the amount of change in the face detection reliability extracted by the feature amount extraction unit 12 from the face detection reliability in the driver's reference posture. The change amount calculation unit 14 calculates the amount of change for each of the yaw angle, pitch angle, and roll angle of the driver's facial direction. The change amount calculation unit 14 also calculates the amount of change for each of the X coordinate, Y coordinate, and Z coordinate of the driver's head position.

[0032] The change amount calculation unit 14 outputs information indicating the calculated change amount (hereinafter referred to as "change amount information") to the first determination unit 15 together with the feature amount information output from the feature amount extraction unit 12. The change amount information includes information on the change amount of the driver's facial orientation (the change amount of the yaw angle, the change amount of the pitch angle, and the change amount of the roll angle), information on the change amount of the driver's head position (the change amount of the X coordinate, the change amount of the Y coordinate, and the change amount of the Z coordinate), and information indicating the change amount of the face detection reliability.

[0033] The first determination unit 15 determines whether or not the driver's posture is normal by comparing the feature extracted by the feature extraction unit 12 and the change calculated by the change amount calculation unit 14 with the normal posture determination conditions, by comparing the feature extracted by the feature extraction unit 12 with the normal posture determination conditions, or by comparing the change calculated by the change amount calculation unit 14 with the normal posture determination conditions. The process of determining whether or not the driver's posture is normal, which is performed by the first determination unit 15 in the abnormal posture detection device 1 according to the first embodiment, is referred to as the "first posture determination process."

[0034] The normal posture determination conditions are conditions for determining whether the driver's posture is normal, and are set in advance by an administrator or the like and stored in a location that can be referenced by the first determination unit 15. For example, the normal posture determination conditions include the following conditions:

[0035] "Of the following formulas (1-1), (1-2), (2-1), (2-2), (3-1), (3-2), (4-1), (4-2), (5-1), (5-2), (6-1), and (6-2), formula (1-1) or (1-2), formula (2-1) or (2-2), formula (3-1) or (3-2), formula (4-1) or (4-2), formula (5-1) or (5-2), and formula (6-1) or (6-2) must all be satisfied.・First yaw angle threshold < amount of change in face direction yaw angle from face direction yaw angle in reference posture < second yaw angle threshold ... (1-1) ・Third yaw angle threshold < face direction yaw angle < fourth yaw angle threshold ... (1-2) ・First pitch angle threshold < amount of change in face direction pitch angle from face direction pitch angle in reference posture < second pitch angle threshold ... (2-1) ・Third pitch angle threshold < face direction pitch angle < fourth pitch angle threshold ... (2-2) ・First roll angle threshold < amount of change in face direction roll angle from face direction roll angle in reference posture < second roll angle threshold ... (3-1) ・Third roll angle threshold < face direction roll angle < fourth roll angle threshold ... (3-2) ・First X coordinate threshold < amount of change in head position X coordinate from head position X coordinate in reference posture < second X coordinate threshold ... (4-1) ・Third X coordinate threshold < head position X coordinate < Fourth X coordinate threshold (4-2) First Y coordinate threshold < amount of change in Y coordinate of head position from the Y coordinate of the head position in the reference posture < second Y coordinate threshold (5-1) Third Y coordinate threshold < Y coordinate of head position < fourth Y coordinate threshold (5-2) First coordinate Z threshold < amount of change in Z coordinate of head position from the Z coordinate of the head position in the reference posture < second Z coordinate threshold (6-1) Third coordinate Z threshold < Z coordinate of head position < fourth Z coordinate threshold (6-2) Note that, first yaw angle threshold < second yaw angle threshold, third yaw angle threshold < fourth yaw angle threshold, first pitch angle threshold < second pitch angle threshold, third pitch angle threshold < fourth pitch angle threshold, first roll angle threshold < second roll angle threshold, third roll angle threshold < fourth roll angle threshold, first X coordinate threshold < second X coordinate threshold, third X coordinate threshold < fourth X coordinate threshold, First Y coordinate threshold < second Y coordinate threshold, third Y coordinate threshold < fourth Y coordinate threshold, first Z coordinate threshold < second Z coordinate threshold, third Z coordinate threshold < fourth Z coordinate threshold.

[0036] The first determination unit 15 determines that the driver's posture is normal if the feature extracted by the feature extraction unit 12 or the change calculated by the change calculation unit 14 satisfies the normal posture determination condition. The first determination unit 15 determines that the driver's posture is not normal if the feature extracted by the feature extraction unit 12 or the change calculated by the change calculation unit 14 does not satisfy the normal posture determination condition.

[0037] Here, the significance of the first judgment unit 15 judging whether the driver's posture is normal or not based on the feature extracted by the feature extraction unit 12, in other words, the feature based on the current captured image, and the change calculated by the change calculation unit 14, in other words, the relative change in the feature from the reference posture feature will be explained.

[0038] 2 is a diagram for explaining the significance of the first determination unit 15 determining whether or not the driver's posture is normal based on a feature amount based on a current captured image and a relative change in the feature amount in Embodiment 1. Note that Fig. 2 is a simplified diagram showing an example of the concept of the first determination unit 15 determining whether or not the driver's posture is normal.

[0039] As described above, in the first embodiment, the reference posture is a posture in which the driver is considered to be facing forward with respect to the traveling direction of the vehicle. For example, as shown on the left side of FIG. 2 , a posture in which the driver is driving with his / her elbows on the ground and the roll angle of the face is 40 degrees can also be estimated as the reference posture. Now, for example, as shown on the right side of FIG. 2 , assume that the driver stops driving with his / her elbows on the ground and starts driving with a straight posture, i.e., the roll angle of the face becomes 0 degrees. In this case, if the first determination unit 15 determines whether the driver's posture is normal based only on the change in the feature amount from the reference posture feature amount, the first determination unit 15 may determine that the driver's posture in the state shown on the right side of FIG. 2 is not normal. The first determination unit 15 determines whether the driver's posture is normal or not by taking into consideration not only the change in the feature from the reference posture feature but also the feature based on the current captured image. Therefore, even if the reference posture is estimated to be a posture such as that shown on the left side of Figure 2, the first determination unit 15 can determine that the driver's posture, which should be considered a normal posture as shown on the right side of Figure 2, is a normal posture.

[0040] In the explanation of the example using Figure 2, the driver's facial direction was used as an example, but the same can be said for the driver's head position. By taking into consideration not only the relative change in the feature from the reference posture feature but also the feature based on the current captured image, it is possible to determine whether the driver's posture is normal, and a driver's posture that should be considered normal can be determined to be normal.

[0041] The first determination unit 15 outputs a determination result as to whether or not the driver's posture is determined to be normal (hereinafter referred to as the "normal posture determination result") to the second determination unit 16. At this time, the first determination unit 15 outputs the change amount information output from the change amount calculation unit 14 to the second determination unit 16 together with the normal posture determination result.

[0042] When the first determination unit 15 determines that the driver's posture is not normal, the second determination unit 16 determines whether the driver has poor posture based on the amount of change calculated by the change amount calculation unit 14 and the machine learning model. The machine learning model is a trained model that receives as input the amount of change in feature from a reference posture feature and outputs information indicating whether the driver has poor posture. The amount of change is, in detail, the amount of change in the driver's facial direction from the facial direction in the reference posture, the amount of change in the driver's head position from the head position in the reference posture, and the amount of change in face detection reliability from the face detection reliability in the reference posture. In the first embodiment, the information indicating whether the driver has poor posture output by the machine learning model includes information indicating whether the driver has poor posture and, if the driver has poor posture, information indicating the type of poor posture (hereinafter referred to as the "posture type").

[0043] The machine learning model is generated in advance and stored in a location that can be referenced by the second determination unit 16, such as the storage unit 18. For example, an administrator or the like generates a machine learning model by having multiple drivers test drive the vehicle, and stores the model in the storage unit 18 or the like.

[0044] Note that a plurality of posture deviation types have been defined in advance as types of posture deviation that may occur when a driver is experiencing poor posture. Figure 3 (partially modified from "Basic Design Document for Driver Abnormality Automatic Detection System" by the Advanced Safety Vehicle Promotion Study Group, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, March 2018) is a diagram for explaining the plurality of posture deviation types. As shown in Figure 3, there are a plurality of posture deviation types. 3 shows "head down" in which the driver leans forward and maintains a posture with their face close to the steering wheel, "head down" in which the driver maintains a posture with their face facing down, "backward lean" in which the driver's upper body leans backward and their face remains facing up, "backward lean" in which the driver's upper body leans up and their face remains facing up, "head-down" in which the driver's face remains tilted to the left or right, "head-down" in which the driver's upper body leans to the left or right and their face remains tilted in the same direction, and "leaning to the side" in which the driver's upper body remains tilted to the left or right. In the present disclosure, "head down," "head down," "backward lean," "head-down lean," "head-down lean," "head-down lean," "side-down lean," and "leaning to the side" shown in FIG. 3 are defined as a plurality of preset posture disorders. Information regarding a plurality of posture deviation types is stored in a location that can be referenced by the second determination unit 16 .

[0045] The second determination unit 16 inputs the amount of change into a machine learning model to obtain information indicating whether or not the driver has poor posture, thereby determining whether or not the driver has poor posture and, if so, the type of poor posture.

[0046] If the first determination unit 15 determines that the driver's posture is normal, the second determination unit 16 does not determine whether the driver has poor posture using the machine learning model. That is, the second determination unit 16 determines that the driver has no poor posture, in other words, that the driver is not experiencing poor posture.

[0047] The second determination unit 16 outputs the determination result of whether or not the driver is experiencing poor posture (hereinafter referred to as the "poor posture determination result") to the detection unit 17. The poor posture determination result is information that associates information indicating whether or not the driver is experiencing poor posture with information indicating the type of poor posture if the driver is experiencing poor posture.

[0048] The process of determining whether the driver is experiencing poor posture, performed by the second determination unit 16 in the abnormal posture detection device 1 of embodiment 1 as described above, is referred to as the "second posture determination process."

[0049] The detection unit 17 detects whether the driver's posture is abnormal or not based on the posture collapse determination result output from the second determination unit 16. In the abnormal posture detection device 1 according to the first embodiment, the process performed by the detection unit 17 to detect whether the driver's posture is abnormal or not is referred to as "abnormal posture detection process."

[0050] The detection unit 17 detects that the driver's posture is abnormal when the state in which the second determination unit 16 has determined that the driver has poor posture continues for a preset time (hereinafter referred to as the "abnormality detection time"). The detection unit 17 detects that the driver's posture is not abnormal when the second determination unit 16 has determined that the driver has not had poor posture, or when the state in which the second determination unit 16 has determined that the driver has had poor posture does not continue for the abnormality detection time.

[0051] In detail, the detection unit 17 first determines whether or not the second determination unit 16 has determined that the driver has poor posture. From the posture deviation determination result output from the second determination unit 16, the detection unit 17 can determine whether or not the second determination unit 16 has determined that the driver has poor posture.

[0052] If the second judgment unit 16 judges that the driver has no posture imbalance, that is, if a posture imbalance judgment result indicating that the driver has not experienced posture imbalance is output, the detection unit 17 detects that the driver's posture is not abnormal.

[0053] If the second determination unit 16 determines that the driver has poor posture, that is, if the second determination unit 16 outputs a posture error determination result indicating that the driver is experiencing poor posture, the detection unit 17 counts up a counter for counting the abnormality detection time (hereinafter referred to as the "abnormal posture detection counter"). On the other hand, if the second determination unit 16 does not determine that the driver has poor posture, that is, if the second determination unit 16 outputs a posture error determination result indicating that the driver is not experiencing poor posture, the detection unit 17 resets the abnormal posture detection counter.

[0054] Next, the second determination unit 16 determines whether the abnormal posture detection counter has reached a preset threshold (hereinafter referred to as the "abnormality detection threshold"). If the abnormal posture detection counter has reached the abnormality detection threshold, the detection unit 17 determines that the state in which the second determination unit 16 determined that the driver is experiencing poor posture has continued for the abnormality detection time, and detects that the driver's posture is abnormal. The detection unit 17 outputs information indicating that the driver's posture is abnormal (hereinafter referred to as "abnormal posture detection information") to the driving control system 300. The abnormal posture detection information includes information indicating that the driver's posture is abnormal and information indicating the posture deviation type. The detection unit 17 can identify the posture deviation type based on the posture deviation determination result output from the second determination unit 16.

[0055] The detection unit 17 may output the abnormal posture detection information to an output device (not shown) as information for analysis in case some unexpected situation occurs.

[0056] If the abnormal posture detection counter has not reached the abnormality detection threshold, the detection unit 17 determines that the state in which the second determination unit 16 determined that the driver is experiencing poor posture has not continued for the abnormality detection time, and does not detect that the driver's posture is abnormal. In other words, the detection unit 17 detects that the driver's posture is not abnormal. The detection unit 17 may output information indicating that the driver's abnormal posture was not detected as abnormal posture detection information to the driving control system 300 or to an output device.

[0057] The storage unit 18 stores various information such as feature information, reference posture feature information, and machine learning models. Note that, in Fig. 1, the storage unit 18 is provided in the abnormal posture detection device 1, but this is merely an example. The storage unit 18 may be provided in a location outside the abnormal posture detection device 1 that can be referenced by the abnormal posture detection device 1.

[0058] FIG. 4 is a diagram showing an example of the configuration of a driving control system 300 according to the first embodiment. In the following description, a vehicle equipped with the abnormal attitude detection system 100 is referred to as a first vehicle, and a vehicle other than the first vehicle is referred to as a second vehicle. Note that there are cases where the first vehicle and the second vehicle are not distinguished from each other and are each referred to as a vehicle. In the following, an example in which the abnormal attitude detection system 100 is used as part of the driving control system 300 will be described, but the abnormal attitude detection system 100 may be configured to be able to communicate with the driving control system 300 wirelessly or via a wire, as shown in FIG. 1.

[0059] The driving control system 300 includes a driving control device C3 that controls the driving of the first vehicle by outputting a control signal based on abnormal attitude detection information from the abnormal attitude detection system 100 and controlling the steering mechanism 3 and braking / driving mechanism 4 of the first vehicle. The driving control system 300 also includes a map information storage device C4 that stores map information used for automatic driving of the vehicle, a surrounding condition monitoring device C5 that monitors the surrounding conditions of the vehicle, a vehicle condition acquisition device C6 that acquires information indicating the condition of the first vehicle, and an alarm control device C7 that controls warnings or notifications to the driver.

[0060] In addition, the abnormal posture detection system 100, driving control device C3, map information storage device C4, surrounding conditions monitoring device C5, vehicle status acquisition device C6, and alarm control device C7 provided in the driving control system 300 are each connected to a communication bus C8, and data can be sent and received via the communication bus C8.

[0061] The steering mechanism 3 is a mechanism provided on the first vehicle for determining the traveling direction of the first vehicle, and includes, for example, a steering column, a steering shaft, a rack, a pinion, and a steering actuator 31. The braking / driving mechanism 4 is a mechanism for controlling the traveling speed of the first vehicle and for switching between forward and reverse driving, and includes, for example, an accelerator, a brake, a shift, and a braking / driving actuator 41. The steering actuator 31 that controls the steering mechanism 3 is composed of, for example, an EPS (Electric Power Steering) motor, and the braking / driving actuator 41 that controls the braking / driving mechanism 4 is composed of, for example, an electronically controlled throttle, a brake actuator, etc.

[0062] The map information storage device C4 is a storage medium that stores map information, including road connectivity, the number of lanes, intersection location information, and railroad crossing location information. Here, the map information storage device C4 is assumed to be mounted on the first vehicle. However, the map information storage device C4 may be configured, for example, as a server that transmits map data to the driving control device C3 via communications. When the map information storage device C4 is configured as the server, for example, a storage medium may be provided in the driving control system 300, and map information acquired from the map information storage device C4 via the on-board communication device C52 (described later) may be stored in the storage medium.

[0063] The surroundings monitoring device C5 monitors the surroundings of the first vehicle, and includes a GPS (Global Positioning System) receiver C51, an in-vehicle communication device C52, an outside-vehicle sensor C53, and a navigation system C54.

[0064] The GPS receiver C51 receives signals transmitted from GPS positioning satellites and detects the current position of the first vehicle.

[0065] The outside vehicle sensor C53 is composed of, for example, at least one of a camera that captures images outside the vehicle, a millimeter-wave radar, a LiDAR, and an ultrasonic sensor, and detects the position of a second vehicle, pedestrian, obstacle, etc. that is present in the vicinity of the first vehicle or the distance from the first vehicle.

[0066] The navigation system C54 calculates a route from the current position of the first vehicle to a destination and provides guidance along the calculated route. The navigation system C54 has a display unit, for example, a liquid crystal display, and is housed in an instrument panel. The navigation system C54 also has an operation unit, such as a touch panel or physical buttons, and is configured to be able to accept operations by the occupant. The operation unit of the navigation system C54 may also be configured with a microphone or the like to be able to accept operations via voice uttered by the occupant.

[0067] The on-board communication device C52 will be described. The on-board communication device C52 is, for example, a wireless communication device connected to an antenna for wireless communication. The on-board communication device C52 acquires information on the location of the second vehicle or pedestrians, traffic information, etc. by communicating with a communication device of a second vehicle or a communication device installed on the road. Here, traffic information includes, for example, congestion information, traffic regulation information, construction section information, etc.

[0068] The on-board communication device C52 can also perform inter-vehicle communication via wireless communication with the on-board communication device C52 of a second vehicle located around the first vehicle. Additionally, the on-board communication device C52 may perform mobile communication with a base station outside the first vehicle. Furthermore, the on-board communication device C52 is configured to be able to transmit information about the first vehicle output onto the communication bus C8 to the second vehicle and a call center, etc. The on-board communication device C52 can also output information received from the second vehicle and information received from the call center, etc., to the communication bus C8. The on-board communication device C52 may also be configured to acquire information from a server, etc., located outside the vehicle, such as a map information storage device C4 configured as a server that transmits map data to the driving control device C3.

[0069] The vehicle state acquisition device C6 acquires information indicating the state of the first vehicle, and includes a steering angle sensor C61, a vehicle speed sensor C62, a steering torque sensor C63, an accelerator position sensor C64, and a brake position sensor C65.

[0070] The steering angle sensor C61 is provided, for example, on an EPS motor or a steering wheel, and detects the steering angle of the first vehicle. The vehicle speed sensor C62 is provided, for example, on a wheel, and detects the traveling speed of the first vehicle.

[0071] The steering torque sensor C63 is provided on the steering wheel, for example, and detects the magnitude of the steering wheel operation force applied by the driver. The accelerator position sensor C64 detects the amount of accelerator pedal depression by the driver. The brake position sensor C65 detects the amount of brake pedal depression by the driver.

[0072] The warning control device C7 acquires abnormality detection information from the abnormal posture detection system 100. When the warning control device C7 acquires abnormality detection information indicating that the driver's posture is abnormal, it outputs a warning to the driver via the warning unit C71. The warning unit C71 is, for example, an audio device mounted on the first vehicle, and the warning is a voice or notification sound output from the audio device. Alternatively, the warning unit C71 may be configured as a display unit of the navigation system C54, and the warning may be a warning message displayed on the display unit of the navigation system C54. As described above, the warning may be any warning that can be recognized by the driver of the first vehicle. Note that the driving control device C3 may output a control signal to cause the warning control device C7 to issue a warning.

[0073] Furthermore, the alarm control device C7 stops the alarm if the driver responds to the alarm after issuing the alarm. Here, the case where the driver responds to the warning means a case where the driver is recognized to have made a conscious action in response to the warning, such as when the abnormal posture detection system 100 outputs information indicating that the driver's posture is not abnormal, or when an operation unit provided in the navigation system C54 or a switch provided on the steering wheel receives an operation by the driver. The alarm control device C7 may also output a signal indicating that the driver has responded to the warning to the abnormal posture detection system 100.

[0074] Furthermore, when abnormal posture detection information indicating that the driver's posture is abnormal is output from the abnormal posture detection system 100, the alarm control device C7 activates an alarm unit C72 such as a turn signal, a hazard lamp, and a headlight that are provided on the exterior of the vehicle so as to be visible from the second vehicle, to notify people outside the first vehicle, such as occupants of the second vehicle, that the driver of the first vehicle is in an abnormal posture. For example, when the driver does not respond to the alarm for a predetermined period after the alarm is issued, the alarm control device C7 activates the alarm unit C72 to notify people outside the first vehicle that the driver of the first vehicle is in an abnormal posture.

[0075] In addition, the warning control device C7 may use the above-mentioned in-vehicle communication device C52 to communicate with the vehicle to notify the occupants of the second vehicle that the posture of the driver of the first vehicle is abnormal, for example by displaying an alert message on the display unit of the navigation system C54 of the second vehicle.

[0076] Furthermore, when driving assistance is performed by the driving control device C3, the notification control unit may use the vehicle state acquired by the vehicle state acquisition unit or the signal output from the driving control device C3 to notify people outside the first vehicle, such as occupants of the second vehicle, of the direction of travel of the first vehicle or whether or not it will change lanes, using the notification unit C72, which is composed of the above-mentioned lighting fixtures, etc., provided on the vehicle exterior, etc.

[0077] The driving control device C3 of the driving control system 300 controls the steering actuator 31 or the braking / driving actuator 41 mounted on the first vehicle to assist the driver in driving. The driving control device C3 performs automatic driving control of the first vehicle, for example, by outputting a signal to the steering actuator 31 or the braking / driving actuator 41. The driving control device C3 performs braking / driving control of the first vehicle, such as by controlling the braking / driving actuator 41, which is composed of an electronically controlled throttle and a brake actuator, to operate the brakes to decelerate or stop the first vehicle. The driving control device C3 performs steering control of the first vehicle, such as by controlling the steering actuator 31, which is composed of an EPS motor, to maintain the lane in which the first vehicle is traveling.

[0078] Furthermore, the driving control device C3 has a plurality of driving assistance functions that assist or substitute for the driver's driving operation by controlling the driving force, braking force, steering force, etc. of the first vehicle. For example, the driving assistance functions include a cruise control function and a lane departure prevention function. In the following description, the cruise control function will be referred to as ACC (Adaptive Cruise Control), and the lane departure prevention function will be referred to as LKA (Lane Keeping Assist).

[0079] When the driving control device C3 executes ACC, it controls the traveling speed of the first vehicle by adjusting the driving force and braking force based on monitoring information of the vehicle in front acquired from the surrounding conditions monitoring device C5. If no vehicle in front is detected, the ACC causes the first vehicle to travel at a constant speed at a target speed preset by the driver or the like. On the other hand, if a vehicle in front is detected, the ACC causes the first vehicle to follow the vehicle in front while maintaining a distance from the vehicle in front.

[0080] Furthermore, when the driving control device C3 executes the LKA, it controls the steering force and the steering holding force based on the shape information of the lane markings in the traveling direction acquired from the surrounding conditions monitoring device C5. The LKA applies a steering force to the steering wheel in a direction that prevents the first vehicle from approaching the lane markings, thereby causing the first vehicle to travel along the lane. Note that road information output by the surrounding conditions monitoring device C5 may be used for vehicle control by the ACC and the LKA.

[0081] Furthermore, when the driving control device C3 executes the emergency evacuation function, it can perform automatic evacuation control (evacuation processing) to automatically stop the first vehicle. When the automatic evacuation control is started, the driving control device C3 causes the surrounding condition monitoring device C5 to search for an evacuation location where the first vehicle should be stopped. The driving control device C3 then moves the first vehicle to the evacuation location set by the surrounding condition monitoring device C5 and stops the first vehicle at the evacuation location. Note that, on expressways, the above-mentioned evacuation location may be a shoulder outside the lane in which the vehicle is traveling. On ordinary roads, the evacuation location may be a shoulder outside the lane in which the vehicle is traveling, as well as an intersection, a railroad crossing, a sidewalk, or any other location that avoids a high probability of the presence of a moving object such as a second vehicle, a train, or a pedestrian.

[0082] Here, if the driver's posture is abnormal, it is estimated that the driver is in an abnormal state. If the driver is in an abnormal state, there is a high possibility that the operation by the driver is an erroneous operation. For example, if the driver's physical condition suddenly changes and the driver leans forward on the steering wheel, there is a high possibility that the steering wheel is operated but this operation is an erroneous operation. Also, if the driver's physical condition suddenly changes and the driver leans back, there is a possibility that the accelerator pedal is accidentally depressed.

[0083] Therefore, when the driving control device C3 executes the driving assistance function due to the driver's abnormal posture, it may be possible to disable an erroneous operation by the driver while validating a driving operation (hereinafter referred to as an override operation) aimed at preventing an accident, such as an intentional driving operation by the driver or a driving operation by a passenger in place of the driver. The vehicle state acquisition device C6 may be provided with an override operation detection unit C66 that detects an override operation.

[0084] The override operation will be described below. Regarding vehicle acceleration, if the driver is in an abnormal state, the driver may lose posture and accidentally press the accelerator pedal. Therefore, when the driving assistance function of the driving control device C3 is being executed, the accelerator operation may be disabled. In other words, the accelerator operation does not need to be included in the override operation.

[0085] On the other hand, even if the driver is in an abnormal state regarding deceleration of the vehicle, the driver may become drowsy and stop the first vehicle in an attempt to avoid a collision with an obstacle. Therefore, the override operation detection unit C66 acquires the amount of brake pedal depression from the brake position sensor C65, and if the braking force obtained by depression of the brake pedal is greater than the braking force in the driving assistance function of the driving control device C3, detects the brake pedal operation as an override operation.

[0086] Furthermore, when changing the vehicle's course, even if the driver is in an abnormal state, a passenger may operate the steering wheel in place of the driver. However, when the driver is in an abnormal state, the driver's posture may become unstable and the driver may fall onto the steering wheel. Therefore, the override operation detection unit C66 does not need to detect this erroneous operation as an override operation. Therefore, the override operation detection unit C66 detects, as an override operation, a steering wheel operation performed when the abnormal posture detection system 100 obtains a detection result indicating that the driver's posture is not unstable.

[0087] When the override operation detection unit C66 detects an override operation while the driving assistance function is being executed, the driving control system 300 may substitute the control of the steering actuator 31 or the braking / driving actuator 41 with the driving operation by the passenger or the driver. In other words, while the driving assistance function is being executed, the driving control of the vehicle may be performed based on the override operation detected by the override operation detection unit C66.

[0088] The operation of the abnormal posture detection device 1 according to the first embodiment will be described.

[0089] 5 is a flowchart for explaining the operation of the reference posture feature calculation process by the abnormal posture detection device 1 according to the first embodiment. The operation of the reference posture feature calculation process by the abnormal posture detection device 1 shown in the flowchart of FIG. 5 is repeatedly performed, for example, in accordance with instructions from a control unit (not shown), while the reference posture feature calculation completion flag is "0" until the reference posture feature calculation completion flag becomes "1." For example, when the ignition of the vehicle is turned on, the control unit refers to the reference posture feature calculation completion flag. Upon confirming that the reference posture feature calculation completion flag is "0," the control unit instructs the image acquisition unit 11, feature extraction unit 12, and reference posture feature calculation unit 13 of the abnormal posture detection device 1 to perform the reference posture feature calculation process.

[0090] The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST1). The image acquisition unit 11 outputs the acquired captured image to the feature extraction unit 12.

[0091] The feature extraction unit 12 extracts feature information to be used for determining the driver's posture (step ST2) based on the captured image acquired by the image acquisition unit 11 in step ST1. The feature extraction unit 12 outputs the feature information to the reference posture feature calculation unit 13 and stores the feature information in chronological order in the storage unit 18.

[0092] The reference posture feature amount calculation unit 13 performs a reference posture feature amount calculation process to estimate the driver's reference posture and calculate the reference posture feature amount (step ST3). In the reference posture feature amount calculation process in step ST3, the reference posture feature amount calculation unit 13 first estimates whether the driver's posture is the reference posture. If the reference posture feature amount calculation unit 13 estimates that the driver's posture is the reference posture, the reference posture feature amount calculation unit 13 stores the feature amount information output from the feature amount extraction unit 12 in the storage unit 18 or an internal storage area of ​​the reference posture feature amount calculation unit 13 as candidate feature amount information for calculating the reference posture feature amount. If the predetermined number of candidate feature amount information has not been stored, the reference posture feature amount calculation unit 13 does not set the reference posture feature amount calculation completion flag to "1" and ends the process of step ST3. Then, the operation of the abnormal posture detection device 1 returns to the process of step ST1. In step ST3, when a preset number of pieces of candidate feature information are stored, reference posture feature calculation unit 13 calculates the driver's reference posture feature and face detection reliability from the stored preset number of pieces of candidate feature information. Reference posture feature calculation unit 13 stores the reference posture feature information in storage unit 18 and sets a reference posture feature calculation completion flag to "1". At this time, reference posture feature calculation unit 13 may delete the reference posture feature information stored in storage unit 18 or an internal storage area of ​​reference posture feature calculation unit 13.

[0093] 6 is a flowchart for explaining the operations of the change amount calculation process, the first attitude determination process, the second attitude determination process, and the abnormal attitude detection process performed by the abnormal attitude detection device 1 according to embodiment 1. The operations of the change amount calculation process, the first attitude determination process, the second attitude determination process, and the abnormal attitude detection process performed by the abnormal attitude detection device 1 shown in the flowchart of FIG. 6 are repeatedly performed after the reference attitude feature calculation process described using the flowchart of FIG. 5 is completed, for example, until the vehicle ignition is turned off. When the reference attitude feature calculation process is completed and the reference attitude feature calculation completion flag is set to "1," the control unit confirms this and instructs the image acquisition unit 11, the feature extraction unit 12, the change amount calculation unit 14, the first determination unit 15, the second determination unit 16, and the detection unit 17 of the abnormal attitude detection device 1 to perform the change amount calculation process, the first attitude determination process, the second attitude determination process, or the abnormal attitude detection process.

[0094] The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST11). The image acquisition unit 11 outputs the acquired captured image to the feature extraction unit 12.

[0095] The feature extraction unit 12 extracts feature information to be used for determining the driver's posture based on the captured image acquired by the image acquisition unit 11 in step ST11 (step ST12). The feature extraction unit 12 outputs the feature information to the change amount calculation unit 14 and stores it in chronological order in the storage unit 18.

[0096] The change amount calculation unit 14 performs a change amount calculation process to calculate the amount of change from the reference posture feature amount of the feature amount extracted by the feature amount extraction unit 12 in step ST12 (step ST13). The change amount calculation unit 14 outputs the change amount information to the first determination unit 15 together with the feature amount information output from the feature amount extraction unit 12.

[0097] The first determination unit 15 performs a first posture determination process to determine whether the driver's posture is normal or not by comparing the feature extracted by the feature extraction unit 12 in step ST12 with the change calculated by the change calculation unit 14 in step ST13 with the conditions for determining normal posture, or by comparing the feature extracted by the feature extraction unit 12 with the conditions for determining normal posture, or by comparing the change calculated by the change calculation unit 14 with the conditions for determining normal posture (step ST14).

[0098] FIG. 7 is a flowchart for explaining the detailed operation of the first attitude determination process by the first determination unit 15, which is performed in step ST14 of FIG.

[0099] The first determination unit 15 compares the feature extracted by the feature extraction unit 12 and the change calculated by the change calculation unit 14 with the normal posture determination conditions, compares the feature extracted by the feature extraction unit 12 with the normal posture determination conditions, or compares the change calculated by the change calculation unit 14 with the normal posture determination conditions (step ST110), and determines whether the feature extracted by the feature extraction unit 12 and the change calculated by the change calculation unit 14 satisfy the normal posture determination conditions, whether the feature extracted by the feature extraction unit 12 satisfies the normal posture determination conditions, or whether the change calculated by the change calculation unit 14 satisfies the normal posture determination conditions (step ST120).

[0100] If the feature extracted by the feature extraction unit 12 or the change calculated by the change amount calculation unit 14 satisfies the normal posture determination condition, if the feature extracted by the feature extraction unit 12 satisfies the normal posture determination condition, or if the change calculated by the change amount calculation unit 14 satisfies the normal posture determination condition ("YES" in step ST120), the first determination unit 15 determines that the driver's posture is normal (step ST130). The first determination unit 15 outputs a normal posture determination result indicating that the driver's posture is normal to the second determination unit 16. At this time, the first determination unit 15 outputs the change amount information output from the change amount calculation unit 14 together with the normal posture determination result to the second determination unit 16.

[0101] If the feature extracted by the feature extractor 12 and the change calculated by the change amount calculator 14 do not satisfy the normal posture determination condition ("NO" in step ST120), the first determination unit 15 does not determine that the driver's posture is normal. That is, the first determination unit 15 determines that the driver's posture is not normal. The first determination unit 15 outputs a normal posture determination result indicating that the first determination unit 15 did not determine that the driver's posture is normal, in other words, that the first determination unit 15 determined that the driver's posture is not normal, to the second determination unit 16. At this time, the first determination unit 15 outputs the change amount information output from the change amount calculator 14 together with the normal posture determination result to the second determination unit 16.

[0102] Returning to the explanation of the flowchart in Fig. 6, the second determination unit 16 performs a second posture determination process to determine whether or not the driver is experiencing poor posture, based on the normal posture determination result output from the first determination unit 15 in step ST14 (step ST15).

[0103] FIG. 8 is a flowchart for explaining the detailed operation of the second attitude determination process by the second determination unit 16, which is performed in step ST15 of FIG.

[0104] In step ST14 of Figure 6, if the first judgment unit 15 determines that the driver's posture is normal (in the case of "YES" in step ST210), the second judgment unit 16 determines that the driver's posture is not poor, in other words, that the driver is not experiencing poor posture, and outputs a posture poor judgment result indicating that the driver is not experiencing poor posture to the detection unit 17, thereby ending the processing shown in the flowchart of Figure 8.

[0105] 6, if the first determination unit 15 determines that the driver's posture is not normal ("NO" in step ST210), the second determination unit 16 determines whether the driver is experiencing poor posture based on the amount of change calculated by the change amount calculation unit 14 in step ST13 of Fig. 6 and the machine learning model (step ST220). The second determination unit 16 outputs a posture error determination result indicating that the driver is experiencing poor posture to the detection unit 17.

[0106] Returning to the explanation of the flowchart in Fig. 6, the detection unit 17 performs an abnormal posture detection process to detect whether the driver's posture is abnormal or not based on the posture collapse determination result output from the second determination unit 16 in step ST15 (step ST16).

[0107] FIG. 9 is a flowchart for explaining the detailed operation of the abnormal posture detection process by the detection unit 17, which is performed in step ST16 of FIG.

[0108] The detection unit 17 determines whether or not the second determination unit 16 has determined in step ST15 of FIG. 6 that the driver's posture has deteriorated (step ST310).

[0109] If the second judgment unit 16 judges that the driver has poor posture (if the answer is "YES" in step ST310), that is, if the second judgment unit 16 outputs a posture deviation judgment result indicating that the driver is experiencing poor posture, the detection unit 17 counts up the abnormal posture detection counter (step ST320).

[0110] On the other hand, if the second judgment unit 16 does not determine that the driver has poor posture (if the answer is "NO" in step ST310), that is, if the second judgment unit 16 outputs a posture deviation judgment result indicating that the driver is not experiencing poor posture, the detection unit 17 resets the abnormal posture detection counter (step ST330).

[0111] The second determination unit 16 determines whether the abnormal posture detection counter has reached the abnormality detection threshold value (step ST340).

[0112] If the abnormal posture detection counter reaches the abnormality detection threshold ("YES" in step ST340), the detection unit 17 determines that the state in which the second determination unit 16 determined that the driver is experiencing poor posture has continued for the abnormality detection time, and detects the driver's abnormal posture (step ST350). The detection unit 17 outputs abnormal posture detection information indicating that the driver's abnormal posture has been detected to the driving control system 300. The detection unit 17 may output the abnormal posture detection information to an output device as information for analysis in case of any unexpected situation.

[0113] If the abnormal posture detection counter has not reached the abnormality detection threshold ("NO" in step ST340), the detection unit 17 determines that the state in which the second determination unit 16 determined that the driver is experiencing poor posture has not continued for the abnormality detection time, and does not detect an abnormal posture of the driver. In other words, the detection unit 17 detects that the driver's posture is not abnormal. The detection unit 17 may output information that an abnormal posture of the driver has not been detected to the driving control system 300 or to an output device.

[0114] In this way, the abnormal posture detection device 1 according to the first embodiment calculates the change amount from the reference posture feature amount of the feature amount extracted based on the captured image capturing the area where the driver's face should be. The abnormal posture detection device 1 first compares the extracted feature amount and the calculated change amount with the normal posture determination conditions, and determines whether the driver's posture is normal by comparing the extracted feature amount with the normal posture determination conditions or by comparing the calculated change amount with the normal posture determination conditions. If the abnormal posture detection device 1 determines that the driver's posture is not normal, it then determines whether the driver is experiencing poor posture based on the calculated change amount and the machine learning model. If the state in which it has been determined that the driver is experiencing poor posture continues for the abnormality detection time, the abnormal posture detection device 1 detects the driver's abnormal posture.

[0115] By using a machine learning model to determine whether an occupant's posture is out of alignment, it is possible to detect an abnormal occupant's posture with high accuracy. On the other hand, in a scene where feature values ​​such as facial orientation cannot be reliably extracted from a captured image, the presence or absence of an occupant's posture is determined based on low-accuracy feature values ​​and a machine learning model. In this case, when determining whether an occupant's posture is out of alignment using a machine learning model, the occupant may be determined to be in an out-of-alignment state even though they are not. Here, FIGS. 10A and 10B are diagrams illustrating an example of a scene where feature values ​​such as facial orientation cannot be reliably extracted from a captured image. FIG. 10A shows an example of a case where the driver's face is no longer captured in the captured image because the driver has assumed a "prone" position. FIG. 10B shows an example of a case where the driver's face is partially obscured in the captured image because the driver touches his or her face with his or her hand. In FIGS. 10A and 10B, the captured image is indicated by Im, and the driver is indicated by Dr. In the first embodiment, the upper left corner of the captured image is the origin, the right direction of the captured image is the positive direction of the x-axis, and the downward direction of the captured image is the positive direction of the y-axis.

[0116] For example, if the driver falls into a prone position, the driver's face will not be captured in the captured image, and the driver's facial orientation and other feature values ​​will not be extracted (see FIG. 10A).

[0117] For example, if the driver is touching his / her face with his / her hand, part of the driver's face is covered in the captured image, and the feature amount such as the driver's facial direction cannot be stably extracted from the captured image (see FIG. 10B). In the example of FIG. 10B, the driver's eyes are covered, and therefore the facial direction cannot be stably extracted.

[0118] It is assumed that the amount of change in the feature from the reference posture feature will be large in both the scene shown in Figure 10A and the scene shown in Figure 10B. As a result, for example, if the amount of change is input into a machine learning model that determines whether the driver has poor posture, it may be determined that the driver has poor posture in both scenes. In fact, it should not be determined that the driver has poor posture in the scene shown in Figure 10B. In determining whether the driver has poor posture using a machine learning model, in a scene in which the feature is not stably extracted, an unintended change in the feature from the reference posture feature may occur, increasing the possibility of erroneously determining whether the driver has poor posture.

[0119] In contrast, as described above, the abnormal posture detection device 1 according to the first embodiment first determines whether the driver's posture is normal by comparing the extracted feature amount and the calculated change amount with the normal posture determination conditions, or by comparing the extracted feature amount with the normal posture determination conditions, or by comparing the calculated change amount with the normal posture determination conditions. After that, if the abnormal posture detection device 1 determines that the driver's posture is not normal, it determines whether the driver is suffering from poor posture based on the calculated change amount and the machine learning model, and if the state in which it has been determined that the driver is suffering from poor posture continues for the abnormality detection time, it detects the driver's abnormal posture.

[0120] Therefore, the abnormal posture detection device 1 can detect an abnormal posture of the driver even when a scene occurs in which feature amounts cannot be stably extracted from the captured image.

[0121] 11 is a flowchart for explaining an example of the operation of the driving assistance function by the driving control system 300 according to the first embodiment. In the following explanation, an example will be given in which the driving control system 300 performs an emergency evacuation function for the vehicle as the driving assistance function when the abnormal posture detection system 100 detects that the driver's posture is abnormal. Note that the operation of the driving control system 300 is started, for example, after the ignition of the vehicle is turned on.

[0122] First, the driving control device C3 of the driving control system 300 acquires abnormal posture detection information from the abnormal posture detection system 100 (step ST401) and determines whether or not an abnormal posture of the driver has been detected (step ST402). If the abnormal posture detection information does not indicate that an abnormal posture of the driver has been detected ("NO" in step ST402), the operation of the driving control system 300 proceeds to the processing of ST401.

[0123] On the other hand, if the abnormal posture detection information indicates that the driver's abnormal posture has been detected, the warning control device C7 activates the warning unit C71 to start warning the driver (step ST403). Here, the warning to the driver by the warning unit C71 is, for example, displaying a warning message on the display unit of the navigation system C54 installed in the first vehicle, or outputting sound from an audio device installed in the first vehicle.

[0124] Next, the warning control device C7 determines whether or not a response has been received from the driver in response to the warning (step ST404). Here, the determination of whether or not a response has been received from the driver in response to the warning is made based on, for example, whether or not abnormal posture detection information indicating that an abnormal posture of the driver has been detected has been received from the abnormal posture detection system 100 within a predetermined time period since the warning was issued by the warning unit C71.

[0125] That is, if abnormal posture detection information indicating that an abnormal posture of the driver has been detected is not received from the abnormal posture detection system 100 before a predetermined time has elapsed since the warning unit C71 was activated, the warning control device C7 determines that a response has been received from the driver to the warning. On the other hand, if a predetermined time has elapsed since the warning unit C71 was activated while abnormal posture detection information indicating that an abnormal posture of the driver has been detected remains received from the abnormal posture detection system 100, the warning control device C7 determines that a response has not been received from the driver to the warning. Note that the determination of whether a response has been received from the driver to the warning may be made, for example, based on whether or not the driver has operated the operating unit to cancel the warning before a predetermined time has elapsed since the warning unit C71 was activated. The predetermined time is, for example, 3 seconds.

[0126] If the driver responds to the warning ("YES" in step ST404), the warning control device C7 stops the warning from the warning unit C71 (step ST405), and the operation of the driving control system 300 proceeds to the processing of step ST401. On the other hand, if the driver does not respond to the warning ("NO" in step ST404), the warning control device C7 activates the notification unit C72 to start notifying people outside the first vehicle (step ST406). Here, notifying the second vehicle means notifying people outside the first vehicle, such as occupants of the second vehicle, that the driver of the first vehicle is not in a state where they can continue driving, or notifying that the first vehicle will start a driving assistance function such as an emergency evacuation function.

[0127] Then, the driving control device C3 starts the emergency evacuation function. First, the driving control device C3 acquires information about the vehicle state from the vehicle state acquisition device C6 (step ST407). The vehicle state is information indicating the state of the first vehicle and is used for vehicle control in driving assistance functions such as the emergency evacuation function.

[0128] Next, the driving control device C3 acquires information about the surrounding conditions of the first vehicle from the surrounding conditions monitoring device C5 (step ST408) and sets a location to which the first vehicle should be evacuated (step ST409). Hereinafter, the location to which the first vehicle should be evacuated is referred to as the evacuation location. Note that the evacuation location may be within a range of 150 m from the current location of the first vehicle, or a location to which the first vehicle can travel from its current location within 60 seconds. In this way, it is possible to prevent the first vehicle from entering the intersection more than necessary while traveling to the evacuation location.

[0129] For example, the driving control device C3 searches for a position where the first vehicle can be safely stopped from the surrounding situation monitoring device C5, and sets the safest position from among the candidate positions where the first vehicle can be safely stopped as the evacuation position. Here, the evacuation position is a position where there is no possibility of collision with a moving object such as a second vehicle or a pedestrian, or an obstacle, and where there is no possibility of collision with the moving object or obstacle on the route from the current position of the first vehicle to the evacuation position. Examples of evacuation positions include a road edge such as a road shoulder, or within a lane other than an intersection or a railroad crossing. Furthermore, if the evacuation position is a road edge such as a road shoulder, space may be secured to allow passengers to escape from the first vehicle.

[0130] After setting the evacuation position, the driving control device C3 performs vehicle control to stop the first vehicle at the evacuation position. After setting the evacuation position, the driving control device C3 may display a route from the current position of the first vehicle to the set evacuation position on the display unit of the navigation system C54.

[0131] First, the driving control device C3 determines whether a lane change is required before moving the first vehicle to the evacuation position (step ST410). If a lane change is not required before moving the first vehicle to the evacuation position (if "NO" in step ST410), that is, if the evacuation position is set within the lane in which the first vehicle is currently traveling, the driving control device C3 controls the steering actuator 31 and the braking / driving actuator 41 to cause the first vehicle to travel within the lane in which the first vehicle is currently traveling (step ST411). Then, the driving control device C3 controls the steering actuator 31 and the braking / driving actuator 41 to decelerate the first vehicle and then stop it at the evacuation position (step ST412).

[0132] In the process of step ST411, the driving control device C3 may drive the first vehicle at a speed at which an emergency stop can be made, such as 10 km / h. In addition, if it is not necessary to continue driving the first vehicle, such as when the evacuation position is set near the first vehicle, the process of step ST411 can be omitted. Furthermore, in the processes of steps ST411 and ST412, when the driving control device C3 decelerates the first vehicle, the speed may be set to, for example, 3 m / s to prevent the passenger from falling. ２ It may be the following:

[0133] Then, after the driving control device C3 stops the first vehicle at the evacuation position, a report is sent to a call center or the like outside the vehicle via the on-board communication device C52 (step ST413). The report sent via the on-board communication device C52 includes, for example, a message indicating that the driver of the first vehicle is in an abnormal posture and may be unable to continue driving, the current position of the first vehicle acquired by the surrounding situation monitoring device C5, and the evacuation position set by the driving control device C3. Note that even when a report is sent via the on-board communication device C52, the driving control device C3 may continue to keep the first vehicle stopped, and the alarm control device C7 may continue to alert people outside the first vehicle.

[0134] The following describes a case where the driving control device C3 determines in the process of step ST410 that a lane change is required before the first vehicle is moved to the evacuation position ("YES" in step ST410). Note that lane change includes a case where the first vehicle is moved to a lane adjacent to the lane in which the first vehicle is traveling, such as a lane change, and a case where the first vehicle is deviated from the lane in which the first vehicle is traveling so as to pull over to the side of the road.

[0135] When the driving control device C3 determines that a lane change is necessary, it controls the steering actuator 31 and the braking / driving actuator 41 to change the lane of the first vehicle (step ST414). Here, the lateral movement speed of the vehicle during lane change may be approximately 0.3 m / s so that a second vehicle or a pedestrian, etc., present at the lane change destination can recognize the lane change of the first vehicle and avoid a collision. Furthermore, when a lane change is to be performed, the warning control device C7 may notify a person outside the first vehicle that the first vehicle will start changing lane via the notification unit C72 at least a predetermined time before the lane change is initiated.

[0136] Next, if no response is obtained from the driver in response to the warning from the alarm unit, a notification is made to the second vehicle via the notification unit C72, indicating that the posture of the driver of the first vehicle is abnormal and that the driver may be in a state where he or she cannot continue driving. Note that the notification to the second vehicle by the notification unit C72 is not limited to a notification indicating that the posture of the driver of the first vehicle is abnormal, but may also be a notification indicating that the emergency evacuation function of the vehicle will be started or a notification indicating that the emergency evacuation function of the vehicle is being executed, which is performed in subsequent processing.

[0137] Then, similar to the process of step ST412, the drive control device C3 controls the steering actuator 31 and the braking / driving actuator 41 to decelerate the first vehicle and then stop it at the evacuation position (step ST415). After the drive control device C3 has stopped the first vehicle at the evacuation position, the operation of the drive control system 300 proceeds to the process of step ST413, and a report is sent to a call center or the like outside the vehicle via the in-vehicle communication device C52.

[0138] Furthermore, if an override operation is performed while the driving support function is being executed, the driving control system 300 may substitute the control of the steering actuator 31 or the braking / driving actuator 41 with a driving operation by the passenger or the driver. That is, for example, if the override operation detection unit C66 detects an override operation during the processing of steps ST406 to ST415 in FIG. 11, the vehicle may be driven based on the detected override operation.

[0139] In the above example, when the abnormal posture detection system 100 detects that the driver's posture is abnormal, the driving control system 300 performs an emergency evacuation function for the vehicle as a driving assistance function. However, the driving control system 300 may also perform ACC or LKA as a driving assistance function. Furthermore, driving assistance functions may be combined, such as by having the driving control system 300 perform ACC and LKA and then execute the emergency evacuation function when safety in the surrounding area is ensured. Driving assistance functions can be combined as appropriate.

[0140] In the above-described first embodiment, the abnormal posture detection device 1 includes the reference posture feature amount calculation unit 13 and has the function of estimating the driver's reference posture and calculating the reference posture feature amount for the estimated reference posture. However, this is merely an example. For example, the function of estimating the driver's reference posture and calculating the reference posture feature amount for the reference posture may be provided by a device (not shown) external to the abnormal posture detection device 1 and connected to the abnormal posture detection device 1. In this case, the abnormal posture detection device 1 does not necessarily include the reference posture feature amount calculation unit 13. Furthermore, the operation of the abnormal posture detection device 1, such as the reference posture feature amount calculation process described using the flowchart shown in FIG. 5, can be omitted. Furthermore, for example, an administrator or the like may set a reference posture in advance based on the facial orientation, head position, and face detection reliability of a typical driver and store the reference posture in a location accessible by the abnormal posture detection device 1, such as the storage unit 18. In this case, the abnormal posture detection device 1 does not necessarily include the reference posture feature amount calculation unit 13, and the operation of the reference posture feature amount calculation process described using the flowchart shown in FIG. 5 can be omitted. However, the abnormal posture detection device 1 can estimate a reference posture and calculate reference posture feature amounts that are more tailored to an individual when the reference posture feature calculation unit 13 or an external device has a function of estimating the reference posture, compared to when the reference posture feature calculation unit 13 or an external device does not have a function of estimating the reference posture. As a result, the abnormal posture detection device 1 can more accurately detect whether the driver is in an abnormal posture.

[0141] Furthermore, in the first embodiment, the change amount used as an input to the machine learning model includes the change amount of face detection reliability. However, this is merely an example. The change amount used as an input to the machine learning model may not include the change amount of face detection reliability. However, including face detection reliability in the change amount used as an input to the machine learning model allows the abnormal posture detection device 1 to more accurately determine whether posture is poor compared to when face detection reliability is not included. Recently, imaging devices 2 mounted on vehicles have been miniaturized. Miniaturized imaging devices 2 often employ narrow-angle lenses. When the imaging device 2 is a so-called narrow-angle camera, the imaging range is narrow. That is, the likelihood that the driver's facial features will be out of frame increases. In other words, the ease of detecting the driver's facial features decreases. In this case, even if the driver's posture is not abnormal, the plausibility of the detected driver's face decreases. By using the face detection reliability as an input to the machine learning model, overdetection of posture is prevented, and it is expected that information indicating the presence or absence of a driver's posture obtained from the machine learning model can be obtained as a more accurate inference result.

[0142] In the first embodiment, the vehicle occupant for which the abnormal posture detection device 1 detects whether the vehicle is in an abnormal posture is the driver of the vehicle, but this is merely an example. The abnormal posture detection device 1 can detect whether any vehicle occupant other than the driver is in an abnormal posture.

[0143] In the first embodiment described above, the abnormal posture detection device 1 is an on-board device mounted on a vehicle, and the image acquisition unit 11, the feature amount extraction unit 12, the reference posture feature amount calculation unit 13, the change amount calculation unit 14, the first determination unit 15, the second determination unit 16, and the detection unit 17 are provided in the on-board device. However, without being limited to this, some of the image acquisition unit 11, the feature amount extraction unit 12, the reference posture feature amount calculation unit 13, the change amount calculation unit 14, the first determination unit 15, the second determination unit 16, and the detection unit 17 may be mounted in the on-board device of the vehicle, and the rest may be provided in a server connected to the on-board device via a network, so that a system is configured by the on-board device and the server. In addition, the image acquisition unit 11, the feature extraction unit 12, the reference posture feature calculation unit 13, the change amount calculation unit 14, the first determination unit 15, the second determination unit 16, and the detection unit 17 may all be provided in the server.

[0144] 12A and 12B are diagrams illustrating an example of the hardware configuration of the abnormal posture detection device 1 according to the first embodiment. In the first embodiment, the functions of the image acquisition unit 11, the feature extraction unit 12, the reference posture feature calculation unit 13, the change amount calculation unit 14, the first determination unit 15, the second determination unit 16, the detection unit 17, and a control unit (not shown) are realized by a processing circuit 101. That is, the abnormal posture detection device 1 includes the processing circuit 101 for controlling the detection of whether the posture of a vehicle occupant is abnormal based on a captured image. The processing circuit 101 may be dedicated hardware as shown in FIG. 12A or a processor 104 that executes a program stored in a memory 105 as shown in FIG. 12B.

[0145] When processing circuitry 101 is dedicated hardware, processing circuitry 101 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0146] When the processing circuit is processor 104, the functions of image acquisition unit 11, feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first determination unit 15, second determination unit 16, detection unit 17, and a control unit (not shown) are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 105. Processor 104 reads and executes the program stored in memory 105 to execute the functions of image acquisition unit 11, feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first determination unit 15, second determination unit 16, detection unit 17, and a control unit (not shown). In other words, abnormal posture detection device 1 includes memory 105 for storing a program that, when executed by processor 104, results in the execution of steps ST1 to ST3 in FIG. 5 and steps ST11 to ST16 in FIG. 6 described above. In addition, it can also be said that the program stored in memory 105 causes a computer to execute the processing procedures or methods of the image acquisition unit 11, the feature extraction unit 12, the reference posture feature calculation unit 13, the change amount calculation unit 14, the first determination unit 15, the second determination unit 16, the detection unit 17, and a control unit (not shown). Here, the memory 105 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

[0147] Note that the functions of the image acquisition unit 11, feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first determination unit 15, second determination unit 16, detection unit 17, and a control unit (not shown) may be partially implemented by dedicated hardware and partially implemented by software or firmware. For example, the image acquisition unit 11 may be implemented by a processing circuit 101 as dedicated hardware, while the functions of the feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first determination unit 15, second determination unit 16, detection unit 17, and a control unit (not shown) may be implemented by a processor 104 reading and executing a program stored in a memory 105. The storage unit 18 may be, for example, the memory 105. The abnormal posture detection device 1 also includes an input interface device 102 and an output interface device 103 that communicate via wired or wireless communication with various devices, such as the imaging device 2 or the driving control system 300.

[0148] As described above, according to the first embodiment, the abnormal posture detection device 1 includes an image acquisition unit 11 that acquires a captured image in which an area in which the face of a vehicle occupant should be present is captured, a feature extraction unit 12 that extracts feature amounts used to determine the posture of the occupant based on the captured image acquired by the image acquisition unit 11, a change amount calculation unit 14 that calculates a change amount of the feature amount extracted by the feature extraction unit 12 from a feature amount in a reference posture of the occupant, and a normal posture determination condition by comparing the feature amount extracted by the feature extraction unit 12 and the change amount calculated by the change amount calculation unit 14 with the normal posture determination condition. The abnormal posture detection device 1 is configured to include a first determination unit 15 that determines whether the posture of the occupant is normal by comparing the amount of change calculated by the change amount calculation unit 14 with a normal posture determination condition, a second determination unit 16 that, when the first determination unit 15 determines that the posture of the occupant is not normal, determines whether the occupant has lost their posture based on the amount of change calculated by the change amount calculation unit 14 and a machine learning model that inputs the amount of change and outputs information indicating whether the occupant has lost their posture, and a detection unit 17 that detects an abnormal posture of the occupant when the state in which the second determination unit 16 has determined that the occupant has lost their posture continues for an abnormality detection time. Therefore, the abnormal posture detection device 1 can detect an abnormal posture of the occupant even when a scene occurs in which feature amounts cannot be stably extracted from a captured image.

[0149] In addition, in the present disclosure, any of the components of the embodiments may be modified or omitted.

[0150] The abnormal posture detection device of the present disclosure can detect an abnormal posture of an occupant even when a scene occurs in which feature amounts cannot be stably extracted from a captured image.

[0151] 1 Abnormal posture detection device, 11 Image acquisition unit, 12 Feature extraction unit, 13 Reference posture feature calculation unit, 14 Change amount calculation unit, 15 First judgment unit, 16 Second judgment unit, 17 Detection unit, 18 Memory unit, 100 Abnormal posture detection system, 2 Imaging device, 300 Driving control system, 3 Steering mechanism, 31 Steering actuator, 4 Braking / driving mechanism, 41 Braking / driving actuator, 101 Processing circuit, 102 Input interface device, 103 Output interface device, 104 Processor, 105 Memory, C3 Driving control device, C4 Map information storage device, C5 Surrounding situation monitoring device, C51 GPS receiver, C52 In-vehicle communication device, C53 Outside vehicle sensor, C54 Navigation system, C6 Vehicle state acquisition device, C61 Steering angle sensor, C62 Vehicle speed sensor, C63 Steering torque sensor, C64 Accelerator position sensor, C65 brake position sensor, C66 override operation detection unit, C7 alarm control device, C71 warning unit, C72 notification unit.

Claims

1. an image acquisition unit that acquires a captured image in which an area in which a face of a vehicle occupant should be present is captured; a feature extraction unit that extracts a feature used to determine a posture of the occupant based on the captured image acquired by the image acquisition unit; a change amount calculation unit that calculates a change amount of the feature amount extracted by the feature amount extraction unit from the feature amount in a reference posture of the occupant; a first determination unit that determines whether or not the posture of the occupant is normal by comparing the feature extracted by the feature extraction unit and the change calculated by the change amount calculation unit with a normal posture determination condition, by comparing the feature extracted by the feature extraction unit with the normal posture determination condition, or by comparing the change calculated by the change amount calculation unit with the normal posture determination condition; a second determination unit that, when the first determination unit determines that the posture of the occupant is not normal, determines whether or not the occupant has lost their posture based on the amount of change calculated by the change amount calculation unit and a machine learning model that receives the amount of change as an input and outputs information indicating whether or not the occupant has lost their posture; a detection unit that detects an abnormal posture of the occupant when the state in which the second determination unit has determined that the occupant is in a bad posture continues for an abnormality detection time; An abnormal posture detection device comprising:

2. a reference posture characteristic amount calculation unit that estimates the reference posture of the occupant and calculates the characteristic amount in the reference posture; The abnormal posture detection device according to claim 1 , further comprising:

3. The feature amount includes a facial direction of the occupant and a head position of the occupant.

3. The abnormal attitude detection device according to claim 1 or 2.

4. The feature amount includes a facial direction of the occupant, a head position of the occupant, and a face detection reliability.

3. The abnormal attitude detection device according to claim 1 or 2.

5. An image acquisition unit acquires a captured image in which an area in which a face of a vehicle occupant should be present is captured; a feature amount extraction unit extracting a feature amount used for determining a posture of the occupant based on the captured image acquired by the image acquisition unit; a change amount calculation unit calculating an amount of change of the feature amount extracted by the feature amount extraction unit from the feature amount in a reference posture of the occupant; a step of a first determination unit determining whether or not the posture of the occupant is normal by comparing the feature extracted by the feature extraction unit and the change calculated by the change amount calculation unit with a normal posture determination condition, by comparing the feature extracted by the feature extraction unit with the normal posture determination condition, or by comparing the change calculated by the change amount calculation unit with the normal posture determination condition; a second determination unit, when the first determination unit determines that the posture of the occupant is not normal, determining whether or not the occupant has lost their posture based on the amount of change calculated by the change amount calculation unit and a machine learning model that receives the amount of change as an input and outputs information indicating whether or not the occupant has lost their posture; A step in which a detection unit detects an abnormal posture of the occupant when a state in which the second determination unit has determined that the occupant has lost his / her posture continues for an abnormality detection time. An abnormal posture detection method comprising:

6. A vehicle control system comprising the abnormal attitude detection device according to claim 1 and a driving control device that controls driving of the vehicle, the abnormal posture detection device outputs a detection result of whether or not the occupant is in the abnormal posture to the operation control device; The driving control device outputs a control signal for outputting a warning to the occupant or for performing an evacuation process of the vehicle based on the detection result to an alarm control device, a steering mechanism, or a braking / driving mechanism mounted on the vehicle. A vehicle control system comprising: