Crew recognition device and crew recognition method

Abstract

This allows for a more accurate assessment of the crew's posture. [Solution] The occupant recognition device 30 includes an image acquisition unit 32 that acquires a first image and a second image, an estimation unit 33 that estimates the occupant's posture based on a third image, a calculation unit 35 that calculates the width of the ear shadows around the occupant based on the difference between the first image and the second image, a correction unit 36 ​​that corrects the estimated occupant's posture based on the width of the ear shadows around the occupant, and a notification unit 37 that outputs a predetermined notification based on the corrected occupant's posture.

Description

Technical Field

[0001] The present disclosure relates to an occupant recognition device and an occupant recognition method.

Background Art

[0002] There is known a technique for detecting an occupant's posture based on an image of the interior of a vehicle. Patent Document 1 discloses a technique for correcting numerical coordinates obtained by digitizing the surface three-dimensional shape of an occupant with a reference plane set based on vehicle seat information, and deriving information regarding the occupant based on the corrected numerical coordinates. Patent Document 2 discloses a technique for illuminating an occupant's head with near-infrared light and detecting the movement of the occupant's head based on the light intensity level of the near-infrared light detected by a near-infrared light sensor provided on a headrest. Patent Document 3 discloses a technique for estimating the position of the top of an occupant's head based on the position of landmarks of the occupant's face detected from an image, and estimating the seat height of the occupant based on the estimation result. Development of a technique for more accurately recognizing an occupant's posture is desired.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] [[ID=四十二]] The present disclosure aims to more accurately recognize an occupant's posture.

Means for Solving the Problems

[0005] To achieve the above objective, the occupant recognition device of this disclosure includes: an image acquisition unit that acquires a first image of the occupant taken by illuminating the inside of the vehicle with infrared light from a first position and a second image of the occupant taken by illuminating the inside of the vehicle with infrared light from a second position; an estimation unit that estimates the occupant's posture based on the images of the occupant; a calculation unit that calculates the width of the shadow cast around the occupant based on the difference between the first image and the second image; a correction unit that corrects the estimated occupant's posture based on the width of the shadow cast around the occupant; and a notification unit that outputs a predetermined notification based on the corrected occupant's posture. [Effects of the Invention]

[0006] According to this disclosure, the posture of the occupants can be recognized more accurately. [Brief explanation of the drawing]

[0007] [Figure 1] A block diagram showing the functional configuration of a crew recognition system equipped with a crew recognition device according to the first embodiment. [Figure 2A] A perspective view of the external appearance of the imaging device. [Figure 2B] A schematic diagram of an image captured by an imaging device. [Figure 3] (a) A diagram showing the general outline of the crew's two-dimensional skeleton, (b) the three-dimensional skeleton converted from the two-dimensional skeleton, and (c) the corrected three-dimensional skeleton. [Figure 4] A diagram illustrating an example of the transition in calibration states. [Figure 5] A diagram illustrating an example of the procedure for creating a difference image by the calculation unit. [Figure 6] A diagram illustrating an example of the scanning procedure for difference images performed by the calculation unit. [Figure 7] A diagram illustrating an example of the procedure for calculating the distance between the ear and the headrest using the correction unit. [Figure 8] A diagram illustrating an example of the procedure for calculating calibration values ​​using the correction unit. [Figure 9] This diagram illustrates an example of the procedure for correcting skeletal points and facial feature points using the correction unit. [Figure 10]A diagram illustrating an example of the notification determination procedure by the notification unit. [Figure 11] A diagram illustrating the procedure for calculating the position of the ear by the calculation unit of the occupant recognition device according to the second embodiment. [Figure 12] A flowchart illustrating an example of the image acquisition process performed by the crew recognition device. [Figure 13] A flowchart illustrating an example of the crew recognition process performed by the crew recognition device. [Figure 14] A flowchart showing the details of step S11, as shown in Figure 13. [Figure 15] A flowchart showing the details of step S27, as shown in Figure 14. [Figure 16] A flowchart showing the details of step S12, as shown in Figure 13. [Modes for carrying out the invention]

[0008] (First Embodiment) The occupant recognition system 100, which includes the occupant recognition device 30 according to the first embodiment, will be described as follows with reference to the drawings. The occupant recognition system 100 is mounted on a vehicle. As shown in Figure 1, the occupant recognition system 100 mainly comprises an imaging device 10, a position sensor 20, and an occupant recognition device 30. The occupant recognition system 100 may further include a display device for displaying images, an operating unit for user operation, etc. The occupant recognition system 100 is connected to an external device 50 via a communication line. The external device 50 consists of, but is not limited to, an alarm device 51 that provides various alarms, a vehicle control device 52 that controls vehicle driving, parking, etc.

[0009] The imaging device 10 is a digital camera that photographs the interior of the vehicle, including the driver and other occupants. The imaging device 10 is installed near the rearview mirror and facing into the vehicle, but it can be installed in any position as long as it can photograph the interior of the vehicle. As shown in Figure 2A, the imaging device 10 mainly comprises an imaging unit 11, a first illumination unit 12, and a second illumination unit 13. The imaging unit 11 includes an optical system with a lens 14, an image sensor (not shown), a processor, and memory, etc.

[0010] The first lighting unit 12 and the second lighting unit 13 are arranged side by side on the left and right so as to be line-symmetrical with respect to the lens 14 along the left-right axis of the imaging element of the imaging unit 11. The first lighting unit 12 is arranged on the left side facing the inside of the vehicle. The second lighting unit 13 is arranged on the right side facing the inside of the vehicle. The position of the first lighting unit 12 is set as the first position L, the position of the second lighting unit 13 is set as the second position R, and the position of the lens 14 is set as the third position P. The first position L, the second position R, and the third position P are, for example, predetermined points on the optical axes of the first lighting unit 12, the second lighting unit 13, and the lens 14, respectively. The position information of the first position L, the second position R, and the third position P is converted three-dimensionally in the three-dimensional vehicle model of the host vehicle and stored in the storage unit 40 in advance. The first lighting unit 12 and the second lighting unit 13 are not limited to the configuration arranged along the left-right axis, and may be configured to be arranged vertically with the lens 14 interposed therebetween along the up-down axis of the imaging element.

[0011] The first lighting unit 12 and the second lighting unit 13 are LEDs that irradiate infrared light as illumination light. The imaging device 10 captures an infrared image inside the vehicle illuminated by the infrared light. Since the illumination light is infrared light, the influence of sunlight, interior lights, etc. is suppressed, and the imaging device 10 can appropriately capture an image inside the vehicle.

[0012] The imaging device 10 lights up the first lighting unit 12 in accordance with a shooting instruction signal from the occupant recognition device 30, irradiates the inside of the vehicle with infrared light from the first position L, captures a first image, and outputs it to the occupant recognition device 30. The imaging device 10 lights up the second lighting unit 13 in accordance with a shooting instruction signal from the occupant recognition device 30, irradiates the inside of the vehicle with infrared light from the second position R, captures a second image, and outputs it to the occupant recognition device 30.

[0013] The imaging device 10 lights up the first illumination unit 12 and the second illumination unit 13 to illuminate the inside of the vehicle with infrared light, acquires a third image of the occupants, and outputs it to the occupant recognition device 30. Figure 2B is a schematic diagram of the third image taken by the imaging device 10. The imaging device 10 takes the third image at a frame rate appropriate to its performance, for example, at intervals of 10 msec or 20 msec, but is not limited to this interval. In addition to the imaging device 10 that takes the first and second images, there may be an imaging device 10 that takes the third image. The third image may be an image taken with visible light.

[0014] The position sensor 20 detects the position information of the seat in which the occupant is seated. The position sensor 20 detects the position of the seat as it moves forward or backward due to the occupant's operation and outputs it to the occupant recognition device 30. The position sensor 20 outputs the three-dimensional coordinates of the seat as the seat position to the occupant recognition device 30. The position sensor 20 may output a value that can identify the seat position to the occupant recognition device 30, and the occupant recognition device 30 may identify the three-dimensional coordinates of the seat based on that information. The position sensor 20 is placed on each seat of the driver's seat and passenger seat, but it may also be placed on the rear seats. In addition to the position sensor 20, the occupant recognition system 100 may also include sensors that detect the angle of the seat back, sensors that detect the position or angle of the headrest and footrest, etc. The occupant recognition device 30 can recognize the occupant more appropriately based on the information detected by each sensor.

[0015] The occupant recognition device 30 recognizes the occupant's posture based on images input from the imaging device 10, three-dimensional coordinates input from the position sensor 20, a vehicle model pre-stored in the memory unit 40, regression parameters, etc. The occupant recognition device 30 is composed of an ECU having a processing unit such as a CPU and GPU, and a storage device such as RAM and ROM. The occupant recognition device 30 can be composed of a single ECU, or it can be composed of multiple ECUs, distributing the functions of the control unit 31 and distributing the data to be stored. The occupant recognition device 30 can also be configured to include an FPGA, ASIC, etc.

[0016] As shown in Figure 1, the crew recognition device 30 functions as a control unit 31 and a storage unit 40. The control unit 31 is mainly composed of a processing unit such as a CPU in the ECU, and controls the operation of the entire crew recognition device 30 by loading a predetermined program stored in ROM into RAM and executing it. The storage unit 40 is mainly composed of a storage device such as RAM in the ECU, but it can also be equipped with an external server or database.

[0017] The memory unit 40 stores control programs for operating the occupant recognition device 30, application programs including occupant recognition programs, etc. The memory unit 40 also temporarily or permanently stores thresholds and parameters used in the control unit 31. The memory unit 40 pre-stores the three-dimensional vehicle model of the vehicle, the three-dimensional coordinates of the vehicle model such as the first lighting unit 12, the second lighting unit 13, the lens 14, the headrest, and the region 60, regression parameters, etc. The memory unit 40 temporarily stores the first image, the second image, the third image, the two-dimensional coordinates of the occupant's skeletal points and facial feature points on the image, the three-dimensional coordinates of the occupant's skeletal points and facial feature points on the vehicle model, provisional calibration values, calibration values, calibration status, etc. The two-dimensional coordinates are, for example, the positions of the occupant's skeletal points and facial feature points, identified on an image coordinate system defined with the upper left of the image as the origin, the X-axis in the left-right direction from the origin, and the Y-axis in the up-down direction from the origin. Two-dimensional coordinates are used to estimate the position of skeletal points and other elements on a two-dimensional image. Three-dimensional coordinates, for example, are the positions of occupant skeletal points and facial feature points identified on a vehicle coordinate system defined by, for example, the vehicle's rear axle center as the origin, the vehicle's width as the y-axis, the vehicle's length as the x-axis, and the vehicle's height as the z-axis. For example, the vehicle coordinate system is used to estimate or correct the position of skeletal points and other elements in a vehicle model that represents the vehicle in three dimensions.

[0018] The "Calibration Status" transitions according to the execution stage of the crew recognition process, as shown in Figure 4. The initial value of the calibration status is set to "Uncalibrated". When a crew member is detected from the third image, the calibration status changes to "Calibration Requested". Subsequently, when predetermined conditions are met, the calibration status changes to "Calibration Image Requested", which requests the capture of the first and second images. Once the images are acquired, the status changes to "Calibrating" in order to calculate the calibration value. Once the calibration value is calculated, the calibration status changes to "Calibrated". Subsequently, when predetermined conditions are met, the calibration status changes to "Applied" in order to apply the calibration value. When a crew member is no longer detected during the process, the calibration status changes to "Uncalibrated".

[0019] The occupant recognition device 30 recognizes the occupant's posture and corrects it if it is not appropriate. The necessity of posture correction is explained below with reference to Figure 3. Figure 3(a) is a schematic diagram showing the occupant's two-dimensional skeleton and the position of the headrest, acquired based on a third image captured by the imaging device 10. The two-dimensional skeleton is the occupant's skeleton estimated from the image and is represented as points or a collection of points on a two-dimensional coordinate system, for example, the image coordinate system. Figure 3(b) is a side view of the three-dimensional skeleton created based on the two-dimensional skeleton. The three-dimensional skeleton is the occupant's skeleton estimated from the two-dimensional skeleton and is represented as points or a collection of points on a three-dimensional coordinate system, for example, the vehicle coordinate system. Figure 3(c) is a side view of the corrected three-dimensional skeleton. In Figure 3, 1' is the two-dimensional skeleton of the occupant's entire body, 2' is the two-dimensional skeleton of the head, and 3' is a line representing the position of the headrest. 1,1A,1B represent the three-dimensional skeleton of the entire occupant's body, 2,2A,2B represent the three-dimensional skeleton of the head, and 3 is a line indicating the position of the headrest.

[0020] Since the images captured by the imaging device 10 are two-dimensional images, errors may occur in the coordinates in the depth direction when estimating the three-dimensional skeleton of the occupant based on the two-dimensional skeleton of the occupant obtained from the two-dimensional image. Due to this error, the estimated three-dimensional skeleton 1A, shown as a dashed line in Figure 3(b), is generated with a shifted head position compared to the original three-dimensional skeleton 1, shown as a solid line. In other words, when creating the three-dimensional skeleton, the occupant's waist is used as a reference point, and the three-dimensional skeleton is created by applying it to the normal posture. As a result, even though the occupant is actually slightly bent forward with their head in front, as shown by the solid line, a three-dimensional skeleton is generated that appears as if the head is in contact with the headrest, affecting the accuracy of the occupant's posture recognition.

[0021] The crew recognition device 30 corrects the estimated three-dimensional skeleton 1A based on a part of the body, specifically the position of the ear, to generate a three-dimensional skeleton 1B, shown by the dashed line in Figure 3(c), which is closer to the original three-dimensional skeleton 1. This enables the crew recognition device 30 to recognize the crew's posture more accurately. Details of the configuration of the crew recognition device 30 and the processing of each part are described below with reference to the drawings.

[0022] The control unit 31 functions as an image acquisition unit 32, an estimation unit 33, a determination unit 34, a calculation unit 35, a correction unit 36, and a notification unit 27. When each unit of the control unit 31 performs processing using the first to third positions, the first to third points, the first to third images, thresholds, regression parameters, etc., these are treated as data for processing.

[0023] The image acquisition unit 32 sequentially outputs instructions to the imaging device 10 to capture a first image and an instruction to capture a second image. The image acquisition unit 32 acquires a first image taken from the imaging device 10 by illuminating the interior of the vehicle with infrared light from a first position L and capturing the occupants, and a second image taken from the imaging device 10 by illuminating the interior of the vehicle with infrared light from a second position R and temporarily stores them in the storage unit 40. The image acquisition unit 32 acquires a third image output from the imaging device 10 at predetermined time intervals and temporarily stores it in the storage unit 40.

[0024] The estimation unit 33 estimates the posture of the occupants based on a third image taken of the occupants. Specifically, the estimation unit 33 acquires a third image from the memory unit 40 as shown in Figure 2B. The estimation unit 33 performs image recognition on the acquired third image and detects the presence of occupants. The estimation unit 33 detects the presence of occupants seated in each seat of the first row. Upon detecting the presence of occupants, the estimation unit 33 estimates the two-dimensional coordinates of the skeletal points and facial feature points of each occupant on the image, as shown in Figure 3(a), based on the third image. The estimation unit 33 temporarily stores the estimated two-dimensional coordinates of the skeletal points and facial feature points for each occupant in the memory unit 40 and changes the calibration status to "calibration requested".

[0025] When the calibration status is "uncalibrated," "calibration image requested," "calibration requested," or "calibrated," the estimation unit 33 estimates the three-dimensional coordinates of the crew member's skeletal points and facial feature points based on the third image using known skeletal estimation techniques. The estimation unit 33 temporarily stores the estimated three-dimensional coordinates of the skeletal points and facial feature points in the storage unit 40 as the three-dimensional coordinates of the uncorrected skeletal points and facial feature points.

[0026] The determination unit 34 determines whether the difference between the position of the crew member in an image taken at a first time and the position of the crew member in an image taken at a second time is greater than a threshold. The determination unit 34 repeatedly retrieves a third image stored at predetermined intervals from the storage unit 40 and detects the position of the crew member's head through image analysis. For example, the determination unit 34 calculates the difference between the head position detected from a third image taken at a certain time and the head position detected from a third image taken at a time earlier than that time. The difference in the crew member's position between the first and second time points can also be called the change in the crew member's position. For example, a certain time point corresponds to the first time point, and a time point earlier than that corresponds to the second time point.

[0027] Based on the determination result, the determination unit 34 determines whether it is possible to perform calibration processing on the three-dimensional coordinates of the estimated skeletal points and facial feature points. The determination unit 34 repeatedly determines whether the amount of change in the occupant's position is greater than a threshold, and if the determination that the amount of change in the occupant's position is less than or equal to the threshold is performed for a predetermined number of consecutive times, it determines that calibration processing is possible and changes the calibration status to "calibration image requested". If the determination unit 34 has not performed judgments of less than or equal to the threshold multiple times, it determines that calibration processing is not possible because the position of the head is not stable. Thus, when the difference between the occupant's position in the image taken of the occupant at the first time and the occupant's position in the image taken of the occupant at the second time is greater than a threshold, the correction unit 36 ​​does not correct the occupant's position estimated by the estimation unit 33.

[0028] Based on the determination result, the determination unit 34 also determines whether to apply the calibration value. The determination unit 34 repeatedly determines whether the amount of change in the occupant's position is greater than a threshold, and if the determination that the amount of change in the occupant's position is less than or equal to the threshold is made for a predetermined number of consecutive times, it determines to apply the calibration value and changes the calibration status to "apply". If the determination unit 34 determines that the amount of change in the occupant's position is less than or equal to the threshold is made for fewer than a certain number of times, it determines not to apply the calibration value and changes the calibration status to "calibration requested". In this way, by checking the head movement state before calibration and before application, calibration can be performed when the head position is stable, and the occupant recognition device 30 can recognize the occupant's posture more accurately.

[0029] The calculation unit 35 calculates the width of the shadow cast around the occupant based on the difference between the first image and the second image. In this embodiment, the width of the shadow cast around the occupant is the width of the shadow of the occupant's ear projected onto the headrest, but it is not limited to the width of the ear shadow; it can be the width of the shadow of any part of the body, such as the head, face, arms, or torso. When the calculation unit 35 detects the occupant in the estimation unit 33, it instructs the image acquisition unit 32 to acquire the first and second images. The calculation unit 35 retrieves the first and second images stored by the image acquisition unit 32 from the storage unit 40 and creates a difference image between them. Figure 5 is a schematic diagram showing the process of creating a difference image by subtracting the first and second images. The image shown in Figure 5(a) is a schematic diagram of each image taken by the imaging device 10. The images shown in Figure 5(b) are images cropped from the first and second images near the occupant's face. In the example in Figure 5, the occupant is on the left side facing the interior of the vehicle, so a shadow is formed in the first image, which is taken by illuminating the interior of the vehicle with the left LED, which is the first lighting unit 12. When the occupant is on the right side facing the interior of the vehicle, a shadow is formed in the second image, which is taken by illuminating the interior of the vehicle with the right LED, which is the second lighting unit 13. By creating a difference image, the dark areas common to both images other than the shadow are removed, and the shadowed areas are emphasized.

[0030] The calculation unit 35 obtains the image coordinates of a first point Et above the occupant's ear and a second point Eb below the ear, as shown in Figure 6, from the skeletal points and facial feature points of the occupant on the third image estimated by the estimation unit 33. In Figure 6, the vertical direction of the paper is the vertical axis of the image, and the horizontal direction of the paper is the horizontal axis of the image. Based on the image coordinates of the first point Et and the second point Eb, the calculation unit 35 scans pixels within a predetermined scanning range from left to right, with a predetermined width, at predetermined intervals and a predetermined number of times below and above the first point Et and the second point Eb, respectively. Through this scanning, the calculation unit 35 measures the width of the shadows above and below the ear, respectively. The unit of shadow width is pixels [px]. The number of scans is, for example, 3, and the vertical spacing between multiple scan lines is, for example, several pixels to several tens of pixels.

[0031] The scanning direction of the image is determined according to the positions of the first illumination unit 12 and the second illumination unit 13. When the first illumination unit 12, the second illumination unit 13 and the lens 14 are arranged side by side along the left-right axis of the image sensor of the imaging unit 11, as in this embodiment, the shadow cast on the image extends in the left-right axis direction of the image. For this reason, the calculation unit 35 scans the difference image in the left-right axis direction. When the first illumination unit 12, the second illumination unit 13 and the lens 14 are arranged side by side along the up-down axis of the image sensor of the imaging unit 11, the shadow cast on the image extends in the up-down axis direction of the image. For this reason, the calculation unit 35 scans the difference image in the up-down axis direction.

[0032] The calculation unit 35 averages the width of each shadow measured a predetermined number of times in the upper and lower parts of the ear obtained by scanning, for each point Et and Eb. The calculation unit 35 sets the average value as the first width Lt and second width Lb of the shadow at the first point Et and the second point Eb.

[0033] The calculation unit 35 estimates the distance [m] between the ear and the headrest based on the calculated shadow widths Lt and Lb of the upper and lower parts of the ear. Referring to Figure 7, the calculation unit 35 obtains regression parameters from the storage unit 40 and calculates the distance [m] between the ear and the headrest corresponding to the shadow width [px] of the ear. For the first point Et and the second point Eb, the calculation unit 35 calculates the first distance Dt and the second distance Db to the headrest based on the first width Lt and the second width Lb and the regression parameters. The calculation unit 35 obtains seat position information from the position sensor 20, corrects the first distance Dt and the second distance Db based on the position information, and temporarily stores it in the storage unit 40. Since the front-to-back position of the headrest changes when the front-to-back position of the seat changes, it is preferable to perform such correction. Since the position of the headrest also changes depending on the angle of the seat backrest, it is also preferable for the calculation unit 35 to obtain the angle of the backrest and correct the first distance Dt and the second distance Db based on the angle.

[0034] The regression parameters are calculated in advance according to the vehicle type, etc., and are stored in the memory unit 40 when the occupant recognition system 100 is introduced. Figure 7(a) is a diagram showing the positional relationship between the first lighting unit 12, the lens 14, the ear, and the headrest, and is a top view of these. Figure 7(b) is a graph plotting the width of the ear shadow [px] and the distance between the ear and the headrest [m]. Based on the regression line calculated from this graph, the regression parameters are obtained and stored in the memory unit 40.

[0035] When the calibration state is "calibrating," the correction unit 36 ​​calculates a calibration value s to correct the estimated occupant's posture based on the width of the shadow cast around the occupant. The correction unit 36 ​​calculates the calibration value s using the first distance Dt above the ear and the second distance Db below the ear, which are calculated by the calculation unit 35 based on the shadow of the ear. The correction unit 36 ​​calculates the calibration value s by following the procedure described in Figure 8. In Figure 8, the points where straight lines extending from the first position L of the first illumination unit 12, which is the left LED, toward the first point Et above the ear and the second point below the ear intersect the surface of the headrest 3 are designated as Ht and Hb, respectively. Ht and Hb correspond to the right ends of the shadows cast by the first point Et and the second point Eb, and are therefore called the first point of impact and the second point of impact.

[0036] The correction unit 36 ​​obtains from the memory unit 40 the three-dimensional coordinates of the first position L representing the position of the first lighting unit 12 in the vehicle model, and the three-dimensional coordinates of the point cloud constituting the vehicle, including the headrest 3. Based on the obtained three-dimensional coordinates, the correction unit 36 ​​converts the two-dimensional coordinates on the image of the first landing point Ht and the second landing point Hb into three-dimensional coordinates on the vehicle model. The correction unit 36 ​​calculates the three-dimensional coordinates of the points moved from the three-dimensional coordinates of the first landing point Ht and the second landing point Hb to the three-dimensional coordinates of the first position L by the first distance Dt and the second distance Db between the upper and lower parts of the ear and the headrest. The correction unit 36 ​​uses the calculated three-dimensional coordinates as the three-dimensional coordinates of the first point Et above the ear and the second point Eb below the ear.

[0037] The correction unit 36 ​​calculates the distance between the first point Et and the second point Eb based on the three-dimensional coordinates, and takes the calculated distance as the size of the ear e. The correction unit 36 ​​obtains the three-dimensional coordinates of the uncorrected skeletal points and facial feature points that were previously estimated based on the third image from the storage unit 40. From the obtained uncorrected three-dimensional coordinates, the correction unit 36 ​​obtains the three-dimensional coordinates of the first point Et' at the top of the ear and the second point Eb' at the bottom of the ear, and calculates the distance e' between the first point Et' and the second point Eb'. The correction unit 36 ​​calculates the calibration value s using the following formula.

[0038] s = e / e'

[0039] When the calibration status is "applied," the correction unit 36 ​​corrects the three-dimensional coordinates of the skeletal points and facial feature points estimated in the procedure shown in Figure 9 using the calibration value s. This correction process may be performed after calculating all the uncorrected skeletal points of the occupant's body, or it may be performed immediately after calculating the three-dimensional coordinates of each skeletal point.

[0040] As shown in Figure 9(a), the correction unit 36 ​​corrects the three-dimensional coordinates of the uncorrected points F' of the skeletal points around the body parts used to calculate the calibration value s (hereinafter referred to as "first corrected skeletal points") and calculates the three-dimensional coordinates of the corrected point F. In this embodiment, since the calibration value s is calculated based on the width of the shadow of the ear, the skeletal points around the ear, such as the eyes, nose, and mouth, are considered to be the uncorrected points F'. The three-dimensional coordinates of the corrected point F are calculated as coordinates extended by xs, which is the distance x between the three-dimensional coordinates of the third position P, which is the position of a predetermined point on the lens 14, towards the uncorrected points F', multiplied by the calibration value s. The correction unit 36 ​​corrects the three-dimensional coordinates of the uncorrected points F' of skeletal points that are not around the body parts used to calculate the calibration value s, such as the arms, torso, and legs (hereinafter referred to as "second corrected skeletal points") and calculates the three-dimensional coordinates of the corrected point F.

[0041] Because the uncorrected point F' of the first corrected skeletal point is corrected to the corrected point F of the first corrected skeletal point, an inappropriate positional relationship may occur between the corrected point F of the first corrected skeletal point and the uncorrected point F' of the second corrected skeletal point. The three-dimensional skeleton 1A shown by the dashed line in Figure 9(b) is the three-dimensional skeleton of the corrected skeletal point. In this three-dimensional skeleton 1A, the position of the neck is unnaturally bent, resulting in a posture that cannot occur in reality. The correction unit 36 ​​corrects the estimated posture of the occupant so that a part of the occupant's body is positioned at a position corresponding to the width of the shadow cast around the occupant. The correction unit 36 ​​corrects the estimated posture of the occupant so that a part of the occupant's body is positioned at a position corresponding to the width of the shadow cast around the occupant by multiplying the position of the part of the body before correction by a calibration value s, which is a value corresponding to the width of the shadow cast.

[0042] Specifically, the inherent tolerance ranges for lengths and angles between skeletal points are pre-set and stored in the memory unit 40. Examples of tolerance ranges include the length between the head and torso, and the angle of the neck. The correction unit 36 ​​detects skeletal points that exceed the tolerance range, for example, the skeletal point at the base of the head and the base of the neck where the neck angle exceeds the tolerance range, and corrects the three-dimensional coordinates of the uncorrected skeletal points, for example, the three-dimensional coordinates of the base of the neck, to a position that does not exceed the tolerance range. As a result, as shown in Figure 9(c), the corrected three-dimensional skeleton 1B takes on a posture close to that of the actual three-dimensional skeleton 1.

[0043] The notification unit 37 outputs a predetermined notification based on the corrected occupant's posture. The notification unit 37 determines whether the occupant's body is located in area 60, which is an area where it is not recommended for any part of the occupant's body to be present, and outputs a predetermined notification. The notification unit 37 outputs a predetermined notification when the occupant is located in an area where it is not recommended for any part of the occupant's body to be present. For example, area 60 is shown by a dashed line in Figure 10. In Figure 10, 1B is the corrected three-dimensional skeleton, and 3, 4, and 5 are simplified headrest, backrest, and seat cushion, respectively. Area 60 is a closed, three-dimensional area defined in the three-dimensional coordinate system of the vehicle model. The three-dimensional coordinates of area 60 are pre-stored in the memory unit 40. If the notification unit 37 determines that a three-dimensional skeletal point representing a specific part of the occupant's body, for example, the skeletal point BP of the chest, is located within area 60 as shown in Figure 10(b), it outputs a notification signal to the external device 50 indicating this. If the notification unit 37 determines that the skeletal point BP in the chest is not located within area 60 as shown in Figure 10(a), it does not notify the external device 50. For example, area 60 is the area occupied by the airbag when the airbag deploys, and it is not recommended that any part of the occupant's body be in this area.

[0044] Upon receiving the notification signal, the external device 50 uses an alerting device 51 to inform the occupant that their chest is located in area 60 and / or that they should adjust their posture appropriately, by displaying information on a display unit and / or outputting sound from a speaker. If area 60 is the area occupied by the airbag when the airbag deploys, the vehicle control device 52 controls the deployment of the airbag, the movement of the seat, and / or the driving of the vehicle.

[0045] An example of the operation of the crew recognition system 100 according to the first embodiment will be described below with reference to the flowcharts in Figures 12 to 16. The flowcharts in Figures 12 to 16 show an example of the operation performed by the crew recognition device 30. The operation of the crew recognition system 100 and the crew recognition device 30 is not limited to the operation described below. The crew recognition device 30 mainly performs the image acquisition process shown in Figure 12 and the crew recognition process shown in Figure 13. The image acquisition process and the crew recognition process are each executed repeatedly in parallel.

[0046] The image acquisition process is described below with reference to the flowchart in Figure 12. The processing of Loop 1 in steps S1 to S8 in Figure 12 is repeated while the vehicle's engine is running. In step S1, the control unit 31 determines whether any of the calibration states in each seat of the first row is "calibration image requested". If the determination result is YES, the control unit 31 advances the program to step S2. If the determination result is NO, the control unit 31 advances the program to step S7. The calibration state is set by the occupant recognition process described later. The calibration state is changed to "calibration image requested" when the amount of movement of the skeletal points of the head is determined to be below a threshold a predetermined number of times consecutively.

[0047] Step S2 is executed when the calibration status of any seat in the first row is "Calibration image requested". In step S2, the image acquisition unit 32 sends a signal to the imaging device 10 instructing it to take the first image. In step S3, the image acquisition unit 32 acquires the first image from the imaging device 10 and temporarily stores it in the storage unit 40. In step S4, the image acquisition unit 32 sends a signal to the imaging device 10 instructing it to take the second image. In step S5, the image acquisition unit 32 acquires the second image from the imaging device 10 and temporarily stores it in the storage unit 40. In step S6, the image acquisition unit 32 changes the calibration status of the seats whose calibration status is "Calibration image requested" to "Calibration in progress". The program then returns to step S1 and the processing of loop 1 is repeated.

[0048] Step S7 is executed when the calibration status is anything other than "Calibration Image Request". In step S7, the image acquisition unit 32 sends a signal to the imaging device 10 instructing it to take a third image. In step S8, the image acquisition unit 32 acquires the third image from the imaging device 10 and temporarily stores it in the storage unit 40. The program then returns to step S1, and the processing of loop 1 is repeated.

[0049] The occupant recognition process is described below with reference to the flowchart in Figure 13. The processing in Loop 2 of steps S10 to S13 in Figure 13 is repeated while the vehicle's engine is running. In step S10, the control unit 31 determines whether the calibration status of any of the seats in the first row is "calibrating". If the result is Yes, the program proceeds to the calibration process in step S12. If the result is No, the program proceeds to the estimation and calculation process in step S11.

[0050] The details of the estimation and calculation processes in step S11 are explained below with reference to the flowchart in Figure 14. In step S20, the estimation unit 33 acquires a third image from the memory unit 40. In step S21, the estimation unit 33 performs image recognition on the acquired third image and detects the presence of occupants. The processing of loop 3 from steps S22 to S36 is performed for all seats in the first row detected in step S21.

[0051] In step S22, the estimation unit 33 determines whether the presence of an occupant has been detected for the seat in question. If the determination is Yes, the presence of an occupant has been detected, so the control unit 31 advances the program to step S23 to proceed with the calibration process. If the determination is No, there is no occupant, and there is no need to perform calibration, so the control unit 31 advances the program to step S36, changes the calibration status of the seat to "uncalibrated", and terminates the processing of loop 3 for that seat. The program then returns to step S22, and the processing of loop 3 for the next seat is repeated.

[0052] In step S23, the control unit 31 determines whether the calibration status is "uncalibrated". If the determination is Yes, the control unit 31 proceeds to step S24 and changes the calibration status of the seat to "calibration requested". The program then proceeds to step S25. If the determination is No, the calibration status is one of "calibrated", "calibration requested", or "calibration image requested", so the control unit 31 skips step S24 and proceeds to step S25.

[0053] In step S25, the estimation unit 33 estimates the two-dimensional coordinates of the skeletal points and facial feature points on the image of the occupant seated in the seat, based on the third image. In step S26, the estimation unit 33 temporarily stores the estimated two-dimensional coordinates of the skeletal points and facial feature points in the storage unit 40. The program then proceeds to step S27, which involves starting the calibration and finalizing process.

[0054] The details of the calibration start and confirmation process in step S27 are explained below with reference to the flowchart in Figure 15. In step S40, the control unit 31 determines whether the calibration status is "Calibration Request". If the determination is Yes, the control unit 31 advances the program to step S41 to determine whether calibration can be performed. If the determination is No, the control unit 31 advances the program to step S44 to determine whether the calibration results can be applied.

[0055] In step S41, the determination unit 34 acquires a third image from the memory unit 40 and calculates the change in the occupant's position. In step S42, the determination unit 34 determines whether calibration is possible. Specifically, if the determination that the change in the occupant's position is below a threshold is made for a predetermined number of consecutive times, the determination unit 34 determines that calibration is possible and verifies "Yes". The program then proceeds to step S43. If the determination unit 34 determines that the change in the occupant's position is below a threshold is made for fewer than a certain number of times, it determines that calibration is not possible and verifies "No". The program then proceeds to termination, and the processing in step S27 is completed. In step S43, the control unit 31 changes the calibration status to "Calibration image requested". The program then proceeds to termination, and the processing in step S27 is completed. When the calibration status is changed to "Calibration image requested", the determination in step S1 of the image acquisition process becomes "Yes", and the processes of acquiring the first and second images in steps S2 to S6 are executed.

[0056] In step S44, the control unit 31 determines whether the calibration status is "calibrated". If the determination is Yes, a provisional calibration value has been calculated in the calibration process described later, so the control unit 31 proceeds to step S45 to determine whether the calibration value can be applied. If the determination is No, the calibration status is "applied" or "calibration image requested", and the applicability of the calibration is not determined, the program proceeds to termination, and the processing in step S27 ends. In step S45, the determination unit 34 acquires a third image from the storage unit 40 and calculates the change in the occupant's position. In step S46, the determination unit 34 determines whether to apply the calibration result. Specifically, if the determination that the change in the occupant's position is below a threshold has been made for a predetermined number of consecutive times, the determination unit 34 determines Yes, meaning that the calibration result will be applied. The program then proceeds to step S47, and the control unit 31 changes the calibration status to "applied". In step S48, the determination unit 34 stores the provisional calibration value in the storage unit 40 as the official calibration value. The program then proceeds to termination, and the processing in step S27 ends.

[0057] In step S46, the determination unit 34 determines "No" if the change in the occupant's position is less than or equal to the threshold number of times, and therefore does not apply the calibration value. The program then proceeds to step S49, and the control unit 31 changes the calibration status to "Calibration Request". The program then proceeds to termination, and the process in step S27 is completed.

[0058] Returning to Figure 14, in step S28, the control unit 31 determines whether the calibration status is "applied". If the determination is Yes, the control unit 31 advances the program to step S29 in order to correct the three-dimensional skeleton using the calibration values. If the determination is No, the control unit 31 advances the program to step S34.

[0059] In step S29, the correction unit 36 ​​obtains calibration values ​​s from the memory unit 40. In step S30, the correction unit 36 ​​uses the calibration values ​​s to estimate the three-dimensional coordinates of the skeletal points and facial feature points of the occupant on the vehicle model. Specifically, as described above, the correction unit 36 ​​corrects the three-dimensional coordinates of skeletal points such as the eyes, nose, and mouth, which are around the ears, and if an inappropriate positional relationship occurs, it corrects the three-dimensional coordinates of skeletal points such as the neck, waist, and arms, which are not around the ears. The program then proceeds to step S31.

[0060] In step S34, the estimation unit 33 estimates the three-dimensional coordinates of uncorrected skeletal points and facial feature points without using calibration values ​​s, based on the third image, using known skeletal estimation techniques. In step S35, the estimation unit 33 temporarily stores the estimated three-dimensional coordinates of uncorrected skeletal points and facial feature points in the storage unit 40. This completes the processing of loop 3, and the program returns to step S22, where the processing of loop 3 is repeated for the next seat.

[0061] In step S31, the notification unit 37 obtains the three-dimensional coordinates of area 60 from the memory unit 40. In step S32, the notification unit 37 determines whether the chest skeletal point BP is located within area 60. If the determination is Yes, the notification unit 37 outputs a notification signal to the external device 50 indicating that the chest skeletal point BP is located within area 60. Upon receiving the notification signal, the external device 50 notifies the occupant as described above or controls the vehicle based on the notification. This completes the processing of loop 3, and the program returns to step S22, where the processing of loop 3 is repeated for the next seat. Once the processing of loop 3 has been executed for all seats, the program proceeds to termination, and the processing of step S11 is completed.

[0062] Returning to Figure 13, the details of the calibration process in step S12 are explained below with reference to the flowchart in Figure 16. In steps S50 and S51, the calculation unit 35 acquires the first and second images from the storage unit 40. In step S52, the calculation unit 35 creates a difference image between the first and second images (see Figure 5). The processing of loop 4 in the next steps S53 to S64 is performed for all seats in the first row where occupants are detected.

[0063] In step S53, the control unit 31 determines whether the calibration status is "calibrating". If the determination is Yes, the program proceeds to step S54 to execute the calibration process. If the determination is No, there is no need to perform the calibration process, so loop 4 for that seat ends, the program returns to step S53, and loop 4 is repeated for the next seat.

[0064] In step S54, the calculation unit 35 obtains the two-dimensional coordinates of skeletal points and facial feature points on the image from the storage unit 40. These two-dimensional coordinates were estimated by the estimation unit 33 in steps S25 to S26 of the estimation and calculation processes and stored in the storage unit 40. The calculation unit 35 obtains the image coordinates of a first point Et above the crew member's ear and a second point Eb below the crew member's ear from the crew member's skeletal points and facial feature points (see Figure 6). In step S55, the calculation unit 35 calculates the scanning range for scanning the image based on the first point Et and the second point Eb. In step S56, the calculation unit 35 performs a scan of the difference image. In step S57, the calculation unit 35 averages the width of each shadow measured a predetermined number of times above and below the ear obtained by scanning for each of the first point Et and the second point Eb, and calculates the first width Lt and the second width Lb of the shadow at the first point Et and the second point Eb.

[0065] In step S58, the calculation unit 35 obtains regression parameters from the storage unit 40. In step S59, the calculation unit 35 calculates a first distance Dt and a second distance Db, which are the distances between the upper part of the ear and the headrest, based on the regression parameters and the width of the ear shadows above and below the ear (see Figure 7). In step S60, the calculation unit 35 obtains the front-rear position of the seat from the position sensor 20 and corrects the first distance Dt and the second distance Db based on the front-rear position.

[0066] In step S61, the correction unit 36 ​​obtains the three-dimensional coordinates of the uncorrected skeletal points and facial feature points of the crew from the storage unit 40. These three-dimensional coordinates were estimated by the correction unit 36 ​​and stored in the storage unit 40 in steps S34 and S35 of the estimation and calculation processes. In step S62, the correction unit 36 ​​calculates the calibration value s based on the obtained three-dimensional coordinates of the uncorrected skeletal points and facial feature points of the crew, referring to Figure 8 and following the calculation procedure described above.

[0067] In step S63, the correction unit 36 ​​temporarily stores the calculated calibration value s as a provisional calibration value in the storage unit 40. The provisional calibration value becomes the official calibration value when it is determined to apply the calibration result in steps S45 to S48 of the calibration start and confirmation process. The program then returns to step S53, and the processing of loop 4 is repeated for the next seat. Once the processing of loop 4 has been executed for all seats, the program proceeds to termination, and the processing of step S12 is completed.

[0068] Returning to Figure 13, the program returns to step S10, and the processing of loop 2 is repeated. The image acquisition process in Figure 12 and the occupant recognition process in Figure 13 are repeated while the vehicle's engine is running, and when the engine stops, the control unit 31 proceeds to terminate each program. This concludes the explanation of the operation of the occupant recognition system 100.

[0069] (Second Embodiment) The crew recognition system 100, which includes the crew recognition device 30 according to the second embodiment, has the same basic configuration and functions as those of the first embodiment shown in Figure 1, etc. In the crew recognition device 30 of the first embodiment, the calculation unit 35 scans the difference image to calculate the width of the ear shadow. In contrast, in the crew recognition device 30 of the second embodiment, the calculation unit 35 scans the difference image to calculate the two-dimensional coordinates on the image of the starting point Hs of the upper or lower part of the ear and the ending point He of the ear. Details of this calculation procedure will be explained below with reference to Figure 11.

[0070] The starting point Hs corresponds to the point in the difference image where a dark pixel that does not represent a shadow changes to a bright pixel that represents a shadow through scanning. The ending point He corresponds to the point in the difference image where a bright pixel that represents a shadow changes to a dark pixel that does not represent a shadow through scanning. In real space, the starting point Hs and ending point He represent the left and right edges of the shadow cast on the headrest, respectively. Hereafter, when distinguishing between the upper and lower parts of the ear, as shown in Figure 11(a), the starting and ending points of the upper part of the ear are designated as the first starting point Hs1 and the first ending point He1, and the starting and ending points of the lower part of the ear are designated as the second starting point Hs2 and the second ending point He2.

[0071] The calculation unit 35 uses the two-dimensional coordinates of the starting point Hs and ending point He on the image, which are the endpoints of the shadow cast on the headrest, to calculate the three-dimensional coordinates of the first point Et representing the upper part of the ear and the second point Eb representing the lower part of the ear, according to the following procedure.

[0072] The calculation unit 35 obtains from the storage unit 40 the three-dimensional coordinates of a third position P representing the position of the lens within the vehicle model, the three-dimensional coordinates of a first position L representing the position of the first illumination unit 12, and the three-dimensional coordinates of the point cloud constituting the vehicle, including the headrest 3. Based on these three-dimensional coordinates, the calculation unit 35 converts the two-dimensional coordinates on the difference image of the start point Hs and end point He into three-dimensional coordinates on the vehicle model.

[0073] As shown in Figure 11(b), the calculation unit 35 calculates a first line d1 connecting the starting point Hs and a third position P representing the position of the lens, and a second line d2 connecting the ending point He and a first position L representing the position of the first illumination unit 12. The first line d1 and the second line d2 are close together in the three-dimensional coordinate system around the three-dimensional coordinate of point E representing the ear. Figure 11(b) is a plan view from above of the first illumination unit 12, the lens 14, the ear, and the headrest, so the first line d1 and the second line d2 intersect at the position of the ear, but they do not necessarily intersect when viewed from the side. The calculation unit 35 calculates the position in the three-dimensional coordinate system where the first line d1 and the second line d2 are closest together, in other words, the position where they intersect in a plan view, and sets the three-dimensional coordinate of the position where the distance from this position to each line d1 and d2 is shortest as the three-dimensional coordinate of point E representing the ear. The calculation unit 35 calculates the three-dimensional coordinates of the first point Et above the ear using the first starting point Hs1 and the first ending point He1 above the ear. The calculation unit 35 calculates the three-dimensional coordinates of the second point Eb below the ear using the second starting point Hs2 and the second ending point He2 below the ear. Using the calculated three-dimensional coordinates of the first point Et and the second point Eb, the calculation unit 35 calculates a calibration value s using the same procedure and formula as described in the first embodiment with reference to Figure 8. In this way, the correction unit 36 ​​corrects the posture of the occupant estimated by the estimation unit 33 so that a part of the occupant's body is located at the position where the first line d1, which connects the position Hs of the first point which is the edge of the shadow and the position P of the lens which represents the position of the imaging device that photographed the occupant, and the second line d2, which connects the position He of the second point which is the edge of the shadow and the first position L which represents the position of the first illumination unit 12, are closest to each other inside the vehicle.

[0074] As described above, the occupant recognition device 30 according to each embodiment includes: an image acquisition unit 32 that acquires a first image of the occupant taken by illuminating the inside of the vehicle with infrared light from a first position L and a second image of the occupant taken by illuminating the inside of the vehicle with infrared light from a second position R; an estimation unit 33 that estimates the occupant's posture based on the images of the occupant; a calculation unit 35 that calculates the width of the shadow cast around the occupant based on the difference between the first image and the second image; a correction unit 36 ​​that corrects the estimated occupant's posture based on the width of the shadow cast around the occupant; and a notification unit 37 that outputs a predetermined notification based on the corrected occupant's posture.

[0075] The occupant recognition method performed by the control unit 31 of the occupant recognition device 30 according to each of the above embodiments includes: an image acquisition step of acquiring a first image of the occupant taken by illuminating the inside of the vehicle with infrared light from a first position L and a second image of the occupant taken by illuminating the inside of the vehicle with infrared light from a second position; an estimation step of estimating the occupant's posture based on the images of the occupant; a calculation step of calculating the width of the shadow cast around the occupant based on the difference between the first image and the second image; a correction step of correcting the estimated occupant's posture based on the width of the shadow cast around the occupant; and a notification step of outputting a predetermined notification based on the corrected occupant's posture.

[0076] With the above configuration, the occupant recognition device 30 and occupant recognition method according to the above embodiment can recognize the occupant's posture more accurately. Since the occupant recognition device 30 and occupant recognition method according to each of the above embodiments use a single imaging device 10 placed inside the vehicle, the occupant's posture can be recognized more accurately with a lower cost and simpler configuration.

[0077] The occupant recognition device 30 in each of the above embodiments further includes a determination unit 34 that determines whether the amount of change between the occupant's position in an image taken of the occupant at a first time and the occupant's position in an image taken of the occupant at a second time is greater than a threshold. The correction unit 36 ​​does not correct the estimated occupant's position when the difference between the occupant's position in an image taken of the occupant at a first time and the occupant's position in an image taken of the occupant at a second time is greater than the threshold. In this way, the correction unit 36 ​​does not perform correction when the occupant has moved significantly, thereby preventing the posture from being corrected when the occupant's body position is unstable.

[0078] The correction unit 36 ​​and correction process in the occupant recognition device 30 of the first embodiment described above correct the estimated occupant's posture so that a part of the occupant's body is positioned at a location corresponding to the width of the shadow cast around the occupant. The correction unit 36 ​​and correction process can correct the occupant's posture more efficiently with fewer steps and less information.

[0079] The correction unit 36 ​​and correction process in the occupant recognition device 30 of the second embodiment described above correct the estimated occupant's posture so that a part of the occupant's body is located at the position closest to the first line d1, which connects the starting point Hs, which is the edge of the shadow, and the third position P, which is the position of the imaging device 10 that photographed the occupant, and the second line d2, which connects the ending point He, which is the edge of the shadow, and the first position L, within the vehicle. The correction unit 36 ​​and correction process can correct the posture by calculation without obtaining regression parameters in advance.

[0080] While embodiments of this disclosure have been described in detail above with reference to the drawings, the specific configurations are not limited to these embodiments, and design modifications that do not depart from the gist of this disclosure are included.

[0081] In the embodiments described above, infrared light is used as the illumination light, but as a modification, visible light such as white light may be used. In this case, the imaging device 10 captures a visible image, such as a color image or a black and white image, of the interior of the vehicle illuminated by visible light from both the first illumination unit 12 and the second illumination unit 13, or by visible light from either one of them.

[0082] In each of the above embodiments, the occupant recognition device 30 corrects the occupant's posture based on the shadow of the occupant's ear projected onto the headrest and notifies when the skeletal point BP of the chest, which is a predetermined part of the body, is located in region 60. In contrast, the modified occupant recognition device 30 may correct the occupant's posture based on the shadow of the back, which is a part of the body, projected onto the backrest, and notify when the skeletal point BP of the chest or a part of the body other than the chest is located in region 60.

[0083] In each of the above embodiments, the occupant recognition device 30 notifies whether the skeletal point BP of the chest of the corrected three-dimensional skeleton is located in region 60. In contrast, the modified occupant recognition device 30 may detect and notify whether the degree to which the occupant's legs are off the seat surface is above a threshold, or whether the occupant's legs are on the dashboard, using a position sensor, image recognition, etc. The degree to which the legs are off the seat surface can be determined by whether the occupant's thighs are located above a predetermined distance from the seat surface, etc. [Explanation of Symbols]

[0084] 10: Imaging device, 27: Notification unit, 30: Crew recognition device, 31: Control unit, 32: Image acquisition unit, 33: Estimation unit, 34: Determination unit, 35: Calculation unit, 36: Correction unit, 37: Notification unit, Et: First point, Eb: Second point, L: First position, R: Second position, P: Third position, d1: First line, d2: Second line

Claims

1. An image acquisition unit that acquires a first image of the occupants taken by illuminating the interior of the vehicle with infrared light from a first position, and a second image of the occupants taken by illuminating the interior of the vehicle with infrared light from a second position, An estimation unit that estimates the posture of the crew member based on an image of the crew member, A calculation unit that calculates the width of the shadow cast around the occupant based on the difference between the first image and the second image, A correction unit that corrects the estimated posture of the occupant based on the width of the shadow cast around the occupant, A notification unit that outputs a predetermined notification based on the corrected occupant's posture, Crew recognition device equipped with a crew recognition system.

2. The system further includes a determination unit that determines whether the difference between the position of the occupant in an image taken of the occupant at a first time and the position of the occupant in an image taken of the occupant at a second time is greater than a threshold. The crew recognition device according to claim 1, characterized in that the correction unit does not correct the estimated crew position when the difference between the position of the crew member in the image of the crew member taken at the first time and the position of the crew member in the image of the crew member taken at the second time is greater than the threshold.

3. The occupant recognition device according to claim 1, characterized in that the correction unit corrects the estimated occupant's posture so that a part of the occupant's body is positioned at a location corresponding to the width of the shadow cast around the occupant.

4. The occupant recognition device according to claim 1, characterized in that the correction unit corrects the estimated occupant's posture such that, within the vehicle, a part of the occupant's body is positioned at the closest point to a first line connecting the position of a first point which is the edge of the shadow and the position of the imaging device that photographed the occupant, and a second line connecting the position of a second point which is the edge of the shadow and the first position.

5. A crew recognition method performed in the control unit of a crew recognition device, Image acquisition steps include obtaining a first image obtained by illuminating the interior of the vehicle with infrared light from a first position and photographing the occupants, and a second image obtained by illuminating the interior of the vehicle with infrared light from a second position and photographing the occupants, An estimation step of estimating the posture of the crew member based on an image of the crew member, A calculation step of calculating the width of the shadow cast around the occupant based on the difference between the first image and the second image, A correction step of correcting the estimated posture of the occupant based on the width of the shadow cast around the occupant, A notification step that outputs a predetermined notification based on the corrected occupant's posture, Crew recognition method including

6. The process further includes a determination step of determining whether the difference between the position of the occupant in an image taken of the occupant at a first time and the position of the occupant in an image taken of the occupant at a second time is greater than a threshold, The occupant recognition method according to claim 5, characterized in that the correction step does not correct the estimated occupant's position when the difference between the occupant's position in the image of the occupant taken at the first time and the occupant's position in the image of the occupant taken at the second time is greater than the threshold.

7. The occupant recognition method according to claim 5, characterized in that the correction step corrects the estimated occupant's posture so that a part of the occupant's body is positioned at a location corresponding to the width of the shadow cast around the occupant.

8. The occupant recognition method according to claim 5, characterized in that the correction step corrects the estimated occupant's posture such that, within the vehicle, a part of the occupant's body is positioned at the closest point to a first line connecting the position of a first point which is the edge of the shadow and the position of an imaging device that photographed the occupant, and a second line connecting the position of a second point which is the edge of the shadow and the first position.

Images