Occupant monitoring device, occupant monitoring system, and occupant monitoring method

Abstract

An occupant monitoring device according to the present invention comprises a correction unit (30) having: a target marker detection unit (31) that detects a target marker from a first captured image that is captured by an image-capture device and includes said target marker, which is provided in a vehicle interior and reflects infrared light; a deviation amount calculation unit (32) that calculates a translational deviation amount and a rotational deviation amount with respect to a reference position of the detected target marker; and a correction value calculation unit (33) that calculates, as the correction value on the basis of the calculated translational deviation amount and rotational deviation amount, a translational deviation amount and a rotational deviation amount of the image-capture device with respect to the vehicle. The occupant monitoring device also comprises a monitoring unit (20) that implements a sensing process in which the calculated correction value is used to sense an occupant of the vehicle from a second captured image captured by the image-capture device and including the occupant.

Description

[Technical field]

[0001] The present disclosure relates to occupant monitoring technology. [Background technology]

[0002] Patent Document 1 discloses a camera having an autofocus function fixed in a vehicle cabin; a luminance acquisition unit that acquires a luminance value of a reference position fixed in the vehicle cabin and a luminance value of the user in a first image captured by the camera at a first time point while focusing on a user; a distance acquisition unit that acquires (a) a first distance between the camera and the reference position and (b) a second distance between the camera and the user obtained from the focus position of the camera at the first time point; and a predetermined relational expression that is satisfied by a distance between an image capture target and the camera and a luminance value in an image captured by the camera of the image capture target, The present invention discloses a technology related to an imaging device that includes a calibration unit that calculates a ratio between a first coefficient for the reference position at the first time point and a second coefficient for the user at the first time point using a luminance value of the reference position, a luminance value of the user, the first distance, and the second distance for a relational equation including a coefficient determined at least in part according to an imaging subject, and a calculation unit that calculates a third distance between the camera and the user at the second time point using the luminance values ​​of the reference position and the user in a second image captured by the camera at a second time point different from the first time point, the first distance, and the ratio. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-79712 A Summary of the Invention [Problem to be solved by the invention]

[0004] According to the imaging device of Patent Document 1, calibration is performed using brightness values, so it is possible to calibrate the distance from the imaging device to the object to be imaged, but there is a problem in that it is not possible to calibrate the angle that may occur when the imaging device is installed in a vehicle.

[0005] The present disclosure has been made to solve such problems, and has an object to provide an occupant monitoring technology that is capable of calibrating the angle. [Means for solving the problem]

[0006] An occupant monitoring device according to an embodiment of the present disclosure includes a correction unit including a target marker detection unit that detects a target marker that reflects infrared light and is provided inside a vehicle cabin and is captured by an imaging device, a deviation calculation unit that calculates a translational deviation and a rotational deviation of the detected target marker from a reference position, and a correction value calculation unit that calculates a translational deviation and a rotational deviation of the imaging device relative to the vehicle as correction values ​​based on the calculated translational deviation and rotational deviation, and a monitoring unit that performs a sensing process to sense an occupant of the vehicle in a second captured image including the occupant captured by the imaging device using the calculated correction value. Effect of the Invention

[0007] According to an occupant monitoring device according to an embodiment of the present disclosure, the angle can be calibrated. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the configuration of an occupant monitoring device and an occupant monitoring system according to a first embodiment. [Figure 2A] FIG. 2 is a diagram illustrating an example of a hardware configuration of an occupant monitoring device. [Figure 2B] FIG. 2 is a diagram illustrating an example of a hardware configuration of an occupant monitoring device. [Diagram 3] 1 is a schematic diagram of the interior of a vehicle illustrating an application example of an occupant monitoring device and an occupant monitoring system. [Figure 4] FIG. 2 is a diagram showing an example of a captured image of the inside of a vehicle. [Diagram 5] 4 is a flowchart of an occupant monitoring method relating to a correction process. [Figure 6] 4 is a flowchart of an occupant monitoring method relating to an update process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Various embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that components with the same or similar reference numerals in the drawings have the same or similar configurations or functions, and redundant descriptions of such components will be omitted.

[0010] Embodiment 1 <Configuration> (Occupant monitoring system) As shown in FIG. 1, the passenger monitoring system 1 includes, for example, a passenger monitoring device 10, an imaging device 11, and a light source 12. The light source 12 is, for example, an infrared light emitting diode (IR-LED). The light source 12 irradiates near-infrared light having a wavelength of, for example, 800 nm to 1,000 nm when performing aiming. The near-infrared light irradiated from the light source 12 is reflected by a target marker TM (described later) provided inside the vehicle. The imaging device 11 is an infrared camera, and receives light having a wavelength of, for example, 400 nm to 1,000 nm (visible light to near-infrared light) to generate a captured image. Note that the light source 12 is an optional component, since it is only necessary to detect the target marker TM when performing aiming for the passenger monitoring system 1 including the imaging device 11. Note that aiming refers to correcting an error in assembling the imaging device 11 to the vehicle or an error in the optical axis of the imaging device 11.

[0011] (Occupant monitoring device) As shown in Fig. 1, the passenger monitoring device 10 includes a monitoring unit 20 that monitors the driver and a correction unit 30 that calculates a correction value used in processing by the monitoring unit 20. The monitoring unit 20 is a functional unit used when the vehicle is used for traveling, and detects a vehicle occupant when the vehicle is stopped or traveling after the ignition of the vehicle is turned on. The monitoring unit 20 includes a driver detection unit 21 and a sensing processing unit 22. The correction unit 30 is a functional unit used when inspecting the imaging device 11, and operates upon receiving an aiming start signal from an external device such as an inspection machine 7 described later. The correction unit 30 includes a target marker detection unit 31, a deviation amount calculation unit 32, and a correction value calculation unit 33.

[0012] (Driver detection section) The driver detection unit 21 detects various characteristics of the driver from a captured image (sometimes referred to as a second captured image) including the driver, who is an occupant, captured by the imaging device 11, using a known technique, and supplies the detection result as a driver detection result to the sensing processing unit 22. Examples of the driver detection result include driver characteristics such as face direction, line of sight direction, driver position, eye opening degree, and opening degree of the eyes.

[0013] For example, the driver detection unit 21 detects the gaze and the facial direction by calculating the shape of the facial parts of the occupant's face, such as the face, eyes, and pupils, and the amount of change over time of the facial parts from the captured image. The gaze can be calculated, for example, by using the position of the pupil relative to the positions of the inner corner and outer corner of the eye. The facial direction can be calculated from the relative positional relationship of the facial parts. The driver detection unit 21 may also detect the gaze and the facial direction by inputting the acquired captured image to a trained model (not shown) that has learned the relationship between the captured image and the gaze and the facial direction.

[0014] The degree of eye opening is the degree of opening of the eyes, and the degree of mouth opening is the degree of opening of the mouth. Taking the degree of eye opening as an example, for example, the positions of the inner corner of the eye, the outer corner of the eye, and the highest point of the upper eyelid are detected, and the distance between the straight line connecting the inner corner and the outer corner of the eye and the highest point of the upper eyelid is divided by the distance between the inner corner and the outer corner of the eye to obtain the flattening ratio of the eyes, and the degree of eye opening can be calculated as a value obtained by dividing the flattening ratio by a predetermined reference value. Since the mouth has the same shape as the eyes, the degree of eye opening can be obtained in the same manner as the degree of eye opening.

[0015] In this manner, using known techniques, the driver detection unit 21 detects various characteristics of the driver and supplies the driver detection results to the sensing processing unit 22.

[0016] (Sensing processing section) The sensing processing unit 22 applies the correction value supplied from the correction unit 30 to the driver detection result in the calculation in the driver sensing process. Using the supplied correction value, the sensing processing unit 22 corrects, for example, a two-dimensional image in an image coordinate system. Instead of correcting the two-dimensional image itself, correction may be performed on the detected line of sight or face direction.

[0017] (Target marker detection section) The target marker detection unit 31 detects the target marker TM from a captured image (sometimes referred to as a first captured image) including the target marker TM output from the imaging device 11. Here, an application example of the occupant monitoring system 1 including the occupant monitoring device 10 and characteristics of the target marker TM arranged in the vehicle will be described with reference to Figs. 3 and 4.

[0018] Fig. 3 is a schematic diagram of the interior of a vehicle cabin, in which an instrument panel 2, a steering wheel 3, an A-pillar 4, a B-pillar 5, and a seat 6 are arranged. As shown in Fig. 3, as one example, the occupant monitoring system 1 is arranged near the instrument panel 2. As another example, the occupant monitoring system 1 may be incorporated within the instrument panel 2, or may be provided on a steering column (not shown).

[0019] As shown in FIG. 3, the target marker TM is provided at an appropriate position on a structure in the vehicle interior or on a structure forming the vehicle interior. The target marker TM is a marker to be used when performing aiming. In the example of FIG. 3, the target marker TM is provided on the instrument panel 2, the A-pillar 4, the B-pillar 5, the seat 6, and the roof 8. The appropriate position refers to a position within the angle of view of the imaging device 11 and capable of being imaged by the imaging device 11. Therefore, even if the position is within the angle of view of the imaging device 11, a position that is in the shadow of the steering wheel 3 is not an appropriate position. In addition, the target marker TM is provided in the vehicle interior so that the target marker TM is reflected in a range of the angle of view that is less susceptible to the influence of the lens characteristics of the imaging device 11, such as distortion, vignetting, and reduced resolution, for example, near the center of the angle of view, thereby improving the detection accuracy of the target marker TM. At least one target marker TM may be provided, but multiple target markers TM may be provided. In addition, when the target marker TM is attached to a movable object inside the vehicle cabin, such as a seat 6, such a movable object is moved to a predetermined position under the control of a vehicle-side electronic control unit (ECU: Electronic Control Unit) not shown, or manually by an operator, when aiming is performed.

[0020] The target marker TM is provided, for example, by drawing it at an appropriate location using IR reflective ink that reflects near-infrared light. Since the light reflected by the IR reflective ink is difficult to see with the human eye, the impact of the target marker TM on the design of the vehicle interior can be reduced. Furthermore, by using IR reflective ink, the target marker TM can be placed inside the vehicle without being restricted by the shape of the structures inside the vehicle interior, so that the degree of freedom when placing the target marker TM inside the vehicle can be increased.

[0021] The shape of the target marker TM may be any shape suitable for aiming, i.e., any shape that is easy to detect. For example, a shape such as a cross, an L-shape, a star, or a triangle, which has multiple line segments extending in different directions and makes it easy to detect deviations around a rotation axis, may be adopted as the shape of the target marker TM. Instead of line segments, a shape including curves, such as a semicircle, may also be used. By accurately detecting the target marker TM, it is possible to achieve high aiming accuracy.

[0022] FIG. 4 is a diagram showing an example of an image captured by the imaging device 11 when the occupant monitoring system 1 is placed at the position shown in FIG.

[0023] 3, during the aiming process, the occupant monitoring device 10 is connected to an inspection machine 7 and used. After receiving an aiming start signal from an external device such as the inspection machine 7, the target marker detection unit 31 detects the target marker TM by performing image processing such as edge processing on the captured image output from the imaging device 11.

[0024] (Displacement calculation section) The displacement amount calculation unit 32 calculates the translational displacement amount and the rotational displacement amount of the detected target marker TM from the reference position. These displacement amounts are calculated in the image coordinate system. The translational displacement amount includes the horizontal positional displacement amount and the vertical positional displacement amount of the target marker TM from the reference position. The rotational displacement amount is the displacement amount of the rotation angle. The reference position indicates the position where the target marker TM is located in the image coordinate system, and is stored in advance in a memory (not shown) and referenced by the displacement amount calculation unit 32, or is transmitted from an external device to the displacement amount calculation unit 32 during the aiming process. The displacement amount calculation unit 32 saves the calculated translational displacement amount and rotational displacement amount of the target marker TM in a memory (not shown). Note that a memory 100c described later may be used instead of the memory (not shown). The saved translational displacement amount and rotational displacement amount of the target marker TM are used for processing by the correction value calculation unit 33.

[0025] In addition, the misalignment calculation unit 32 may compare the current translational misalignment and rotational misalignment of the target marker TM redetected at any timing with the translational misalignment and rotational misalignment of the target marker TM that have already been stored based on past detection to calculate a difference, and if the difference is greater than a certain value, update the stored translational misalignment and rotational misalignment of the target marker TM. By performing such an update process, it is possible to deal with the case where the imaging device 11 has been replaced. In other words, the correction value calculation unit 33 can calculate a correction value based on the current translational misalignment and rotational misalignment of the target marker TM after the imaging device 11 has been replaced.

[0026] (Correction value calculation section) The correction value calculation unit 33 accesses a memory (not shown) and acquires the translational misalignment amount and rotational misalignment amount of the target marker TM stored based on the processing by the misalignment amount calculation unit 32. The correction value calculation unit 33 calculates the translational misalignment amount and rotational misalignment amount of the imaging device 11 relative to the vehicle as correction values ​​based on the acquired translational misalignment amount and rotational misalignment amount of the target marker TM. The calculated misalignment amount is the misalignment amount in the vehicle coordinate system based on an arbitrary position inside the vehicle.

[0027] For example, when the target marker TM includes one line segment, the angle between both ends of the line segment and the imaging device 11 is obtained. In addition, the position and shape of the target marker TM are known. Therefore, the curve that the position of the imaging device 11 can take is determined from the angle between both ends of the line segment and the imaging device 11 and the known length of the line segment that constitutes the target marker TM. When the target marker TM includes multiple line segments, the position of the imaging device 11 can be identified as the intersection of multiple curves that the position of the imaging device 11 can take. In this way, the position of the imaging device 11 can be identified by extracting multiple feature points from the target marker TM. Therefore, the translational deviation amount and rotational deviation amount of the imaging device 11 with respect to the vehicle can be obtained based on the translational deviation amount and rotational deviation amount of the target marker TM. The correction value calculation unit 33 stores the calculated correction value in a memory not shown. The sensing processing unit 22 of the monitoring unit 20 accesses the memory not shown to acquire the calculated correction value, and applies the acquired correction value to the calculation in the sensing process.

[0028] Also, when one target marker TM is a marker that is plane-symmetrical with respect to a certain reference plane, a correction value may be calculated by extracting a plurality of feature points that constitute the target marker TM according to a known technique. More specifically, the position vector of each feature point based on the position of the imaging device is obtained, and the normal vector and equation of the reference plane are obtained using the obtained position vector, and the normal vector and equation of the constituent plane that is constituted by the plurality of feature points are obtained. The deviation amount obtained by comparing the normal vector and equation parameters of the reference plane and the normal vector and equation parameters of the constituent plane of the target marker TM used as a reference with the normal vector and equation parameters of the reference plane and the normal vector and equation parameters of the constituent plane of the detected target marker TM is calculated as a correction value.

[0029] According to the occupant monitoring device 10, the translational deviation amount and rotational deviation amount of the imaging device 11 with respect to the vehicle are calculated as correction values, so that it is possible to correct an error in the mounting angle of the imaging device 11 with respect to the vehicle. Therefore, the accuracy of the driver detection result by the occupant monitoring device 10 is improved compared to the accuracy of the driver detection result by the conventional technology.

[0030] <Hardware configuration> Next, the hardware configuration of the occupant monitoring device 10 will be described with reference to FIG. 2A and FIG. 2B. As an example, as shown in FIG. 2A, the occupant monitoring device 10 is realized by a processing circuit 100a. The processing circuit 100a realizes a driver detection unit 21, a sensing processing unit 22, a target marker detection unit 31, a deviation amount calculation unit 32, and a correction value calculation unit 33. The processing circuit 100a is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of the driver detection unit 21, the sensing processing unit 22, the target marker detection unit 31, the deviation amount calculation unit 32, and the correction value calculation unit 33 may be realized by separate processing circuits, or these functions may be realized together by one processing circuit.

[0031] As another example, as shown in FIG. 2B, the occupant monitoring device 10 is realized by a computer including a processor 100b and a memory 100c. The driver detection unit 21, the sensing processing unit 22, the target marker detection unit 31, the deviation amount calculation unit 32, and the correction value calculation unit 33 are realized by reading and executing a program stored in the memory 100c by the processor 100b. The program is realized as software, firmware, or a combination of software and firmware. Examples of the memory 100c include non-volatile or volatile semiconductor memories such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically-EPROM), magnetic disks, flexible disks, optical disks, compact disks, mini disks, and DVDs.

[0032] <Operation> (Correction process) Next, the operation of the passenger monitoring method performed by the passenger monitoring device 10 will be described with reference to Fig. 5. Fig. 5 is a flowchart of the passenger monitoring method relating to the correction process.

[0033] In step ST1, the target marker detection unit 31 detects a target marker TM from a captured image output from the imaging device 11.

[0034] In step ST2, the misalignment calculation unit 32 calculates the misalignment amounts in the horizontal direction, the vertical direction, and the rotation angle between the position of the detected target marker TM and a desired position. That is, the misalignment calculation unit 32 calculates the translational misalignment amount and the rotational misalignment amount of the detected target marker TM from a reference position.

[0035] In step ST3, the correction value calculation unit 33 calculates the correction values ​​of the horizontal direction, the vertical direction, and the rotation angle used in the sensing process. That is, based on the translational shift amount and the rotational shift amount of the target marker TM, the correction value is calculated as the translational shift amount and the rotational shift amount of the imaging device 11 with respect to the vehicle. The translational shift amount and the rotational shift amount of the target marker TM are obtained by accessing a memory (not shown).

[0036] In step ST4, the sensing processing unit 22 applies the correction value supplied from the correction unit 30 to the driver detection result in the calculation in the driver sensing process.

[0037] (Update process of deviation amount) Next, an occupant monitoring method relating to an update process by the occupant monitoring device 10 will be described with reference to FIG. 6. FIG. 6 is a flowchart of an occupant monitoring method relating to an update process of the translational deviation amount and the rotational deviation amount of the target marker TM. Steps ST11, ST12, ST15, and ST16 in FIG. 6 correspond to steps ST1, ST2, ST3, and ST4 in FIG. 5, respectively, and therefore descriptions of these steps in FIG. 6 will be omitted. Below, steps ST13 and ST14 will be described. Note that the update process in FIG. 6 is performed after the process in FIG. 5 has been previously performed and the calculated translational deviation amount and rotational deviation amount of the target marker TM are internally stored.

[0038] In step ST13, the deviation calculation unit 32 compares the translational deviation and rotational deviation of the target marker TM calculated this time with the translational deviation and rotational deviation of the target marker TM calculated in the past and stored in a memory not shown, and calculates the difference.

[0039] In step ST14, if the calculated difference is greater than a predetermined constant value, the misalignment calculation unit 32 updates the previous misalignment amount internally stored in a memory (not shown) with the translational misalignment amount and rotational misalignment amount of the target marker TM calculated this time. In step ST15, the correction value calculation unit 33 calculates a correction value using the updated translational misalignment amount and rotational misalignment amount of the target marker TM.

[0040] <Additional Notes> Certain aspects of the various embodiments described above are summarized in the following notes.

[0041] (Appendix 1) The occupant monitoring device (10) according to Appendix 1 includes a target marker detection unit (31) that detects a target marker that reflects infrared light and is provided inside a vehicle cabin and is captured by an imaging device (11), a correction unit (30) that includes a deviation calculation unit (32) that calculates a translational deviation and a rotational deviation of the detected target marker from a reference position, and a correction value calculation unit (33) that calculates a translational deviation and a rotational deviation of the imaging device relative to the vehicle as correction values ​​based on the calculated translational deviation and rotational deviation, and a monitoring unit (20) that performs a sensing process to sense the occupant in a second captured image including the occupant of the vehicle captured by the imaging device using the calculated correction value.

[0042] (Appendix 2) The occupant monitoring device (10) according to Appendix 2 is the occupant monitoring device described in Appendix 1, wherein when a difference between the currently calculated translational deviation and rotational deviation of the target marker from a reference position and the previously calculated translational deviation and rotational deviation of the target marker from a reference position is greater than a predetermined value, the correction value calculation unit calculates the correction value based on the currently calculated translational deviation and rotational deviation of the target marker from a reference position.

[0043] (Appendix 3) An occupant monitoring system (1) according to appendix 3 includes an occupant monitoring device according to appendix 1 or 2, and the imaging device.

[0044] (Appendix 4) The passenger monitoring system (1) according to appendix 3 is the passenger monitoring system described in appendix 3, further comprising a light source (12) that irradiates infrared light.

[0045] (Appendix 5) The occupant monitoring method according to Supplementary Note 5 is an occupant monitoring method by an occupant monitoring device including a correction unit (30) including a target marker detection unit (31), a deviation amount calculation unit (32), and a correction value calculation unit (33), and a monitoring unit (20), comprising: a step in which the target marker detection unit of the correction unit detects a target marker from a first captured image including a target marker that reflects infrared light and is provided in a vehicle cabin and is captured by an imaging device; a step in which the deviation amount calculation unit of the correction unit calculates a translational deviation amount and a rotational deviation amount of the detected target marker from a reference position; a step in which the correction value calculation unit of the correction unit calculates a translational deviation amount and a rotational deviation amount of the imaging device relative to the vehicle as correction values ​​based on the calculated translational deviation amount and rotational deviation amount; and a step in which the monitoring unit performs a sensing process of sensing the occupant in a second captured image including the occupant of the vehicle captured by the imaging device using the calculated correction value.

[0046] It is possible to combine the embodiments, and to modify or omit each embodiment as appropriate. [Industrial Applicability]

[0047] The occupant monitoring device of the present disclosure can be used as an occupant monitoring device for monitoring an occupant of a vehicle. [Explanation of symbols]

[0048] 1 occupant monitoring system, 2 instrument panel, 3 steering wheel, 4 A-pillar, 5 B-pillar, 6 seat, 7 inspection machine, 8 roof, 10 occupant monitoring device, 11 imaging device, 12 light source, 20 monitoring unit, 21 driver detection unit, 22 sensing processing unit, 30 correction unit, 31 target marker detection unit, 32 deviation amount calculation unit, 33 correction value calculation unit, 100a processing circuit, 100b processor, 100c memory.

Claims

1. a target marker detection unit that detects a target marker that reflects infrared light and is provided in a vehicle interior from a first captured image captured by an imaging device, the target marker being included in the first captured image; a displacement amount calculation unit that calculates a translational displacement amount and a rotational displacement amount of the detected target marker from a reference position; a correction value calculation unit that calculates a translational shift amount and a rotational shift amount of the imaging device with respect to the vehicle as correction values ​​based on the calculated translational shift amount and rotational shift amount; A correction unit comprising: a monitoring unit that performs a sensing process to sense an occupant of the vehicle in a second captured image including the occupant captured by the imaging device using the calculated correction value; and An occupant monitoring device comprising:

2. 2. The occupant monitoring device of claim 1, wherein when a difference between a currently calculated translational deviation and rotational deviation of the target marker from a reference position and a previously calculated translational deviation and rotational deviation of the target marker from a reference position is greater than a predetermined value, the correction value calculation unit calculates the correction value based on the currently calculated translational deviation and rotational deviation of the target marker from the reference position.

3. An occupant monitoring device according to claim 1 or 2; The imaging device; An occupant monitoring system.

4. 4. The passenger monitoring system of claim 3, further comprising a light source that emits infrared light.

5. An occupant monitoring method using an occupant monitoring device including a target marker detection unit, a correction unit including a displacement amount calculation unit and a correction value calculation unit, and a monitoring unit, comprising: a step in which a target marker detection unit of the correction unit detects a target marker from a first captured image captured by an imaging device, the target marker being provided in a vehicle cabin and reflecting infrared light; A step in which a deviation amount calculation unit of the correction unit calculates a translation deviation amount and a rotation deviation amount of the detected target marker from a reference position; a correction value calculation unit of the correction unit calculating, as correction values, translational shift amounts and rotational shift amounts of the imaging device with respect to the vehicle based on the calculated translational shift amounts and rotational shift amounts; a step of performing a sensing process by the monitoring unit using the calculated correction value to sense an occupant in a second captured image including the occupant of the vehicle captured by the imaging device; The occupant monitoring method includes:

Images