Vehicle interior monitoring device and vehicle interior monitoring method

Abstract

The present invention is provided with: an image acquisition unit (201) that acquires, from an imaging device (1), a captured image showing the interior of a vehicle; a vehicle state determination unit (202) that determines the state of the vehicle; a sensing processing unit (203) that, on the basis of the captured image acquired by the image acquisition unit (201), determines whether at least either the angle or the position of the imaging device (1) needs to be changed, and estimates information pertaining to the occupant when it is determined that such change is not needed; and an operation control unit (204) that changes at least either the angle or the position of the imaging device (1) when it is determined by the sensing processing unit (203) that at least either the angle or the position of the imaging device (1) needs to be changed, and also when it is determined, on the basis of the determination result by the vehicle state determination unit (202), that the occupant is not driving the vehicle.

Description

Vehicle interior monitoring device and vehicle interior monitoring method

[0001] The present disclosure relates to an interior monitoring device and an interior monitoring method for monitoring an occupant in a vehicle.

[0002] In the past, there have been known devices that detect the state of a vehicle occupant using an image captured by an imaging device. For example, Patent Document 1 discloses an in-vehicle robot that is equipped with a camera that captures images of the interior or exterior of the vehicle and determines its behavior based on the image captured by the camera.

[0003] Japanese Patent Application Laid-Open No. 2020-009075

[0004] On the other hand, in-vehicle monitoring devices generally aim to contribute to safety by detecting whether occupants are looking away or falling asleep at the wheel, so in order to detect the occupants' line of sight or the degree to which their eyes are open, it is necessary for these in-vehicle monitoring devices to ensure a certain level of resolution for the eyes.

[0005] In addition, some in-vehicle monitoring devices have a function that requires capturing an image of the entire body, i.e., a wide angle of view, such as determining the physique of an occupant and using the image for controlling an airbag. However, if the angle of view is widened for the above function, the resolution of the image in the eye area decreases, which is a problem for in-vehicle monitoring devices.

[0006] The above-mentioned Patent Document 1 describes an in-vehicle robot with a built-in camera, but does not specifically disclose how to realize functions that require wide-area sensing, such as detecting a driver's physique, while realizing functions that cannot be interrupted while driving, such as preventing inattentive driving and dozing.

[0007] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an in-vehicle monitoring device that is capable of sensing over a wider range while maintaining image resolution compared to conventional devices.

[0008] The in-vehicle monitoring device according to the present disclosure is characterized by comprising an image acquisition unit that acquires an image showing the interior of the vehicle from an imaging device; a vehicle state determination unit that determines the state of the vehicle; a sensing processing unit that determines whether it is necessary to change at least one of the angle and position of the imaging device based on the image acquired by the image acquisition unit, and estimates information about the occupant if it is determined that it is not necessary to change at least one of the angle and position of the imaging device; and an operation control unit that changes at least one of the angle and position of the imaging device if the sensing processing unit determines that it is necessary to change at least one of the angle and position of the imaging device and if it is determined based on the determination result by the vehicle state determination unit that the occupant is not driving the vehicle themselves.

[0009] According to the present disclosure, the above-described configuration makes it possible to perform sensing over a wider range while maintaining image resolution compared to conventional techniques.

[0010] 4A and 4B are diagrams showing an example of the operation of the in-vehicle monitoring device according to embodiment 1 when performing physique estimation, in which FIG. 4A is a diagram showing an example of an image captured before changing the angle and position of the imaging device, and FIG. 4B is a diagram showing an example of an image captured after changing at least one of the angle and position of the imaging device. 6A and 6B are diagrams showing an example of the hardware configuration of the in-vehicle monitoring device according to embodiment 1 and embodiment 2.

[0011] Hereinafter, embodiments will be described in detail with reference to the drawings. Embodiment 1. Fig. 1 is a diagram showing an example of the configuration of an interior-of-vehicle monitoring device 2 according to embodiment 1. The interior-of-vehicle monitoring device 2 is a device for monitoring occupants in a vehicle. As shown in Fig. 1, the interior-of-vehicle monitoring device 2 includes an image acquisition unit 201, a vehicle state determination unit 202, a sensing processing unit 203, and an operation control unit 204. In addition to the interior-of-vehicle monitoring device 2, Fig. 1 also shows an imaging device 1.

[0012] Here, the image capturing device 1 and the vehicle interior monitoring device 2 are shown as separate devices. However, the image capturing device 1 and the vehicle interior monitoring device 2 may be integrated into one device. For example, the image capturing device 1 and the vehicle interior monitoring device 2 may be provided in an in-vehicle robot that monitors the interior of a vehicle.

[0013] The imaging device 1 captures an image of the interior of a vehicle. The captured image captured by the imaging device 1 shows the interior of the vehicle and, if an occupant is present in the vehicle, may also show the occupant. The imaging device 1 is normally set at a predetermined angle and position, and is capable of capturing an image showing, for example, the face of an occupant in the driver's seat. The captured image captured by the imaging device 1 has a resolution that allows the interior monitoring device 2 to estimate whether the occupant is looking away or falling asleep, i.e., a resolution that allows the interior monitoring device 2 to perform estimation processing that should be prioritized when the occupant is driving the vehicle. Information indicating the captured image captured by the imaging device 1 is output to the interior monitoring device 2.

[0014] The imaging device 1 is a general term for devices that can represent or acquire the interior conditions of a vehicle as an image, such as an RGB camera, an infrared camera, an RGB-IR camera, a distance image sensor, or a radio wave sensor.

[0015] Furthermore, the imaging device 1 is configured so that at least one of the angle and position can be changed by the in-vehicle monitoring device 2 controlling a motor or the like of the imaging device 1. For example, if the imaging device 1 has an imaging unit and a light source, at least one of the angle and position of at least one of the imaging unit and the light source can be changed by the in-vehicle monitoring device 2. The direction in which the angle and position of the imaging device 1 can be changed is, for example, the vertical direction or the horizontal direction.

[0016] Furthermore, if the imaging device 1 is an imaging device that uses infrared light other than visible light, it is desirable for the imaging device 1 to fix the exposure state when at least one of the angle and position is changed by the in-vehicle monitoring device 2.

[0017] The image acquisition unit 201 acquires a captured image from the imaging device 1. It is desirable that the image acquisition unit 201 constantly performs the above-described acquisition process. Information indicating the captured image acquired by the image acquisition unit 201 is output to the sensing processing unit 203.

[0018] The vehicle state determination unit 202 determines the state of the vehicle. In this case, the vehicle state determined by the vehicle state determination unit 202 is a vehicle state that can determine whether or not the occupant is driving the vehicle. For example, the vehicle state determination unit 202 determines, as the vehicle state, one or more of the following: the vehicle speed is 0, the vehicle is in a gear other than drive or reverse, or the vehicle is in an automatic driving state. Drive includes second and low. In other words, a gear other than drive or reverse means park or neutral. Information indicating the determination result by the vehicle state determination unit 202 is output to the operation control unit 204.

[0019] The vehicle state determination unit 202 may perform the above-mentioned determination process at all times, or may perform the determination process when the sensing processing unit 203 determines that it is necessary to change at least one of the angle and position of the imaging device 1.

[0020] The sensing processing unit 203 determines whether it is necessary to change at least one of the angle and position of the imaging device 1 based on the captured image acquired by the image acquisition unit 201, and estimates information about the occupant if it determines that it is not necessary to change at least one of the angle and position of the imaging device 1.

[0021] For example, the sensing processing unit 203 estimates information that can be estimated from the occupant's face, such as whether the occupant is looking away or dozing, based on captured images acquired by the image acquisition unit 201 when the angle and position of the imaging device 1 are at a predetermined angle and position. Note that the above estimation process is considered to be an estimation process that should be performed preferentially when the occupant is driving the vehicle themselves. Furthermore, for example, the sensing processing unit 203 performs at least one of information that can be estimated from the occupant's face and parts below the face or multiple seats, such as occupant physique estimation, attribute estimation, seating position estimation, and personal recognition, based on captured images acquired by the image acquisition unit 201 before and after at least one of the angle and position of the imaging device 1 is changed. An example of an occupant's attribute is the occupant's gender. Examples of seating positions include the driver's seat, passenger seat, and rear seat. Personal recognition refers to recognizing the identity of the occupant sitting in a seat. It should be noted that the above processing is not processing that should be performed when the occupant himself is driving the vehicle, but processing that may be performed when the occupant himself is not driving the vehicle.

[0022] When the sensing processing unit 203 determines that it is necessary to change at least one of the angle and the position of the imaging device 1, it instructs the operation control unit 204 to change at least one of the angle and the position of the imaging device 1. At this time, the sensing processing unit 203 may also notify the vehicle state determination unit 202 of the above.

[0023] The information about the occupant estimated by the sensing processing unit 203 is output to the outside of the vehicle interior monitoring device 2. In addition to the information indicating the estimation result described above, the sensing processing unit 203 may also output information about the state of the estimation process, such as the start or end of the estimation process, to the outside of the vehicle interior monitoring device 2.

[0024] When the sensing processing unit 203 determines that it is necessary to change at least one of the angle and position of the imaging device 1, and when it is determined based on the determination result by the vehicle state determination unit 202 that the occupant himself / herself is not driving the vehicle, the operation control unit 204 changes at least one of the angle and position of the imaging device 1. In this case, for example, when the vehicle state determination unit 202 determines that the vehicle speed is 0, that the vehicle gear is other than drive or reverse, or that the vehicle is in an autonomous driving state, the operation control unit 204 determines that the occupant himself / herself is not driving the vehicle.

[0025] Furthermore, for example, if the imaging device 1 has an imaging unit and a light source, the operation control unit 204 can change at least one of the angle and position of at least one of the imaging unit and the light source that the imaging device 1 has.

[0026] In addition, when at least one of the angle and position of the imaging device 1 is changed and the estimation process by the sensing processing unit 203 is completed, the operation control unit 204 returns the angle and position of the imaging device 1 to the default angle and position.

[0027] On the other hand, when the operation control unit 204 determines based on the determination result by the vehicle state determination unit 202 that the occupant is driving the vehicle, the operation control unit 204 does not change the angle and position of the imaging device 1 .

[0028] Next, a basic operation example of the in-vehicle monitoring device 2 according to the first embodiment shown in Fig. 1 will be described with reference to Fig. 2. For example, the in-vehicle monitoring device 2 is triggered to operate when the vehicle door is unlocked and an occupant gets in, and starts monitoring the interior of the vehicle. In the basic operation example of the in-vehicle monitoring device 2 according to the first embodiment shown in Fig. 1, as shown in Fig. 2, first, the image acquisition unit 201 acquires an image from the imaging device 1 (step ST101).

[0029] Next, the sensing processing unit 203 extracts information necessary for the estimation process based on the captured image acquired by the image acquisition unit 201 (step ST102).

[0030] Next, the sensing processing unit 203 determines whether at least one of the angle and position of the imaging device 1 needs to be changed (step ST103). At this time, the sensing processing unit 203 determines whether the estimation process is possible from the extracted information. If the sensing processing unit 203 determines that the estimation process is not possible with the extracted information alone, it determines that at least one of the angle and position of the imaging device 1 needs to be changed. On the other hand, if the sensing processing unit 203 determines that the estimation process is possible with the extracted information, it determines that at least one of the angle and position of the imaging device 1 does not need to be changed.

[0031] If the sensing processing unit 203 determines in step ST103 that at least one of the angle and the position of the image capture device 1 needs to be changed, the vehicle state determination unit 202 determines the state of the vehicle (step ST104). At this time, the vehicle state determination unit 202 determines the state of the vehicle, which can determine whether or not the occupant is driving the vehicle. For example, the vehicle state determination unit 202 determines one or more of the following: the vehicle speed is 0, the vehicle gear is other than drive or reverse, or the vehicle is in an autonomous driving state.

[0032] Next, the operation control unit 204 determines whether at least one of the angle and position of the image capture device 1 can be changed based on the determination result by the vehicle state determination unit 202 (step ST105). At this time, the operation control unit 204 determines whether the occupant is not driving the vehicle based on the determination result by the vehicle state determination unit 202. For example, the operation control unit 204 determines that the occupant is not driving the vehicle when the vehicle state determination unit 202 determines that the vehicle speed is 0, that the vehicle gear is other than drive or reverse, or that the vehicle is in an autonomous driving state. Then, when the operation control unit 204 determines that the occupant is not driving the vehicle, it determines that at least one of the angle and position of the image capture device 1 can be changed. On the other hand, when the operation control unit 204 determines that the occupant is driving the vehicle, it determines that at least one of the angle and position of the image capture device 1 cannot be changed.

[0033] In step ST105, the sequence ends when the operation control unit 204 determines that it is not possible to change at least one of the angle and the position of the imaging device 1. In other words, when the vehicle state determination unit 202 determines that the occupant is driving the vehicle, the sensing processing unit 203 should prioritize estimation processing such as whether the occupant is looking away or falling asleep at the wheel, and it is better to refrain from making major changes to the imaging device 1. Therefore, the sequence ends without changing the angle and the position of the imaging device 1.

[0034] On the other hand, if in step ST105 the operation control unit 204 determines that it is possible to change at least one of the angle and position of the imaging device 1, it changes at least one of the angle and position of the imaging device 1 (step ST106).

[0035] Note that, when the imaging device 1 is an imaging device that uses infrared light other than visible light, it is desirable for the imaging device 1 to fix the exposure state when at least one of the angle and position is changed by the operation control unit 204. That is, it is desirable for the imaging device 1 to perform exposure control processing using a fixed value in order to prevent sensing in the in-vehicle monitoring device 2 from becoming unstable due to flickering of light that accompanies a change in at least one of the angle and position by the operation control unit 204.

[0036] Next, the image acquisition unit 201 acquires the captured image from the imaging device 1 (step ST107).

[0037] Next, the sensing processing unit 203 extracts information necessary for the estimation process based on the captured image acquired by the image acquisition unit 201 (step ST108). After that, the sequence returns to step ST103.

[0038] On the other hand, if it is determined in step ST103 that at least one of the angle and position of the imaging device 1 does not need to be changed, the sensing processing unit 203 performs estimation processing using the extracted information (step ST109).

[0039] When at least one of the angle and the position of the imaging device 1 has been changed and the estimation process by the sensing processing unit 203 has ended, the operation control unit 204 returns the angle and the position of the imaging device 1 to the default angle and the default position. Then, the sequence ends.

[0040] Next, a more specific operation example of the in-vehicle monitoring device 2 according to the first embodiment shown in Fig. 1 performing physique estimation will be described with reference to Fig. 3. For example, the in-vehicle monitoring device 2 is triggered to operate when the vehicle door is unlocked and an occupant gets in, and starts monitoring the interior of the vehicle. In the operation example of the in-vehicle monitoring device 2 according to the first embodiment shown in Fig. 1 performing physique estimation, as shown in Fig. 3, first, the image acquisition unit 201 acquires an image from the imaging device 1 (step ST201).

[0041] Next, the sensing processing unit 203 extracts skeleton points of the occupant based on the captured image acquired by the image acquisition unit 201 (step ST202).

[0042] Next, the sensing processing unit 203 determines whether at least one of the angle and position of the image capture device 1 needs to be changed (step ST203). At this time, the sensing processing unit 203 determines whether the occupant's physique can be estimated from the extracted skeletal points of the occupant. If the sensing processing unit 203 determines that the occupant's physique cannot be estimated using only the extracted skeletal points of the occupant, it determines that at least one of the angle and position of the image capture device 1 needs to be changed. On the other hand, if the sensing processing unit 203 determines that the occupant's physique can be estimated using the extracted skeletal points of the occupant, it determines that at least one of the angle and position of the image capture device 1 does not need to be changed.

[0043] For example, Fig. 4A shows an example of a captured image before the angle and position of the imaging device 1 are changed. In Fig. 4A, the upper body (face and chest) of the occupant is captured, but the lower body (waist) of the occupant is not captured. Therefore, the sensing processing unit 203 can extract skeletal points of the occupant's upper body from this captured image, but cannot extract skeletal points of the occupant's lower body, and therefore cannot estimate the occupant's physique based solely on the skeletal points of the occupant's upper body. In Fig. 4A, reference numeral 11a denotes the imaging area of ​​the imaging device 1 before the angle and position are changed, and reference numeral 12a denotes the non-imaging area of ​​the imaging device 1 before the change.

[0044] If the sensing processing unit 203 determines in step ST203 that at least one of the angle and the position of the image capture device 1 needs to be changed, the vehicle state determination unit 202 determines the state of the vehicle (step ST204). At this time, the vehicle state determination unit 202 determines the state of the vehicle, which can determine whether or not the occupant is driving the vehicle. For example, the vehicle state determination unit 202 determines one or more of the following: the vehicle speed is 0, the vehicle gear is other than drive or reverse, or the vehicle is in an autonomous driving state.

[0045] Next, the operation control unit 204 determines whether at least one of the angle and position of the image capture device 1 can be changed based on the determination result by the vehicle state determination unit 202 (step ST205). At this time, the operation control unit 204 determines whether the occupant is not driving the vehicle based on the determination result by the vehicle state determination unit 202. For example, the operation control unit 204 determines that the occupant is not driving the vehicle when the vehicle state determination unit 202 determines that the vehicle speed is 0, that the vehicle gear is other than drive or reverse, or that the vehicle is in an autonomous driving state. Then, when the operation control unit 204 determines that the occupant is not driving the vehicle, it determines that at least one of the angle and position of the image capture device 1 can be changed. On the other hand, when the operation control unit 204 determines that the occupant is driving the vehicle, it determines that at least one of the angle and position of the image capture device 1 cannot be changed.

[0046] In step ST205, the sequence ends when the operation control unit 204 determines that it is not possible to change at least one of the angle and the position of the imaging device 1. In other words, when the vehicle state determination unit 202 determines that the occupant is driving the vehicle, the sensing processing unit 203 should prioritize estimation processing such as whether the occupant is looking away or dozing, and it is better to refrain from making major changes to the imaging device 1. Therefore, the sequence ends without changing the angle and the position of the imaging device 1.

[0047] On the other hand, if the operation control unit 204 determines in step ST205 that at least one of the angle and position of the image capturing device 1 can be changed, it changes at least one of the angle and position of the image capturing device 1 (step ST206). Note that when the sensing processing unit 203 performs physique estimation, it changes at least one of the angle and position of the image capturing device 1 vertically (downward). That is, the operation control unit 204 changes at least one of the angle and position of the image capturing device 1 so that the occupant's lower body (waist area) is captured in the captured image.

[0048] Note that, when the imaging device 1 is an imaging device that uses infrared light other than visible light, it is desirable for the imaging device 1 to fix the exposure state when at least one of the angle and position is changed by the operation control unit 204. That is, it is desirable for the imaging device 1 to perform exposure control processing using a fixed value in order to prevent sensing in the in-vehicle monitoring device 2 from becoming unstable due to flickering of light that accompanies a change in at least one of the angle and position by the operation control unit 204.

[0049] Next, the image acquisition unit 201 acquires the captured image from the imaging device 1 (step ST207).

[0050] Next, the sensing processing unit 203 extracts skeleton points of the occupant based on the captured image acquired by the image acquisition unit 201 (step ST208), after which the sequence returns to step ST203.

[0051] On the other hand, if the sensing processing unit 203 determines in step ST203 that it is not necessary to change at least one of the angle and position of the imaging device 1, it performs estimation processing using the extracted skeleton points of the occupant (step ST209).

[0052] For example, FIG. 4B shows an example of a captured image after at least one of the angle and position of the image capture device 1 is changed. In FIG. 4B, at least one of the angle and position of the image capture device 1 is changed downward, so that the occupant's lower body (waist area) is captured. Therefore, the sensing processing unit 203 can extract skeletal points of the occupant's lower body from this captured image. The sensing processing unit 203 can estimate the occupant's physique by using the skeletal points of the occupant's upper body (face and chest area) already extracted using the captured image of FIG. 4A and the skeletal points of the occupant's lower body additionally extracted using the captured image of FIG. 4B. In FIG. 4B, reference numeral 11b denotes the captured area of ​​the image capture device 1 after at least one of the angle and position is changed, and reference numeral 12b denotes the non-captured area of ​​the image capture device 1 after the change. The physique estimation process performed by the sensing processing unit 203 is similar to existing physique estimation processes, and therefore, a description thereof will be omitted.

[0053] When at least one of the angle and the position of the imaging device 1 has been changed and the estimation process by the sensing processing unit 203 has ended, the operation control unit 204 returns the angle and the position of the imaging device 1 to the default angle and the default position. Then, the sequence ends.

[0054] The above describes the case where the in-vehicle monitoring device 2 performs physique estimation. In contrast, when the in-vehicle monitoring device 2 performs attribute estimation, seating position estimation, or personal recognition, the captured image required for the estimation process can be acquired by laterally changing at least one of the angle and position of the image capture device 1. That is, when the in-vehicle monitoring device 2 performs attribute estimation, seating position estimation, or personal recognition, at least one of the angle and position of the image capture device 1 is changed so that not only the driver's seat but also seats such as the passenger seat and rear seats are captured. This enables the in-vehicle monitoring device 2 to estimate the attributes of the occupants sitting in each seat, the seating positions of each occupant, or the identities of the occupants sitting in each seat. For example, when the in-vehicle monitoring device 2 performs personal recognition, it first performs personal recognition of the occupant sitting in the driver's seat using a captured image showing the driver's seat, then changes at least one of the angle and position of the imaging device 1, and estimates the presence or absence of an occupant sitting in a seat other than the driver's seat (e.g., the passenger seat or a rear seat) using a captured image showing the seat, and then performs personal recognition of the occupant if it is estimated that an occupant is present. Furthermore, when the in-vehicle monitoring device 2 estimates whether the occupant is looking away or dozing, there is no need to change the angle and position of the imaging device 1. That is, the in-vehicle monitoring device 2 can estimate whether the occupant is looking away or dozing using a captured image showing the occupant's face, for example, as shown in FIG. 4A.

[0055] As described above, in the in-vehicle monitoring device 2 according to the first embodiment, when the estimation process by the sensing processing unit 203 requires a change in at least one of the angle and position of the image capturing device 1, if it is determined from the state of the vehicle that the occupant is not driving the vehicle themselves, the in-vehicle monitoring device 2 according to the first embodiment changes at least one of the angle and position of the image capturing device 1. As a result, in consideration of actual use cases for in-vehicle use, the in-vehicle monitoring device 2 according to the first embodiment can achieve functions that require wide-range sensing, such as body size detection, while maintaining the resolution of the eye area necessary to achieve high-performance detection of inattentiveness or drowsiness.

[0056] That is, the process of estimating whether the occupant is looking away or dozing off requires high resolution for the eyes and is necessary when the occupant is driving the vehicle. Therefore, in the in-vehicle monitoring device 2 according to embodiment 1, when it is determined based on the vehicle state that the occupant is driving the vehicle, the angle and position of the image capture device 1 cannot be changed, thereby enabling the process of estimating whether the occupant is looking away or dozing off. On the other hand, processes such as estimating the occupant's physique, attributes, seating position, and personal recognition require wide-area sensing, but can be performed even when the occupant is not driving the vehicle. Therefore, in the in-vehicle monitoring device 2 according to embodiment 1, when it is determined based on the vehicle state that the occupant is not driving the vehicle, the angle and position of the image capture device 1 can be changed, thereby enabling the process of estimating the occupant's physique, attributes, seating position, and personal recognition.

[0057] As described above, according to this embodiment 1, the vehicle interior monitoring device 2 includes an image acquisition unit 201 that acquires an image showing the interior of the vehicle from the imaging device 1, a vehicle state determination unit 202 that determines the state of the vehicle, a sensing processing unit 203 that determines whether it is necessary to change at least one of the angle and position of the imaging device 1 based on the image acquired by the image acquisition unit 201, and estimates information about the occupant if it is determined that it is not necessary to change at least one of the angle and position of the imaging device 1, and an operation control unit 204 that changes at least one of the angle and position of the imaging device 1 if it is determined by the sensing processing unit 203 that it is necessary to change at least one of the angle and position of the imaging device 1 and if it is determined based on the determination result by the vehicle state determination unit 202 that the occupant himself is not driving the vehicle. Furthermore, according to this embodiment 1, the vehicle state determination unit 202 determines whether the vehicle speed is 0, whether the vehicle gear is other than drive or reverse, or whether the vehicle is in an autonomous driving state, and the operation control unit 204 determines that the occupant is not driving the vehicle when the vehicle state determination unit 202 determines that the vehicle speed is 0, whether the vehicle gear is other than drive or reverse, or whether the vehicle is in an autonomous driving state. As a result, the in-vehicle monitoring device 2 according to embodiment 1 is capable of sensing over a wider range while maintaining image resolution compared to conventional devices.

[0058] Furthermore, according to this first embodiment, the sensing processing unit 203 estimates whether or not the occupant is looking away or dozing, based on the captured image acquired by the image acquisition unit 201 when the angle and position of the imaging device 1 are at a predetermined angle and position. As a result, the in-vehicle monitoring device 2 according to the first embodiment is able to estimate whether or not the occupant is looking away or dozing, while maintaining image resolution and enabling sensing over a wider range than conventional devices.

[0059] Furthermore, according to this first embodiment, the sensing processing unit 203 estimates at least one of the occupant's physical size, attribute, seating position, and personal recognition based on the captured images acquired by the image acquisition unit 201 before and after a change in at least one of the angle and position of the imaging device 1. As a result, the in-vehicle monitoring device 2 according to the first embodiment is able to perform sensing over a wider range while maintaining image resolution compared to conventional devices, and is also able to perform at least one of the occupant's physical size, attribute, seating position, and personal recognition.

[0060] Furthermore, according to this embodiment 1, the imaging device 1 fixes the exposure state when at least one of the angle and the position is changed by the operation control unit 204. As a result, the in-vehicle monitoring device 2 according to embodiment 1 can maintain an exposure state suitable for estimation processing by the sensing processing unit 203 even when at least one of the angle and the position of the imaging device 1 is changed.

[0061] Furthermore, according to this embodiment 1, the imaging device 1 has an imaging unit and a light source, and when it is determined that the occupant is not driving the vehicle based on the determination result by the vehicle state determination unit 202, the operation control unit 204 changes at least one of the angle and position of at least one of the imaging unit and the light source of the imaging device 1. As a result, in the vehicle interior monitoring device 2 according to embodiment 1, even if it is difficult to change both the imaging unit and the light source, it can be addressed by changing either the imaging unit or the light source. For example, the light source may be capable of irradiating a wide area, and at least one of the angle and position of the imaging unit may be changed without changing the angle and position.

[0062] Furthermore, according to this embodiment 1, the vehicle interior monitoring method includes the steps of: an image acquisition unit 201 acquiring an image showing the interior of the vehicle from the imaging device 1; a vehicle state determination unit 202 determining the state of the vehicle; a sensing processing unit 203 determining whether it is necessary to change at least one of the angle and position of the imaging device 1 based on the image acquired by the image acquisition unit 201, and estimating information about the occupant if it is determined that it is not necessary to change at least one of the angle and position of the imaging device 1; and an operation control unit 204 changing at least one of the angle and position of the imaging device 1 if it is determined by the sensing processing unit 203 that it is necessary to change at least one of the angle and position of the imaging device 1 and if it is determined based on the determination result by the vehicle state determination unit 202 that the occupant is not driving the vehicle. As a result, the vehicle interior monitoring method according to embodiment 1 enables sensing over a wider range while maintaining image resolution compared to conventional methods.

[0063] Embodiment 2. In the in-vehicle monitoring device 2 according to embodiment 1, the angle and position of the image capture device 1 are prohibited from being changed when it is determined from the vehicle state that the occupant is driving the vehicle. However, it is also possible that the occupant drives the vehicle before or immediately after the estimation process by the sensing processing unit 203 is completed while at least one of the angle and position of the image capture device 1 has been changed. In this case, if the angle and position of the image capture device 1 are prohibited from being changed, the angle and position of the image capture device 1 will not be returned to the default angle and position, and the sensing processing unit 203 will be unable to perform estimation processing, such as whether the occupant is looking away or falling asleep. Therefore, an example configuration of the in-vehicle monitoring device 2 according to embodiment 2 will be described, which takes into account the above-described case.

[0064] Fig. 5 is a diagram showing an example of the configuration of an in-vehicle monitoring device 2 according to embodiment 2. In the example of the configuration of the in-vehicle monitoring device 2 according to embodiment 2 shown in Fig. 5, a notification unit 205 is added to the example of the configuration of the in-vehicle monitoring device 2 according to embodiment 1 shown in Fig. 1. Other components of the in-vehicle monitoring device 2 according to embodiment 2 shown in Fig. 5 are the same as the example of the configuration of the in-vehicle monitoring device 2 according to embodiment 1 shown in Fig. 1, and the same reference numerals are used to denote the same components, and only the different parts will be described.

[0065] In addition, in embodiment 2, the operation control unit 204 allows at least one of the angle and position of the imaging device 1 to be changed even if it is determined that the occupant himself is driving the vehicle based on the determination result by the vehicle state determination unit 202, if the angle and position of the imaging device 1 are not returned to the default angle and position after at least one of the angle and position of the imaging device 1 is changed.

[0066] At this time, if the estimation process by the sensing processing unit 203 has finished, the operation control unit 204 makes changes to return the angle and position of the imaging device 1 to the default angle and position. If the estimation process by the sensing processing unit 203 has not finished, the operation control unit 204 makes changes to at least one of the angle and position of the imaging device 1 so that the sensing processing unit 203 can perform the estimation process, and then makes changes to return the angle and position of the imaging device 1 to the default angle and position.

[0067] When the operation control unit 204 determines that the occupant is driving the vehicle, and when a change is made to return the angle and position of the image capture device 1 to the default angle and position, or when a change is made to at least one of the angle and position of the image capture device 1 so that the sensing processing unit 203 can perform estimation processing, the notification unit 205 notifies an external device that the change is being made. Examples of this notification unit 205 include a display device or an audio output device.

[0068] 5 shows a case where the notification unit 205 is provided in the vehicle interior monitoring device 2 according to embodiment 2. However, the notification unit 205 is not an essential component of the vehicle interior monitoring device 2 according to embodiment 2, and does not have to be provided in the vehicle interior monitoring device 2.

[0069] As described above, the in-vehicle monitoring device 2 according to the second embodiment makes it possible, as an exception, to change the angle and position of the imaging device 1 when the occupant drives the vehicle while the angle and position of the imaging device 1 have not yet returned to the default angle and position. As a result, the in-vehicle monitoring device 2 according to the second embodiment makes it possible to return the angle and position of the imaging device 1 to the default angle and position even when the occupant drives the vehicle before or immediately after the processing of physique estimation by the sensing processing unit 203 is completed, thereby enabling the sensing processing unit 203 to perform estimation processing such as whether the occupant is looking away or falling asleep at the wheel.

[0070] As described above, according to the second embodiment, if the angle and position of the imaging device 1 have not been returned to the default angle and position after at least one of the angle and position of the imaging device 1 has been changed, the operation control unit 204 makes a change to return the angle and position of the imaging device 1 to the default angle and position, even if it is determined that the occupant is driving the vehicle based on the determination result by the vehicle state determination unit 202, or makes a change to return the angle and position of the imaging device 1 to the default angle and position after making a change to at least one of the angle and position of the imaging device 1 so that the sensing processing unit 203 can perform estimation processing. As a result, the in-vehicle monitoring device 2 according to the second embodiment can appropriately respond to a case where the occupant drives the vehicle before or immediately after the above processing is completed, in addition to the effects of the first embodiment.

[0071] Furthermore, according to the second embodiment, the in-vehicle monitoring device 2 includes a notification unit 205 that notifies the outside when the operation control unit 204 determines that the occupant is driving the vehicle and when a change is made to return the angle and position of the image capture device 1 to a default angle and position, or when a change is made to at least one of the angle and position of the image capture device 1 so that the sensing processing unit 203 can perform estimation processing. As a result, in addition to the effects of the first embodiment, the in-vehicle monitoring device 2 according to the second embodiment can notify the occupant that, even when the occupant is driving the vehicle, the angle and position of the image capture device 1 have not returned to the default angle and position for estimation processing by the sensing processing unit 203, and estimation processing such as inattentive driving or dozing cannot be immediately performed.

[0072] Finally, with reference to FIG. 6 , an example of the hardware configuration of the vehicle interior monitoring device 2 according to the first and second embodiments will be described. Here, an example of the hardware configuration of the vehicle interior monitoring device 2 according to the first embodiment is shown, but the same applies to the example of the hardware configuration of the vehicle interior monitoring device 2 according to the second embodiment. The functions of the image acquisition unit 201, the vehicle state determination unit 202, the sensing processing unit 203, and the operation control unit 204 in the vehicle interior monitoring device 2 are realized by a processing circuit 51. The processing circuit 51 may be dedicated hardware as shown in FIG. 6A , or may be a CPU (also referred to as a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 52 that executes a program stored in a memory 53 as shown in FIG. 6B .

[0073] When the processing circuitry 51 is dedicated hardware, the processing circuitry 51 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of the image acquisition unit 201, the vehicle state determination unit 202, the sensing processing unit 203, and the operation control unit 204 may be realized individually by the processing circuitry 51, or the functions of each unit may be realized collectively by the processing circuitry 51.

[0074] When the processing circuit 51 is a CPU 52, the functions of the image acquisition unit 201, vehicle state determination unit 202, sensing processing unit 203, and operation control unit 204 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 53. The processing circuit 51 realizes the functions of each unit by reading and executing the programs stored in the memory 53. That is, the vehicle interior monitoring device 2 includes the memory 53 for storing programs that, when executed by the processing circuit 51, result in the execution of, for example, each step shown in FIG. 2 . It can also be said that these programs cause a computer to execute the procedures and methods of the image acquisition unit 201, vehicle state determination unit 202, sensing processing unit 203, and operation control unit 204. Here, examples of the memory 53 include non-volatile or volatile semiconductor memories such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically EPROM), as well as magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs (Digital Versatile Discs).

[0075] It is also possible to realize some of the functions of the image acquisition unit 201, the vehicle state determination unit 202, the sensing processing unit 203, and the operation control unit 204 with dedicated hardware and some with software or firmware. For example, the function of the image acquisition unit 201 can be realized by the processing circuit 51 as dedicated hardware, and the functions of the vehicle state determination unit 202, the sensing processing unit 203, and the operation control unit 204 can be realized by the processing circuit 51 reading and executing programs stored in the memory 53.

[0076] In this way, the processing circuitry 51 can realize each of the above-described functions by hardware, software, firmware, or a combination of these.

[0077] It should be noted that the embodiments may be freely combined, or any of the components in each embodiment may be modified, or any of the components in each embodiment may be omitted.

[0078] The in-vehicle monitoring device according to the present disclosure is capable of sensing over a wider range while maintaining image resolution compared to conventional devices, and is suitable for use as an in-vehicle monitoring device for monitoring passengers in a vehicle.

[0079] REFERENCE SIGNS LIST 1 Imaging device, 2 Vehicle interior monitoring device, 51 Processing circuit, 52 CPU, 53 Memory, 201 Image acquisition unit, 202 Vehicle state determination unit, 203 Sensing processing unit, 204 Operation control unit, 205 Notification unit

Claims

1. An image acquisition unit that acquires captured images of the vehicle's interior from an imaging device, A vehicle state determination unit that determines the state of the vehicle, A sensing processing unit determines, based on the captured image acquired by the image acquisition unit, whether it is necessary to change at least one of the angle and position of the imaging device, and if it is determined that it is not necessary to change at least one of the angle and position of the imaging device, it estimates information about the occupant. The system includes an operation control unit that performs an operation to change at least one of the angle and position of the imaging device when the sensing processing unit determines that it is necessary to change at least one of the angle and position of the imaging device, and when the vehicle state determination unit determines, based on the determination result, that the occupant is not driving the vehicle. The sensing processing unit performs at least one of the following based on the captured images acquired by the image acquisition unit before and after at least one of the angle and position of the imaging device is changed: estimating the occupant's physique, estimating their attributes, estimating their seating position, and recognizing them as an individual. An in-vehicle monitoring device characterized by the following features.

2. The vehicle state determination unit determines whether the vehicle's speed is 0, whether the vehicle's gear is other than drive or reverse, or whether the vehicle is in an automatic driving state. The operation control unit determines that the vehicle is not being driven by an occupant if the vehicle state determination unit determines that the vehicle's speed is 0, the vehicle's gear is other than drive or reverse, or the vehicle is in an automatic driving state. The in-vehicle monitoring device according to claim 1, characterized by the features described above.

3. The sensing processing unit estimates whether at least one of the occupant is distracted or asleep, based on the captured image acquired by the image acquisition unit when the angle and position of the imaging device are at a predetermined angle and position. The in-vehicle monitoring device according to claim 1 or 2, characterized by the features described above.

4. The imaging device fixes the exposure state when at least one of the angle and position is changed by the motion control unit. The in-vehicle monitoring device according to claim 1, characterized by the features described above.

5. The imaging device has an imaging unit and a light source, If the operation control unit determines, based on the determination result by the vehicle state determination unit, that the occupant is not driving the vehicle, it changes at least one of the angle and position of at least one of the imaging unit and light source of the imaging device. The in-vehicle monitoring device according to claim 1, characterized by the features described above.

6. If the motion control unit has not returned the angle and position of the imaging device to the default angle and position after changing at least one of the angle and position of the imaging device, it will either change the angle and position of the imaging device to the default angle and position, or make changes to allow the sensing processing unit to perform estimation processing on at least one of the angle and position of the imaging device, and then change the angle and position of the imaging device to the default angle and position. The in-vehicle monitoring device according to claim 1 or 2, characterized by the features described above.

7. If the operation control unit determines that the occupant is driving the vehicle, and changes are made to return the angle and position of the imaging device to the default angle and position, or if changes are made to at least one of the angle and position of the imaging device for estimation processing by the sensing processing unit, the system is provided with a notification unit that notifies the outside that such changes have been made. The in-vehicle monitoring device according to claim 6, characterized in that it is a vehicle interior monitoring device.

8. The image acquisition unit acquires an image of the interior of the vehicle from the imaging device. The vehicle status determination unit performs the steps of determining the status of the vehicle, The sensing processing unit determines, based on the captured image acquired by the image acquisition unit, whether it is necessary to change at least one of the angle and position of the imaging device, and if it determines that it is not necessary to change at least one of the angle and position of the imaging device, it estimates information about the occupant. The motion control unit has the step of changing at least one of the angle and position of the imaging device when the sensing processing unit determines that it is necessary to change at least one of the angle and position of the imaging device, and when the vehicle state determination unit determines, based on the determination result, that the occupant is not driving the vehicle. The sensing processing unit performs at least one of the following based on the captured images acquired by the image acquisition unit before and after at least one of the angle and position of the imaging device is changed: estimating the occupant's physique, estimating their attributes, estimating their seating position, and recognizing them as an individual. A vehicle interior monitoring method characterized by the following features.