Vehicle interior condition recognition device

Abstract

To continuously recognize the state inside a cabin including the state of a driver by using cameras provided on a steering wheel.SOLUTION: A state inside cabin recognition device of the present disclosure comprises: a plurality of cameras that is provided on a steering wheel and can photograph the inside of a cabin; and an image processing apparatus that is connected with the plurality of cameras. The image processing apparatus acquires photographed images from the plurality of cameras. The image processing apparatus subsequently determines a shielded state of each of the plurality of cameras from the state of each of the photographed images from the plurality of cameras. The image processing apparatus further selects the photographed image from the minimum shielded camera that is shielded in the smallest degree, of the plurality of cameras. The image processing apparatus then executes recognition processing for recognizing the state inside the cabin on the photographed image from the minimum shielded camera.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a device for recognizing the state inside a vehicle cabin using a camera mounted on a steering wheel. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2010-013090 discloses a driving condition monitoring system that uses a camera mounted on the steering wheel to capture an image of the driver's face. This driving condition monitoring system is configured to rotate the captured image by the angle of the steering wheel to correct the tilt, and then extract features of the driver's face from the corrected image to determine the driver's condition. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-013090 Summary of the Invention [Problem to be solved by the invention]

[0004] The system disclosed in the above publication has a single camera located near the center of the steering wheel. If the single camera is blocked, it cannot capture an image of the driver's face. When a camera is mounted on the steering wheel, the camera is frequently blocked by a person's hand or arm, so it is expected that the system disclosed in the above publication will often be unable to determine the driver's condition.

[0005] The present disclosure has been made in view of the above-mentioned problems, and one objective of the present disclosure is to enable continuous recognition of the state of the interior of a vehicle, including the state of the driver, using a camera mounted on the steering wheel. [Means for solving the problem]

[0006] The present disclosure provides a vehicle interior condition recognition device for achieving the above-mentioned object. The vehicle interior condition recognition device of the present disclosure includes a plurality of cameras mounted on a steering wheel capable of capturing images of the interior of the vehicle interior, and an image processing device connected to the plurality of cameras. The image processing device is configured to execute the following processes: a first process is to acquire captured images from each of the plurality of cameras; a second process is to determine the occlusion state of each of the plurality of cameras based on the state of the captured images from each of the plurality of cameras; a third process is to select an image captured by a minimally occluded camera with the smallest degree of occlusion from among the plurality of cameras; and a fourth process is to perform recognition processing on the image captured by the minimally occluded camera to recognize the state of the vehicle interior. [Effects of the Invention]

[0007] The vehicle interior condition recognition device disclosed herein has multiple cameras mounted on the steering wheel, rather than just one, that can capture images of the vehicle interior. Even if some of the cameras are blocked by a person's hand or arm, the remaining cameras can still capture images of the vehicle interior. The vehicle interior condition recognition device disclosed herein uses an image processing device to perform recognition processing to recognize the vehicle interior condition for images captured by the least blocked camera among the multiple cameras. This allows the vehicle interior condition, including the driver's condition, to be continuously recognized. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating a configuration of a vehicle interior state recognition device according to an embodiment of the present disclosure. [Figure 2] 2A and 2B are diagrams for explaining the selection of a camera to be used for recognizing the state inside the vehicle cabin. [Figure 3] 3A and 3B are diagrams for explaining the selection of a camera to be used for recognizing the state inside the vehicle cabin. [Figure 4] 10A and 10B are diagrams illustrating a correction process when switching between cameras used to recognize the interior state of a vehicle. [Figure 5]4 is a flowchart showing a first example of the operation of the vehicle interior state recognition device. [Figure 6] 10 is a flowchart showing a second example of the operation of the vehicle interior state recognition device. DETAILED DESCRIPTION OF THE INVENTION

[0009] 1. Overview Hereinafter, an embodiment of the vehicle interior state recognition device of the present disclosure will be described with reference to the drawings. Hereinafter, the term vehicle interior state recognition device refers to the vehicle interior state recognition device according to this embodiment.

[0010] The vehicle interior condition recognition device is a device that captures images of the interior of the vehicle cabin using multiple cameras and recognizes the state of the interior of the vehicle cabin from the captured images. In other words, the vehicle interior condition recognition device is a type of monitoring system that uses cameras. However, the vehicle interior condition recognition device is not a system that monitors the interior of the vehicle cabin from multiple directions using multiple cameras. The monitoring performed by the vehicle interior condition recognition device is fixed-point monitoring from a single fixed point. The fixed point for monitoring in the vehicle interior condition recognition device is the steering wheel. The multiple cameras are provided on the steering wheel. The vehicle interior condition recognition device appropriately selects a camera to be the fixed point for monitoring from the multiple cameras provided on the steering wheel. The camera selected is the camera that captures the object to be recognized most clearly, specifically, the camera whose field of view is not obstructed or the camera whose field of view is least obstructed.

[0011] The cameras on the steering wheel may be installed on the rim, on the central hub, or on the spokes connecting the rim and hub. However, it is preferable that at least one camera is installed in a location that is located to the left of the center of the steering wheel when the steering wheel is in the neutral position. It is also preferable that at least one camera is installed in a location that is located to the right of the center of the steering wheel when the steering wheel is in the neutral position. It is unlikely that the cameras installed on the left and right of the center of the steering wheel will be blocked at the same time, and it is highly likely that at least one of the cameras will be able to recognize the interior of the vehicle.

[0012] Furthermore, if three or more cameras are installed, it is preferable that at least one camera be installed in a location that is located near the longitudinal center line of the steering wheel when the steering wheel is in the neutral position. More specifically, it is preferable that the camera be installed in a location that is located below or above the center of the steering wheel when the steering wheel is in the neutral position. By distributing the cameras in this way, it is possible to reduce the possibility that all cameras will be blocked at the same time and increase the possibility that one of the cameras will be able to continuously monitor the interior of the vehicle.

[0013] Vehicles to which the in-vehicle state recognition device is applicable include manually driven vehicles driven by a driver on board the vehicle, autonomously driven vehicles driven by an autonomous driving system, and remotely driven vehicles remotely driven by a remote operator. The interior state of the vehicle that is the target of recognition by the in-vehicle state recognition device includes the state of the driver sitting in the driver's seat. Examples of the driver's state that can be recognized include the driver's facial direction, line of sight angle, line of sight movement, eyelid opening, and eyelid movement. In addition, the state of passengers other than the driver and the in-vehicle environment are also examples of the interior state that is the target of recognition by the in-vehicle state recognition device. In particular, in the case of autonomously driven vehicles and remotely driven vehicles, the driver is not necessarily sitting in the driver's seat, so it is possible to recognize the in-vehicle environment behind the driver's seat using a camera installed on the steering wheel.

[0014] 2. Configuration of vehicle interior condition recognition device FIG. 1 is a diagram illustrating a configuration of a vehicle interior state recognition device 2 according to an embodiment of the present disclosure. The vehicle interior state recognition device 2 includes multiple cameras 61, 62, and 63 provided on a steering wheel 4. In the example shown in FIG. 1, three cameras 61, 62, and 63 are provided on the rim of the steering wheel 4. Specifically, the first camera 61 is provided at a position located to the left of the center of the steering wheel 4 when the steering wheel 4 is in a neutral position. The second camera 62 is provided at a position located to the right of the center of the steering wheel 4 when the steering wheel 4 is in a neutral position. The third camera 63 is provided at a position located directly below the center of the steering wheel 4 when the steering wheel 4 is in a neutral position.

[0015] The three cameras 61, 62, and 63 have the same field of view and are mounted facing in a direction that allows them to capture images of the interior of the vehicle. Specifically, the field of view of the cameras 61, 62, and 63 is set so that they can capture an image of the driver's face when the driver is sitting in the driver's seat, and can capture an image of the interior environment behind the driver's seat when the driver is not sitting in the driver's seat. The specifications of the cameras 61, 62, and 63, including the resolution and frame rate, are all the same.

[0016] The vehicle interior state recognition device 2 includes an image processing device 10. The image processing device 10 is connected to cameras 61, 62, and 63 via an in-vehicle network such as LVDS. Images captured by the cameras 61, 62, and 63 are captured by the image processing device 10.

[0017] The image processing device 10 includes an interface 12, image memories 141, 142, and 143, a processor 16, and a program memory 18. The interface 12 receives images transmitted from cameras 61, 62, and 63 via an in-vehicle network. The images received by the interface 12 are temporarily stored in the image memories 141, 142, and 143.

[0018] Image memories 141, 142, and 143 are frame memories that store image data, and are provided for each camera. That is, a first image memory 141 is provided for the first camera 61, a second image memory 142 is provided for the second camera 62, and a third image memory 143 is provided for the third camera 63. However, image memories 141, 142, and 143 may each be independent hardware (memory devices) or may be different memory areas of the same memory device. The images temporarily stored in image memories 141, 142, and 143 are read into processor 16.

[0019] The processor 16 may be, for example, a CPU, a GPU, an FPGA, or an ASIC. Alternatively, the processor 16 may be a combination of two or more of the CPU, the GPU, the FPGA, and the ASIC. The program memory 18 stores a plurality of instructions 20 executable by the processor 16. The processor 16 reads and executes the instructions 20 from the program memory 18. When the instructions 20 stored in the program memory 18 are executed by the processor 16, the processor 16 performs processing for recognizing the interior state of the vehicle on images read from the image memories 141, 142, and 143.

[0020] The image processing device 10 may be configured to connect to a communication network using a communication device (not shown) and communicate with an external monitoring center. By transmitting the recognition result of the interior state of the vehicle from the image processing device 10 to the monitoring center, the interior state of the vehicle can be remotely monitored at the monitoring center.

[0021] 3. Operation of the vehicle interior condition recognition device Next, the operation of the vehicle interior state recognition device 2 configured as described above will be described. The image processing device 10 selects the least occluded camera among the three cameras 61, 62, and 63, and performs recognition processing on the image captured by the least occluded camera. Examples of the occlusion states of the cameras 61, 62, and 63 are shown in Figures 2A, 2B, 3A, and 3B.

[0022] In Figure 2A, the first camera 61 is blocked by the left hand 8L, and the second camera 62 is blocked by the right hand 8R. However, the third camera 63 is not blocked. In the state shown in Figure 2A, the third camera 63 is the least blocked camera.

[0023] Suppose the steering wheel 4 is turned from the state shown in Fig. 2A to the state shown in Fig. 2B. In Fig. 2B, the first camera 61 is blocked by the left hand 8L, and the third camera 63 is blocked by the right hand 8R. However, the second camera 63 is not blocked. In the state shown in Fig. 2B, the second camera 62 is the least blocked camera.

[0024] 2A to the state shown in FIG. 2B, the least obstructed camera is switched from the third camera 63 to the second camera 62. When switching the cameras, the image processing device 10 performs a correction process to suppress discontinuities in the recognition results. The details of the correction process will be described later.

[0025] In FIG. 3A, the first camera 61 is blocked by the left hand 8L. However, the second camera 62 and the third camera 63 are not blocked. In the state shown in FIG. 3A, the second camera 62 and the third camera 63 are both minimally blocked cameras. The image captured by the second camera 62, which is a minimally blocked camera, and the image captured by the third camera 63, which is also a minimally blocked camera, are both candidate images that can be used to recognize the interior state of the vehicle. The image processing device 10 selects one image from the candidate images using a method that will be described later.

[0026] Assume that the steering wheel 4 is turned from the state shown in FIG. 3A to the state shown in FIG. 3B. In FIG. 3B, the third camera 63 is blocked by the right hand 8R. However, the first camera 61 and the second camera 62 are not blocked. In the state shown in FIG. 3B, the first camera 61 and the second camera 62 are both minimally blocked cameras. The image captured by the first camera 61, which is a minimally blocked camera, and the image captured by the second camera 62, which is also a minimally blocked camera, are both candidate images that can be used to recognize the state inside the vehicle cabin. The image processing device 10 selects one image from the candidate images.

[0027] If the camera used for recognition in Fig. 3A is the second camera 62 and the camera used for recognition in Fig. 3B is also the second camera 62, the least occluded camera does not change when transitioning from the state shown in Fig. 3A to the state shown in Fig. 3B. However, if, for example, the camera used for recognition in Fig. 3A is the third camera 63 and the camera used for recognition in Fig. 3B is the first camera 61, the least occluded camera changes when transitioning from the state shown in Fig. 3A to the state shown in Fig. 3B. In this case, image processing device 10 performs correction processing on the image captured by third camera 63 to suppress discontinuities in the recognition results.

[0028] Fig. 4 is a diagram illustrating the correction process when switching cameras used to recognize the interior conditions of a vehicle. Fig. 4 shows an example of the change over time in the recognition results of the interior conditions of a vehicle. As an example of the recognition results of the interior conditions of a vehicle, Fig. 4 shows the driver's line of sight angle. The recognition results shown by the solid line in Fig. 4 indicate the recognition results of the line of sight angle before correction, and the recognition results shown by the dotted line in Fig. 4 indicate the recognition results of the line of sight angle after correction.

[0029] The cameras 61, 62, and 63 are mounted facing the same direction. The image processing device 10 corrects the tilt by rotating the images captured by each of the cameras 61, 62, and 63 by the rotation angle of the steering wheel 4. However, as shown by the solid line in FIG. 4, there may be a discontinuity between the recognition result of the gaze angle before switching the camera and the recognition result of the gaze angle after switching the camera. In order to eliminate the discontinuity in the recognition results of the gaze angle before and after switching, the image processing device 10 performs a correction process on the recognition result of the gaze angle after switching. An example of the correction process is a Kalman filter.

[0030] The operation of the vehicle interior state recognition device 2 is represented by a flowchart. Fig. 5 is a flowchart showing a first example of the operation of the vehicle interior state recognition device, and Fig. 6 is a flowchart showing a second example of the operation of the vehicle interior state recognition device. Whether the vehicle interior state recognition device 2 operates in the first example or the second example is determined by the process executed by the image processing device 10, more specifically, the contents of the instructions 20 executed by the processor 16.

[0031] First, a first example of the operation of the vehicle interior state recognition device 2 will be described with reference to Fig. 5. In the first example, the vehicle interior state recognition device 2 executes a series of processes shown in the flowchart of Fig. 5 for each frame of the cameras 61, 62, and 63.

[0032] In step S11, cameras 61, 62, and 63 capture images of the interior of the vehicle. The images captured by each of cameras 61, 62, and 63 are captured by image processing device 10. Image processing device 10 stores the captured images captured by each of cameras 61, 62, and 63 in corresponding image memories 141, 142, and 143.

[0033] In step S12, the image processing device 10 determines the occlusion state of each of the cameras 61, 62, and 63 based on the state of the images captured by each of the cameras 61, 62, and 63. For example, an image that is too dark overall or an image in which uniformly bright portions account for a predetermined percentage or more of the entire image can be determined to be an image captured by an occluded camera. Note that the concept of the occlusion state includes not only whether or not an image is occluded, but also the degree of occlusion calculated from the percentage of uniformly dark or bright portions of the image.

[0034] In step S13, the image processing device 10 selects candidate images that can be used for recognition processing from the images captured by each of the cameras 61, 62, and 63 based on the occlusion state of each of the cameras 61, 62, and 63. The candidate images are images captured by the least occluded camera with the smallest degree of occlusion. In the example shown in FIG. 2A, the candidate images are images captured by the third camera 63. In the example shown in FIG. 2B, the candidate images are images captured by the second camera 62. In the example shown in FIG. 3A, the candidate images are images captured by the second camera 62 and the third camera 63. In the example shown in FIG. 3B, the candidate images are images captured by the first camera 61 and the second camera 62.

[0035] In step S14, the image processing device 10 calculates the usefulness of each candidate image. The usefulness refers to the usefulness in the recognition process. An image that is advantageous for recognizing the state inside the vehicle cabin is an image with a high usefulness. For example, the sharpness of the image, the distance from the camera to the subject, and the number of feature points of the subject included in the image can be used as the usefulness.

[0036] In step S15, the image processing device 10 selects the image with the highest usefulness among the candidate images. In the example shown in Fig. 3A, the image with the highest usefulness is selected from the images captured by the second camera 62 and the third camera 63. In the example shown in Fig. 2A, the candidate images are only the images captured by the third camera 63, so the image captured by the third camera 63 is selected. However, if there is only one candidate image as in this example, the processes of steps S14 and S15 may be omitted.

[0037] In step S16, the image processing device 10 performs a recognition process for recognizing the state of the interior of the vehicle on the image selected in step S16. For the recognition process, for example, machine learning including deep learning is used.

[0038] In step S17, the image processing device 10 acquires the recognition result of the interior state of the vehicle through the recognition process executed in step S16.

[0039] In step S18, the image processing device 10 performs a correction process on the recognition result acquired in step S17 to ensure continuity with the recognition result of the previous frame. An example of the content of the correction process is as described with reference to FIG.

[0040] Then, in step S19, the image processing device 10 outputs the recognition result corrected in step S18 as the recognition result for this frame. The recognition result output from the image processing device 10 is used, for example, in a driving assistance system. If the recognition result indicates that the driver's alertness or attention has decreased, the driving assistance system executes control to ensure safety. Examples of control to ensure safety include alerting the driver, stopping the vehicle in a safe place, and switching from manual driving to automatic driving.

[0041] Next, a second example of the operation of the vehicle interior state recognition device 2 will be described with reference to Fig. 6. In the second example, the vehicle interior state recognition device 2 executes a series of processes shown in the flowchart of Fig. 6 for each frame of the cameras 61, 62, and 63.

[0042] In step S21, the image processing device 10 performs the same process as in step S11 in the first example. In step S22, the image processing device 10 performs the same process as in step S12 in the first example. Then, in step S23, the image processing device 10 performs the same process as in step S13 in the first example.

[0043] In step S24, the image processing device 10 executes a recognition process for recognizing the interior state of the vehicle with respect to each of the candidate images selected in step S23.

[0044] In step S25, the image processing device 10 acquires the recognition result of the vehicle interior state obtained by the recognition process executed in step S24 for each candidate image.

[0045] In step S26, the image processing device 10 calculates the reliability of the recognition result for each candidate image acquired in step S25. The method for calculating the reliability of the recognition result varies depending on the recognition target. For example, if the recognition target is a driver, the gaze angle is determined from the position of the pupils, so the larger the size of the recognized driver's eyes, the higher the reliability is determined. Also, if the recognition target is an occupant in the vehicle, the higher the accuracy of the bone structure estimation, the higher the reliability is determined.

[0046] In step S27, the image processing device 10 selects the most reliable recognition result from among the recognition results. In the example shown in Fig. 3A, the more reliable recognition result is selected from the recognition result obtained from the image captured by the second camera 62 and the recognition result obtained from the image captured by the third camera 63. In the example shown in Fig. 2A, the only recognition result is the recognition result obtained from the image captured by the third camera 63, so the recognition result obtained from the image captured by the third camera 63 is selected. However, if there is only one recognition result as in this example, the processes of steps S26 and S27 may be omitted.

[0047] In step S28, the image processing device 10 performs a correction process on the recognition result acquired in step S27 to ensure continuity with the recognition result of the previous frame.

[0048] Then, in step S29, the image processing device 10 outputs the recognition result corrected in step S28 as the recognition result for this frame. According to the second example, the state of the vehicle interior can be recognized continuously as in the first example, and the state of the vehicle interior can be recognized with substantially the same quality as in the first example. [Explanation of symbols]

[0049] 2. Vehicle interior condition recognition device 4 Steering Wheel 10 Image processing device 61, 62, 63 Camera

Claims

1. A plurality of cameras that are provided at different positions on the steering wheel and are capable of photographing the interior of the vehicle; an image processing device connected to the plurality of cameras, The image processing device includes: acquiring a photographed image from each of the plurality of cameras; determining an occlusion state of each of the plurality of cameras from a state of an image captured by each of the plurality of cameras; selecting an image captured by a least-obscured camera having the smallest degree of obscuration from among the plurality of cameras; performing a recognition process on the image captured by the minimally occluded camera to recognize a state inside the vehicle; When the minimum occlusion camera is switched among the plurality of cameras, a correction process is performed on the recognition result obtained by the recognition process to suppress discontinuity before and after the switching. A vehicle interior condition recognition device characterized by:

2. A plurality of cameras that are provided at different positions on the steering wheel and are capable of photographing the interior of the vehicle; an image processing device connected to the plurality of cameras, The image processing device includes: acquiring a photographed image from each of the plurality of cameras; determining an occlusion state of each of the plurality of cameras from a state of an image captured by each of the plurality of cameras; selecting an image captured by a least-obscured camera having the smallest degree of obscuration from among the plurality of cameras; When there are a plurality of the minimally occluded cameras, calculating the usefulness of each of the images captured by the plurality of minimally occluded cameras; performing a recognition process for recognizing a state inside the vehicle cabin for the captured image having the highest usefulness; When the camera that captured the image to be recognized is switched between the plurality of cameras, a correction process is performed on the recognition result of the recognition process to suppress discontinuity before and after the switching. A vehicle interior condition recognition device characterized by:

3. A plurality of cameras that are provided at different positions on the steering wheel and are capable of photographing the interior of the vehicle; an image processing device connected to the plurality of cameras, The image processing device includes: acquiring a photographed image from each of the plurality of cameras; determining an occlusion state of each of the plurality of cameras from a state of an image captured by each of the plurality of cameras; selecting an image captured by a least-obscured camera having the smallest degree of obscuration from among the plurality of cameras; When there are a plurality of the minimally occluded cameras, a recognition process is performed on each of the images captured by the plurality of minimally occluded cameras to recognize a state inside the vehicle interior. Calculating the reliability of a recognition result for each of the images captured by the plurality of minimally occluded cameras; selecting a recognition result with the highest reliability as a recognition result for the state inside the vehicle; When the camera that captured the image that is the basis of the selected recognition result is switched between the multiple cameras, a correction process is performed on the selected recognition result to suppress discontinuity before and after the switch. A vehicle interior condition recognition device characterized by:

4. In the vehicle interior state recognition device according to any one of claims 1 to 3, At least one of the plurality of cameras is provided at a location located to the left of the center of the steering wheel when the steering wheel is in a neutral position, At least one of the plurality of cameras is provided at a position located to the right of the center of the steering wheel when the steering wheel is in the neutral position. A vehicle interior condition recognition device characterized by:

5. In the vehicle interior state recognition device according to any one of claims 1 to 3, the plurality of cameras includes at least three cameras; At least one of the plurality of cameras is provided at a location located to the left of the center of the steering wheel when the steering wheel is in a neutral position, at least one of the plurality of cameras is provided at a position located to the right of the center of the steering wheel when the steering wheel is in the neutral position, At least one of the plurality of cameras is provided at a location that is located near a center line of the steering wheel in the longitudinal direction when the steering wheel is in the neutral position. A vehicle interior condition recognition device characterized by:

Images