System and method for detecting face of driver in cabin camera image for vehicle

Abstract

To accurately detect the face of a driver from movements of a face image detected in images continuously captured within a cabin.SOLUTION: In detection of a face image of a driver in images captured in a cabin of a vehicle 10, a method includes: capturing images in the cabin a plurality of times with a camera 12; detecting an image of a human face in each of the images captured in the cabin a plurality of times; extracting a representative point in an existing area of the image of each face detected in each image; performing clustering processing of classifying into clusters the representative points of the image of the face on the basis of position coordinates of the representative point of the image of each face extracted in each image; identifying as a driver cluster the cluster determined as the cluster of the representative points of the image of the face of the person seated on a driver seat of the vehicle on the basis of a coordinate range of the existing area of the cluster in a coordinate space; and detecting the image of the face of the person being in the existing area of the driver cluster as the image of the face of the driver.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a system and method for monitoring the interior of a vehicle using an in-vehicle camera image, and more particularly to a system and method for automatically detecting the driver's face in an in-vehicle camera image. [Background technology]

[0002] In vehicles such as automobiles, various techniques have been proposed for detecting the face of the driver in an image taken by a camera capturing an image of the interior of the vehicle in order to monitor the state of the occupants, particularly the driver. For example, Patent Document 1 discloses a system for enabling a driver who has exited the vehicle to understand the movements of fellow passengers, in which a person detected within a preset range in an image taken by a camera capturing an image of the interior of the vehicle is determined to be the driver. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-72559 Summary of the Invention [Problem to be solved by the invention]

[0004] Technology for detecting and recognizing human faces in images has already been put to practical use. It is easy to detect and recognize the faces of each occupant in an image of the interior of a vehicle captured by a camera, and their positions can also be identified. Therefore, it is possible to identify which face image in the image belongs to the driver based on the detected face position. However, because the position and orientation of a face image change from moment to moment, it is not always possible to accurately identify the driver's face from only the face image detected at a certain moment. In other words, in order to accurately identify the driver's face in an image of the interior of a vehicle, it is necessary to sequentially detect face images in images captured over an appropriate period of time (video or multiple still images captured at appropriately set times), track their movement, and determine which image belongs to the driver's face.

[0005] In view of the above circumstances, the main object of the present invention is to provide a technology for accurately detecting the driver's face by tracking the movement of the facial image detected in images taken continuously or intermittently multiple times inside the vehicle cabin. [Means for solving the problem]

[0006] According to one aspect of the present invention, the above problem is solved by a system for detecting an image of a driver's face in an image captured inside a vehicle cabin, comprising: an imaging means for imaging the interior of the vehicle multiple times; a face detection means for detecting an image of a human face in each of the plurality of images of the interior of the vehicle obtained by the imaging means taking images a plurality of times; a facial image representative point extraction means for extracting a predetermined point in each of the detected regions in each of the images in which the human face image exists as a representative point of the facial image; a clustering processing means for performing a clustering process to classify the representative points of the facial images extracted from each of the images into clusters based on the position coordinates of each of the representative points of the facial images extracted from each of the images in a coordinate space set for the images; a driver image detection means for identifying a cluster that is determined to be a cluster of representative points of an image of a face of a person sitting in the driver's seat of the vehicle based on the coordinate range of the existence area of ​​the cluster in the coordinate space as a driver cluster, and detecting an image of a face of a person that is present in the existence area of ​​the driver cluster as an image of the face of the driver; This is achieved by a system including:

[0007] In the above configuration, the "imaging means" may be a camera that is normally used to capture images of the interior of a vehicle. The imaging means that captures images multiple times may capture video (continuous images) of the interior of the vehicle, or may be configured to capture still images intermittently at an appropriately set timing. The "face detection means" that detects the image of a human face in each image of the interior of the vehicle may be a means that executes a process to detect the image of a human face in an image using an arbitrary algorithm. In detecting such a human face, typically, an area (bounding box) that surrounds the image of the human face in the image is detected. Box) is defined. The "face image representative point extraction means" extracts a predetermined point in the area (which may be the area of ​​a bounding box) of the human face image detected in each image by the face detection means as the representative point of the face image, and the predetermined point may be any position set in the area of ​​the area, such as the center position, upper right corner, upper left corner, or center of gravity of the area. The "clustering processing means" classifies the representative points of the face images in multiple captured images extracted by the face image representative point extraction means into clusters using an arbitrary algorithm based on their coordinate positions in a state where they are scattered in the coordinate space of images captured inside the vehicle cabin (since the position of even a single face image changes from moment to moment, the representative point of the same face image is located in a different position for each image). In this classification, Generally speaking, representative points that are close to each other in the coordinate space of the image are classified to belong to the same cluster according to the algorithm used (when there is only one occupant in the vehicle cabin, all representative points are usually classified into one cluster, and such a case also falls within the scope of the present invention). In the "driver image detection means", among the clusters of representative points generated by the above clustering process, a cluster that is determined to be a cluster of representative points of the image of the face of a person sitting in the driver's seat of the vehicle based on the coordinate range of the existence area of ​​each cluster in the coordinate space of the image is identified as the driver cluster. The "cluster of representative points of the image of the face of a person sitting in the driver's seat of the vehicle" may be a cluster that is within the range in which a representative point of the existence area of ​​the image of the face of such person can exist when a person is sitting in the driver's seat of the vehicle.Then, a face image corresponding to a representative point belonging to the cluster identified as the driver cluster is detected as the face image of the driver. The "face detection means," "facial image representative point extraction means," "clustering processing means," and "driver image detection means" are realized by the operation of a computer device in accordance with a program. The computer device may be a cloud system outside the vehicle or may be an on-board computer.

[0008] The system of the present invention described above makes it possible to detect the image of a driver's face captured in a series of images obtained by photographing the interior of a vehicle multiple times. With this configuration, even if the driver is seated in the driver's seat, the image of the driver's face can be detected even if the image of the driver's face cannot be detected in an image captured at a certain moment due to the position or orientation of the driver's face in that image.

[0009] In the driver image detection means of the above configuration, identifying a cluster determined to be a cluster of representative points of the image of the face of a person sitting in the driver's seat of the vehicle based on the coordinate range of the cluster existence area in the coordinate space of the image as the driver cluster may be achieved in several ways. Specifically, in one aspect, when a plurality of clusters are detected by the clustering processing means, the driver image detection means may be configured to identify a cluster located in an area in the coordinate space of the image that is closest to the coordinates of the area where the driver's seat of the vehicle exists as the driver cluster. In another aspect, the driver image detection means may be configured to identify a cluster located within a predetermined range in the coordinate space of the image as the driver cluster. Here, the "predetermined range" may be appropriately set to be near the driver's seat of the vehicle.

[0010] According to the system of the present invention, a novel method for accurately detecting the face of a driver by tracking the movement of facial images detected in images continuously taken inside a vehicle cabin is achieved. Thus, according to another aspect of the present invention, the above-mentioned object is to provide a method for detecting the face of a driver in images taken inside a vehicle cabin, comprising: A step of capturing an image of the interior of the vehicle cabin multiple times by an imaging means; detecting an image of a human face in each of the images of the interior of the vehicle captured multiple times; extracting a predetermined point in each of the detected regions in each of the images in which the human face image exists as a representative point of the face image; a step of performing a clustering process to classify the representative points of the facial images extracted from each of the images into clusters based on the position coordinates of each of the representative points of the facial images extracted from each of the images in a coordinate space set for the images; a step of identifying a cluster that is determined to be a cluster of representative points of a face image of a person sitting in the driver's seat of the vehicle based on the coordinate range of the existence area of ​​the cluster in the coordinate space as a driver cluster, and detecting a face image of a person that is present in the existence area of ​​the driver cluster as an image of the face of the driver; This is achieved by a method comprising:

[0011] In the above-described configuration of the present invention, the step of capturing images of the interior of the vehicle multiple times is performed using a camera that is normally used to capture images of the interior of the vehicle, and the other steps are realized by operating a computer device according to a program, as in the above-described system of the present invention.

[0012] In the above-described configuration of the method of the present invention, identifying a cluster determined to be a cluster of representative points of the image of the face of a person sitting in the driver's seat of a vehicle based on the coordinate range of the cluster's existence area in the coordinate space of the image as the driver cluster may be achieved in several ways. Specifically, in one aspect, when multiple clusters are detected, the cluster located in the area closest to the coordinates of the existence area of ​​the driver's seat of the vehicle in the coordinate space of the image may be identified as the driver cluster. Furthermore, the method may be configured to identify a cluster located in a predetermined range that may be appropriately set near the driver's seat of the vehicle in the coordinate space of the image as the driver cluster. [Effects of the Invention]

[0013] Thus, the configuration of the present invention enables more accurate detection of the driver's face by tracking the movement of a facial image detected in images captured continuously or intermittently multiple times within the vehicle cabin. Because the present invention enables more accurate detection of the driver's facial image, even if multiple faces are detected in the image, facial images of passengers other than the driver or pedestrians outside the vehicle can be excluded as noise. Furthermore, since the configuration of the present invention detects the driver's facial image using the presence area of ​​the human facial image, brightness information outside the presence area of ​​the human facial image in the image may be discarded. The clustering process and the process of identifying clusters of the driver's facial image in the present invention use representative points of the presence area of ​​the human facial image, which is advantageous in that the processing load is significantly reduced compared to processes using image brightness data.

[0014] Other objects and advantages of the present invention will become apparent from the following description of preferred embodiments of the invention. [Brief explanation of the drawings]

[0015] [Figure 1] Fig. 1(A) is a diagram illustrating an overview of a vehicle driver's face detection system according to this embodiment, and Fig. 1(B) is a block diagram showing the configuration of the system according to this embodiment. [Figure 2] FIG. 2 is a flowchart showing the process of detecting the face of a vehicle driver according to this embodiment. [Figure 3]3A to 3G are diagrams illustrating the processing of camera images taken inside a vehicle cabin up to the detection of the driver's face according to this embodiment. (A) is a schematic diagram of a camera image taken inside the vehicle cabin, (B) is a schematic diagram of a bounding box displayed around a human face detected in the camera image taken inside the vehicle cabin, and (C) is a schematic diagram of representative points extracted from the bounding box displayed in the camera image taken inside the vehicle cabin. (D) is a schematic diagram of representative points of the bounding boxes in each of multiple camera images displayed in the coordinate space of the image, and (E) is a schematic diagram of the representative points distributed in the coordinate space of the image in (D) after clustering processing has been performed to classify each representative point into a cluster. (F) is a schematic diagram of the rightmost cluster in the coordinate space of the image identified as the driver cluster. (G) is a schematic diagram of a state in which a cluster contained within the region in the coordinate space of the image where the face of a person sitting in the driver's seat is expected to exist is identified as the driver cluster. [Explanation of symbols]

[0016] 10...vehicle, 12...camera, 30...computer device (cloud), D...driver, P...occupant BEST MODE FOR CARRYING OUT THE INVENTION

[0017] The present invention will now be described in detail with reference to some preferred embodiments thereof with reference to the accompanying drawings, in which like reference numerals indicate like parts.

[0018] System configuration In the vehicle driver's face detection system according to this embodiment, as shown in FIG. 1(A), an image of passengers D and P inside a vehicle 10 such as an automobile, captured by a camera 12, is sent to a computer device 30, where a process is executed to detect the face of the driver P in the image. As shown in the figure, the computer device 30 may be located in an external system, such as a cloud computing system, and the camera image may be transmitted from the on-board camera 12 to the computer device 30 via any type of communication system. If the vehicle 10 or the camera 12 has the processing capability to perform the vehicle driver's face detection according to this embodiment, the driver's face detection process may be executed in the vehicle, and the results may be transmitted to an external system.

[0019] In the computer device 30 that executes the driver's face detection process from the vehicle interior images captured by the camera 12, specifically, as shown in FIG. 1B, images uploaded from the camera 12 are first stored in the image data server 31. The images may be video (continuous images) or multiple still images captured at appropriate intervals. The images stored in the image data server 31 are then sent to the face detection processing unit 32, where, as described below, a face detection framework is used to detect the human face image in each image. The face detection framework may be of any type or task as long as it can detect the human face image. Typically, the position coordinates of the area where the human face image exists in a coordinate space set in the image and the brightness of each pixel within that area are recorded as face image data for each image, and such data may be stored in the inference result storage server 33. Image data other than the face image data may be deleted to reduce data volume and costs. Furthermore, the facial image data may be output as an intermediate file, or may be stored in memory without being output as an intermediate file. Next, the facial image coordinate extraction unit 34 uses the facial image data for each pixel to determine a representative point in the facial image area for each pixel, and extracts its position coordinates. The representative point may be a point appropriately set in the facial image area, such as the center point or the upper right corner point of the facial image area. This allows the position coordinates of the representative points of the facial images in the multiple images in the image coordinate space to be obtained. In other words, the coordinates of each position of the group of representative points of the facial images in the image coordinate space are obtained. Thereafter, in the clustering processing unit 35, using the position coordinates of the group of representative points of the face image in the coordinate space of the image, clustering processing is performed on the group of representative points of the face image, as described below, where clusters are generated into which the group of representative points of the face image are classified, and in the driver image determination unit 36, one of the generated clusters that is determined to be a cluster of the face image of a person sitting in the driver's seat is determined to be a driver cluster, and the face image of the person in that cluster is detected as the face image of the driver.

[0020] The respective units 32, 34 to 36 in the computer device 30 are realized by the operation of the computer device in accordance with a program. The data servers 32, 33 may be ordinary data storage devices used in this field.

[0021] System Operation In the driver's face detection process in this embodiment, as already mentioned, the image of the driver's face is detected not in an image of a certain moment, but in images captured continuously or images captured intermittently multiple times. As a result, even if the image of the driver's face cannot be identified in an image of a certain moment because the driver looks to the side or down, the image of the driver's face can be identified using images captured continuously or images captured multiple times, so that the image of the driver's face can be detected more reliably.

[0022] Specifically, referring to FIG. 2, in the driver's face detection process (S0), first, as shown in FIG. 3A, images (frame 1, frame 2, frame 3, frame 4, etc.) captured by the camera 12 consecutively or at appropriately set intervals are acquired and stored in the image data server 31 (step 1). Note that several hundred to several thousand images may be acquired and stored for one detection of the driver's face. Then, detection of a human face image is performed for each of these images (face detection processing unit 32, step 2). Typically, the presence area of ​​the human face image is detected, and a bounding box bx is set for each of these areas, as shown in FIG. 3B. In this regard, as shown in the illustrated example, a bounding box bx may not be set for a face facing sideways or downwards, i.e., the presence area may not be detected (frames 3 and 4). Furthermore, the position and size of the bounding box bx change depending on the face orientation (frame 4). Once a bounding box bx is set for each face image, a representative point rp of each bounding box bx is determined, as shown in Figure 3(C), and the position coordinates of each representative point rp in the coordinate space set in the image are extracted (face image coordinate extraction unit 34, step 3). As already mentioned, the representative point of the bounding box bx may be a point appropriately set in the area where the face image exists, such as the center point or the upper right corner point of the bounding box bx.

[0023] Thereafter, in a state where the position coordinates of the representative points rp of each face image are distributed in a coordinate space set in the image as shown in Fig. 3(D), a clustering process is performed (clustering processing unit 35, step 4), and the group of representative points of each face image is classified into clusters (α, β, γ, ...) as shown in Fig. 3(E). This clustering process may be performed using any algorithm (e.g., DBSCAN), and generally speaking, points that are close to each other are classified to belong to the same cluster. Note that when there is only one occupant in the vehicle cabin, it may happen that all representative points are classified into one cluster.

[0024] As described above, once the groups of representative points of each facial image are classified into clusters, one of the clusters generated therein that is determined to be a cluster of the facial image of a person sitting in the driver's seat is selected as the driver cluster (driver image determination unit 36, step 5). The selection of such a driver cluster may be performed in various ways. In one aspect, as shown in FIG. 3(F), among the clusters generated by the clustering process, the one closest to the driver's seat (in a vehicle where the driver's seat is on the right, the rightmost cluster α) may be selected as the driver cluster. In another aspect, as shown in FIG. 3(G), the cluster α existing within the range X assumed to be the area where the facial image of a person sitting in the driver's seat of the vehicle is present may be selected as the driver cluster. In this way, if an image belonging to a cluster other than the driver cluster is captured, it can be excluded as an image (noise) of the face of a passenger other than the driver or a passerby outside the vehicle. Then, the facial images belonging to the cluster selected as the driver cluster are recognized as the facial images of the driver (step 6).

[0025] Thus, according to the configuration of the present embodiment, the driver's face is detected by tracking the movement of the facial image detected in images captured continuously or intermittently multiple times within the vehicle cabin. This makes it possible to detect the driver's face more accurately than when attempting to detect the driver's face by referring only to images captured at a certain moment. The detection of the driver's face according to this embodiment may be performed at an appropriately set timing while the vehicle is being driven. The driver's facial image detected in the images may be used for any purpose, such as monitoring the driver while driving, for example, to confirm driving safety by monitoring facial expressions and posture.

[0026] The above description has been made in relation to the embodiments of the present invention, but it will be apparent that many modifications and changes will be readily apparent to those skilled in the art, and the present invention is not limited to the above-described exemplary embodiments, but can be applied to various devices without departing from the concept of the present invention.

Claims

1. A system for detecting an image of a driver's face in an image captured inside a vehicle, an imaging means for imaging the interior of the vehicle multiple times; a face detection means for detecting an image of a human face in each of the plurality of images of the interior of the vehicle obtained by the imaging means taking images a plurality of times; a facial image representative point extraction means for extracting a predetermined point in each of the detected regions in each of the images in which the human face image exists as a representative point of the facial image; a clustering processing means for performing a clustering process to classify the representative points of the facial images extracted from each of the images into clusters based on the position coordinates of each of the representative points of the facial images extracted from each of the images in a coordinate space set for the images; a driver image detection means for identifying a cluster that is determined to be a cluster of representative points of an image of a face of a person sitting in the driver's seat of the vehicle based on the coordinate range of the existence area of ​​the cluster in the coordinate space as a driver cluster, and detecting an image of a face of a person that is present in the existence area of ​​the driver cluster as an image of the face of the driver; A system including:

2. 2. The system of claim 1, wherein when a plurality of clusters are detected by the clustering processing means, the driver image detection means is configured to identify, as the driver cluster, the cluster located in an area in the coordinate space of the image that is closest to the coordinates of an area where the driver's seat of the vehicle is located.

3. 2. The system according to claim 1, wherein the driver image detection means is configured to identify a cluster existing within a predetermined range in the coordinate space of the image as the driver cluster.

4. A method for detecting an image of a driver's face in an image captured inside a vehicle, comprising: A step of capturing an image of the interior of the vehicle cabin multiple times by an imaging means; detecting an image of a human face in each of the images of the interior of the vehicle captured multiple times; extracting a predetermined point in each of the detected regions in each of the images in which the human face image exists as a representative point of the face image; a step of performing a clustering process to classify the representative points of the facial images extracted from each of the images into clusters based on the position coordinates of each of the representative points of the facial images extracted from each of the images in a coordinate space set for the images; a step of identifying a cluster that is determined to be a cluster of representative points of a face image of a person sitting in the driver's seat of the vehicle based on the coordinate range of the existence area of ​​the cluster in the coordinate space as a driver cluster, and detecting a face image of a person that is present in the existence area of ​​the driver cluster as an image of the face of the driver; A method comprising:

5. 5. The method of claim 4, wherein when a plurality of clusters are detected by the clustering process, the cluster located in the area in the coordinate space of the image closest to the coordinates of the area where the driver's seat of the vehicle is located is identified as the driver cluster.

Images