Onboard image processing device and onboard image processing method

Abstract

Onboard image processing device, comprising an image acquisition section (101) which acquires an image from an imaging device (20) in which an area inside a vehicle is captured, which includes an area in which the face of a driver sitting in the driver's seat should be located, a face detection section (102) which, based on the image acquired by the image acquisition section (101), detects the driver's face and the driver's facial area on that image, a luminance calculation section (104) that calculates the average luminance of the image elements in the driver's face area detected by the face detection section (102), a luminance evaluation section (105) that evaluates whether the average luminance calculated by the luminance calculation section (104) is greater than or equal to a luminance evaluation threshold or not, an optical adjustment control section (106) which, in the event that the luminance assessment section (105) has assessed that the average luminance is below the luminance assessment threshold, performs a control of the optical adjustment of the imaging device according to the average luminance, a misdetection evaluation section (107) which, based on the result of the control performed by the optical adjustment control section (102), evaluates whether the face detection section (106) has misdetected the driver's face or not, and a re-control arrangement section (108) which, in the event that the misdetection evaluation section (107) has evaluated that the face detection section (102) has misdetected the driver's face, outputs a re-control arrangement to the optical setting control section (106) by which a control is carried out resetting the optical setting of the imaging device (20) to an appropriate value, wherein the optical adjustment control section (106) performs an amplification adjustment of the imaging device (20), wherein the face detection section (102) detects the driver's face and the face area of ​​the driver in question in the face detection area set in the image, and The misdetection evaluation section (107) assesses whether, as a consequence of the gain adjustment performed by the optical setting control section (106), an overexposure area in a region (601) where a higher probability of an image of the driver's face in the face detection area is suspected, meets an evaluation condition or not, and in the event that the overexposure area meets the evaluation condition, assesses a misdetection of the driver's face by the face detection section (102), and the assessment condition checks whether the overexposure area in the area (601) of the presumed face detection is greater than or equal to a preset threshold.

Description

Technical field

[0001] The invention relates to an onboard image processing device and an onboard image processing method for determining the face of a driver based on an image. Background technology

[0002] A conventional technique is known in which a driver's face is detected based on an image of the driver acquired by an imaging device, and the optical settings of the imaging device are adjusted based on this detection result so that the image is better suited for detecting the driver's face. The optical settings of the imaging device include exposure control, gain adjustment, etc. For example, patent document 1 discloses an onboard image processing device with the function of instructing a CCD controller to detect the driver's facial area from an image received by a camera and to calculate an exposure time that results in optimal image brightness in the detected facial area.

[0003] Patent document 2 discloses an in-vehicle image processing device with a means for receiving a facial image. Patent document 3 discloses an image-based device for assessing the condition of a vehicle driver. Furthermore, patent documents 4 to 6 were identified as providing technological background. State-of-the-art documents, patent documents Patent Document 1: JP 2009-116742 A Patent document 2: DE 11 2008 002 646 T5 Patent document 3: DE 11 2017 007 251 T5 Patent document 4: DE 10 2019 106 262 A1 Patent document 5: JP 2005- 66 023 A Patent document 6: DE 11 2015 002 934 B4 Overview of the invention Problem to be solved by the invention

[0004] With a conventional onboard image processing device, such as the one disclosed in patent document 1, the problem is that the possibility of the face of an occupant other than the driver being depicted in the image cannot be taken into account. If the image shows the face of an occupant other than the driver, the conventional onboard image processing device may therefore misdetect the face of an occupant as the driver's face. Controlling the optical settings based on a misdetected driver's face does not provide adequate control.

[0005] The invention was made to solve the above problem and aims to prevent the control of an unsuitable optical setting based on a misdetected face of the driver. Means of solving the task

[0006] The aforementioned problem and further issues are solved by the subject matter of claims 1 and 9. Further advantageous embodiments of the invention are specified in the dependent claims. A corresponding onboard image processing device comprises, firstly, an image acquisition section that acquires an image from an imaging device in which an area inside a vehicle is captured, including an area where the face of a driver sitting in the driver's seat should be located; a face detection section that detects the driver's face and the driver's facial area in the image based on the image acquired by the image acquisition section; a luminance calculation section that calculates the average luminance of the image elements in the driver's facial area detected by the face detection section; and a luminance evaluation section that evaluateswhether the average luminance calculated by the luminance calculation section is greater than or equal to a luminance assessment threshold or not; an optical adjustment control section which, in the event that the luminance assessment section has assessed that the average luminance is below the luminance assessment threshold, performs a control of the optical adjustment of the imaging device according to the average luminance; a misdetection assessment section which, based on the result of the control performed by the optical adjustment control section, assesses whether the face detection section has misdetected the driver's face or not; and a re-control arrangement section which, in the event that the misdetection assessment section has detected that the face detection section has misdetected the driver's face,outputs a recalibration arrangement to the optical setting control section, by which a control is performed that resets the optical setting of the imaging device to an appropriate value. Effects of the invention

[0007] According to the invention, the control of an unsuitable optical setting based on a misdetected face of the driver can be prevented. Brief explanation of the characters [ Fig. 1] Fig. Figure 1 is a view showing a structural example of an onboard image processing device according to a first embodiment. [ Fig. 2] Fig. 2A and Fig. 2B are views that illustrate examples of pictorial representations of the installation position and the recording area of ​​an imaging device in the first embodiment. [ Fig. 3] Fig. Figure 3 is a view illustrating a pictorial representation of the method by which, in the first embodiment, a face detection section determines the driver's face. [ Fig. 4] Fig. Figure 4 is a view illustrating a pictorial representation of an example of the condition in a vehicle when, in the first embodiment, the driver's face is falsely detected by the face detection section. [ Fig. 5] Fig. Figure 5 is a view showing a pictorial representation in which, in the first embodiment, overexposure has occurred in the image in the vicinity of the driver's face area. [ Fig. 6] Fig. Figure 6 is a view illustrating a pictorial representation of a face presence presumption area in the first embodiment. [ Fig. 7] Fig. 7A and Fig. Figure 7B shows illustrations of an example of the face detection area, which in the first embodiment was reduced by an area reduction section. [ Fig. 8] Fig. Figure 8 is a flowchart to explain the operation of the onboard image processing device according to the first embodiment. [ Fig. 9] Fig. 9 is a flowchart to illustrate the operation in steps ST804 to ST807 in the first embodiment. Fig. 8 to explain in more detail. [ Fig. 10] Fig. Figure 10 is a flowchart to illustrate the operation of steps ST804 to ST807 in the onboard image processing device according to a second embodiment. Fig. 8 to explain in more detail. [ Fig. 11] Fig. Figure 11 is a view showing a structural example of an onboard image processing device according to a third embodiment. [ Fig. 12] Fig. Figure 12 is a flowchart to illustrate the operation of steps ST804 to ST807 in the onboard image processing device according to the third embodiment. Fig. 8 to explain in more detail. [ Fig. 13] Fig. 13A and Fig. Figure 13B shows examples of the hardware structure of an onboard image processing device according to the first to third embodiments. Forms for carrying out the invention

[0008] The following section provides a detailed explanation of the methods for implementing the invention, using the figures as an example. First embodiment

[0009] Fig. Figure 1 is a view showing a structural example of an onboard image processing device 10 according to a first embodiment.

[0010] The onboard image processing device 10 is installed in a vehicle (not shown).

[0011] The onboard image processing device 10 is connected to an imaging device 20 and an illumination device 30.

[0012] In the first embodiment, it is assumed that the imaging device 20 is used together with an imaging device comprising a so-called "driver monitoring system" and installed in a vehicle to observe the driver's condition inside the vehicle. This is merely an example, where the imaging device 20 can be a camera, etc., installed for the purpose of monitoring the vehicle interior and is installed at least in such a way as to capture an area inside the vehicle that includes the area where the face of a driver seated in the driver's seat should be located. The area where the face of a driver seated in the driver's seat should be located is, for example, an area corresponding to the space in the immediate vicinity of the driver's seat headrest.The imaging device 20 can, for example, also be a visible light camera or an infrared camera. Fig. 2A and Fig. 2B are views that illustrate examples of pictorial representations of the installation position and the recording area of ​​the imaging device 20 in the first embodiment. Fig. 2A and Fig. 2B are views that show pictorial representations of the interior of a vehicle from above. As in Fig. 2A and Fig. As shown in Figure 2B, in the first embodiment it is specified that there are also occupants other than the driver inside the vehicle, wherein in Fig. 2A and Fig. 2B shows a driver with 21a, a rear passenger with 21b and a front passenger with 21c.

[0013] In the first embodiment, for example, it is assumed that the imaging device 20, as in Fig. 2A is shown, installed in the center of the vehicle's dashboard (hereinafter referred to as the "dashboard") and depicts the side of the driver's seat from the center of the dashboard.

[0014] Furthermore, in the first embodiment, it is assumed that the imaging device 20, as in Fig. 2B is shown, installed in the center of the vehicle's dashboard and depicts the driver's seat and the passenger seat from the center of the dashboard.

[0015] In the following explanation, it is assumed that the imaging device 20 is installed such that the installation position and the imaging area are as shown in Fig. 2A shows the result.

[0016] The lighting device 30 is a light-emitting illuminator, such as an LED (Light Emitting Diode), etc. The lighting device 30 is installed near the imaging device 20. In the first embodiment, the imaging device 20 is provided with the lighting device 30. Fig. 2A and Fig. In 2B, the illustration of the lighting device 30 is omitted.

[0017] The onboard image processing device 10 acquires an image from the imaging device 20 and detects the driver's face and the driver's facial area within this image. Based on the average luminance of the image elements in the facial area, the optical setting of the imaging device 20 is adjusted. The facial area is a specific region within the image that is adjusted based on the detected position of parts of the driver's face in the image. Details of the facial area are explained below.

[0018] If the optical setting of the imaging device 20 is adjusted, the onboard image processing device 10 assesses, based on the result of the adjustment, whether the driver's face was misdetected or not (hereinafter referred to as the "misdetection assessment"). The onboard image processing device 10 adjusts the optical setting of the imaging device 20 to an appropriate value if the misdetection assessment determines that the driver's face was misdetected. Adjusting the optical setting of the imaging device 20 to an appropriate value, for example, sets the optical setting of the imaging device 20 to a preset state.In the first embodiment, the onboard image processing device 10 is configured to control the optical setting of the imaging device 20 to a preset state when it has been determined that the driver's face has been misdetected. The onboard image processing device 10 then performs a new detection of the driver's face based on an image acquired by the imaging device 20, whose optical setting has been reset to its initial state.

[0019] In the first embodiment, the optical setting of the imaging device 20 is, for example, the exposure time of the imaging device 20, the switch-on time of the lighting device 30, or the gain value of the imaging device 20. Bringing the optical setting into its preset state involves setting the initial value to which the exposure time is preset, the initial value to which the switch-on time is preset, or the initial value to which the gain value is preset. The respective initial values ​​are set to values ​​that are assumed to be sufficient to clearly image the driver's face under certain conditions.Furthermore, if the false detection assessment determines that the driver's face has not been falsely detected, the onboard image processing device 10 performs processing in which the driver's condition is monitored based on the detected driver's face, a warning is issued according to the driver's condition, etc.

[0020] This prevents the onboard image processing device 10 from controlling the optical setting of the imaging device 20 according to the average luminance of the image elements of a facial area that includes the face of an occupant other than the driver. Furthermore, it prevents the onboard image processing device 10 from issuing a warning, etc., that is not based on the driver's condition.

[0021] The onboard image processing device 10 comprises, as shown in Fig. Figure 1 shows an image acquisition section 101, a face detection section 102, a driver monitoring section 103, a luminance calculation section 104, a luminance evaluation section 105, an optical setting control section 106, a fault detection evaluation section 107, a recontrol arrangement section 108, a range reduction section 109 and a redetection arrangement section 110.

[0022] The optical setting control section 106 includes an exposure control section 1061 and an image processing section 1062.

[0023] The image acquisition section 101 acquires an image from the imaging device 20 in which an area inside the vehicle is depicted, which includes the area in which the face of a driver sitting in the driver's seat should be located.

[0024] The imaging device 20 converts light acquired from inside the vehicle into electrical signals and generates image data based on these electrical signals. The imaging device 20 outputs the generated image data to the image acquisition section 101. The image acquisition section 101 acquires the image data output by the imaging device 20.

[0025] The imaging device 20 can also convert light acquired from inside the vehicle into electrical signals and output them to the image acquisition section 101. In this case, the image acquisition section 101 acquires the image data by converting the electrical signals output by the imaging device 20.

[0026] The statement that the image acquisition section 101 “acquires an image from the imaging device 20” includes, for example, when the image acquisition section 101 acquires image data output by the imaging device 20 as specified above, or when the image acquisition section 101 acquires image data by converting the electrical signals output by the imaging device 20.

[0027] The image acquisition section 101 outputs the acquired image to the face detection section 102 and the luminance calculation section 104.

[0028] The face detection section 102 detects the driver's face and a specific area of ​​the driver's face within the image acquired by the image acquisition section 101. Specifically, the face detection section 102 detects the driver's face, for example, by detecting parts of the face within a defined face detection area in the image. The face detection section 102 can detect these facial parts using a known image recognition technology. The face detection area is a region in the image that is defined as the target area for the driver's face detection by the face detection section 102. The initial value of the face detection area is preset, for example, by a user. Data relating to the face detection area is then stored in a memory section (not shown).

[0029] The face detection area is defined by the face detection area data, which includes data that allows the face detection area to be specified within the image, such as coordinates, etc., to specify a frame indicating the face detection area. Face detection section 102, with reference to the memory section, specifies the face detection area and detects the driver's face.

[0030] The face detection section 102 also detects the driver's face. In the first embodiment, the face area is defined, for example, as a rectangle that passes over the upper end of the face, the lower end of the face, the left end of the face, and the right end of the face, with the position of the eyebrows defining the upper end, the position of the mouth the lower end, and the outer corners of the left and right eyes defining the left and right ends, respectively. The face area can also be defined as a rectangle in which the protruding rectangle, which passes over the upper end of the face, the lower end of the face, the left end of the face, and the right end of the face, is enlarged by a preset ratio, etc.

[0031] When the following explanation refers to the fact that the face detection section 102 detects the driver's face, the detection of the driver's face also includes the detection of the driver's facial area.

[0032] If multiple faces are detected in the face detection area, the face detection section 102 determines the face with the highest reliability level as the driver's face, based on the reliability level of the detected faces.

[0033] In the first embodiment, for example, the face detection section 102 considers the largest face in the face area as the face with the highest degree of reliability.

[0034] This is Fig. 3 a view to illustrate a pictorial representation of the method by which, in the first embodiment, the face detection section 102 determines the face of the driver.

[0035] In Fig. 3 is an image acquired by the image acquisition section 101 with 200 and an area for face detection in the image 200 with 201 is shown.

[0036] It is now assumed that the face detection section 102 in the face detection area 201 in image 200 detected two faces, namely a first face 202a and a second face 202b. Furthermore, it is assumed that the face detection section 102 detected a face area of ​​the first face 202a (hereinafter referred to as "first face area 203a") and a face area of ​​the second face 202b (hereinafter referred to as "second face area 203b").

[0037] In this case, the face detection section 102 evaluates the larger face 202a, corresponding to the first face area 203a, from the first face area 203a and the second face area 203b, as having the highest degree of reliability and identifies it as the driver's face. That is, the face detection section 102 detects the first face 202a as the driver's face from the detected first face 202a and second face 202b.

[0038] For example, if the face detection area is set so large that the driver's face can be detected even when the driver's face moves to different positions, it can happen that the face detection area, as in e.g. in Fig. Figure 3 shows that the area for face detection also includes the face of a passenger sitting in the back seat. If the face of a passenger sitting in the back seat is included in the area for face detection, it is possible that the face detection section 102 will detect the second face 202b, which is the face of this passenger, together with the first face 202a, which is the face of the driver. Normally, the driver sits closer to the imaging device 20, so the driver's face is captured larger in the image.Consequently, even if both the second face 202b, which is the occupant's face, and the first face 202a, which is the driver's face, have been detected by the face detection section 102, the first face 202a, which has the larger facial area, judges the first face 202a, with the larger facial area, to be the face with the highest degree of reliability and determines the first face 202a to be the driver's face.

[0039] As an example, it was assumed here that the face detection section 102 assesses the reliability of the detected face based on the size of the facial area; this is merely an example. The face detection section 102 can, for instance, also assess the reliability of the face according to the level of reliability of the detected parts of the face. The reliability of the parts of a face refers, for example, to the extent to which the eyes are typical of eyes or the nose is typical of a nose. The face detection section 102 can detect the reliability of the parts of the face using a known image recognition technology such as pattern matching, etc.

[0040] The face detection section 102 outputs data regarding the detected face of the driver (hereinafter referred to as "face data") to the driver monitoring section 103, the luminance calculation section 104, and the false detection evaluation section 107. The driver's face data includes data regarding the parts of the driver's face and data regarding the driver's facial area. More specifically, the data regarding the parts of the driver's face includes, for example, data that represent parts of the driver's face, show the position of these parts, and relate them to coordinates in the image. Furthermore, the data regarding the driver's facial area includes, for example, coordinates in the image used to specify a frame showing the driver's facial area.The face detection section 102 may also be one in which, together with the image output by the image acquisition section 101, an output is sent to the driver monitoring section 103, the luminance calculation section 104 and the false detection evaluation section 107.

[0041] Furthermore, the face detection section 102 allows the detected face data of the driver to be stored in the storage section.

[0042] Back to the explanation of Fig. 1. The driver monitoring section 103 monitors the driver's condition based on facial data output by the face detection section 102 when face detection presence data is output by the false detection evaluation section 107. In the first embodiment, the driver's condition monitored by the driver monitoring section 103 is defined as a state in which the driver is not concentrating on driving, such as microsleep, looking away, etc., and which may impair driving ability. The false detection evaluation section 107 is described below.

[0043] The driver monitoring section 103, for example, monitors whether the driver is experiencing microsleep, looking away, etc., based on the degree of eye opening or the angle of the bridge of the nose. The technology that calculates the degree of eye opening, the angle of the bridge of the nose, etc., based on data relating to parts of the face is generally known, so a detailed explanation is omitted. It is sufficient for the driver monitoring section 103 to monitor whether the driver is in a state that could impair their ability to drive.

[0044] If the driver monitoring section 103 assesses that the driver is in a condition that may impair driving ability, it outputs warning output command data to a warning output control section (not shown).

[0045] When the driver monitoring section 103 issues warning output command data, the warning output control section causes a warning to be issued by an output device (not shown). The output device is, for example, an audio output device with which a vehicle is equipped and is connected to the onboard image processing device 10. The warning output control section causes the output device to, for example, emit a warning tone.

[0046] The luminance calculation section 104 calculates the average luminance of the image elements in the driver's face area (hereinafter referred to as "driver face area image elements"), which was detected by the face detection section 102, in the image acquired by the image acquisition section 101.

[0047] The luminance calculation section 104 outputs data regarding the calculated average luminance of the driver face area image elements (hereinafter referred to as "face area luminance data") to the luminance evaluation section 105.

[0048] The Luminance Assessment Section 105 assesses, based on the face area luminance data output by the Luminance Calculation Section 104, whether the average luminance of the driver's face area image elements is greater than or equal to a threshold value (hereinafter referred to as the "Luminance Assessment Threshold"). For example, the Luminance Assessment Threshold is set at the time of product delivery, etc., using an image previously taken under standard lighting conditions, based on the average luminance of the driver's face area image elements that can be conceivably determined if the driver's face were adequately detected. This threshold is, for example, the lowest conceivable average luminance value for the driver's face area image elements.

[0049] The Luminance Assessment Section 105 outputs data to the Exposure Control Section 1061 of the Optical Setting Control Section 106 or to the Image Processing Section 1062 of the Optical Setting Control Section 106 if the average luminance of the driver face area image elements is below the threshold for luminance assessment.

[0050] Specifically, the luminance assessment section 105 first outputs data to the exposure control section 1061 indicating that the average luminance of the driver's face area pixels is below the luminance assessment threshold. The exposure control section 1061, upon receiving this data, then performs exposure control. The exposure control section 1061 is described in detail below. If the exposure control section 1061 performs exposure control, the luminance assessment section 105 evaluates whether the average luminance of the driver's face area pixels after exposure control is greater than or equal to the luminance assessment threshold. The luminance assessment section 105 can acquire the average luminance of the driver's face area image elements after exposure control by instructing the luminance calculation section 104 to calculate it.

[0051] Luminance Assessment Section 105 outputs data to Image Processing Section 1062 indicating that the average luminance of the driver face area image elements is below the luminance assessment threshold after exposure control. Image Processing Section 1062 then performs a gain adjustment. Image Processing Section 1062 is described in detail below. After Image Processing Section 1062 performs a gain adjustment, Luminance Assessment Section 105 evaluates whether the average luminance of the driver face area image elements after the gain adjustment is greater than or equal to the luminance assessment threshold.The luminance assessment section 105 can determine the average luminance of the driver's face area image elements after the gain setting by instructing the luminance calculation section 104 to calculate it. In this way, the luminance assessment section 105 ensures that changes to the optical setting are made in the sequence of the exposure control section 1061 and the image processing section 1062.

[0052] In the event that the average luminance of the driver face area image elements after exposure control is assessed to be greater than or equal to the luminance assessment threshold, the luminance assessment section 105 does not output any data to the optical setting control section 106 indicating that the average luminance of the driver face area image elements is below the luminance assessment threshold.

[0053] If the luminance assessment section 105 has assessed that the average luminance of the driver face area image elements is below the threshold for luminance assessment, the optical adjustment control section 106 performs a control of the optical adjustment of the imaging device 20 according to the average luminance of the driver face area image elements calculated by the luminance calculation section 104.

[0054] Exposure control section 1061 of optical setting control section 106 performs exposure control when luminance assessment section 105 outputs data indicating that the average luminance of the driver's face area image elements is below the luminance assessment threshold. Specifically, exposure control section 1061 extends the exposure time of the imaging device 20. Furthermore, exposure control section 1061 extends the on-time of the lighting device 30, if necessary, in accordance with the changed exposure time. The extent to which exposure time or on-time is extended by exposure control section 1061 is considered to be pre-defined according to the average luminance of the driver's face area image elements.

[0055] The image processing section 1062 of the optical setting control section 106 performs gain control when the luminance assessment section 105 outputs data indicating that the average luminance of the driver's face area image elements after exposure control is below the threshold for luminance assessment. Specifically, the image processing section 1062 increases the gain value of the imaging device 20, thus brightening the image. The extent to which the gain value is increased by the image processing section 1062 is predefined according to the average luminance of the driver's face area image elements.

[0056] Furthermore, the optical adjustment control device 106 performs a control operation by which the optical adjustment of the imaging device 20 is reset to its initial value when a re-adjustment order has been issued by the re-adjustment arrangement section 108. Details of the re-adjustment arrangement section 108 are explained below.

[0057] When a new control instruction is issued, the exposure control section 1061 performs an exposure control operation to return the exposure time, or the exposure time and the power-on time, to the initial value.

[0058] When outputting a new control arrangement, the image processing section 1062 performs a gain adjustment to bring it into the preset state by resetting the gain value to the initial value.

[0059] Based on the result of the control performed by the optical adjustment control section 106, the misdetection evaluation section 107 assesses whether the driver's face was misdetected by the face detection section 102, i.e., it performs a misdetection evaluation. Specifically, the misdetection evaluation section 107 assesses whether the data relating to the imaging device 20, obtained as a result of the exposure control performed by the exposure control section 1061 of the optical adjustment control section 106, meet face misdetection conditions.Furthermore, the misdetection evaluation section 107 assesses whether the data relating to the imaging device 20, obtained as a result of performing the gain adjustment when a gain adjustment of the image processing section 1062 of the optical setting control section 106 has been performed, meet face misdetection conditions or not. In the first embodiment, the data relating to the imaging device 20, obtained as a result of the exposure control by the exposure control section 1061 or as a result of the gain adjustment by the image processing section 1062, include data on the luminance of the image acquired by the imaging device 20 or data on the gain value of the imaging device 20.

[0060] The misdetection evaluation section 107 evaluates that the face detection section 102 misdetected the driver's face if the data relating to the imaging device 20 meet the face misdetection conditions. Conversely, the misdetection evaluation section 107 evaluates that the face detection section 102 correctly detected the driver's face if the data relating to the imaging device 20 do not meet the face misdetection conditions.

[0061] That is to say, in the first embodiment, the face misdetection conditions are conditions for assessing whether the face detection section 102 has misdetected the driver's face or not.

[0062] Fig. Figure 4 is a view to illustrate a pictorial representation of an example of the condition in a vehicle when, in the first embodiment, the face of the driver is falsely detected by the face detection section 102.

[0063] For example, if the occupants of a vehicle are in a condition that is indicative of a serious injury or death, and the vehicle is in a state of health that could lead to serious injury or death, then the vehicle's occupants may be in a state of health that could lead to serious injury or death. Fig. As shown in Figure 3, the face detection section 102 detects the first face 202a as the driver's face. Since the first face 202a is indeed the driver's face, the face detection section 102 was able to correctly detect the driver's face.

[0064] It is now assumed that the driver subsequently recovered from the state he was in. Fig. As shown in image 3, a hand is moved to a position which, as seen from the imaging device 20, is in front of the driver's face.

[0065] This means, as in Fig. As shown in Figure 4, only the second face, 202b, which is the face of the occupant in the back seat, is detected as a face in the area for face detection 201. While face detection section 102 can no longer detect parts of the first face, 202a, which is the driver's face and which it had been able to detect previously, it can detect parts of the second face, 202b, in the area for face detection 201. Consequently, face detection section 102 detects the second face, 202b, as the driver's face. Therefore, face detection section 102 may misdetect the driver's face.

[0066] If the face detection section 102 has falsely detected the driver's face, the optical adjustment control section 106 can perform an optical adjustment based on the average luminance of the driver's face area pixels. Specifically, if the face detection section 102 falsely detects the driver's face, the luminance calculation section 104 calculates the average luminance of the falsely detected driver's face area pixels. If the luminance evaluation section 105 then determines that the average luminance of the falsely detected driver's face area pixels is below the threshold for luminance evaluation, the optical adjustment control section 106 performs an optical adjustment of the imaging device 20 based on the falsely detected driver's face area pixels.

[0067] In the above by means of Fig. As explained in example 4, this means that the optical setting control section 106 performs a control of the optical setting of the imaging device 20 according to the average luminance of the image elements of the second face area 203b when the second face 202b is misdetected as the driver's face.

[0068] Since, for example, light from the lighting device 30 normally reaches the rear seat with more difficulty than the driver's seat, rear seat occupants appear dark in the image. If the face detection section 102 misidentifies a rear seat occupant's face as the driver's, the average luminance of the second face area 203b, calculated by the luminance calculation section 104, may fall below the threshold for luminance assessment because the second face 202b is depicted darker. In this case, the optical setting control section 106 adjusts the optical setting to reflect the second face area 203b, which is the face of a different occupant than the driver.

[0069] If the optical setting is controlled in a manner tailored to the second facial area 203b, this results in the first facial area 203a, which should actually have been detected as the driver's facial area, being rendered brighter than necessary in the image, since it is an area that is already rendered brighter than the second facial area 203b. Specifically, for example, if exposure control is performed by the exposure control section 1061 of the optical setting control section 106, the average luminance of the image elements of the first facial area 203a becomes significantly higher than the average luminance considered to be the average luminance of the image elements of the driver's facial area.Furthermore, for example, if gain control is performed by the image processing section 1062 of the optical setting control section 106, the gain value of the first face area 203a becomes significantly larger than the gain value considered sufficient for a clear image of the driver's face. If, for example, the image processing section 1062 performs a gain adjustment, the increase in the gain value of the first face area 203a also results in so-called "overexposure" in that area. Fig. Figure 5 is a view showing a pictorial representation in which, in the first embodiment, overexposure has occurred in the area surrounding the driver's face in the image acquired by the imaging device 20.

[0070] Even if, for example, after a condition in which parts of the driver's face cannot be detected temporarily (cf. Fig. 4) the state in which the parts of the driver's face can be detected has returned (cf. Fig. 3), therefore, because the luminance of the image elements of the first face area 203a, which is the face area of ​​the driver, is not adequate, re-detection of the first face 202a, which is the face of the driver, as a face by the face detection section 102 becomes difficult.

[0071] The face detection section 102 then continuously detects the second face 202b, which is the face of the rear-seat occupant, as the driver's face. Because the face detection section 102 continuously detects the second face 202b as the driver's face, the driver monitoring section 103 continuously monitors the status of the second face 202b. Consequently, if the driver monitoring section 103 determines that the rear-seat occupant is experiencing microsleep, a warning is issued by the output device. This warning is unnecessary, as it is not actually a warning regarding the driver's microsleep.

[0072] Therefore, based on the result of the control performed by the optical setting control section 106, the misdetection evaluation section 107 performs a misdetection evaluation, taking into account that the face detection section 102 may detect the face of an occupant other than the driver as the driver's face (cf. Fig. 4) As described above, the misdetection evaluation section 107 performs a misdetection evaluation in which it is assessed whether, due to the exposure control performed by the exposure control section 1061 or due to the gain setting by the image processing section 1062, the data relating to the imaging device 20 meet the face misdetection conditions or not.

[0073] If exposure control was performed by the exposure control section 1061, the face misdetection condition is, for example, "whether the average luminance of the image elements of an area where a higher probability of an image of the driver's face is presumed to be in the area for face detection (hereinafter referred to as the "face presence presumption area") satisfies a preset condition (hereinafter referred to as the "first further evaluation condition") or not".

[0074] The area of ​​presumed facial presence in the first embodiment will now be explained.

[0075] Fig. Figure 6 is a view illustrating a pictorial representation of the face presence presumption area in the first embodiment.

[0076] Fig. Figure 6, as an example, is a view showing a pictorial representation of the driver's face area in an image 200, which was taken by the image processing section 1062 after an amplification setting adapted to the second face area 203b, when, as in Fig. Figure 4 shows that the second face 202b, which is the face of the rear seat occupant, was detected as the driver's face by face detection section 102 because the driver's face was temporarily in an undetectable state. For the sake of simplicity, in Fig. 6. The representation of the area for face detection 201 has been omitted. Furthermore, in Fig. 6 of the face presence presumption area is represented by 601. Furthermore, in Fig. 6 divided by 600 represents a headrest.

[0077] The face presence presumption area 601, as described above, is an area where a higher probability of the driver's face being captured within the face detection area is presumed. For example, the face detection area 201 may be set so large that the driver's face can be detected even if the driver's face moves to different positions. On the other hand, the driver's posture while driving is often nearly constant, so the position of the driver's face also frequently remains in a specific position. Consequently, even with a large face detection area setting 201, sections within the face detection area 201 will exist where the probability of capturing the driver's face is relatively high and sections where the probability is relatively low.The face presence presumption area 601 is set taking this circumstance into account, being an area set as a section of higher probability of an image of the driver's face in the face detection area 201.

[0078] The face presence prediction area 601 can, for example, be predicted by the exposure control section 1061 based on the position of the headrest 600. In this case, the face detection section 1061 can first detect the position of the headrest using a known image recognition technology such as pattern matching. Furthermore, data regarding the relationship between the position of the driver's face and the position of the headrest are pre-stored in a memory section with which the onboard image processing device 10 is equipped, in a state where a driver of normal stature is looking forward in a normal posture and in a normal position in the driver's seat.The exposure control section 1061 can infer the face presence guessing area 601 based on the detected position of the headrest and the data stored in the memory section regarding the relationship between the position of the driver's face and the position of the headrest.

[0079] The exposure control section 1061 can further divide the area for face detection 201 in image 200 into a left and right half, and from the halved area specify the area of ​​the side near the steering wheel as the face presence presumption area. For example, in a right-hand drive vehicle, which is assumed in the first embodiment, the area of ​​the left side of the halved area is specified as the face presence presumption area 601.

[0080] Furthermore, the exposure control section 1061 can, for example, with reference to the storage section, specify the facial area that was detected before the facial area of ​​the driver currently detected by the facial detection section 102 as the facial presence presumption area 601 based on stored facial data. For example, the facial detection section 102 detects, in the event that the condition in a vehicle that is in the Fig. The image shown in section 3 depicts a state similar to that shown in Fig. The image shown in Figure 4 has been changed, and the previously detected face of the driver (first face 202a) is no longer detected, so the face of the rear seat occupant (second face 202b) is assumed to be the driver's face. The exposure control section 1061 can also specify the previously detected first face area 203a of the first face 202a as the face presence presumption area 601.

[0081] Assuming that the driver's face is not behind the headrest, the exposure control section 1061 can also calculate the position of the headrest based on the adjustment range of the seat and infer the face presence guessing range 601.

[0082] The face presence presumption area 601 may also be an area on the image that was predefined for the image 200 taken by the recording device 20 with reference to the position of the driver's face in a state in which a driver of normal stature in a normal posture in a normal position of the driver's seat is looking forward.

[0083] The exposure control section 1061 should be able to specify the face presence presumption area 601 in an image using a method.

[0084] As a further initial evaluation condition, it may also be necessary to determine whether the average luminance of the image elements of the face presence presumption area 601 (hereinafter referred to as "presumption area image elements") is greater than or equal to a preset value (hereinafter referred to as the "driver's seat luminance evaluation threshold"). For example, the driver's seat luminance evaluation threshold could be set to the lowest luminance value that is considered too bright for detecting the driver's face.

[0085] In the event that the average luminance of the guess area image elements is greater than or equal to the threshold for the driver's seat luminance assessment, the face misdetection condition is deemed to be met, and the misdetection assessment section 107 assesses that the face detection section 102 misdetected the driver's face.

[0086] If the average luminance of the presumption area image elements is greater than or equal to the threshold for the driver's seat luminance assessment, it is assumed with a high probability that the exposure control of the imaging device 20 is carried out according to the average luminance of the image elements of the face area of ​​an occupant other than the driver who is located in a darker area than the driver's seat.

[0087] If the image processing device 1062 has performed a gain adjustment, the face misdetection condition is, for example, "whether the gain value of the imaging device 20 satisfies a preset condition (hereinafter referred to as the "second further evaluation condition") or not". The misdetection evaluation section 107 should be able to acquire data on the gain value of the imaging device 20, for example, from the image processing section 1062.

[0088] As a second further evaluation condition, it can also be specified whether, for example, the gain value of the imaging device 20 has become greater than or equal to a preset threshold value (hereinafter referred to as the "gain assessment threshold"). The gain assessment threshold is set as a gain value that is considered too bright for a clear image of the driver's face.

[0089] For example, if the gain value of the imaging device 20 is greater than or equal to the gain assessment threshold, the face misdetection condition is deemed to be met, and the misdetection assessment section 107 assesses that the face detection section 102 misdetected the driver's face.

[0090] If the gain value of the imaging device 20 is greater than or equal to the threshold for gain assessment, it is assumed with a high probability that the gain setting of the imaging device 20 is carried out according to the average luminance of the image elements of the face area of ​​an occupant other than the driver, who is located in a darker place than the driver's seat.

[0091] Furthermore, if the image processing section 1062 has adjusted the gain, the face false detection condition can be specified as whether "an overexposed area of ​​the driver's face meets a preset condition (hereinafter referred to as the "judgment condition") or not." The image processing section 1062 can determine the face presence presumption area 601 using the same method as the exposure control section 1061 uses to determine the face presence presumption area 601. The method for determining the face presence presumption area 601 has already been explained, so a detailed explanation is omitted here.

[0092] The assessment condition can, for example, be specified as whether the overexposure area of ​​the facial presence presumption area 601 is greater than or equal to a preset threshold (hereinafter referred to as the "overexposure assessment threshold") or whether the overexposure area of ​​the facial presence presumption area 601 is large or not. The overexposure assessment threshold is a preset overexposure area at which it can be assumed that parts of the driver's face cannot be detected.

[0093] In the event that the overexposure area of ​​the face presumption area 601 is greater than or equal to the overexposure assessment threshold, the face misdetection condition is deemed to be met, and the misdetection assessment section 107 assesses that the face detection section 102 has misdetected the driver's face.

[0094] If the overexposure area of ​​the driver's face is greater than or equal to the threshold for overexposure assessment, it is assumed with a high probability that the gain setting of the imaging device 20 will be carried out according to the average luminance of the image elements of the face area of ​​an occupant other than the driver who is located in a darker area than the driver's seat.

[0095] The misdetection evaluation section 107 can also assess, by combining at least two arbitrary conditions from a plurality of face misdetection conditions such as those described above, whether the data relating to the imaging device 20 meet the face misdetection conditions or not.

[0096] For example, if exposure control was performed by the exposure control section 1061 and gain adjustment was also performed by the image processing section 1062, the misdetection evaluation section 107 can assess "whether the average luminance of the presumption area image elements meets the first further evaluation condition or not" and further assess "whether the overexposure area of ​​the face presence presumption area 601 meets the evaluation condition or not". If the average luminance of the presumption area image elements meets the first further evaluation condition and the overexposure area of ​​the face presence presumption area 601 also meets the evaluation condition, the misdetection evaluation section 107 assesses that the face detection section 102 misdetected the driver's face.

[0097] If the false detection evaluation section 107 has determined that the data relating to the imaging device 20 meet the face false detection conditions, it considers that the face detection section 102 has falsely detected the driver's face and issues a range reduction order to the range reduction section 109, reducing the area for face detection. Furthermore, the false detection evaluation section 107 issues a recalibration order to the recalibration order section 108 to instruct the optical adjustment control section 106 to perform a control action resetting the exposure time, the on-time, or the gain value to its initial value.

[0098] On the other hand, the misdetection evaluation section 107 assesses that the face detection section 102 was able to correctly detect the driver's face and outputs face detection presence data to the driver monitoring section 103 if the data relating to the imaging device 20 do not meet the face misdetection conditions.

[0099] If a recalibration message is issued by the fault detection evaluation section 107, the recalibration arrangement section 108 issues a recalibration arrangement to the optical adjustment control section 106 so that control is carried out by which the optical adjustment of the imaging device 20 is reset to an appropriate value. In the first embodiment, the appropriate value of the optical adjustment of the imaging device 20 is the initial value of the optical adjustment of the imaging device 20.

[0100] For example, in the re-control section 108, it can be specified whether, based on the re-control message, the exposure control section 1061 is instructed to perform an exposure control that resets the exposure time to the initial value, or the image processing section 1062 is instructed to perform an gain adjustment that resets the gain value to the initial value. In the fault detection evaluation section 107, the re-control message contains data about which control should be performed: either the exposure control or the gain adjustment.

[0101] The area reduction section 109 reduces the area for face detection when an area reduction order has been issued by the false detection evaluation section 107. The area reduction section 109 reduces the area for face detection such that it results in an area that does not include the area of ​​the driver's face that was falsely detected by the face detection section 102. For this to occur, it is sufficient for the area reduction section 109 to acquire the face data issued by the face detection section 102 via the false detection evaluation section 107 and, based on this acquired face data, to specify the area of ​​the driver's face that was falsely detected by the face detection section 102.

[0102] Fig. 7A and Fig. Figure 7B shows illustrations of an example of the face detection area, which in the first embodiment was reduced by the area reduction section 109. In the Fig. 7A and Fig. Image 200 shown in 7B was taken from the state shown in the image as in Fig. Figure 4 shows a return to the state in which the parts of the first face 202a, which is the face of the driver, are depicted.

[0103] The area reduction section 109 narrows the area for face detection, for example, by moving an edge of the right side of a rectangle that defines the face detection area to the left. In doing so, the area reduction section 109 narrows the area for face detection until it results in an area that does not include the area of ​​the driver's face that was falsely detected by the face detection section 102 (see...). Fig. 7A).

[0104] Furthermore, the area reduction section 109, for example, narrows the face detection area from the lower right node of a rectangle that defines the face detection area to the upper left node, while maintaining the length-to-width ratio of the face detection area. In doing so, the area reduction section 109 narrows the face detection area until it results in an area that does not include the driver's face area that was falsely detected by the face detection section 102 (see Figure 102). Fig. 7B).

[0105] For example, the area reduction section 109 updates the face detection area defined by the face detection area data stored in the memory section to the face detection area after reduction.

[0106] Then the area reduction section 109 outputs the data that the area for face detection has been reduced to the new detection arrangement section 110.

[0107] If the area reduction section 109 outputs data indicating that the area for face detection has been reduced, the re-detection arrangement section 110 causes the face detection section 102 to perform a re-detection of the driver's face in the area for face detection after the reduction by the area reduction section 109.

[0108] The onboard image processing device 10 according to the first embodiment has the structure described above.

[0109] It is now assumed that the onboard image processing device 10 does not include the false detection evaluation section 107, and that the onboard image processing device 10 has no function to evaluate whether the face detection section 102 has falsely detected the driver's face or not. That is, as with the conventional technique described above, the possibility that the face detected by the image is other than the driver's face is not taken into account.

[0110] In this case, the face detection section 102, if the condition described above, which is shown in the image of Fig. As shown in Figure 4, for example, a situation arises in which, due to some external influences, parts of the driver's face cannot be temporarily detected, and the face of an occupant other than the driver is detected as the driver's face. Then, data regarding the face of the occupant other than the driver, which is the face of the occupant other than the driver that has been identified as the driver's face, is output by the face detection section 102 to the driver monitoring section 103 and the luminance calculation section 104.

[0111] As described above, light from the lighting device 30, for example, reaches the rear seat with more difficulty than the driver's seat, so that rear seat occupants appear dark in the image. If the face detection section 102 misidentifies the face of a rear seat occupant as the driver's face, the average luminance of the driver's face area image elements, calculated by the luminance calculation section 104, will consequently be much lower than the average luminance considered to be the average luminance of the driver's face area image elements. As a result, the optical adjustment control section 106 may adjust the optical adjustment to reflect the face area of ​​the rear seat occupant.Even if, for example, after a state in which parts of the driver's face cannot be detected temporarily, the state in which parts of the driver's face can be detected has returned (cf. . Fig. 3) Therefore, because the luminance of the image elements in the driver's face area is not adequate, re-detection of the driver's face by the face detection section 102 becomes difficult. Specifically, the area around the driver's seat, which is better illuminated by the light from the lighting device 30, etc., than the rear seat, becomes too bright, resulting in, for example, overexposure in the image. This makes it more difficult for the face detection section 102 to detect the driver's face than in an image taken with the optical settings of the preset state. Consequently, the face detection section 102 continues to detect the face of the occupant in the rear seat as the driver's face.

[0112] Furthermore, the driver monitoring section 103 continuously monitors the facial features of the rear seat occupant. For example, if the driver monitoring section 103 determines that the rear seat occupant is experiencing microsleep, a warning is issued by the output device. This is an unnecessary warning, as it is not actually a warning regarding the driver's microsleep.

[0113] In contrast, the onboard image processing device 10 according to the first embodiment as described above comprises the misdetection evaluation section 107. The misdetection evaluation section 107 evaluates whether the driver's face has been misdetected or not by assessing, after performing a control of the optical setting according to the average luminance of the driver's face area image elements detected by the face detection section 102, whether the data relating to the imaging device 20 meet the face misdetection conditions or not.If the misdetection evaluation section 107 then evaluates a misdetection of the driver's face, it issues a recalibration instruction to the optical setting control section 106 to instruct the recalibration instruction section 108 to execute a control action that resets the exposure time, the on-time, or the gain value to its initial value. Therefore, the onboard image processing device 10 can prevent the control of an unsuitable optical setting based on a misdetection of the driver's face. Furthermore, the onboard image processing device 10 can prevent a state from persisting after the control of an unsuitable optical setting based on a misdetection of the driver's face, in which the detection of the actual driver's face becomes difficult.

[0114] The operation of the onboard image processing device 10 according to the first embodiment is explained.

[0115] Fig. Figure 8 is a flowchart to explain the operation of the onboard image processing device 10 according to the first embodiment.

[0116] The onboard image processing device 10 repeats the process shown in the flowchart while the vehicle is in motion. Fig. 8 shown operations.

[0117] The image acquisition section 101 acquires an image from the imaging device 20 (step ST801).

[0118] The image acquisition section 101 outputs the acquired image to the face detection section 102 and the luminance calculation section 104.

[0119] The face detection section 102 detects the driver's face within a defined face detection area in the image acquired by the image acquisition section 101 in step ST801 (if "YES" is selected in step ST802). The face detection section 102 outputs the detected driver's facial data to the driver monitoring section 103, the luminance calculation section 104, and the false detection evaluation section 107. Furthermore, the face detection section 102 stores the detected driver's facial data in the storage section.

[0120] If the face detection section 102 cannot detect the driver's face in the face detection area (at "NO" in step ST802), the processing by the onboard image processing device 10 returns to step ST801.

[0121] Luminance Calculation Section 104 calculates the average luminance of the driver's face area image elements (step ST803) in the image acquired in step ST801 by Image Acquisition Section 101. Luminance Calculation Section 104 outputs face area luminance data to Luminance Evaluation Section 105.

[0122] The Luminance Assessment Section 105 assesses, based on the face area luminance data output by the Luminance Calculation Section 104, whether the average luminance of the driver's face area pixels is greater than or equal to a luminance assessment threshold (step ST804). If, in step ST804, the Luminance Assessment Section 105 determines that the average luminance of the driver's face area pixels is below the luminance assessment threshold (in the case of "NO" in step ST804), it outputs data indicating that the average luminance of the driver's face area pixels is below the luminance assessment threshold to the Optical Adjustment Control Section 106.

[0123] If the luminance assessment section 105 in step ST804 has assessed that the average luminance of the driver face area image elements is greater than or equal to the luminance assessment threshold (if “YES” in step ST804), the processing of the onboard image processing device 10 ends.

[0124] The optical setting control section 106 performs a control of the optical setting of the imaging device 20 according to the average luminance of the driver face area image elements, which was calculated by the luminance calculation section 104 (step ST805).

[0125] The fault detection evaluation section 107 performs a fault detection evaluation (step ST806) based on the result of the control performed in step ST805 by the optical adjustment control section 106.

[0126] If the false detection evaluation section 107 has determined that the face detection section 102 has falsely detected the driver's face (if "YES" in step ST807), it issues a range reduction order to the range reduction section 109, reducing the area for face detection. Furthermore, the false detection evaluation section 107 issues a recontrol message to the recontrol order section 108.

[0127] On the other hand, if the misdetection evaluation section 107 has evaluated that the face detection section 102 has not misdetected the driver's face (at "NO" in step ST807), it outputs face detection presence data to the driver monitoring section 103, and the processing of the onboard image processing device 10 ends.

[0128] If a recontrol message was issued by the fault detection evaluation section 107 in step ST807, the recontrol arrangement section 108 issues a recontrol arrangement to the optical setting control section 106.

[0129] If a re-control instruction was issued by the re-control arrangement section 108, the optical setting control device 106 performs a control by which the optical setting of the imaging device 20 is reset to the initial value (step ST808).

[0130] If a range reduction order was issued by the false detection evaluation section 107 in step ST807, the range reduction section 109 reduces the area for face detection (step ST809). The range reduction section 109 then outputs the data indicating that the area for face detection has been reduced to the new detection order section 110.

[0131] If, in step ST809, the area reduction section 109 outputs data indicating that the face detection area has been reduced, the re-detection arrangement section 110 causes the face detection section 102 to re-detect the driver's face in the face detection area after the reduction by the area reduction section 109 (step ST810). The processing by the onboard image processing device 10 then ends.

[0132] In the by means of Fig. 8. The operation of the onboard image processing device 10, as explained, provides the area reduction section 109 when the processing of the onboard image processing device 10 ends, in the event that the area for face detection has been reduced (see step ST809 in Fig. 8) in the data relating to the face detection area stored in the memory section, reset the face detection area to its initial value.

[0133] Regarding the operation of the onboard image processing device 10, which is described above by means of Fig. As explained in section 8, it is also possible that, if the area reduction section 109 has reduced the area for face detection, it resets the area for face detection to its initial value if, even after a certain time has elapsed since the reduction of the area for face detection, the face detection section 102 does not detect the driver's face (see step ST802 of section 8). Fig. 8) In this case, for example, it can be ensured that the area reduction section 109 acquires data from the face detection section 102 about whether the driver's face has been detected or not.

[0134] Fig. 9 is a flowchart to illustrate the operation in steps ST804 to ST807 in Fig. 8 to explain in more detail.

[0135] In step ST901, the luminance assessment section 105 evaluates based on the values ​​provided by the luminance calculation section 104 in step ST803. Fig. 8 output face area luminance data, whether the average luminance of the driver face area pixels is greater than or equal to a luminance assessment threshold or not (step ST901). The operation in step ST901 is the operation of step ST804 in Fig. 8.

[0136] If the luminance assessment section 105 in step ST901 has assessed that the average luminance of the driver face area image elements is greater than or equal to the luminance assessment threshold (if “YES” in step ST901), the processing of the onboard image processing device 10 ends.

[0137] If the luminance assessment section 105 has assessed in step ST901 that the average luminance of the driver face area image elements is below the threshold for luminance assessment (if "NO" in step ST901), it outputs data that the average luminance of the driver face area image elements is below the threshold for luminance assessment to the exposure control section 1061, and the exposure control section 1061 performs an exposure control of the imaging device 20 (step ST902).

[0138] If the exposure control section 1061 performs exposure control in step ST902, the luminance assessment section 105 assesses whether the average luminance of the driver face area image elements after exposure control is greater than or equal to the luminance assessment threshold (step ST903).

[0139] If the luminance assessment section 105 has assessed in step ST903 that the average luminance of the driver face area image elements after exposure control is greater than or equal to the threshold for luminance assessment (if “YES” in step ST903), the processing by the onboard image processing device 10 proceeds to step ST906. If the luminance assessment section 105 has assessed in step ST903 that the average luminance of the driver face area image elements after exposure control is below the threshold for luminance assessment (if "NO" in step ST903), it outputs data that the average luminance of the driver face area image elements is below the threshold for luminance assessment to the image processing section 1062, and the image processing section 1062 performs a gain adjustment (step ST904).

[0140] If the image processing section 1062 performs a gain adjustment in step ST904, the luminance assessment section 105 assesses whether the average luminance of the driver face area image elements after the gain adjustment is greater than or equal to the luminance assessment threshold (ST905).

[0141] If the luminance assessment section 105 in step ST905 has assessed that the average luminance of the driver face area image elements after the gain setting is below the threshold for luminance assessment (if "NO" in step ST905), it outputs data that the average luminance of the driver face area image elements is below the threshold for luminance assessment to the exposure control section 1061, and the processing by the onboard image processing device 10 returns to step ST902.

[0142] If, in step ST905, the Luminance Evaluation Section 105 has determined that the average luminance of the driver face area image elements after gain adjustment is greater than or equal to the luminance evaluation threshold (in the case of "YES" in step ST905), the Misdetection Evaluation Section 107 performs a misdetection evaluation based on the result of the control performed by the Optical Adjustment Section 106 (step ST906). Specifically, the Misdetection Evaluation Section 107 assesses whether the data relating to the imaging device 20, obtained as a result of performing exposure control when exposure control was performed by the Exposure Control Section 1061 in step ST902, meet face misdetection conditions.Furthermore, the misdetection evaluation section 107 assesses whether the data relating to the imaging device 20, which are obtained as a result of performing the gain adjustment when a gain adjustment was performed by the image processing section 1062 in step ST904, meet face misdetection conditions or not.

[0143] If the data relating to the imaging device 20, obtained as a result of the exposure control, meet the face misdetection conditions, or if the data relating to the imaging device 20, obtained as a result of the gain adjustment, meet the face misdetection conditions (at "YES" in step ST907), the misdetection evaluation section 107 judges that the face detection section 102 has misdetected the driver's face and issues a face detection area reduction order to the area reduction section 109. Furthermore, the misdetection evaluation section 107 issues a recalibration order to the recalibration order section 108. Then, processing by the onboard image processing device 10 proceeds to step ST808 in Fig. 8 over.

[0144] In the event that the data relating to the imaging device 20, obtained as a result of performing the exposure control, do not meet the face misdetection conditions, and furthermore, the data relating to the imaging device 20, obtained as a result of performing the gain adjustment, do not meet the face misdetection conditions (in case of "NO" in step ST907), the misdetection evaluation section 107 assesses that the face detection section 102 has not misdetected the driver's face and outputs face detection presence data to the driver monitoring section 103. Then the processing of the onboard image processing device 10 ends.

[0145] In the first embodiment described above, the onboard image processing device 10 comprises the area reduction section 109, wherein the area reduction section 109 causes a reduction of the area for face detection when an area reduction order is issued by the false detection evaluation section 107. However, there is no limitation to this, and a configuration is also possible in which the onboard image processing device 10 does not include the area reduction section 109.In this case, if the optical setting control section 106 performs a control based on the re-control arrangement issued by the re-control arrangement section 108, thereby resetting the optical setting of the imaging device 20 to the initial value, the re-control arrangement section 108 causes the face detection section 102 to re-detect the driver's face in the area for face detection of an image after resetting the optical setting to the initial value.

[0146] However, because the onboard image processing device 10 includes the area reduction section 109, the detection of the driver's face can be carried out more reliably when performing the re-detection of the driver's face after the assessment that the driver's face was falsely detected.

[0147] Furthermore, in the foregoing first embodiment, it was assumed that an area for face detection is set in the image, with no limitation on this, and that setting an area for face detection in the image is not absolutely necessary.

[0148] Furthermore, in the aforementioned first embodiment, it is specified that when the optical setting of the imaging device 20 is controlled to an appropriate value by the onboard image processing device 10, if the misdetection assessment has determined that the driver's face has been misdetected, the optical setting of the imaging device 20 is controlled to a preset state.However, there is no limitation here, and the onboard image processing device 10 can be used to control the optical setting of the imaging device 20 to an appropriate value, which is carried out when, during the misdetection assessment, it has been judged that the driver's face has been misdetected; also to control the exposure time of the imaging device 20 to the exposure time that, chronologically speaking, was set one step earlier; to control the switch-on time of the imaging device 20 to the switch-on time that, chronologically speaking, was set one step earlier; or to control the gain value of the imaging device 20 to the gain value that, chronologically speaking, was set one step earlier.Since the imaging device 20 imaged the driver's face until shortly before the driver's face was falsely detected using the optical setting with these values, it is conceivable that even if the exposure time is set to the exposure time that was set one step earlier, chronologically speaking, the switch-on time to the switch-on time that was set one step earlier, chronologically speaking, or the gain value to the gain value that was set one step earlier, chronologically speaking, the driver's face can be adequately imaged.

[0149] The appropriate value of the optical setting in the first embodiment includes, apart from the initial value of the exposure time, the initial value of the switch-on time or the initial value of the gain value as described above, also the exposure time that was set one step earlier, the switch-on time that was set one step earlier, or the gain value that was set one step earlier.

[0150] According to the foregoing, the setup comprises an onboard image processing device 10 according to the first embodiment, an image acquisition section 101 which acquires an image from an imaging device 20 in which an area inside a vehicle is captured, which includes an area in which the face of a driver sitting in the driver's seat should be located, a face detection section 102 which, based on the image acquired by the image acquisition section 101, detects the driver's face and the driver's facial area in this image, a luminance calculation section 104 which calculates the average luminance of the image elements in the facial area of ​​the driver detected by the face detection section 102, and a luminance evaluation section 105 which evaluates,whether the average luminance calculated by the luminance calculation section 104 is greater than or equal to a luminance assessment threshold or not, an optical adjustment control section 106, which, in the event that the luminance assessment section 105 has assessed that the average luminance is below the luminance assessment threshold, performs a control of the optical adjustment of the imaging device 20 according to the average luminance, a misdetection assessment section 107, which, based on the result of the control performed by the optical adjustment control section 106, assesses whether the face detection section 102 has misdetected the driver's face or not, and a re-control arrangement section 108, which, in the event that the misdetection assessment section 107 assesses that the face detection section 102 misdetected the driver's face,The optical setting control section 106 outputs a recalibration order, which resets the optical setting of the imaging device 20 to an appropriate value. Therefore, the onboard image processing device 10 can prevent the control of an unsuitable optical setting based on a misdetected driver's face. Second embodiment

[0151] In the second embodiment, a form of implementation is explained in which the precision of the false detection assessment can be improved by taking into account the position of the driver's face detected in the image.

[0152] The structure of the onboard image processing device 10 according to the second embodiment is the same as the structure of the onboard image processing device 10 according to the first embodiment, so a duplicate explanation is omitted.

[0153] The onboard image processing device 10 according to the second embodiment and the onboard image processing device 10 of the first embodiment differ in the specific operation of the fault detection evaluation section 107.

[0154] In the second embodiment, the false detection evaluation section 107 performs a false detection evaluation of the driver's face based on the result of the control performed by the optical adjustment control section 106 and based on the position of the driver's face as detected by the face detection section 102. Specifically, the false detection evaluation section 107 assesses whether the data relating to the imaging device 20, obtained as a result of the control performed by the optical adjustment control section 106, meet the face false detection conditions and whether, furthermore, the position of the driver's face, as detected by the face detection section 102, is in a non-driver area or not.

[0155] In the second embodiment, the non-driver area is an area where it can be assumed with a high degree of probability that the face of an occupant other than the driver's face is imaged in the face detection area. The non-driver area is, for example, preset by a user, etc., to match the face detection area, which is set taking into account the imaging area of ​​the imaging device 20.

[0156] For example, it is assumed that the installation position and imaging area of ​​the imaging device 20 are the installation position and imaging area as shown in Fig. Figure 2A illustrates this. The imaging area is generally set to capture the driver's face in a substantially central position in the left-right direction of the image, for example, with the headrest in the center. However, in the image containing the imaging area, the face detection area may be set so extensively that the driver's face, which is normally captured in the center of the image, can be detected even if the driver's face has moved to different positions. If, for example, the face of a rear-seat passenger appears in such a face detection area, there is a high probability that the rear-seat passenger's face will appear at an edge of the face detection area.Consequently, in this case, the non-driver area is set to the edge area of ​​the face detection area. The non-driver area is related to the face detection area and stored in a memory section.

[0157] One possible reason for the extensive changes in exposure time, on-time, or gain value due to the control performed by the optical adjustment control section 106 is the occurrence of a phenomenon that makes it difficult for the light from the lighting device 30 to reach the driver's face. For example, the driver's face may appear dark if the driver is wearing a hat, and depending on the angle, the light from the lighting device 30 may be blocked by the hat. In this case, the optical adjustment control section 106 adjusts the settings based on the correctly detected area of ​​the driver's face, so that, for example, the average luminance of the driver's face image elements may temporarily become greater than or equal to the threshold for driver seat luminance assessment.Even if the average luminance of the driver's face area image elements temporarily reaches the threshold for driver's seat luminance assessment, it cannot always be definitively stated that control has been adapted to the face area of ​​an occupant other than the driver. Therefore, in the second embodiment, the misdetection assessment section 107, in addition to the face misdetection conditions, also considers the position of the driver's face detected by the face detection section 102 to determine whether the face detection section 102 has misdetected the driver's face or not.

[0158] The false detection evaluation section 107 can calculate the position of the driver's face, detected by the face detection section 102, within the image based on the facial data. For example, the false detection evaluation section 107 can use the coordinates of the center of the facial area, based on the facial data, as the position of the driver's face. Furthermore, the false detection evaluation section 107 can also use a frame indicating the driver's facial area, based on the facial data output by the face detection section 102, as the position of the driver's face.

[0159] If the position of the driver's face, as detected by face detection section 102, is within the non-driver area, it is highly likely that the face detected by face detection section 102 is the face of another occupant. Conversely, if the position of the driver's face is not within the non-driver area, the driver's face has not moved excessively, and it can be said with a high degree of probability that the optical adjustment control section 106 has performed the necessary control adjustment adapted to the correctly detected area of ​​the driver's face.

[0160] The misdetection evaluation section 107 evaluates that the face detection section 102 has misdetected the driver's face if, as a result of the control by the optical setting control section 106, the face misdetection conditions are met, and furthermore, the position of the driver's face detected by the face detection section 102 is in the non-driver area.

[0161] The operation of the onboard image processing device 10 according to the second embodiment is explained. The basic operation of the onboard image processing device 10 according to the second embodiment is the same as the operation described in the first embodiment by means of Fig. 8 was explained, so a duplicate explanation is omitted. Fig. Figure 10 is a flowchart to illustrate the operation of steps ST804 to ST807 in the onboard image processing device 10 according to the second embodiment. Fig. 8 to explain in more detail. Unlike the first embodiment, the detailed operation of the false detection evaluation section 107 in the second embodiment is explained by means of Fig. 10 explained.

[0162] The specific operation of steps ST901 to ST907 in Fig. 10 is identical to the specific operation of steps ST901 to ST907 in Fig. 9, which was explained in the first embodiment, so that a double explanation is omitted.

[0163] In the event that in step ST907 the data relating to the imaging device 20 obtained as a result of performing the exposure control meet the face misdetection conditions, or the data relating to the imaging device 20 obtained as a result of performing the gain adjustment meet the face misdetection conditions (if “YES” is given in step ST907), the misdetection evaluation section 107 assesses whether the position of the driver’s face, which was detected by the face detection section 102, is in the non-driver area or not (step ST1001).

[0164] If step ST1001 determines that the driver's face is located in the non-driver area ("YES" in step ST1001), the false detection evaluation section 107 determines that the face detection section 102 falsely detected the driver's face. The false detection evaluation section 107 then issues an area reduction order to the area reduction section 109, reducing the area for face detection. Furthermore, the false detection evaluation section 107 issues a recalibration message to the recalibration order section 108. Processing by the onboard image processing device 10 then proceeds to step ST808. Fig. 8 over.

[0165] If, on the other hand, step ST1001 determined that the driver's face was not in the non-driver area ("NO" in step ST1001), the false detection evaluation section 107 evaluates that the face detection section 102 did not falsely detect the driver's face. The false detection evaluation section 107 then outputs face detection presence data to the driver monitoring section 103. The processing by the onboard image processing device 10 then ends.

[0166] Therefore, based on the result of the control performed by the optical adjustment control section 106 and the driver's position detected by the face detection section 102, the misdetection evaluation section 107 performs a misdetection evaluation of the driver's face. This improves the precision of the misdetection evaluation of the driver's face.

[0167] In the foregoing explanation, a false detection assessment is carried out by the false detection assessment section 107 assessing whether, due to the control effected by the optical setting control section 106, the face false detection conditions are met, and whether the position of the driver's face, detected by the face detection section 102, is in the non-driver area or not.

[0168] However, the assessment, which is carried out in addition to the assessment of whether the result of the control carried out by the optical setting control section 106 meets the face misdetection conditions or not, is not limited to the assessment described above.

[0169] For example, the false detection evaluation section 107 compares the position of the driver's face, based on the current face data output by face detection section 102 (which triggered the control action by optical setting control section 106), with the position of the driver's face based on face data chronologically one point earlier than that output by face detection section 102. The false detection evaluation section 107 can then perform processing to assess whether, as a result of this comparison, the difference in the position coordinates of the driver's face based on both sets of face data is greater than or equal to a preset threshold, in addition to assessing whether the face false detection conditions are met.Furthermore, in this case, for example, the misdetection evaluation section 107 may not compare the positions of the driver's face, but may compare specific points of the driver's face or compare the size of the frame showing the driver's facial area, and make an assessment of whether the difference between the specific points of the face is greater than or equal to a threshold, or of whether the difference in the size of the frame showing the driver's facial area is greater than or equal to a threshold, in addition to assessing whether the face misdetection conditions are met.

[0170] If the false detection evaluation section 107, in addition to evaluating whether the face detection conditions are met or not, performs an evaluation regarding the position of the driver's face or an evaluation regarding specific points of the driver's face, the precision of a false detection evaluation may be affected if, in the event that the setting position and imaging area of ​​the imaging device 20 are similar to the setting position and imaging area in Fig. 2B are set so that the face of the occupant in the front passenger seat is misdetected as the driver's face. It can be said that if there is a large change in the position of the driver's face or specific points on the driver's face, there is a high probability that the face detection section 102 will misdetect the face of the occupant in the front passenger seat as the driver's face. That is, it can be said that the control carried out by the optical setting control section 106 is, with a high probability, a control carried out according to the average luminance of the image elements within the face area of ​​the occupant in the front passenger seat, if the data relating to the imaging device 20 meet the face misdetection conditions.Because the misdetection evaluation section 107, in addition to evaluating whether the face detection conditions are met or not, performs an evaluation regarding the position of the driver's face or an evaluation regarding specific points of the driver's face, it can therefore be judged that the face detection section 102 misdetected the face of the occupant in the front passenger seat as the driver's face.

[0171] As with the first embodiment, a configuration is also possible in the second embodiment in which the onboard image processing device 10a does not include the area reduction section 109. Furthermore, setting an area for face detection in the image is not absolutely necessary.

[0172] As described above, the onboard image processing device 10 according to the second embodiment comprises the same structure as the onboard image processing device 10 according to the first embodiment, and the misdetection evaluation section 107 is configured to perform the misdetection evaluation based on the result of the control performed by the optical adjustment control section 106 and based on the position of the driver's face, which was detected by the face detection section 102. Therefore, the onboard image processing device 10a can prevent the control of an inappropriate optical adjustment based on a misdetected driver's face, and the precision of the misdetection evaluation can be improved by taking into account the position of the detected driver's face when performing the misdetection evaluation. Third embodiment

[0173] The third embodiment describes a form of implementation in which the precision of the false detection assessment can be improved by taking into account the unwinding length of a safety belt.

[0174] Fig. Figure 11 is a view showing a structural example of an onboard image processing device 10a according to the third embodiment.

[0175] In Fig. 11 structures are identical to those of the onboard image processing device 10, which are by means of Fig. 1 as explained in the first embodiment, provided with the same reference numerals and a double explanation omitted.

[0176] The onboard image processing device 10a according to the third embodiment differs from the onboard image processing device 10 of the first embodiment in that it includes a roll-off length detection section 111.

[0177] The roll-off length detection section 111 detects the roll-off length of a seat belt mounted on the driver's seat.

[0178] The unwinding length detection section 111, for example, detects the unwinding length of the seat belt in an image acquired by the image acquisition section 101, based on the amount of movement of the position of a marker provided on the seat belt. The marker was pre-placed on the seat belt. The marker can be detected by the imaging device 20. If, for example, the imaging device 20 is an infrared camera, the marker is a marker detectable by an infrared camera.

[0179] Furthermore, it can also be a roll-off length detection section 111, which detects the roll-off length of the seat belt by calculating the roll-off length of the seat belt based on a rotation quantity detected by a roll-off sensor (not shown), which is mounted, for example, near the seat belt mounted on the driver's seat and has a section that rotates according to the amount of movement of the seat belt.

[0180] The roll-out length of the seat belt detected by the roll-out length detection section 111 is, for example, the roll-out length based on the seat belt in a non-used state (hereinafter referred to as "roll-out length based on the non-use state").

[0181] Furthermore, the roll-off length of the seat belt detected by the roll-off length detection section 111 can, for example, also be the roll-off length based on the seat belt in a fastened state with the driver in a normal seating position (hereinafter referred to as "roll-off length based on the

[0182] (referred to as "state of use"). If the driver changes their posture from the normal sitting position for any purpose, the seat belt's roll-off length can also change. If the roll-off length detection section 111 detects the roll-off length based on the state of use, it can detect the seat belt's roll-off length according to a more accurate change in the driver's posture, taking into account the driver's stature, than if the roll-off length were detected based on the non-state of use.

[0183] The roll-off length detection section 111 outputs the detected roll-off length to the false detection evaluation section 107.

[0184] The roll-off length detection section 111 repeatedly performs the detection operation at least at the same time as the face detection section 102 performs a detection operation. The detected roll-off length is related to the face data and stored in a memory section. For example, when storing the face data in the memory section, the face detection section 102 acquires the roll-off length data from the roll-off length detection section 111 and stores it related to the face data in the memory section.

[0185] The misdetection evaluation section 107 performs a misdetection evaluation based on the result of the control carried out by the optical setting control section 106 and based on the roll length detected by the roll length detection section 111.

[0186] Specifically, the false detection evaluation section 107 compares, for example, the roll-off length corresponding to the current face data output by the face detection section 102, which triggered the control operation by the optical setting control section 106, with the roll-off length corresponding to the face data chronologically one point prior to the face data output by the face detection section 102. Then, in addition to assessing whether the face false detection conditions are met, the false detection evaluation section 107 also assesses whether the difference in the relevant roll-off length is less than or equal to a preset threshold value (hereinafter referred to as the "first threshold for roll-off length assessment").

[0187] The misdetection assessment section 107 assesses that the face detection section 102 has misdetected the driver's face if the result of the control by the optical setting control section 106 meets the face misdetection conditions and, in addition, the difference in rolling length is less than or equal to the first threshold for rolling length assessment.

[0188] On the other hand, the misdetection evaluation section 107 judges that the face detection section 102 has not misdetected the driver's face if the result of the control by the optical setting control section 106 has met the face misdetection conditions, but the difference in rolling length is greater than the first threshold for rolling length evaluation.

[0189] Specifically, the misdetection evaluation section 107 can also assess whether, for example, the result of the control carried out by the optical setting control section 106 meets the face misdetection conditions and whether, in addition, the roll-off length detected by the roll-off length detection section 111 is less than or equal to a threshold value (hereinafter referred to as the "second threshold for roll-off length evaluation").

[0190] The false detection evaluation section 107 assesses that the face detection section 102 has falsely detected the driver's face if the result of the control by the optical adjustment control section 106 meets the face false detection conditions and, furthermore, the rolling distance is less than or equal to the second threshold for rolling distance assessment. Conversely, the false detection evaluation section 107 assesses that the face detection section 102 has not falsely detected the driver's face if the result of the control by the optical adjustment control section 106 meets the face false detection conditions and, furthermore, the rolling distance is greater than the second threshold for rolling distance assessment.

[0191] One possible reason for the extensive changes in exposure time, on-time, or gain value due to the control performed by the optical adjustment control section 106 is excessive body movement by the driver. Excessive body movement by the driver also leads to excessive movement of the driver's face. For example, if the driver moves their body excessively to visually check behind the vehicle, it is conceivable that their face will also move excessively backward, where the light from the lighting device 30 has difficulty reaching.Even if the average luminance of the driver's face area image elements has temporarily reached the threshold for driver's seat luminance assessment, it cannot necessarily always be said that control adapted to the face area of ​​an occupant other than the driver has taken place.

[0192] Therefore, in the third embodiment, the false detection evaluation section 107 performs a false detection evaluation that takes into account not only the face false detection conditions but also the unwind length of the driver's seat belt. As described above, it is conceivable that excessive body movement by the driver could also cause excessive movement of the driver's face. If the driver then moves their body from the normal sitting position, the driver's seat belt unwinds, and the unwind length detection section 111 detects the unwind length corresponding to the movement of the seat belt.If the roll-over detection section 111 detects the roll-over based on the non-use state, the roll-over length will be greater than the roll-over length in the normal seating position if there is excessive movement of the face from the driver's normal seated position. Consequently, the difference between the roll-over length at one time and the roll-over length at another time can be used to assess whether the driver's face has moved excessively. Alternatively, if the roll-over length is greater than a certain value, it can be used to assess whether the face has moved excessively from its normal seated position.

[0193] If the rollover detection section 111 detects the rollover based on the usage condition, the rollover is set to a reference value of zero for a normal driver seating position. For example, if the driver's face moves excessively from its normal position, the rollover will exceed the reference value. In this case, the difference between the rollover at one time and at another can be used to determine whether the driver's face has moved excessively. Alternatively, if the rollover exceeds a certain value, it can be used to determine whether the driver's face has moved excessively from its normal position.

[0194] If it could be detected that the driver had moved their face excessively, it is also conceivable that, as described above, the optical setting control section 106 performed a control action due to the excessive facial movement, thereby excessively changing the optical setting. That is to say, if it could be determined that the driver had moved their face excessively, this does not necessarily mean that the driver's face was misdetected. Conversely, if it could be detected that the driver had not moved their face excessively, it is conceivable that the optical setting control section 106 performed a control action that excessively changed the optical setting because the driver's face was misdetected.

[0195] Consequently, the misdetection evaluation section 107 assesses that the face detection section 102 has misdetected the driver's face if, for example, the result of the control by the optical adjustment control section 106 meets the face misdetection conditions, and furthermore, the difference between the current rolling distance and the chronologically considered rolling distance one step earlier is less than or equal to the first threshold for rolling distance evaluation. If the difference between the current rolling distance and the chronologically considered rolling distance one step earlier is less than or equal to the first threshold for rolling distance evaluation, it is conceivable that the driver's face was not moved excessively.

[0196] On the other hand, the misdetection evaluation section 107 assesses that the face detection section 102 has not misdetected the driver's face if the result of the control by the optical adjustment control section 106 fulfills the face misdetection conditions, but the difference in rolling distance is greater than the first threshold for rolling distance evaluation. If the difference between the rolling distances is greater than the first threshold for rolling distance evaluation, it is conceivable that the driver's face was moved excessively.

[0197] Furthermore, the misdetection evaluation section 107 assesses that the face detection section 102 has misdetected the driver's face if, for example, the result of the control by the optical adjustment control section 106 meets the face misdetection conditions and, in addition, the rolling distance is less than or equal to the second threshold for the rolling distance evaluation. If the rolling distance is less than or equal to the second threshold for the rolling distance evaluation, it is conceivable that the driver's face was not moved excessively.

[0198] On the other hand, the misdetection evaluation section 107 assesses that the face detection section 102 has not misdetected the driver's face if the result of the control by the optical setting control section 106 fulfills the face misdetection conditions, but the rolling distance is greater than the second threshold for rolling distance evaluation. If the rolling distance is greater than the second threshold for rolling distance evaluation, it is conceivable that the driver's face was moved excessively.

[0199] The operation of the onboard image processing device 10a according to the third embodiment is explained.

[0200] The basic operation of the onboard image processing device 10a according to the third embodiment is identical to the operation that is carried out by means of Fig. 8 was explained in the first embodiment, so a double explanation is omitted. Fig. Figure 12 is a flowchart to illustrate the operation of steps ST804 to ST807 in the onboard image processing device 10a according to the third embodiment. Fig. 8 to explain in more detail. Unlike the first embodiment, the detailed operation of the false detection evaluation section 107 in the third embodiment is described by means of Fig. 12 explained.

[0201] The specific operation of steps ST901 to ST907 in Fig. 12 is identical to the specific operation of steps ST901 to ST907 in Fig. 9, which was explained in the first embodiment, so that a duplicate explanation is omitted. Furthermore, it is specified that the false detection evaluation section 107 in the one described in Fig. 12 operation shown is based on the result of the control carried out by the optical setting control section 106, and is based on the difference between the current rolling length and the chronologically considered rolling length one before performing a misdetection assessment.

[0202] In the event that in step ST907 the data relating to the imaging device 20 obtained as a result of performing the exposure control meet the face misdetection conditions, or the data relating to the imaging device 20 obtained as a result of performing the gain adjustment meet the face misdetection conditions (if “YES” is given in step ST907), the misdetection evaluation section 107 assesses whether the difference between the roll-off lengths is less than or equal to the first roll-off length evaluation threshold or not (step ST1201).

[0203] If, in step ST1201, it was determined that the difference between the roll lengths is less than or equal to the first threshold for roll length assessment (in case of "YES" in step ST1201), the false detection assessment section 107 assesses that the face detection section 102 has falsely detected the driver's face. The false detection assessment section 107 then issues a range reduction order to the range reduction section 109, reducing the area for face detection. Furthermore, the false detection assessment section 107 issues a re-control message to the re-control order section 108. The processing by the onboard image processing device 10 then proceeds to step ST808 in Fig. 8 over.

[0204] On the other hand, if step ST1201 determines that the difference between the roll lengths is greater than the threshold for roll length assessment (in case of "NO" in step ST1201), the false detection assessment section 107 determines that the face detection section 102 did not falsely detect the driver's face. The false detection assessment section 107 then outputs face detection presence data to the driver monitoring section 103. The processing of the onboard image processing device 10 then ends.

[0205] Thus, the false detection evaluation section 107 performs a false detection evaluation based on the result of the control carried out by the optical adjustment control section 106 and based on the unwinding length of the seat belt detected by the unwinding length detection section 111. This improves the precision of the false detection evaluation.

[0206] As with the first and second embodiments, a configuration is also possible in the third embodiment in which the onboard image processing device 10a does not include the area reduction section 109. Furthermore, setting an area for face detection in the image is not absolutely necessary.

[0207] As described above, the onboard image processing device 10a according to the third embodiment, in addition to the structure of the onboard image processing device 10 according to the first embodiment, comprises the roll-off length detection section 111 for detecting the roll-off length of the seat belt mounted on the driver's seat, wherein the false detection evaluation section 107 is configured to perform the false detection evaluation based on the result of the control performed by the optical adjustment control section 106 and based on the roll-off length detected by the roll-off length detection section 111.Therefore, according to the onboard image processing device 10a, the control of an unsuitable optical setting based on a misdetected face of the driver can be prevented, and the precision of the misdetection assessment can be improved by taking into account the rollover length of the driver's seat belt when performing the misdetection assessment.

[0208] Fig. 13A and Fig. Figure 13B shows examples of the hardware structure of an onboard image processing device 10, 10a according to the first to third embodiments.

[0209] In the first to third embodiments, the functions of the image acquisition section 101, the face detection section 102, the driver monitoring section 103, the luminance calculation section 104, the luminance evaluation section 105, the optical setting control section 106, the fault detection evaluation section 107, the area reduction section 109, the new detection arrangement section 110 and the rolling length detection section 111 are implemented by means of a processing circuit 1301. That is, the onboard image processing devices 10, 10a include the processing circuit 1301, which performs a misdetection assessment to determine whether the driver's face has been misdetected, in order to control the optical setting of the imaging device 20 based on the result of the misdetection assessment.

[0210] The 1301 processing circuit can be a special piece of hardware such as in Fig. 10A may be shown, but it can also be like in Fig. Figure 10B shows a CPU (Central Processing Unit) 1305 that executes programs stored in a memory 1306.

[0211] If the processing circuit 1301 is a special piece of hardware, the processing circuit 1301 corresponds, for example, to a simple circuit, a complex circuit, a programmed processor, parallel programmed processors, ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or a combination thereof.

[0212] If the processing circuit 1301 is the CPU 1305, the functions of the image acquisition section 101, the face detection section 102, the driver monitoring section 103, the luminance calculation section 104, the luminance evaluation section 105, the optical setting control section 106, the fault detection evaluation section 107, the area reduction section 109, the new detection arrangement section 110 and the rolling length detection section 111 are implemented by means of software, firmware or a combination of software and firmware.That is, the image acquisition section 101, the face detection section 102, the driver monitoring section 103, the luminance calculation section 104, the luminance evaluation section 105, the optical setting control section 106, the fault detection evaluation section 107, the area reduction section 109, the new detection arrangement section 110 and the roll length detection section 111 are implemented by a processing circuit such as the CPU 1305, a system LSI (Large-Scale Integration) etc., which executes programs stored in an HDD (Hard Disk Drive) 1302, the memory 1306 etc. Furthermore, it can also be said that the programs located in HDD 1302, memory 1306, etc.are stored, for the execution of the sequence or procedure of the image acquisition section 101, the face detection section 102, the driver monitoring section 103, the luminance calculation section 104, the luminance evaluation section 105, the optical setting control section 106, the false detection evaluation section 107, the area reduction section 109, the new detection arrangement section 110 and the rolling length detection section 111 on a computer. The memory 1306 can be a non-volatile or volatile semiconductor memory such as e.g. Examples include RAM (Random Access Memory), ROM (Read Only Memory), Flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), etc., a magnetic disk, a floppy disk, an optical disk, a compact disc, a minidisc, a DVD (Digital Versatile Disc), etc.

[0213] It is also possible to implement some of the functions of the image acquisition section 101, the face detection section 102, the driver monitoring section 103, the luminance calculation section 104, the luminance assessment section 105, the optical setting control section 106, the fault detection assessment section 107, the area reduction section 109, the new detection arrangement section 110 and the rolling length detection section 111 using special hardware and to implement some of them using software or firmware.For example, the processing circuit 1301, as special hardware, can implement the functions of the image acquisition section 101 and the roll-off length detection section 111, and the functions of the face detection section 102, the driver monitoring section 103, the luminance calculation section 104, the luminance evaluation section 105, the optical setting control section 106, the fault detection evaluation section 107, the area reduction section 109 and the new detection arrangement section 110 can be implemented by a processing circuit reading and executing programs stored in the memory 1306.

[0214] Furthermore, the onboard image processing devices 10, 10a include an input interface device 1303 and an output interface device 1304 for carrying out wired or wireless communication with the imaging device 20, the illumination device 30, etc.

[0215] In the foregoing first and second embodiments, the onboard image processing devices 10, 10a were mounted in a vehicle, although this is only an example.

[0216] For example, some or all of the components of the onboard image processing devices 10, 10a, which are connected by means of Fig. 1 to Fig. As explained in section 11, they are also available on a server.

[0217] Furthermore, in the first to third embodiments described above, the optical setting control section 106 in the onboard image processing devices 10, 10a could perform both exposure control and gain control as control of the optical setting; this is merely an example. The optical setting control section 106 can also perform only either exposure control or gain control as control of the optical setting.

[0218] In the invention of the present application, within the scope of the invention, an unrestricted combination of the respective forms of embodiment or any modification of a component of the respective forms of embodiment is possible, or any components of the respective forms of embodiment can be omitted. Industrial applicability

[0219] The onboard image processing device according to this invention is designed in such a way that an unsuitable control of the optical setting based on a misdetected face of the driver can be prevented, so that it is suitable for application to an onboard image processing device by which the face of the driver of the vehicle is detected based on an image in which occupants in a vehicle are depicted. Explanation of reference symbols 10, 10a Onboard image processing device 101 Image Acquisition Section 102 Face Detection Section 103 Driver Monitoring Section 104 Luminance Calculation Section 105 Luminance Assessment Section 106 1061 Optical setting control section Exposure control section 1062 Image Processing Section 107 False Detection Assessment Section 108 Re-control arrangement section 109 Area Reduction Section 110 New Detection Arrangement Section 111 Roll-off length detection section 20 Imaging device 30 Lighting device 1301 Processing circuit 1302 HDD 1303 Input interface device 1304 Output interface device 1305 CPU 1306 storage spaces.

Claims

[1] Onboard image processing device comprising an image acquisition section (101) which acquires from an imaging device (20) an image in which an area inside a vehicle is recorded, which includes an area in which the face of a driver sitting in the driver's seat should be located, a face detection section (102) which detects the driver's face and the driver's facial area on the image acquired by the image acquisition section (101), a luminance calculation section (104) that calculates the average luminance of the picture elements in the face area of ​​the driver detected by the face detection section (102); a luminance judgment section (105) that judges whether or not the average luminance calculated by the luminance calculation section (104) is greater than or equal to a luminance judgment threshold value; an optical adjustment control section (106) which, in the event that the luminance judgment section (105) has judged that the average luminance is below the luminance judgment threshold, carries out control of the optical adjustment of the imaging device in accordance with the average luminance, a misdetection judging section (107) which judges whether or not the face detection section (106) has misdetected the driver's face based on the result of the control performed by the optical adjustment control section (102), and a recontrol arrangement section (108) which, in the event that the misdetection judging section (107) has judged that the face detection section (102) has misdetected the driver's face, outputs to the optical adjustment control section (106) a recontrol arrangement by which a control resetting the optical adjustment of the imaging device (20) to an appropriate value is carried out, wherein the optical adjustment control section (106) performs a gain adjustment of the imaging device (20), wherein the face detection section (102) detects the driver's face and the face area of ​​the driver in question in the face detection area set in the image, and the misdetection judging section (107) judges whether, as a consequence of the gain adjustment performed by the optical adjustment control section (106), an overexposure area in a region (601) in which a higher probability of imaging the driver's face in the face detection area is suspected satisfies a judgment condition or not, and in the event that the overexposure area satisfies the judgment condition, judges misdetection of the driver's face by the face detection section (102), and wherein the judgment condition checks whether the overexposure area in the region (601) of suspected face detection is greater than or equal to a preset threshold. [2] The onboard image processing device according to claim 1, wherein the optical adjustment control section (106) performs exposure control of the imaging device (20). [3] The onboard image processing device according to claim 1 or 2, wherein the face detection section (102) detects the driver's face and the face portion of the driver in question in a face detection area set in the image, the misdetection judging section (107) judges whether or not, as a consequence of the exposure control performed by the optical setting control section (106), the average luminance of the picture elements in an area suspected of having a higher probability of imaging the driver's face in the face detection area satisfies a first further judgment condition, and in the case that the average luminance satisfies the first further judgment condition, judges misdetection of the driver's face by the face detection section (102). [4] The onboard image processing device according to any one of claims 1 to 3, wherein the misdetection judging section (107) judges whether or not the gain value of the imaging device (20) satisfies a second further judgment condition as a consequence of the gain adjustment performed by the optical adjustment control section (106), and the misdetection judging section (107) judges misdetection of the driver's face by the face detection section (102) when the gain value of the imaging device (20) satisfies the second further judgment condition. [5] An onboard image processing device according to any one of claims 1 to 4, wherein the face detection section (102) detects the driver's face and the face area of ​​the driver in question in the face detection area set in the image, and the onboard image processing device comprises an area reduction section (109) for reducing the area for face detection in the event that the misdetection judging section (107) judges that the face detection section (102) has misdetected the driver's face, and a re-detection arranging section (108) that causes the face detection section (102) to re-detect the driver's face and the driver's face area in the face detection area after reduction by the area reduction section. [6] The onboard image processing device according to claim 5, wherein the area reducing section (109) reduces the area for face detection to provide an area not including the driver's face misdetected by the face detecting section (102). [7] The onboard image processing apparatus according to any one of claims 1 to 6, wherein the misdetection judging section (107) performs the misdetection judgment (102) based on the result of the control performed by the optical adjustment control section (106) and based on the position of the driver's face detected by the face detection section. [8] An onboard image processing device according to any one of claims 1 to 7, comprising a unwinding length detection section (111) for detecting the unwinding length of a seat belt mounted on the driver's seat, wherein the misdetection judging section (107) performs the misdetection judgment based on the result of the control performed by the optical adjustment control section (106) and based on the unwinding length detected by the unwinding length detection section (111). [9] Onboard image processing method, comprising a step (ST801) in which an image acquisition section (101) acquires from an imaging device (20) an image in which an area inside a vehicle is captured, which includes an area in which the face of a driver sitting on the driver's seat should be located, a step (ST802) in which a face detection section (102) detects the driver's face and the driver's face area in this image based on the image acquired by the image acquisition section (101), a step (ST803) in which a luminance calculation section (104) calculates the average luminance of the picture elements in the face area of ​​the driver detected by the face detection section (102), a step (ST804) in which a luminance judgment section (105) judges whether the average luminance calculated by the luminance calculation section (104) is greater than or equal to a threshold value for the luminance judgment or not, a step (ST805) in which an optical adjustment control section (106) carries out control of the optical adjustment of the imaging device (20) in accordance with the average luminance, if the luminance judgment section (105) has judged that the average luminance is below the threshold value for the luminance judgment, a step (ST806) in which a misdetection judging section (107) judges whether or not the face detection section (102) has misdetected the driver's face based on the result of the control performed by the optical adjustment control section (106), a step in which a recontrol arrangement section (108), in the event that the misdetection judging section (107) has judged that the face detection section (102) has misdetected the driver's face (ST807), outputs to the optical adjustment control section (106) a recontrol arrangement by which a control resetting the optical adjustment of the imaging device (20) to an appropriate value is carried out (ST808), a step in which the optical adjustment control section (106) performs a gain adjustment (ST904) of the imaging device (20), a step (ST802) in which the face detection section (102) detects the driver's face and the face area of ​​the driver in question in the face detection area set in the image, a step in which the misdetection judging section (107) judges (ST905) whether, as a consequence of the gain adjustment (ST904) performed by the optical adjustment control section (106), an overexposure area in a region (601) where a higher probability of imaging the driver's face in the face detection region is suspected satisfies a judgment condition or not, and, in the event that the overexposure area satisfies the judgment condition, judges (ST907) misdetection of the driver's face by the face detection section (102), wherein the judgment condition checks whether the overexposure area in the suspected face detection region (601) is greater than or equal to a preset threshold value.

Images