Imaging device, driver monitoring system, brightness adjustment method, and program

Abstract

To provide an imaging device that can perform brightness adjustment suitable for a plurality of image processes other than display, in addition to display.SOLUTION: An imaging device (2) includes an image selection unit (21) that selects frame images that constitute a captured video, and brightness adjustment units (22-1 to 22-N) that perform brightness adjustment for display on frame images captured at a first frame period among the frame images selected from the captured video by the image selection unit (21), and perform brightness adjustment for a plurality of different image processing operations on a plurality of frame images that are each captured at a plurality of frame periods that are different from the first frame period and are different from each other.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an imaging device, a driver monitoring system, a brightness adjustment method, and a program. [Background technology]

[0002] For specific image processing, adjusting the image brightness to a level appropriate for that processing can improve processing accuracy. For example, in image processing to detect the face of a vehicle driver captured in a captured image, it is necessary to dynamically adjust the brightness according to the brightness in the vehicle cabin and continuously detect each feature that makes up the face. For example, Patent Document 1 describes an imaging device that outputs frame images suitable for display and frame images suitable for image processing alternately in time series from a single camera unit. For example, this imaging device sets exposure conditions for odd-numbered frame images so as to obtain a video signal with a brightness level for display, and captures and outputs the images under the set exposure conditions. For even-numbered frame images, it sets exposure conditions so as to obtain a video signal with a brightness level appropriate for image processing, and outputs the images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-13777 Summary of the Invention [Problem to be solved by the invention]

[0004] The conventional technology described in Patent Document 1 adjusts odd-numbered frame images to a brightness level suitable for display, and adjusts even-numbered frame images to a brightness level suitable for one image processing. As a result, the conventional technology has the problem of being unable to adjust brightness suitable for multiple image processing.

[0005] The present disclosure is intended to solve the above-mentioned problems, and has an object to provide an imaging device that can perform brightness adjustment suitable for multiple image processes other than display, in addition to display. [Means for solving the problem]

[0006] The imaging device according to the present disclosure includes an image selection unit that acquires captured video and selects frame images that constitute the acquired captured video, and a brightness adjustment unit that performs brightness adjustment for display on frame images captured at a first frame period among the frame images selected from the captured video by the image selection unit, and performs brightness adjustment for a plurality of different image processing operations on a plurality of frame images that are each captured at a plurality of different frame periods that are different from the first frame period. [Effects of the Invention]

[0007] According to the present disclosure, of frame images selected from a captured video, a frame image captured at a first frame period is subjected to brightness adjustment for display, and a plurality of frame images captured at a plurality of different frame periods different from the first frame period are subjected to brightness adjustment for a plurality of different image processes. This allows the imaging device according to the present disclosure to perform brightness adjustment suitable for a plurality of image processes other than display, in addition to brightness adjustment for display. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of a driver monitoring system according to a first embodiment. [Figure 2] 3 is a block diagram showing the configuration of a brightness adjustment unit included in the imaging device according to the first embodiment. FIG. [Figure 3] 4 is a flowchart showing a luminance adjustment method according to the first embodiment. [Figure 4] FIG. 2 is a schematic diagram showing an outline of brightness adjustment in the first embodiment. [Figure 5] FIG. 10 is a schematic diagram showing an overview of a first modification of brightness adjustment in the first embodiment. [Figure 6] FIG. 10 is a schematic diagram showing an overview of a second modification of brightness adjustment in the first embodiment. [Figure 7] FIG. 10 is a schematic diagram showing an overview of a third modification of brightness adjustment in the first embodiment. [Figure 8] 8A and 8B are block diagrams showing an example of a hardware configuration that realizes the functions of the imaging device according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Embodiment 1 The imaging device according to the first embodiment is mounted on a moving body and is used by a driver monitoring system (hereinafter referred to as DMS) to detect the state of an occupant of the moving body, including the facial direction or line of sight of the occupant. In the first embodiment, the moving body is a "vehicle." Note that the moving body may be a railroad vehicle, an aircraft, or a ship, in addition to a vehicle. In the following description, it is assumed that the occupant whose state is detected by the DMS according to the first embodiment is the "driver."

[0010] (Driver Monitoring System) Fig. 1 is a block diagram showing the configuration of a DMS1 according to a first embodiment. In Fig. 1, the DMS1 is a system that is mounted on a vehicle, detects the driver's state by analyzing captured images of the vehicle driver, and issues a warning or provides information to the driver based on the detected driver's state. The DMS1 includes an imaging device 2, an imaging unit 3, a display unit 4, an image analysis unit 5, and a speaker 6, and, for example, displays warning information for the driver on the display unit 4 and outputs a warning sound for the driver from the speaker 6. The warning issued to the driver may be a warning that the seat belt is not fastened, or a warning requesting the driver to hold the steering wheel when switching from autonomous driving to manual driving. The information provided to the driver may include, for example, information indicating route guidance such as turning right or left, information inquiring about whether or not to change the route, information recommending a break, or information guiding the driver to road signs.

[0011] For example, the functions of DMS1 include a driver face detection function (1), a driver fatigue driving detection function (2), a driver gaze tracking function (3), a driver hand movement detection function (4), and a driver ecology detection function (5). The driver's face detection function (1) is a function that detects and recognizes the driver's face from, for example, a captured image of the driver's face. This function can identify whether the occupant in the driver's seat is a registered driver. The driver fatigue detection function (2) is a function that, for example, detects the driver's facial features from a captured image of the driver's face and analyzes the driver's fatigue level based on the movement of the facial features. For example, it can identify signs of fatigue based on the frequency and length of blinking, or head movement, and issue a warning to the driver. The driver's gaze tracking function (3) is a function that monitors the direction of the driver's gaze by detecting and tracking the driver's gaze from a captured image of the driver's face, for example. This function can issue a warning if the driver becomes distracted. The driver's hand movement detection function (4) is a function that detects the position and movement of the driver's hands while driving from, for example, a captured image of the driver's hands, and checks whether the driver is gripping the steering wheel properly or not. This function makes it possible to check whether the driver is controlling the vehicle properly. The driver's ecology detection function (5) is a function that, for example, identifies changes in the color tone or brightness of the face caused by minute vibrations or pulsations of the facial skin from a captured image of the driver's face, and estimates the driver's heart rate by analyzing the identified changes in the surface. This function makes it possible to recognize the driver's health condition while driving.

[0012] For example, the brightness adjustments of the captured image suitable for the functions (1), (2), and (3) that require detection of the driver's facial features may be similar to one another. Furthermore, the brightness adjustment of the captured image suitable for function (4) of detecting the driver's hands may be different from the brightness adjustment for functions (1), (2), and (3). Furthermore, the brightness adjustment of the captured image suitable for function (5) that requires detecting changes in the color tone or brightness of the face may be different from the brightness adjustment for the other functions (1) to (4).

[0013] (Overview of imaging device) The imaging device 2 acquires captured images, which are multiple frame images captured by the imaging unit 3 and output in chronological order, and performs display brightness adjustment on the frame images captured at a first frame period among the captured images. The imaging device 2 performs multiple different brightness adjustments for image processing on the multiple frame images captured at different frame periods different from the first frame period among the captured images acquired from the imaging unit 3. Here, the first frame period is a period during which the imaging device 2 selects frame images to be subjected to display brightness adjustment from the multiple frame images output in chronological order from the imaging unit 3. For example, each function of the imaging device 2 is realized by a computer connected to the imaging unit 3 and to an external device equipped with a display unit 4, an image analysis unit 5, and a speaker 6. Note that the imaging device 2 may be a device equipped with at least one of the imaging unit 3, the display unit 4, the image analysis unit 5, and the speaker 6.

[0014] As a brightness adjustment for display, the imaging device 2 may adjust the brightness of the image according to the brightness of the environment in which the display unit 4 is placed. For example, the imaging device 2 compares the illuminance inside the vehicle cabin detected by an illuminance sensor (not shown in FIG. 1 ) with preset upper and lower limit values ​​for illuminance, and determines that the vehicle cabin is a bright environment if the illuminance inside the vehicle cabin is equal to or greater than the upper limit value, and determines that the vehicle cabin is a dark environment if the illuminance inside the vehicle cabin is less than the lower limit value. If the imaging device 2 determines that the vehicle cabin is a bright environment, it adjusts the brightness of the frame image displayed on the display unit 4 to increase it, and if it determines that the vehicle cabin is a dark environment, it adjusts the brightness of the frame image displayed on the display unit 4 to decrease it.

[0015] Furthermore, as brightness adjustment for display, the imaging device 2 may perform brightness adjustment based on the content of the image to be displayed on the display unit 4. For example, the imaging device 2 compares the brightness of the frame image acquired from the imaging unit 3 with preset upper and lower brightness limit values, and if the brightness of the frame image is equal to or greater than the upper limit value, determines that the frame image was captured in a bright location, and if the brightness of the frame image is less than the lower limit value, determines that the frame image was captured in a dark location. If the imaging device 2 determines that the frame image was captured in a bright location, it adjusts the brightness of the image to be displayed on the display unit 4 to decrease it, and if it determines that the frame image was captured in a dark location, it adjusts the brightness of the image to be displayed on the display unit 4 to increase it.

[0016] The multiple image processing operations correspond to various functions of the DMS 1. For example, the above-mentioned functions (1), (2), (3), (4), and (5) do not all perform the same image processing on the captured image, but each has its own brightness adjustment that is suited to them. For example, there is image processing for detecting the driver's facial features, image processing for detecting the driver's hands, and image processing for detecting the driver's pulse. In image processing for detecting the driver's facial features, for example, it is necessary to dynamically adjust the brightness according to the brightness around the face and continue to detect each facial feature. Furthermore, in image processing for detecting the driver's hands, it is necessary to dynamically adjust the brightness according to the brightness around the driver's hands and continue to detect each facial feature. When detecting a driver's pulse using a captured image, it is necessary to capture subtle changes in brightness on the surface of the face, and correcting the brightness of the captured image at short time intervals makes accurate detection more difficult. For this reason, image processing for detecting the driver's pulse requires control that does not change brightness significantly, except when the brightness around the face changes significantly.

[0017] (imaging unit) The imaging unit 3 captures images of vehicle occupants. For example, the imaging unit 3 is provided with a CCD image sensor or a CMOS image sensor, and an optical system that focuses incident light on the image sensor. The imaging unit 3 obtains an electrical signal corresponding to the image focused on the image sensor by the optical system. The electrical signal output from the image sensor is A / D converted to output a frame image, which is a video signal for one frame. Furthermore, the image capturing unit 3 may be an infrared camera that captures images using near-infrared light, or may be one that captures images using visible light.

[0018] (Display) The display unit 4 is a display device that displays captured images output from the imaging device 2, and displays, for example, captured images whose brightness has been adjusted for display by the imaging device 2. The display unit 4 may be a display unit included in the imaging device 2, or may be a display unit included in an external device that is communicatively connected to the imaging device 2 via a wired or wireless signal line. For example, the display unit 4 may be an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display.

[0019] (Image analysis section) The image analysis unit 5 acquires captured video whose brightness has been adjusted for multiple image processing by the imaging device 2, and performs multiple image processing operations on the acquired captured video. For example, the image analysis unit 5 performs image processing on the frame images that make up the captured video to detect the driver's state, and then warns or provides information to the driver based on the detected driver's state. The image analysis unit 5 is provided in an information processing device connected to the imaging device 2. Alternatively, the image analysis unit 5 may be provided in the imaging device 2.

[0020] (Speaker) The speaker 6 is provided in an information processing device equipped with the image analysis unit 5, and outputs audio information output from the image analysis unit 5. For example, the speaker 6 outputs an audio warning to the driver output from the image analysis unit 5. The audio warning to the driver may be an audio warning that the seat belt is not fastened, or an audio requesting the driver to hold on to the steering wheel.

[0021] (Details of the imaging device) The imaging device 2 includes an image selection unit 21 and brightness adjustment units 22-1, 22-2, ..., 22-N. For example, the imaging device 2 is realized by a computer. A memory included in the computer stores a program constituting an information processing application for realizing each function of the image selection unit 21 and brightness adjustment units 22-1, 22-2, ..., 22-N. A processor included in the computer executes the information processing application read from the memory, thereby realizing each function of the image selection unit 21 and brightness adjustment units 22-1, 22-2, ..., 22-N. Here, N is an integer equal to or greater than 2.

[0022] (Image selection section) The image selection unit 21 acquires the captured video output from the imaging unit 3 and selects frame images that make up the acquired captured video. For example, the image selection unit 21 is set with a plurality of different frame periods and output destination information that specifies a plurality of brightness adjustment units to which the frame images captured at these periods are respectively output. When N=3, the image selection unit 21 is set with the first to third frame periods and output destination information that identifies the brightness adjustment units 22-1, 22-2, and 22-3 that output the frame images captured at the first to third frame periods, respectively. The image selection unit 21 starts counting the first to third frame periods, triggered by the time when the frame image output in chronological order from the imaging unit 3 is first acquired. Image selection unit 21 selects frame images captured at a first frame period from the multiple frame images output in chronological order from imaging unit 3 and outputs them to brightness adjustment unit 22-1 based on the output destination information. Similarly, image selection unit 21 selects frame images captured at a second frame period from the frame images output in chronological order and outputs them to brightness adjustment unit 22-2 based on the output destination information. Image selection unit 21 selects frame images captured at a third frame period from the frame images output in chronological order and outputs them to brightness adjustment unit 22-3 based on the output destination information.

[0023] (Brightness adjustment section) The brightness adjustment units 22-1, 22-2, ..., 22-N perform brightness adjustment for display on frame images captured at a first frame period among the frame images selected from the captured video by the image selection unit 21, and perform brightness adjustment for a plurality of different image processing operations on a plurality of frame images each captured at a plurality of different frame periods that are different from the first frame period. For example, when N=3, the imaging device 2 includes a brightness adjustment unit 22-1, a brightness adjustment unit 22-2, and a brightness adjustment unit 22-3. In this case, the brightness adjustment unit 22-1 adjusts the brightness for display on the frame images captured in the first frame period among the frame images selected from the captured video by the image selection unit 21. The brightness adjustment unit 22-2 performs brightness adjustment for the first image processing on frame images captured at a second frame period different from the first frame period. The brightness adjustment unit 22-3 performs brightness adjustment for second image processing, which is different from the first image processing, on frame images captured at a third frame period that is different from both the first frame period and the second frame period.

[0024] (Details of brightness adjustment section) 2 is a block diagram showing the configuration of the brightness adjustment unit 22-k included in the imaging device 2. In FIG. Here, k is an integer between 1 and N. The adjustment unit 221 uses the luminance correction value calculated by the correction value calculation unit 222 to adjust the luminance of the input frame image. For example, the adjustment unit 221 adjusts the brightness of the entire frame image by increasing or decreasing the overall brightness level of the frame image by the correction value acquired from the correction value calculation unit 222. Furthermore, the adjustment unit 221 may adjust the brightness of the entire frame image by changing the luminance value of each pixel in the frame image using a gamma function value acquired as a correction value from the correction value calculation unit 222. If the gamma function value is smaller than 1, the image will be darker, and if the gamma function value is 1 or greater, the image will be brighter.

[0025] The correction value calculation unit 222 calculates a correction value used for adjusting the brightness of the frame image by the adjustment unit 221. For example, the correction value calculation unit 222 calculates a correction value used for adjusting the brightness of the currently acquired frame image based on the result of brightness adjustment performed on the previously acquired frame image out of the frame images selected by the image selection unit 21. For example, correction value calculation section 222 compares the brightness of the acquired frame image with preset upper and lower brightness limit values, and if the brightness of the frame image is equal to or greater than the upper limit value, determines that the frame image was captured in a bright location, and if the brightness of the frame image is less than the lower limit value, determines that the frame image was captured in a dark location. If correction value calculation section 222 determines that the frame image was captured in a bright location, it calculates a correction value to adjust the brightness of the frame image in a direction to decrease it, and if it determines that the frame image was captured in a dark location, it calculates a correction value to adjust the brightness in a direction to increase it.

[0026] Furthermore, in brightness adjustment for image processing for detecting the driver's facial features, the correction value calculation unit 222 compares the brightness of the image area containing the driver's face extracted from the frame image with preset upper and lower brightness limits, and determines that the area around the driver's face is bright if the brightness of the image area is equal to or greater than the upper limit, and determines that the area around the driver's face is dark if the brightness of the image area is less than the lower limit. If the correction value calculation unit 222 determines that the area around the driver's face is bright, it calculates a correction value to adjust the brightness of the image area in a direction that decreases the brightness, and if it determines that the area around the driver's face is dark, it calculates a correction value to adjust the brightness in a direction that increases the brightness.

[0027] Similarly, in brightness adjustment for image processing to detect the driver's hands, the correction value calculation unit 222 compares the brightness of an image area showing the driver's hands extracted from the frame image with preset upper and lower brightness limits, and determines that the area around the driver's hands is bright if the brightness of the image area is equal to or greater than the upper limit, and determines that the area around the driver's hands is dark if the brightness of the image area is less than the lower limit. If the correction value calculation unit 222 determines that the area around the driver's hands is bright, it calculates a correction value to adjust the image area in a direction that decreases the brightness, and if it determines that the area around the driver's hands is dark, it calculates a correction value to adjust the image area in a direction that increases the brightness.

[0028] In addition, in the brightness adjustment for multiple image processes, brightness adjustment is performed when there is a brightness fluctuation equal to or greater than a preset threshold within a preset range. The correction value calculation unit 222 calculates a correction value for performing brightness adjustment that does not significantly change brightness, except when there is a significant change in brightness around the face, in the brightness adjustment for image processing that detects the driver's pulse. For example, the correction value calculation unit 222 compares the brightness of the image area containing the driver's face extracted from the frame image with a preset upper limit value of brightness, and if the brightness of the image area is less than the upper limit value, calculates a correction value such that the change in brightness value from the previous time is less than a threshold value, and causes the adjustment unit 221 to perform brightness adjustment that does not significantly change the brightness. Furthermore, when the brightness of the image area exceeds the upper limit, the correction value calculation unit 222 calculates a correction value that maintains the current brightness value, and causes the adjustment unit 221 to perform brightness adjustment that follows the change in the brightness value of the image. This allows brightness adjustment that does not change the brightness significantly.

[0029] In addition, in brightness adjustment for image processing for detecting the driver's pulse, the correction value calculation unit 222 may calculate a correction value for the adjustment unit 221 so that brightness adjustment is performed when there is a brightness fluctuation within a predetermined range that is equal to or greater than a predetermined threshold. In this case, the correction value calculation unit 222 checks whether the brightness of the image area showing the driver's face is within a predetermined range, and if it is within the range, determines whether the brightness of the image area is equal to or greater than a predetermined threshold. If the brightness of the image area exceeds an upper limit, the correction value calculation unit 222 calculates a correction value that maintains the current brightness value. This makes it possible to perform brightness adjustment without significantly changing the brightness of the image.

[0030] Next, a luminance adjustment method according to the first embodiment will be described. 3 is a flowchart showing the brightness adjustment method according to Embodiment 1. Below, the case where N=3 is taken as an example. The image selection unit 21 acquires the captured video output from the imaging unit 3 and selects frame images constituting the acquired captured video (step ST1). For example, the image selection unit 21 selects frame images captured at a first frame period from among the multiple frame images output in time series from the imaging unit 3 and outputs them to the brightness adjustment unit 22-1 based on the output destination information. Similarly, the image selection unit 21 selects frame images captured at a second frame period from among the frame images output in time series and outputs them to the brightness adjustment unit 22-2 based on the output destination information. The image selection unit 21 selects frame images captured at a third frame period from among the frame images output in time series and outputs them to the brightness adjustment unit 22-3 based on the output destination information.

[0031] Next, the brightness adjustment unit 22-1 performs brightness adjustment for display on the frame images captured in the first frame period among the frame images selected from the captured video by the image selection unit 21 (step ST2). The brightness adjustment unit 22-2 performs brightness adjustment for the first image processing on the frame images captured at a second frame period different from the first frame period (step ST3). The brightness adjustment unit 22-3 performs brightness adjustment for second image processing, which is different from the first image processing, on the frame images captured at a third frame period that is different from both the first frame period and the second frame period (step ST4).

[0032] The image selection unit 21 checks whether or not the imaging by the imaging unit 3 has finished (step ST5). For example, the image selection unit 21 determines that the imaging by the imaging unit 3 has finished when no captured image is output from the imaging unit 3 for a certain period of time or longer. Alternatively, the image selection unit 21 may acquire vehicle information indicating whether or not driving of the vehicle has finished, and may determine that the imaging by the imaging unit 3 has finished when it is determined from the acquired vehicle information that driving of the vehicle has finished.

[0033] If the imaging unit 3 has finished capturing images (step ST5; YES), the imaging device 2 ends the series of processes shown in Fig. 3. If the imaging unit 3 has not finished capturing images (step ST5; NO), the imaging device 2 returns to the process of step ST1 and repeats the series of processes shown in Fig. 3. In this way, by the imaging device 2 executing the brightness adjustment method shown in Fig. 3, it is possible to perform brightness adjustment suitable for N types of image processing other than display in addition to display.

[0034] An outline of brightness adjustment of an image captured by the imaging device 2 will be described. FIG. 4 is a schematic diagram showing an overview of brightness adjustment in the first embodiment, showing the case where N=3. As shown in FIG. 4, image selection unit 21 selects images (1) and (4) captured in a first frame cycle from among a plurality of time-series frame images and outputs them to brightness adjustment unit 22-1. Furthermore, image selection unit 21 selects images (2) and (5) captured in a second frame cycle from among the plurality of time-series frame images and outputs them to brightness adjustment unit 22-2. Furthermore, image selection unit 21 selects images (3) and (6) captured in a third frame cycle from among the plurality of time-series frame images and outputs them to brightness adjustment unit 22-3.

[0035] The brightness adjustment unit 22-1 adjusts the brightness for display of the images (1) and (4) selected by the image selection unit 21. The brightness-adjusted images (1) and (4) for display are output to the display unit 4. The brightness adjustment unit 22-2 performs brightness adjustment (1) for the first image processing on the images (2) and (5) selected by the image selection unit 21. The brightness-adjusted images (2) and (5) that have undergone brightness adjustment (1) for the first image processing are output to the image analysis unit 5. The brightness adjustment unit 22-3 performs brightness adjustment (2) for the second image processing on the images (3) and (6) selected by the image selection unit 21. The brightness-adjusted images (3) and (6) that have undergone brightness adjustment (2) for the second image processing are output to the image analysis unit 5.

[0036] The display unit 4 displays the brightness-adjusted images (1) and (4) that have been subjected to brightness adjustment suitable for display, thereby enabling easy-to-view image display. Furthermore, the image analysis unit 5 can perform the first image processing with high accuracy by performing the first image processing on the brightness-adjusted images (2) and (5) that have been subjected to brightness adjustment suitable for the first image processing. For example, when the first image processing is to detect the facial features of the driver, the image area containing the face is adjusted to a brightness suitable for detecting the facial features, so that improvement in detection accuracy can be expected. Furthermore, the image analysis unit 5 can perform the second image processing with high accuracy by performing the second image processing on the brightness-adjusted images (3) and (6) that have been subjected to brightness adjustment suitable for the second image processing. For example, if the second image processing is to detect the driver's hands, the image area containing the driver's hands is adjusted to a brightness suitable for detecting the hands, which is expected to improve detection accuracy. In other words, even if the image capturing device 2 uses captured video captured by a single image capturing unit 3, it is possible to perform brightness adjustment suitable for different image processing for different image areas within the frame image.

[0037] (Variation 1) If the interval between frame images to be brightness adjusted becomes long, and if the interval is caused by a change in the brightness of the environment in which the imaging unit 3 is placed or a change in the brightness of the location where the image was captured, then the brightness adjustment for display may actually cause the displayed image to look unnatural. Therefore, the brightness adjustment section 22-1 may acquire frame images captured by the imaging section 3 at intervals of one frame as the first frame period, and perform brightness adjustment for display on the acquired frame images.

[0038] FIG. 5 is a schematic diagram showing an overview of Variation 1 of brightness adjustment in Embodiment 1. As shown in FIG. 5, the first frame period for selecting an image to be subjected to brightness adjustment for display is a period in which one frame is left. Image selection unit 21 selects images (1), (3), (5), and (7) captured in the first frame period from among the plurality of time-series frame images and outputs them to brightness adjustment unit 22-1. Image selection unit 21 selects images (2) and (6) captured in the second frame period from among the plurality of time-series frame images and outputs them to brightness adjustment unit 22-2. Image selection unit 21 selects images (4) and (8) captured in the third frame period from among the plurality of time-series frame images and outputs them to brightness adjustment unit 22-3.

[0039] The brightness adjustment unit 22-1 adjusts the brightness for display of the images (1), (3), (5), and (7) selected by the image selection unit 21. The brightness-adjusted images (1), (3), (5), and (7) after the brightness adjustment for display are output to the display unit 4. The brightness adjustment unit 22-2 performs brightness adjustment (1) for the first image processing on the images (2) and (6) selected by the image selection unit 21. The brightness-adjusted images (2) and (6) that have undergone brightness adjustment (1) for the first image processing are output to the image analysis unit 5. The brightness adjustment unit 22-3 performs brightness adjustment (2) for the second image processing on the images (4) and (8) selected by the image selection unit 21. The brightness-adjusted images (4) and (8) that have undergone brightness adjustment (2) for the second image processing are output to the image analysis unit 5. In this way, frame images are acquired at a first frame period with the shortest frame interval of just one frame, thereby reducing the occurrence of an unnatural feeling in the displayed image caused by a longer interval between frame images.

[0040] (Variation 2) When multiple image processes are added, the image selection unit 21 may select frame images that were allocated for image processes before the addition so that they are distributed to brightness adjustment for the image processes. This allows the imaging device 2 to perform brightness adjustment suitable for multiple image processes even if the number of image processes requiring brightness adjustment increases.

[0041] 6 is a schematic diagram showing an overview of Variation 2 of brightness adjustment in Embodiment 1, and shows a case in which brightness adjustment for the third image processing is added to Variation 1 of imaging device 2 shown in FIG. 5. For example, Variation 2 of imaging device 2 includes one or more spare brightness adjustment units for adding functions in addition to brightness adjustment units 22-1, 22-2, and 22-3. When an additional instruction to further execute the third image processing is input in DMS 1 using an input device not shown in FIG. 1, this additional instruction information is input to imaging device 2. The third image processing is different from the first image processing and the second image processing. The additional instruction information includes at least output destination information for specifying the brightness adjustment unit 22-4 that performs the brightness adjustment for the third image processing, and a fourth frame period for selecting a frame image to be subjected to the brightness adjustment for the third image processing. The fourth frame period is different from any of the first to third frame periods. The imaging device 2 causes one of the auxiliary brightness adjustment units to function as the brightness adjustment unit 22-4 based on the output destination information for identifying the brightness adjustment unit 22-4.

[0042] 6, the image selection unit 21 selects images (1), (3), (5), (7), and (9) captured in a first frame cycle from among the plurality of time-series frame images and outputs them to the brightness adjustment unit 22-1. The image selection unit 21 selects images (2) and (8) captured in a second frame cycle from among the plurality of time-series frame images and outputs them to the brightness adjustment unit 22-2. The image selection unit 21 selects images (4) and (10) captured in a third frame cycle from among the plurality of time-series frame images and outputs them to the brightness adjustment unit 22-3. Furthermore, the image selection unit 21 selects an image (6) captured in a fourth frame cycle from among the plurality of time-series frame images, and outputs the image (6) to the brightness adjustment unit 22-4.

[0043] The brightness adjustment unit 22-1 adjusts the brightness for display of each of the images (1), (3), (5), (7), and (9) selected by the image selection unit 21. The brightness-adjusted images (1), (3), (5), (7), and (9) after the brightness adjustment for display are output to the display unit 4. The brightness adjustment unit 22-2 performs brightness adjustment (1) for the first image processing on the images (2) and (8) selected by the image selection unit 21. The brightness-adjusted images (2) and (8) that have undergone brightness adjustment (1) for the first image processing are output to the image analysis unit 5. The brightness adjustment unit 22-3 performs brightness adjustment (2) for the second image processing on the images (4) and (10) selected by the image selection unit 21. The brightness-adjusted images (4) and (10) that have undergone brightness adjustment (2) for the second image processing are output to the image analysis unit 5. The brightness adjustment unit 22-4 performs brightness adjustment (3) for the third image processing on the image (6) selected by the image selection unit 21. The brightness-adjusted image (6) that has undergone brightness adjustment (3) for the third image processing is output to the image analysis unit 5. This allows the DMS 1 to dynamically respond to diversification of its functions. The cycle for allocating frame images for the added image processing (for example, the fourth frame cycle) may be set so as to maintain the first frame cycle for brightness adjustment for display. This prevents the interval between frame images for which brightness adjustment for display is performed from becoming longer, thereby preventing the display image from looking unnatural due to longer intervals between frame images.

[0044] (Variation 3) When multiple image processes are reduced, the image selection unit 21 may select frame images that were allocated for image processes before the reduction so that they are allocated to brightness adjustment for the image processes. This allows the imaging device 2 to perform brightness adjustment suitable for multiple image processes even if the number of image processes that require brightness adjustment is reduced.

[0045] 7 is a schematic diagram showing an overview of Modification 3 of brightness adjustment in Embodiment 1. This illustrates a case where the second image processing is stopped from Modification 2 of the imaging device 2 shown in FIG. When a stop instruction to stop the second image processing is input to the DMS 1 using an input device not shown in FIG. The stop instruction information also includes at least output destination information for identifying the brightness adjustment unit 22-3 that was performing the brightness adjustment for the second image processing to be stopped, and a third frame period for selecting a frame image for which brightness adjustment for the second image processing is to be performed. The imaging device 2 stops the brightness adjustment unit 22-3 based on the output destination information for identifying the brightness adjustment unit 22-3.

[0046] Image selection unit 21 distributes images (4) and (10) that were output to brightness adjustment unit 22-3, which is stopped, to brightness adjustment unit 22-2 and brightness adjustment unit 22-4. Here, image selection unit 21 changes the second frame period and the fourth frame period so that the first frame period is maintained. As a result, image selection unit 21 selects images (1), (3), (5), (7), and (9) captured at the first frame period from the multiple frame images in time series, as in FIG. 6, and outputs them to brightness adjustment unit 22-1. Furthermore, the image selection unit 21 changes the second frame period to select images (2), (6), and (10) and outputs them to the brightness adjustment unit 22-2. Furthermore, the image selection unit 21 changes the fourth frame period to select images (4) and (8) and outputs them to the brightness adjustment unit 22-4.

[0047] The brightness adjustment unit 22-1 adjusts the brightness for display of each of the images (1), (3), (5), (7), and (9) selected by the image selection unit 21. The brightness-adjusted images (1), (3), (5), (7), and (9) after the brightness adjustment for display are output to the display unit 4. 7, the brightness adjustment unit 22-2 performs brightness adjustment (1) for the first image processing on the images (2), (6), and (10) selected by the image selection unit 21. The brightness-adjusted images (2), (6), and (10) that have undergone brightness adjustment (1) for the first image processing are output to the image analysis unit 5. 7, the brightness adjustment unit 22-4 performs brightness adjustment (3) for the third image processing on the images (4) and (8) selected by the image selection unit 21. The brightness-adjusted images (4) and (8) that have undergone brightness adjustment (3) for the third image processing are output to the image analysis unit 5. In this way, the DMS 1 can dynamically respond even when any function is stopped.

[0048] Furthermore, when the display function for captured images is stopped, the image that has been allocated to brightness adjustment for display may be allocated to brightness adjustment for other image processing. For example, consider a case where the display function of the captured image is stopped during the brightness adjustment shown in Fig. 4. In this case, when a stop instruction to stop the display of the image is input to the DMS 1 using the input device, this stop instruction information is input to the imaging device 2. The stop instruction information also includes at least output destination information for identifying the brightness adjustment unit 22-1 that was performing the display brightness adjustment to be stopped, and a first frame period for selecting a frame image for which the display brightness adjustment is to be performed. The imaging device 2 stops the brightness adjustment unit 22-1 based on the output destination information for specifying the brightness adjustment unit 22-1.

[0049] The image selection unit 21 distributes the image that was being output to the stopped brightness adjustment unit 22-1 to brightness adjustment units 22-2 and 22-3 that perform brightness adjustment for other image processing. For example, the image selection unit 21 changes the second frame period to select images (2), (4), and (5), and outputs them to the brightness adjustment unit 22-2. Furthermore, the image selection unit 21 changes the third frame period to select images (1), (3), and (6), and outputs them to the brightness adjustment unit 22-3.

[0050] The brightness adjustment unit 22-2 performs brightness adjustment (1) for the first image processing on the images (2), (4), and (5) selected by the image selection unit 21. The brightness-adjusted images (2), (4), and (5) that have undergone brightness adjustment (1) for the first image processing are output to the image analysis unit 5. Furthermore, the brightness adjustment unit 22-3 performs brightness adjustment (2) for the second image processing on the images (1), (3), and (6) selected by the image selection unit 21. The brightness-adjusted images (1), (3), and (6) that have undergone brightness adjustment (2) for the second image processing are output to the image analysis unit 5. In this way, the DMS 1 can dynamically respond even when any function is stopped.

[0051] Furthermore, for image processing that requires brightness adjustment that does not cause large changes in image brightness, the frame cycle may be increased so that images to be brightness adjusted are assigned with priority. For example, when allocating a new frame image to a brightness adjustment unit that performs brightness adjustment for image processing, the image selection unit 21 resets the frame period so that the image is allocated to a process with a high priority determined based on the characteristics of the image processing. For example, the brightness adjustment for detecting the driver's pulse needs to be adjusted so that the brightness change does not become too large, as described above. Therefore, the image selection unit 21 assigns a high priority to brightness adjustment for detecting the driver's pulse, and sets the frame period so that brightness adjustment is performed to suppress brightness changes during pulse detection, thereby improving the accuracy of image processing.

[0052] Although the above embodiment has been described with a plurality of brightness adjustment units that respectively perform brightness adjustment for display and brightness adjustment for a plurality of image processes, the present invention is not limited to this. For example, the imaging device 2 may include a single brightness adjustment unit that adjusts the brightness for display and for multiple image processing operations. The imaging device 2 configured in this manner can also achieve the above-described effects.

[0053] Next, the hardware configuration for realizing the functions of the imaging device 2 will be described. The functions of the image selection unit 21 and brightness adjustment units 22-1, 22-2, ..., 22-N included in the imaging device 2 are realized by processing circuits. That is, the imaging device 2 includes a processing circuit for executing the processes of steps ST1 to ST5 shown in Fig. 3. The processing circuit may be dedicated hardware, or may be a CPU (Central Processing Unit) that executes a program stored in a memory.

[0054] Fig. 8A is a block diagram showing a hardware configuration for realizing the functions of the imaging device 2. Fig. 8B is a block diagram showing a hardware configuration for executing software for realizing the functions of the imaging device 2. In Figs. 8A and 8B, image selection unit 21 acquires captured images from imaging unit 3 via input interface 100. Brightness adjustment unit 22-1 outputs the image after brightness adjustment to display unit 4 via output interface 101. Brightness adjustment units 22-2, ..., 22-N output the image after brightness adjustment to image analysis unit 5 via output interface 101.

[0055] 8A, the processing circuit 102 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of the image selection unit 21 and the brightness adjustment units 22-1, 22-2, ..., 22-N included in the imaging device 2 may be realized by separate processing circuits, or these functions may be realized together by a single processing circuit.

[0056] 8B, the functions of the image selection unit 21 and brightness adjustment units 22-1, 22-2, . . . , 22-N included in the imaging device 2 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 104.

[0057] The processor 103 reads and executes programs stored in the memory 104 to realize the functions of the image selection unit 21 and the brightness adjustment units 22-1, 22-2, ..., 22-N included in the imaging device 2. For example, the imaging device 2 includes a memory 104 for storing a program that, when executed by the processor 103, results in the execution of the processes from step ST1 to step ST5 shown in FIG. 3. These programs cause a computer to execute the procedures or methods of the processes performed by the image selection unit 21 and the brightness adjustment units 22-1, 22-2, ..., 22-N. The memory 104 may be a computer-readable storage medium that stores a program for causing a computer to function as the image selection unit 21 and the brightness adjustment units 22-1, 22-2, ..., 22-N.

[0058] Memory 104 may be, for example, a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically-EPROM) (registered trademark), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD, etc.

[0059] Some of the functions of the image selection unit 21 and brightness adjustment units 22-1, 22-2, ..., 22-N included in the imaging device 2 may be realized by dedicated hardware, and other functions may be realized by software or firmware. For example, the function of the image selection unit 21 may be realized by a processing circuit 102 that is dedicated hardware, and the functions of the brightness adjustment units 22-1, 22-2, ..., 22-N may be realized by a processor 103 reading and executing a program stored in a memory 104. In this way, the processing circuit can realize the above functions by hardware, software, firmware, or a combination of these.

[0060] As described above, the imaging device 2 according to the first embodiment includes an image selection unit 21 that selects frame images constituting a captured video, and brightness adjustment units 22-1 to 22-N that perform brightness adjustment for display on frame images captured at a first frame period among the frame images selected from the captured video by the image selection unit 21, and perform brightness adjustment for a plurality of different image processes on a plurality of frame images that are captured at a plurality of different frame periods different from the first frame period. This allows the imaging device 2 to perform brightness adjustment suitable for N types of image processing other than display, in addition to brightness adjustment for display.

[0061] In the imaging device 2 according to the first embodiment, the brightness adjustment units 22-2 to 22-N perform brightness adjustment when there is a brightness fluctuation equal to or greater than a preset threshold within a preset range in brightness adjustment for a plurality of image processes. This enables the imaging device 2 to perform brightness adjustment without significantly changing the brightness of the image.

[0062] In the imaging device 2 according to the first embodiment, the brightness adjustment unit 22-1 acquires frame images captured at intervals of one frame as the first frame period, and performs brightness adjustment for display on the acquired frame images. This allows the imaging device 2 to prevent the display image from looking unnatural due to the longer intervals between frame images.

[0063] In the imaging device 2 according to the first embodiment, when the number of image processes is increased or decreased, the image selection unit 21 selects frame images that were allocated for image processes before the increase or decrease so that they are allocated to brightness adjustment for image processing or display. This allows the imaging device 2 to perform brightness adjustment suitable for multiple image processes even if the number of image processes that require brightness adjustment increases or decreases.

[0064] The DMS1 according to the first embodiment includes an imaging unit 3 that images a vehicle occupant, an imaging device 2 that acquires the captured image captured by the imaging unit 3, a display unit 4 that displays the captured image whose luminance has been adjusted for display by the imaging device 2, and an image analysis unit 5 that acquires the captured image whose luminance has been adjusted for multiple image processing by the imaging device 2 and performs multiple image processing on the acquired captured image. As a result, the DMS1 can provide an imaging device 2 that performs luminance adjustment suitable for N types of image processing other than display in addition to display.

[0065] The brightness adjustment method according to the first embodiment includes the steps (ST1) of image selection unit 21 selecting frame images constituting a captured video, and steps (ST2 to ST4) of brightness adjustment units 22-1 to 22-N performing brightness adjustment for display on frame images captured at a first frame period among the frame images selected from the captured video by image selection unit 21, and performing brightness adjustment for a plurality of different image processes on a plurality of frame images captured at a plurality of different frame periods different from the first frame period. By imaging device 2 executing the above method, brightness adjustment suitable for N types of image processing other than display can be performed in addition to brightness adjustment for display.

[0066] A computer that executes the program according to the first embodiment functions as an image selection unit 21 that selects frame images that make up a captured video, and as brightness adjustment units 22-1 to 22-N that perform brightness adjustment for display on frame images captured at a first frame period among the frame images selected from the captured video by image selection unit 21, and perform brightness adjustment for a plurality of different image processes on a plurality of frame images that are respectively captured at a plurality of different frame periods that differ from the first frame period. By having a computer execute this program, it is possible to perform brightness adjustment suitable for N types of image processing other than display, in addition to brightness adjustment for display.

[0067] Any of the components of the embodiments may be modified or omitted. [Explanation of symbols]

[0068] 1 Driver monitoring system (DMS), 2 Imaging device, 3 Imaging unit, 4 Display unit, 5 Image analysis unit, 6 Speaker, 21 Image selection unit, 22-1, 22-2, 22-3, 22-4, 22-k, 22-N Brightness adjustment unit, 100 Input interface, 101 Output interface, 102 Processing circuit, 103 Processor, 104 Memory, 221 Adjustment unit, 222 Correction value calculation unit.

Claims

1. an image selection unit that acquires a captured image and selects frame images that constitute the acquired captured image; a brightness adjustment unit that performs brightness adjustment for display on the frame images captured at a first frame period among the frame images selected from the captured video by the image selection unit, and performs a plurality of different brightness adjustments for image processing on the plurality of frame images that are respectively captured at a plurality of different frame periods that are different from the first frame period; An imaging device comprising:

2. The brightness adjustment unit performs brightness adjustment when there is a brightness fluctuation equal to or greater than a preset threshold within a preset range in the brightness adjustment for the plurality of image processes.

2. The imaging device according to claim 1.

3. The brightness adjustment unit acquires the frame images captured in a cycle with one frame interval as the first frame cycle, and performs the brightness adjustment for display on the acquired frame images.

2. The imaging device according to claim 1.

4. When the number of image processes is increased or decreased, the image selection unit selects the frame image that was allocated for the image processes before the increase or decrease so that the frame image is allocated to the brightness adjustment for the image processes or the display.

2. The imaging device according to claim 1.

5. an imaging unit that images an occupant of a vehicle; The imaging device according to claim 1 , further comprising: an imaging section configured to acquire the captured image; a display unit that displays the captured image whose luminance has been adjusted for display by the imaging device; an image analysis unit that acquires the captured images that have been brightness-adjusted for the plurality of image processing by the imaging device and performs a plurality of image processing operations on the acquired captured images; A driver monitoring system with

6. A brightness adjustment method performed by an imaging device, comprising: an image selection unit acquiring a captured image and selecting frame images constituting the acquired captured image; a brightness adjustment unit performing brightness adjustment for display on the frame images captured at a first frame period among the frame images selected from the captured video by the image selection unit, and performing a plurality of different brightness adjustments for image processing on a plurality of the frame images captured at a plurality of different frame periods that are different from the first frame period; A brightness adjustment method comprising:

7. Computer, an image selection unit that acquires a captured image and selects frame images that constitute the acquired captured image; and A program for causing the program to function as a brightness adjustment unit that performs brightness adjustment for display on frame images captured at a first frame period among the frame images selected from the captured video by the image selection unit, and performs brightness adjustment for multiple different image processing operations on multiple frame images that are each captured at multiple frame periods that are different from the first frame period and are mutually different from each other.

Images