Seat position estimation device, seat position estimation method, and computer program for seat position estimation

Abstract

To provide a seat position estimation device which can estimate a position of a driver seat from an image showing a cabin of a vehicle.SOLUTION: A seat position estimation device comprises: a mask image generation unit 31 which generates a mask image by masking a region showing a driver in a driver image showing a cabin generated by an imaging unit 2 while a driver is getting on a vehicle 10; a difference image generation unit 32 which generates one or more difference images with a difference between each of one or more reference images with mutually different positions of a driver seat 11 and the mask image; and an estimation unit 33 which obtains an individual estimation value of the position of the driver seat for each difference image by inputting each of the one or more difference images to a discriminator and calculates a statistical representative value of the individual estimation value obtained for each difference image as an estimation value of the position of the driver seat.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a seat position estimation device, a seat position estimation method, and a computer program for seat position estimation, which estimate the position of a driver's seat of a vehicle from an image showing the interior of a vehicle cabin. [Background technology]

[0002] It has been considered to determine whether or not a driver's condition is suitable for driving a vehicle based on an image obtained by capturing an image of the driver's face using an imaging device. In order to determine the driver's condition, it is necessary to accurately detect the driver's face from the image. However, the position and size of the driver's face on the image change depending on the position of the driver's seat. Therefore, in order to accurately detect the driver's face from the image, it is preferable to be able to estimate the position of the driver's seat.

[0003] Meanwhile, a method for determining the position of a vehicle seat has been proposed for an occupant protection system (see Patent Document 1). The method disclosed in Patent Document 1 detects an image area in the passenger compartment of a vehicle by a camera, and transmits image information detected by the camera to an evaluation unit. The method then selects a predetermined portion including an inherent feature of the vehicle seat from the image area, and determines the position of the inherent feature of the vehicle seat from the image area detected in the predetermined portion. In addition, an area including a side of the vehicle seat associated with the camera and / or surrounded by two planes in contact with the side is detected as the predetermined portion. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2004-516470 Summary of the Invention [Problem to be solved by the invention]

[0005] While the driver is in the vehicle, most of the driver's seat is hidden by the driver when viewed from the imaging device. Therefore, even if most of the driver's seat is hidden by the driver in the image, it is required to be able to estimate the position of the driver's seat from the image.

[0006] SUMMARY OF THE PRESENT DISCLOSURE In view of the above, an object of the present invention is to provide a seat position estimation device capable of estimating the position of a driver's seat from an image showing the interior of a vehicle cabin. [Means for solving the problem]

[0007] According to one embodiment, there is provided a seat position estimation device, which includes a storage unit that stores one or more reference images that represent the interior of a vehicle when the driver's seat of the vehicle is in a predetermined position, and the positions of the driver's seat in each of the one or more reference images are different from each other, a mask image generation unit that identifies an area in which the driver is represented in a driver image generated while the driver is riding in the vehicle by an imaging unit provided to capture an image of the interior of the vehicle and generates a mask image by masking the identified area, a difference image generation unit that generates one or more difference images by subtracting the mask image from each of the one or more reference images, and an estimation unit that inputs each of the one or more difference images to a classifier that has been trained in advance to estimate the position of the driver's seat, thereby obtaining an individual estimated value of the driver's seat position for each of the one or more difference images, and calculates a statistical representative value of the individual estimated value for each of the one or more difference images as an estimated value of the driver's seat position.

[0008] In the seat position estimation device, it is preferable that an allowable range of the driver's seat position is set for each of one or more reference images.The estimation unit preferably calculates, as an estimate of the driver's seat position, a statistical representative value of the individual estimated values ​​of the driver's seat position that are included in the allowable range of the reference image corresponding to the differential image, among the individual estimated values ​​of the driver's seat position obtained for each of one or more differential images.

[0009] In addition, in this seat position estimation device, it is preferable that the difference image generation unit generates an edge mask image by applying an edge detection filter to the mask image, and generates one or more difference images based on the difference between the edge mask image and each of one or more edge reference images obtained by applying the edge detection filter to each of the one or more reference images.

[0010] Furthermore, in this seat position estimation device, it is preferable that each of the one or more reference images further represents a steering wheel of the vehicle having a different position from each other. Then, it is preferable that the estimation unit obtains an individual estimated value of the steering wheel position for each of the one or more difference images by inputting each of the one or more difference images to a second classifier that has been trained in advance to estimate the steering wheel position, and further calculates a statistical representative value of the individual estimated value of the steering wheel position for each of the one or more difference images as the estimated value of the steering wheel position.

[0011] According to another aspect, there is provided a seat position estimation method, which includes: identifying an area in which a driver is represented in a driver image generated while the driver is in the vehicle by an imaging unit provided to capture an image of the interior of the vehicle; generating a mask image by masking the identified area; generating one or more difference images by subtracting the mask image from one or more reference images each representing the interior of the vehicle when the driver seat of the vehicle is in a predetermined position; inputting each of the one or more difference images, in which the driver seat position is different from each other in each of the one or more reference images, into a classifier previously trained to estimate the position of the driver seat, to obtain an individual estimate of the position of the driver seat for each of the one or more difference images; and calculating a statistical representative value of the individual estimate for each of the one or more difference images as the estimate of the position of the driver seat.

[0012] According to still another aspect, there is provided a computer program for seat position estimation, the computer program for seat position estimation including instructions for causing a processor mounted on a vehicle to execute the following: identifying an area in which a driver is represented in a driver image generated while the driver is seated in the vehicle by an imaging unit provided to capture an image of the interior of the vehicle, generating a mask image by masking the identified area, generating one or more difference images by subtracting the mask image from each of one or more reference images representing the interior of the vehicle when the driver seat of the vehicle is in a predetermined position, inputting each of the one or more difference images, in which the driver seat position in each of the one or more reference images is different from one another, into a classifier previously trained to estimate the position of the driver seat, to obtain an individual estimate of the position of the driver seat for each of the one or more difference images, and calculating a statistical representative value of the individual estimate for each of the one or more difference images as the estimate of the position of the driver seat. Effect of the Invention

[0013] The seat position estimation device according to the present disclosure has an effect of being able to estimate the position of a driver's seat from an image showing the interior of a vehicle cabin. [Brief description of the drawings]

[0014] [Figure 1] FIG. 2 is an explanatory diagram of a driver's seat position and a steering position that are targets of estimation. [Diagram 2] 1 is a schematic configuration diagram of a vehicle control system including a seat position estimation device. [Diagram 3] FIG. 2 is a hardware configuration diagram of an ECU that is an example of a seat position estimation device. [Figure 4] FIG. 4 is a functional block diagram of a processor of an ECU related to a seat position estimation process. [Diagram 5] FIG. 1 is a diagram for explaining an outline of mask image generation. [Figure 6] 1A to 1C are diagrams illustrating an overview of differential image generation and position estimated value calculation. [Figure 7] FIG. 13 is a diagram illustrating the relationship between the allowable range of the driver's seat position set for each reference image, the estimated value of the position calculated by the first classifier, and the finally estimated driver's seat position. [Figure 8] 4 is an operational flowchart of a seat position estimation process executed by a processor of an ECU of a vehicle. [Figure 9] 4 is an operational flowchart of a vehicle control process executed by a processor of an ECU of the vehicle. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, a seat position estimation device, a seat position estimation method executed by the seat position estimation device, and a computer program for seat position estimation will be described with reference to the drawings. The seat position estimation device estimates the position of the driver's seat and the position of the steering wheel based on an image showing the interior of the vehicle in which the driver is riding, which is generated by a camera for a driver monitor provided in the vehicle. To this end, the seat position estimation device specifies an area in which the driver is shown on the image by a background difference between the image showing the interior of the vehicle in which the driver is riding and a background image showing the interior of the vehicle when the driver is not riding. Then, the seat position estimation device generates a mask image in which the specified area is masked. Next, the seat position estimation device generates a difference image between the mask image and at least one reference image showing the interior of the vehicle in a predetermined seat position and a predetermined steering position. Then, the seat position estimation device obtains an estimated value of the seat position and the steering wheel position by inputting each generated difference image to a discriminator for estimating the seat position and the steering wheel position.

[0016] In the following, an example will be described in which the seat position estimation device is implemented in a vehicle control system that monitors the driver's state and performs driving assistance or the like according to the driver's state.

[0017] FIG. 1 is an explanatory diagram of the driver's seat position and the steering position to be estimated. In this embodiment, the inclination angle of the backrest 11a of the driver's seat 11, which is adjustable in the direction indicated by the arrow 101, and the depth direction position of the seat surface 11b of the driver's seat 11, which is adjustable in the direction indicated by the arrow 102, are the estimation targets of the driver's seat position. In addition, the vertical position of the steering wheel 12, which is adjustable in the direction indicated by the arrow 103, and the depth direction position of the steering wheel 12, which is adjustable in the direction indicated by the arrow 104, are the estimation targets of the steering position. However, without being limited to this, the height direction position of the seat surface 11b of the driver's seat may also be the estimation target of the driver's seat position.

[0018] FIG. 2 is a schematic diagram of a vehicle control system including a seat position estimation device. In this embodiment, a vehicle control system 1 that is mounted on a vehicle 10 and controls the vehicle 10 includes a driver monitor camera 2, a notification device 3, and an electronic control unit (ECU) 4 that is an example of a seat position estimation device. The driver monitor camera 2, the notification device 3, and the ECU 4 are communicably connected to each other via an in-vehicle network that complies with a communication standard such as a controller area network. The vehicle control system 1 may include an outside camera (not shown) that captures an area around the vehicle 10 and generates an image showing the area around the vehicle 10. Alternatively, the vehicle control system 1 may include a distance sensor (not shown) such as a LiDAR or radar that measures the distance from the vehicle 10 to an object present around the vehicle 10. Furthermore, the vehicle control system 1 may include a positioning device (not shown) such as a GPS receiver for measuring the position of the vehicle 10 based on a signal from a satellite. Furthermore, the vehicle control system 1 may include a navigation device (not shown) for searching for a planned driving route to a destination. Furthermore, the vehicle control system 1 may have a storage device (not shown) that stores map information to be referenced in the automatic driving control of the vehicle 10.

[0019] The driver monitor camera 2 is an example of an imaging unit, and has a two-dimensional detector composed of an array of photoelectric conversion elements, such as CCD or C-MOS, that are sensitive to visible light or infrared light, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. The driver monitor camera 2 may further have a light source, such as an infrared LED, for illuminating the driver. The driver monitor camera 2 is provided in the vehicle interior so that the face of the driver seated on the driver's seat of the vehicle 10 is included in the area to be photographed, that is, so that the face of the driver can be photographed. For example, the driver monitor camera 2 is attached to the instrument panel or its vicinity, or to the steering wheel, facing the driver. The driver monitor camera 2 photographs the area to be photographed at a predetermined photographing period (for example, 1 / 30 to 1 / 10 seconds) and generates an image in which the area to be photographed is shown (hereinafter, referred to as a driver image). The driver image obtained by the driver monitor camera 2 is an example of an interior image showing the interior of the vehicle 10, and may be a color image or a gray image. Every time the driver monitor camera 2 generates a driver image, it outputs the generated driver image to the ECU 4 via the in-vehicle network.

[0020] The notification device 3 is provided in the vehicle cabin of the vehicle 10, and is a device that provides a predetermined notification to the driver by light, sound, vibration, text display, or image display. To achieve this, the notification device 3 has at least one of a speaker, a light source, a vibrator, or a display device, for example. When the notification device 3 receives a notification indicating a warning to the driver from the ECU 4, it notifies the driver of the warning by sound from the speaker, light emission or blinking of the light source, vibration of the vibrator, or display of a warning message on the display device.

[0021] The ECU 4 controls the driving of the vehicle 10 according to a driving control level applied to the vehicle 10, or assists the driver in driving the vehicle 10. Furthermore, the ECU 4 monitors the driver based on the driver image received from the driver monitor camera 2 and detects an abnormality occurring in the driver. When the ECU 4 determines that an abnormality has occurred in the driver, the ECU 4 warns the driver or controls the vehicle 10 to make an emergency stop of the vehicle 10. Furthermore, the ECU 4 estimates the position of the driver seat and the position of the steering wheel based on the driver image at a predetermined timing. The ECU 4 reflects the estimation result in the detection conditions for detecting the driver's face from the driver image, thereby improving the detection accuracy of the driver's face and the estimation accuracy of the driver's state.

[0022] Fig. 3 is a hardware configuration diagram of the ECU 4. As shown in Fig. 3, the ECU 4 has a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 may each be configured as a separate circuit, or may be configured integrally as a single integrated circuit.

[0023] The communication interface 21 has an interface circuit for connecting the ECU 4 to an in-vehicle network. Every time the communication interface 21 receives a driver image from the driver monitor camera 2, the communication interface 21 passes the received driver image to the processor 23. Furthermore, when the communication interface 21 receives information to be notified to the driver via the notification device 3, such as a notification indicating a warning to the driver, from the processor 23, the communication interface 21 outputs the information to the notification device 3.

[0024] The memory 22 is an example of a storage unit, and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores various algorithms and various data used in the seat position estimation process executed by the processor 23 of the ECU 4. For example, the memory 22 stores various data and parameters used for estimating the positions of the driver's seat and steering wheel or detecting the driver. Such data includes, for example, a background image representing the interior of the vehicle when the driver is not in the vehicle 10, and one or more reference images representing the interior of the vehicle when the driver's seat and steering wheel are in a predetermined position. Furthermore, the memory 22 stores a detection condition table representing the relationship between the estimated value of the position and the detection condition of the driver's face. Furthermore, the memory 22 stores parameters for defining the judgment condition for judging whether or not an abnormality has occurred in the driver (hereinafter, sometimes simply referred to as the abnormality judgment condition). Furthermore, the memory 22 temporarily stores the driver image and various data generated during the seat position estimation process and the estimated value of the position obtained as a result of the position estimation process. Furthermore, the memory 22 stores various parameters and various data used for driving control of the vehicle 10. Such data includes images generated by an exterior camera, distance measurement signals generated by a distance sensor, positioning signals indicating the position of the vehicle 10 generated by a GPS receiver, a planned driving route generated by a navigation device, and map information.

[0025] The processor 23 has one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further have other arithmetic circuits such as a logical arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. The processor 23 executes vehicle driving control processing including seat position estimation processing at predetermined intervals.

[0026] 4 is a functional block diagram of the processor 23 related to the vehicle driving control process including the seat position estimation process. The processor 23 has a mask image generating unit 31, a difference image generating unit 32, an estimation unit 33, a detection unit 34, a posture detection unit 35, an abnormality determination unit 36, a warning processing unit 37, and a vehicle control unit 38. Each of these units of the processor 23 is, for example, a functional module realized by a computer program running on the processor 23. Alternatively, each of these units of the processor 23 may be a dedicated arithmetic circuit provided in the processor 23. Among these units of the processor 23, the processes executed by the mask image generating unit 31, the difference image generating unit 32, and the estimation unit 33 correspond to the seat position estimation process. The mask image generating unit 31, the difference image generating unit 32, and the estimation unit 33 execute the seat position estimation process on the driver image received by the ECU 4 from the driver monitor camera 2 at a predetermined timing. Furthermore, the detection unit 34 detects the driver's face from the latest driver image at predetermined intervals using the results of the seat position estimation process. Then, the posture detection unit 35, the abnormality determination unit 36, the warning processing unit 37, and the vehicle control unit 38 determine whether or not an abnormality has occurred in the driver based on the detected driver's face, and control the vehicle 10 or issue a warning to the driver according to the determination result.

[0027] The mask image generating unit 31 detects an area in the driver image where the driver is represented (hereinafter, may be referred to as a driver area) by a background difference between the driver image and the background image received by the ECU 4 from the driver monitor camera 2 at a predetermined timing. Then, the mask image generating unit 31 generates a mask image by masking the driver area on the driver image.

[0028] The predetermined timing is preferably the timing when the driver has finished setting the driver seat position and the steering position. This is because it is assumed that the driver seat and steering positions will not be changed frequently after the driver has finished setting the driver seat position and the steering position. Therefore, the predetermined timing can be, for example, the timing when a certain time has elapsed since the ignition switch of the vehicle 10 was turned on. Alternatively, the predetermined timing may be the timing when the speed of the vehicle 10 first exceeds a predetermined speed threshold (for example, 10 km / h) after the ignition switch of the vehicle 10 is turned on. Alternatively, the predetermined timing may be the timing when a predetermined period has elapsed since the previous execution of the seat position estimation process. In addition, if the driver's face cannot be detected from the driver image, there is a possibility that the detection conditions are not appropriate. Therefore, the predetermined timing may be the timing when the ratio of driver images in which the driver's face is not detected among a series of driver images in the most recent predetermined period of time becomes equal to or greater than a predetermined ratio.

[0029] The mask image generating unit 31 calculates the absolute value of the luminance difference between corresponding pixels of the driver image and the background image as the background difference. Then, the mask image generating unit 31 specifies a set of pixels in which the absolute value of the luminance difference is equal to or greater than a predetermined luminance difference threshold as a driver area in which the driver is represented. If the driver image and the background image are expressed in a color system that does not include a direct luminance component, such as the RGB color system, the mask image generating unit 31 converts the color system of the driver image and the background image to the HLS color system. Then, the mask image generating unit 31 may calculate the absolute value of the luminance difference for each corresponding pixel using the luminance component of each pixel of the driver image and the background image. Also, a plurality of background images according to the brightness in the vehicle cabin may be stored in advance in the memory 22. In this case, the mask image generating unit 31 may read out from the memory 22 a background image associated with a brightness closest to a measurement value of the amount of light by a light amount sensor (not shown) mounted on the vehicle 10, and use the readout image for the background difference.

[0030] The mask image generating unit 31 rewrites the value of each pixel included in the driver region in the driver image with a predetermined value, thereby generating a mask image in which the driver region is masked.

[0031] 5 is a diagram for explaining an outline of mask image generation. A driver 501 is depicted in a driver image 500. Therefore, a driver region 502 in which the driver 501 is depicted on the driver image 500 is specified by background difference between the driver image 500 and a background image 510 in which the driver is not depicted. Therefore, a mask image 520 is generated by masking the driver region 502 by rewriting the values ​​of each pixel in the driver region 502 of the driver image 500 to a predetermined value.

[0032] The mask image generating unit 31 may detect the driver area using a plurality of driver images obtained in time series before and after a predetermined timing. For example, the mask image generating unit 31 detects an individual driver area for each driver image by performing background subtraction between each of the plurality of driver images and a background image. Then, the mask image generating unit 31 determines a new driver area as a union or intersection of the individual driver areas for each driver image. The mask image generating unit 31 may generate a mask image by masking the driver area obtained in any one of the series of driver images. This allows the mask image generating unit 31 to more reliably mask the area in the driver image where the driver is likely to be present.

[0033] The mask image generating unit 31 passes the generated mask image to the difference image generating unit 32 .

[0034] The difference image generating unit 32 generates a difference image between the mask image and each of one or more reference images. The reference image shows the interior of the vehicle 10 when the driver's seat and the steering wheel are set to a predetermined position. When a plurality of reference images are prepared, a combination of the driver's seat position and the steering wheel position different from each other is shown for each reference image. In particular, the plurality of reference images preferably include two or more reference images set so that only one of the movable parts of the driver's seat and the steering wheel is in a different position. For example, it is preferable to prepare two or more reference images in which the position of the seat surface of the driver's seat and the position of the steering wheel in the depth direction and the position in the up-down direction are fixed, and only the inclination of the backrest of the driver's seat is different. Alternatively, it is preferable to prepare two or more reference images in which the position of the seat surface of the driver's seat and the inclination of the backrest and the position of the steering wheel in the up-down direction are fixed, and only the position of the steering wheel in the depth direction is different. As a result, in each difference image, the difference between the position shown in the mask image and the position shown in the corresponding reference image for a specific movable part becomes clear.

[0035] The difference image generating unit 32 calculates the absolute value of the difference in pixel values ​​between corresponding pixels in the one or more reference images and the mask image for each of the one or more reference images. The difference image generating unit 32 then generates a difference image for each pixel of the one or more reference images such that the absolute value of the pixel difference between corresponding pixels becomes the value of that pixel. If the reference image and the mask image are color images, the difference image generating unit 32 may generate a difference image by executing the above process for each color component. As a result, the difference image emphasizes the difference between the positions of the driver's seat and steering wheel shown in the reference image and the positions of the driver's seat and steering wheel shown in the mask image.

[0036] Alternatively, the difference image generating unit 32 may apply a predetermined edge detection filter to each reference image and mask image to generate an image representing the strength of an edge. The difference image generating unit 32 may then calculate, for each reference image, the absolute value of the difference in pixel values ​​between corresponding pixels between an edge reference image generated from the reference image and an edge mask image generated from the mask image. In this case, too, the difference image generating unit 32 generates a difference image for each pixel such that the absolute value of the difference in pixel values ​​between corresponding pixels becomes the value of that pixel. Note that the difference image generating unit 32 may use, for example, a Sobel filter, a Prewitt filter, or a Laplacian filter as the predetermined edge detection filter.

[0037] The difference image generating unit 32 passes the difference image generated for each reference image to the estimation unit 33 .

[0038] The estimation unit 33 inputs each of the difference images generated for each reference image to a first classifier that estimates the position of the driver's seat and a second classifier that estimates the position of the steering wheel. As a result, the estimation unit 33 obtains an individual estimated value of the position of the driver's seat and an individual estimated value of the position of the steering wheel for each difference image. Note that the estimation unit 33 can use, for example, a deep neural network (DNN) having a convolutional neural network (CNN) type architecture having multiple convolution layers as the first classifier and the second classifier. Alternatively, the estimation unit 33 can use a DNN having a self attention network (SAN) type architecture as the first classifier and the second classifier. These classifiers are trained in advance according to a predetermined learning method such as an error backpropagation method using a large number of teacher images showing the driver's seat or steering wheel in various positions.

[0039] The range in which the steering wheel may be displayed on the driver image and the range in which the driver seat may be displayed are assumed. As described above, when the driver monitor camera 2 is provided on the instrument panel or in its vicinity, the steering wheel is located closer to the driver monitor camera 2 than the driver seat. Therefore, the range in which the steering wheel is displayed on the driver image is larger than the range in which the driver seat is displayed. Therefore, the estimation unit 33 inputs the entire difference image to the second classifier that estimates the position of the steering wheel. On the other hand, the estimation unit 33 may input only the range in which the driver seat may be displayed, out of the difference image, to the first classifier that estimates the position of the driver seat.

[0040] According to a modified example, the estimation unit 33 may input the entire difference image to both the first classifier and the second classifier. Alternatively, one classifier may be trained in advance to estimate both the driver seat position and the steering wheel position. In this case, the estimation unit 33 can obtain individual estimated values ​​of both the driver seat position and the steering wheel position by inputting the difference image to the one classifier.

[0041] The estimation unit 33 calculates a statistical representative value, for example, an average value or a median value, of the individual estimated values ​​of the driver's seat position estimated from each difference image as an estimated value of the driver's seat position. Similarly, the estimation unit 33 calculates a statistical representative value of the individual estimated values ​​of the steering position estimated from each difference image as an estimated value of the steering position. Note that, when only one reference image is prepared, the individual estimated value of the driver's seat position calculated by the first classifier for the difference image obtained by the difference between the one reference image and the mask image becomes the statistical representative value. That is, the individual estimated value itself becomes the estimated value of the driver's seat position. Similarly, the individual estimated value of the steering position calculated by the second classifier for the difference image obtained by the difference between the one reference image and the mask image becomes the estimated value of the steering position.

[0042] Note that an allowable range of the driver's seat position may be set in advance for each reference image. The estimation unit 33 selects an individual estimated value of the driver's seat position included in the allowable range set for the reference image used to generate the difference image from among the individual estimated values ​​of the driver's seat position estimated from each difference image. The estimation unit 33 may then calculate a statistical representative value of the selected individual estimated value as the estimated value of the driver's seat position. Similarly, an allowable range of the steering position may be set in advance for each reference image. The estimation unit 33 selects an individual estimated value of the steering position included in the allowable range set for the reference image used to generate the difference image from among the individual estimated values ​​of the steering position estimated from each difference image. The estimation unit 33 may then set the statistical representative value of the selected individual estimated value as the estimated value of the steering position.

[0043] FIG. 6 is a diagram for explaining an outline of difference image generation and position estimation value calculation. In this example, n reference images 600-1 to 600-n (n is an integer equal to or greater than 2) with different combinations of driver seat position and steering wheel position are prepared in advance. Then, for each of the reference images 600-1 to 600-n, a difference image between the mask image 601 is generated. That is, a difference image 602-i is generated from the reference image 600-i (i=1, 2, . . . , n) and the mask image 601. The generated difference images 602-1 to 602-n are input one by one to a first classifier 610 and a second classifier 620. Note that, as described above, only an area 603 on the difference image 602-i that may possibly represent the driver seat is input to the first classifier 610, and the entire difference image 602-i may be input to the second classifier 620. As a result, an individual estimated value of the driver seat position and an individual estimated value of the steering wheel position are calculated for each of the difference images 602-1 to 602-n.

[0044] FIG. 7 is a diagram for explaining the relationship between the allowable range of the driver seat position set for each reference image, the individual estimated value of the position calculated by the first classifier, and the finally estimated driver seat position. In this example, it is assumed that the driver seats having different inclination angles of the backrest of the driver seat are represented in each of the reference images 0 to 11. It is also assumed that the estimated value of the inclination angle of the backrest of the driver seat output by the first classifier is output in 5 degree increments in the range of 10 degrees to 40 degrees. The allowable range of the inclination angle is set to 10 degrees to 25 degrees for the reference images 0 to 2. The allowable range of the inclination angle is set to 20 degrees to 35 degrees for the reference images 3 to 5. The allowable range of the inclination angle is set to 25 degrees to 40 degrees for the reference images 6 to 8. The allowable range of the inclination angle is set to 15 degrees to 30 degrees for the reference images 9 to 11.

[0045] In FIG. 7, each star 701 represents an individual estimated value of the inclination angle of the driver's seat calculated by the first classifier from the difference images calculated for each of the reference images 0 to 11. In this example, for reference image 5 and reference image 8, the individual estimated values ​​of the inclination angle of the driver's seat are outside the allowable range. Therefore, in estimating the final inclination angle of the backrest of the driver's seat, the individual estimated values ​​calculated from the difference images corresponding to reference image 5 and reference image 8 are excluded. Then, the final estimated value of the inclination angle of the backrest of the driver's seat is calculated as a statistical representative value of the individual estimated values ​​calculated from the difference images corresponding to reference images 0 to 4, 6 to 7, and 9 to 11.

[0046] The estimation unit 33 stores the estimated value of the driver's seat position and the estimated value of the steering wheel position in the memory 22.

[0047] 8 is an operational flowchart of the seat position estimation process executed by the processor 23. The processor 23 may execute the seat position estimation process in accordance with the following operational flowchart at the predetermined timing described above.

[0048] The mask image generating unit 31 of the processor 23 detects a driver area from the driver image by a background difference between the driver image and the background image (step S101). Then, the mask image generating unit 31 masks the driver area on the driver image, that is, rewrites the value of each pixel in the driver area to a predetermined value to generate a mask image (step S102).

[0049] The difference image generating unit 32 of the processor 23 generates difference images between the mask image and each of the one or more reference images (step S103).

[0050] The estimation unit 33 of the processor 23 inputs each of the difference images generated for each reference image to the first classifier and the second classifier. As a result, the estimation unit 33 calculates an individual estimated value of the driver seat position and an individual estimated value of the steering wheel position for each difference image (step S104). Then, the estimation unit 33 calculates a statistical representative value of the individual estimated values ​​of the driver seat position estimated from each difference image as an estimated value of the driver seat position (step S105). Similarly, the estimation unit 33 calculates a statistical representative value of the individual estimated values ​​of the steering wheel position estimated from each difference image as an estimated value of the steering wheel position (step S106). After step S106, the processor 23 ends the seat position estimation process.

[0051] The detection unit 34 detects, from the driver image received by the ECU 4 from the driver monitor camera 2, an area showing the driver's face (hereinafter referred to as a face area).

[0052] The detection unit 34 detects candidates for a face region from the driver image, for example, by inputting the driver image into a classifier that has been trained in advance to detect the driver's face from the image. The detection unit 34 can use, for example, a DNN having a CNN-type or SAN-type architecture as such a classifier. Alternatively, the detection unit 34 may use, as such a classifier, a classifier based on a machine learning method other than DNN, such as a support vector machine or an AdaBoost classifier. Such a classifier is trained in advance according to a predetermined learning method, such as an error backpropagation method, using a large number of teacher images showing the driver's face.

[0053] The detection unit 34 judges whether the candidate for the face region output by the classifier satisfies the detection condition, and if the candidate for the face region satisfies the detection condition, the candidate is determined as the face region. The detection condition is set as an estimated region in which the driver's face is assumed to exist on the driver image and an allowable range of the size of the face region on the driver image. That is, the detection unit 34 judges that the candidate for the face region output by the classifier satisfies the detection condition if the candidate for the face region is included in the estimated region indicated by the detection condition and the size of the face region is within the allowable range of the size indicated by the detection condition. The detection condition is set according to a combination of the driver's seat position and the steering position. In this embodiment, a detection condition table indicating the detection condition for each combination of the driver's seat position and the steering position is stored in advance in the memory 22. Specifically, the more the seat surface of the driver's seat moves forward or the smaller the inclination angle of the backrest, the closer the position of the driver is to the driver monitor camera 2. And the closer the position of the driver is to the driver monitor camera 2, the larger the area in which the driver's face is represented on the driver image. Furthermore, the closer the driver's position is to the driver monitor camera 2, the greater the amount of movement of the driver on the driver image accompanying the driver's movement. Therefore, the further forward the seat surface of the driver's seat moves, or the smaller the inclination angle of the backrest, the wider the estimated area indicated by the detection conditions is set. Also, the area in which the steering wheel is represented on the driver image changes depending on the steering wheel position. Therefore, the estimated area indicated by the detection conditions is set so as not to include the area in which the steering wheel is represented on the driver image depending on the steering wheel position. The detection unit 34 refers to the detection condition table and specifies the detection conditions corresponding to the combination of the estimated value of the driver's seat position and the estimated value of the steering wheel position obtained by the estimation unit 33. The detection unit 34 then uses the specified detection conditions to detect the face area.

[0054] The detection unit 34 notifies the orientation detection unit 35 of information indicating the position and range of the detected face area (for example, the coordinates of the upper left corner and the coordinates of the lower right corner of the face area).

[0055] The posture detection unit 35 detects the posture of the driver based on the face area detected from the driver image. In this embodiment, the posture detection unit 35 detects the position and direction of the driver's face as information representing the driver's posture.

[0056] The posture detection unit 35 detects a plurality of feature points of the driver's face, such as the corners of the eyes, the tip of the nose, and the corners of the mouth, from the face area of ​​the driver image. In this case, the posture detection unit 35 detects the feature points of the face by inputting the face area to a classifier that has been trained in advance to detect the feature points of the face shown in the image. As such a classifier, the posture detection unit 35 can use, for example, a DNN having a CNN-type architecture, a support vector machine, or an AdaBoost classifier. Note that the classifier for detecting the face area and the classifier for detecting the feature points of the face may be configured integrally. In this case, the detection unit 34 may input the driver image to the classifier to detect the face area and each of the feature points of the face. Alternatively, the posture detection unit 35 may detect the individual feature points of the driver's face from the face area according to template matching between a template representing the feature points of the face and the face area, or another method for detecting the feature points of the face.

[0057] The posture detection unit 35 fits each of the detected facial feature points to a three-dimensional face model that represents the three-dimensional shape of the face. The posture detection unit 35 then detects the facial orientation of the three-dimensional face model when each feature point best fits the three-dimensional face model as the facial orientation of the driver. Note that the posture detection unit 35 may detect the facial orientation of the driver based on the driver image according to another method for determining the facial orientation shown in the image.

[0058] Furthermore, the posture detection unit 35 detects the position of the center of gravity of the face area in the driver image as the position of the driver's face.

[0059] The posture detection unit 35 may detect the posture of the driver based on the driver's gaze direction. In this case, the posture detection unit 35 detects an eye region representing the driver's eyes from the face region. In this case, the posture detection unit 35 may detect the eye region for each of the left and right eyes by a method similar to the above-mentioned method for detecting facial feature points. The posture detection unit 35 further detects the pupil center and the corneal reflection image (Purkinje image) of the light source of the driver monitor camera 2 by template matching for either the left or right eye region. Then, the posture detection unit 35 may detect the driver's gaze direction based on the positional relationship between the pupil center and the Purkinje image. If the posture detection unit 35 cannot detect either the left or right eye region of the driver, it determines that the detection of the gaze direction has failed.

[0060] The posture detection unit 35 notifies the abnormality determination unit 36 ​​of the detection result of the driver's posture with respect to the driver image, i.e., the detection result of the position and orientation of the driver's face. Furthermore, when the driver's line of sight direction is detected, the posture detection unit 35 notifies the abnormality determination unit 36 ​​of the detection result of the line of sight direction.

[0061] The abnormality determination unit 36 ​​determines that an abnormality has occurred in the driver when the driver's posture detected by the posture detection unit 35 satisfies the abnormality determination condition. As described above, in this embodiment, the driver's posture is represented by the position and direction of the driver's face. The abnormality determination condition is that the state in which the driver's face position or direction is out of the normal range continues for a time threshold or more. Therefore, the abnormality determination unit 36 ​​determines whether or not the position and direction of the driver's face are included in a preset normal range each time the posture detection unit 35 notifies the abnormality determination unit 36 ​​of the position and direction of the driver's face. Then, the abnormality determination unit 36 ​​determines that an abnormality has occurred in the driver when the length of the period during which the position or direction of the driver's face is out of the normal range continues to be equal to or greater than the time threshold. In addition, when the driver's line of sight direction has been detected, the abnormality determination unit 36 ​​determines whether or not the line of sight direction is included in a preset normal range each time the posture detection unit 35 notifies the abnormality determination unit 36 ​​of the direction of the driver's line of sight. The abnormality determination unit 36 ​​may determine that an abnormality has occurred in the driver when the length of the period during which the driver's line of sight direction continues to be outside the normal range is equal to or greater than a time threshold value.

[0062] The abnormality determination unit 36 ​​notifies the warning processing unit 37 and the vehicle control unit 38 of the determination result as to whether or not an abnormality has occurred in the driver.

[0063] When the warning processing unit 37 receives a determination result that an abnormality has occurred in the driver from the abnormality determination unit 36, the warning processing unit 37 performs a predetermined warning process. For example, the warning processing unit 37 causes a speaker of the notification device 3 to emit a voice signal or a warning sound requesting the driver to assume a driving posture. Alternatively, the warning processing unit 37 causes a display of the notification device 3 to display a warning message requesting the driver to assume a driving posture. Alternatively, the warning processing unit 37 causes a vibrator of the notification device 3 to vibrate.

[0064] The warning processing unit 37 performs a warning process to request the driver to assume a driving posture via the notification device 3, and then stops executing the warning process when it receives a determination result from the abnormality determination unit 36 ​​that no abnormality has occurred in the driver.

[0065] The vehicle control unit 38 controls the vehicle 10 according to the driving control level applied to the vehicle 10 until receiving a determination result from the abnormality determination unit 36 ​​that an abnormality has occurred in the driver. When the driving control level applied to the vehicle 10 is a driving control level in which the driver is not involved in driving the vehicle 10, the vehicle control unit 38 controls the vehicle 10 so that the vehicle 10 runs along the lane in which the vehicle 10 is running. To this end, the vehicle control unit 38 detects lane markings that separate the vehicle lane from adjacent lanes, and moving objects such as other vehicles running around the vehicle 10, from an image generated by an exterior camera. The vehicle control unit 38 then compares the detected lane markings with map information to estimate the position and attitude of the vehicle 10. The vehicle control unit 38 then controls the vehicle 10 so that the vehicle 10 does not collide with each moving object and runs along the lane in which the vehicle 10 is running, based on the estimated results of the position and attitude of the vehicle 10 and the detection results of each moving object around the vehicle 10.

[0066] Furthermore, when the vehicle control unit 38 continuously receives a determination result from the abnormality determination unit 36 ​​that an abnormality has occurred in the driver for a certain period of time, the vehicle control unit 38 controls the vehicle 10 to make an emergency stop of the vehicle 10. Note that the vehicle control unit 38 may control the vehicle 10 to make an emergency stop of the vehicle 10 immediately upon receiving a determination result from the abnormality determination unit 36 ​​that an abnormality has occurred in the driver. In this case, the vehicle control unit 38 may move the vehicle 10 to the shoulder of the road and then stop the vehicle 10 based on the estimation results of the position and attitude of the vehicle 10, the detection results of each moving object around the vehicle 10, and map information.

[0067] 9 is an operational flowchart of the vehicle control process executed by the processor 23. The processor 23 may execute the vehicle control process in accordance with the following operational flowchart at predetermined intervals.

[0068] The detection unit 34 of the processor 23 detects candidates for face areas from the latest driver image (step S201). The detection unit 34 also sets detection conditions based on a combination of an estimated value of the driver seat position and an estimated value of the steering wheel position (step S202). The detection unit 34 then determines whether the detected candidates for face areas satisfy the detection conditions (step S203).

[0069] If the candidate for the face area does not satisfy the detection condition (step S203-No), the detection unit 34 determines that the detection of the driver's face has failed. Then, the processor 23 ends the vehicle control process. On the other hand, if the candidate for the face area satisfies the detection condition (step S203-Yes), the detection unit 34 determines that the detection of the driver's face has been successful. Then, the posture detection unit 35 of the processor 23 detects the position and direction of the driver's face based on the face area detected from the driver image (step S204). Note that, as described above, the posture detection unit 35 may detect the driver's line of sight direction from the face area.

[0070] The abnormality determination unit 36 ​​of the processor 23 determines whether or not an abnormality has occurred in the driver based on the position and direction of the driver's face (step S205). In this case, the abnormality determination unit 36 ​​may determine whether or not an abnormality has occurred in the driver based on the direction and position of the driver's face or the line of sight direction in a series of driver images in a time series at a predetermined time in the latest. When the abnormality determination unit 36 ​​determines that an abnormality has occurred in the driver (step S205-Yes), the warning processing unit 37 of the processor 23 notifies the driver of a warning requesting the driver to take a driving posture via the notification device 3 (step S206). Furthermore, the vehicle control unit 38 of the processor 23 controls the vehicle 10 to make an emergency stop of the vehicle 10 (step S207).

[0071] On the other hand, in step S205, if the abnormality determination unit 36 ​​determines that no abnormality has occurred in the driver (step S205-No), the vehicle control unit 38 controls the vehicle 10 according to the driving control level applied to the vehicle 10 (step S208).

[0072] After step S207 or step S208, the processor 23 ends the vehicle control process.

[0073] As described above, the seat position estimation device generates a mask image by masking an area showing the driver from an image showing the interior of the vehicle in which the driver is riding. Furthermore, the seat position estimation device generates a difference image between the mask image and at least one reference image showing the interior of the vehicle with the driver's seat in a predetermined position and the steering wheel in a predetermined position. The seat position estimation device then inputs each generated difference image to a classifier to obtain estimated values ​​of the driver's seat position and the steering wheel position. This allows the seat position estimation device to estimate the driver's seat position from the image showing the interior of the vehicle.

[0074] According to a modified example, the mask image generating unit 31 may specify the driver area by a method other than background subtraction. For example, the mask image generating unit 31 detects a candidate area in which the driver is represented by executing a process similar to that of the detection unit 34 for each of a series of driver images obtained in time series. Furthermore, the mask image generating unit 31 obtains an area that is a intersection of the candidate areas detected from each of the series of driver images. Since the area that is this intersection is determined to represent the driver in each driver image, it is highly likely that the driver is actually represented. Therefore, the mask image generating unit 31 may specify the area that is this intersection as the driver area.

[0075] In some vehicles, the driver monitor camera 2 is installed at a position where the steering wheel position does not affect the accuracy of detecting the face area from the driver image. In such a case, the seat position estimation device may estimate only the position of the driver seat without estimating the steering wheel position. In this case, each reference image may represent a driver seat in a different position regardless of the steering wheel position. In addition, in the estimation unit 33, the calculation of the individual estimated value of the steering wheel position by the second classifier and the calculation of the estimated value of the steering wheel position based on each individual estimated value are omitted. According to this modified example, the process of estimating the steering wheel position is omitted, so that the calculation load of the processor 23 is reduced.

[0076] In addition, a computer program for realizing the functions of the processor 23 of the ECU 4 according to the above embodiment or modified example may be provided in a form recorded on a computer-readable portable recording medium, such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

[0077] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0078] 1 Vehicle Control System 2 Driver monitor camera 3 Notification device 4 Electronic control unit (seat position estimation device) 10 Vehicles 11 Driver's seat 12 Steering 21 Communication Interface 22 Memory 23 Processors 31 Mask image generation unit 32 Differential image generation unit 33 Estimation part 34 Detection section 35 Attitude detection unit 36 Abnormality determination section 37 Warning Processing Section 38 Vehicle control unit

Claims

1. A storage unit that stores a plurality of reference images representing the interior of a vehicle, wherein the positions of the driver's seat of the vehicle in each of the plurality of reference images are different from each other; A mask image generation unit that identifies a region in a driver image generated while a driver is riding in the vehicle by an imaging unit provided to image the interior of the vehicle, and generates a mask image by masking the identified region; A difference image generation unit that generates a plurality of difference images based on the difference between the mask image and each of the plurality of reference images; An estimation unit that inputs each of the plurality of difference images into a discriminator pre-trained to estimate the position of the driver's seat, obtains an individual estimated value of the position of the driver's seat for each of the plurality of difference images, and calculates a statistical representative value of the individual estimated values for each of the plurality of difference images as an estimated value of the position of the driver's seat; A seat position estimation device having the above.

2. An allowable range of the position of the driver's seat is set for each of the plurality of reference images; The estimation unit calculates, as the estimated value of the position of the driver's seat, a statistical representative value of the individual estimated values of the position of the driver's seat included in the allowable range of the reference image corresponding to the difference image among the individual estimated values of the position of the driver's seat obtained for each of the plurality of difference images. The seat position estimation device according to Claim 1.

3. The difference image generation unit generates an edge mask image by applying an edge detection filter to the mask image, and generates the plurality of difference images based on the difference between the edge mask image and each of a plurality of edge reference images obtained by applying the edge detection filter to each of the plurality of reference images. The seat position estimation device according to Claim 1.

4. Each of the plurality of reference images further represents the steering wheel of the vehicle having different positions from each other; The estimation unit inputs each of the plurality of difference images into a second discriminator that has been pre-trained to estimate the position of the steering, thereby obtaining an individual estimated value of the position of the steering for each of the plurality of difference images, and further calculating a statistical representative value of the individual estimated values of the position of the steering for each of the plurality of difference images as an estimated value of the position of the steering. The sheet position estimation device according to any one of claims 1 to 3.

5. In a driver image generated while a driver is riding in the vehicle by an imaging unit provided to image the interior of the vehicle, an area in which the driver is represented is specified, and a mask image is generated by masking the specified area. A plurality of difference images are generated by taking the difference between the mask image and each of a plurality of reference images representing the interior of the vehicle, and the positions of the driver's seat of the vehicle in each of the plurality of reference images are different from each other. By inputting each of the plurality of difference images into a discriminator that has been pre-trained to estimate the position of the driver's seat, an individual estimated value of the position of the driver's seat is obtained for each of the plurality of difference images, and a statistical representative value of the individual estimated values for each of the plurality of difference images is calculated as an estimated value of the position of the driver's seat. A sheet position estimation method including this.

6. In a driver image generated while a driver is riding in the vehicle by an imaging unit provided to image the interior of the vehicle, an area in which the driver is represented is specified, and a mask image is generated by masking the specified area. A plurality of difference images are generated by taking the difference between the mask image and each of a plurality of reference images representing the interior of the vehicle, and the positions of the driver's seat of the vehicle in each of the plurality of reference images are different from each other. By inputting each of the plurality of difference images into a discriminator that has been pre-trained to estimate the position of the driver's seat, an individual estimated value of the position of the driver's seat is obtained for each of the plurality of difference images, and a statistical representative value of the individual estimated values for each of the plurality of difference images is calculated as an estimated value of the position of the driver's seat. A computer program for estimating seat position for causing the above to be executed by a processor mounted on the vehicle.

Images