Seat position estimation device, seat position estimation method, and seat position estimation computer program

Abstract

The present application provides a seat positioning estimation device, a seat positioning estimation method, and a seat positioning estimation computer program. The seat positioning estimation device has: a masking image generation section (31) that generates a masking image by masking a region representing a driver in an image representing the driver in a vehicle cabin generated by an imaging section (2) during a period in which the driver is seated in the vehicle (10); a difference image generation section (32) that generates one or more difference images by differencing each of one or more reference images that differ from each other in positioning of a driver's seat (11) and the masking image; and an estimation section (33) that inputs each of the one or more difference images to an identifier, obtains an individual estimation value of the positioning of the driver's seat for each difference image, and calculates a statistical representative value of the individual estimation values obtained for each difference image as an estimation value of the positioning of the driver's seat.

Description

TECHNICAL FIELD

[0001] The present application relates to a seat position estimation device, a seat position estimation method, and a seat position estimation computer program that estimate the position of a driver's seat of a vehicle from an image representing the inside of a vehicle cabin. BACKGROUND

[0002] It is studied whether or not the state of a driver is a state suitable for driving of a vehicle, based on an image obtained by imaging the face of the driver of the vehicle using an imaging device. In order to determine the state of the driver, it is required to detect the face of the driver from the image with high precision. However, the position and size of the face of the driver on the image vary depending on the position of the driver's seat. Therefore, in order to detect the face of the driver from the image with high precision, it is preferable to be able to estimate the position of the driver's seat.

[0003] On the other hand, with respect to an occupant protection system, a method of finding the position of a vehicle seat is proposed (see Japanese Patent Application Laid-Open No. 2004-516470). In the method disclosed in Japanese Patent Application Laid-Open No. 2004-516470, an image area in a passenger compartment of a vehicle is detected by a camera, and image information detected by the camera is transmitted to an evaluation unit. Also, in the method, a predetermined portion containing an inherent feature of a vehicle seat is selected from the image area, and the position of the inherent feature of the vehicle seat is found from the image area detected in the predetermined portion. In addition, as the predetermined portion, an area surrounded by a side surface of the vehicle seat corresponding to the camera and / or two flat surfaces contacting the side surface is detected. SUMMARY

[0004] During the period when the driver is seated in the vehicle, most of the driver's seat is hidden by the driver when viewed from the imaging device. Therefore, it is required to be able to estimate the position of the driver's seat from the image even if most of the driver's seat is hidden by the driver on the image.

[0005] Therefore, an object of the present application is to provide a seat position estimation device that is able to estimate the position of a driver's seat from an image representing the inside of a vehicle cabin.

[0006] According to one embodiment, a seat position estimation device is provided. The seat position estimation device has: a storage section that stores one or more reference images that indicate an interior of a vehicle when a driver's seat of the vehicle is in a predetermined position, the positions of the driver's seat in each of the one or more reference images being different from each other; a masking image generation section that determines a region that indicates the driver in a driver image generated by an imaging section provided in a manner that images the interior of the vehicle during a period in which the driver is seated in the vehicle, and masks the determined region, thereby generating a masking image; a difference image generation section that generates one or more difference images by a difference between the masking image and each of the one or more reference images; and an estimation section that inputs each of the one or more difference images to an identifier that is previously learned in a manner of estimating the position of the driver's seat, obtains an individual estimation value of the position of the driver's seat with respect to each of the one or more difference images, and calculates a statistical representative value of the individual estimation values with respect to each of the one or more difference images as an estimation value of the position of the driver's seat.

[0007] In the seat position estimation device, it is preferable that a permissible range of the position of the driver's seat be set for each of the one or more reference images. Also, it is preferable that the estimation section calculate a statistical representative value of the individual estimation values of the position of the driver's seat that are included in the permissible range of the reference image corresponding to the difference image, among the individual estimation values of the position of the driver's seat obtained with respect to each of the one or more difference images, as the estimation value of the position of the driver's seat.

[0008] Further, in the seat position estimation device, it is preferable that the difference image generation section generate the one or more difference images by generating an edge masking image by applying an edge detection filter to the masking image, and generating the one or more difference images by a difference between the edge masking image and each of one or more edge reference images obtained by applying the edge detection filter to each of the one or more reference images.

[0009] Furthermore, in the seat position estimation device, it is preferable that each of the one or more reference images also indicate a steering device of the vehicle whose position is different from each other. Also, it is preferable that the estimation section input each of the one or more difference images to a second identifier that is previously learned in a manner of estimating the position of the steering device, obtain an individual estimation value of the position of the steering device with respect to each of the one or more difference images, and calculate a statistical representative value of the individual estimation values of the position of the steering device with respect to each of the one or more difference images as the estimation value of the position of the steering device as well.

[0010] According to another aspect, a seat position estimation method is provided. The seat position estimation method includes: determining a region representing a driver in a driver image generated by an imaging section provided so as to image an interior of a vehicle during a period in which the driver is seated in the vehicle, and masking the determined region, thereby generating a masked image; generating one or more difference images by differencing the masked image and each of one or more reference images representing the interior of the vehicle when a driver's seat of the vehicle is in a predetermined position, the positions of the driver's seat in each of the one or more reference images being different from each other; and obtaining individual estimation values of the position of the driver's seat with respect to each of the one or more difference images by inputting each of the one or more difference images to an identifier that is previously learned so as to estimate the position of the driver's seat, and calculating a statistical representative value of the individual estimation values with respect to each of the one or more difference images as an estimation value of the position of the driver's seat.

[0011] According to a further aspect, a seat position estimation computer program is provided. The seat position estimation computer program includes commands for causing a processor mounted on a vehicle to execute the following steps: determining a region representing a driver in a driver image generated by an imaging section provided so as to image an interior of a vehicle during a period in which the driver is seated in the vehicle, and masking the determined region, thereby generating a masked image; generating one or more difference images by differencing the masked image and each of one or more reference images representing the interior of the vehicle when a driver's seat of the vehicle is in a predetermined position, the positions of the driver's seat in each of the one or more reference images being different from each other; and obtaining individual estimation values of the position of the driver's seat with respect to each of the one or more difference images by inputting each of the one or more difference images to an identifier that is previously learned so as to estimate the position of the driver's seat, and calculating a statistical representative value of the individual estimation values with respect to each of the one or more difference images as an estimation value of the position of the driver's seat.

[0012] The seat position estimation device according to the present disclosure has an effect of being able to estimate the position of a driver's seat from an image representing the interior of a vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a diagram of the position of the driver's seat that is a target of estimation and the position of a steering device.

[0014] Figure 2 is a diagram of the position of the driver's seat that is a target of estimation and the position of a steering device.

[0015] Figure 3 is a hardware configuration diagram of an ECU that is an example of the seat position estimation device.

[0016] Figure 4 is a functional block diagram of a processor of an ECU related to seat position estimation processing.

[0017] Figure 5 is a diagram that explains an outline of a masking image generation.

[0018] Figure 6 is a diagram that explains an outline of a difference image generation and a position estimation value calculation.

[0019] Figure 7 is a diagram that explains a relationship of a permissible range of a position of a driver seat set for each reference image, an estimation value of the position calculated by the 1st recognizer, and a finally estimated position of the driver seat.

[0020] Figure 8 is an action flowchart of seat position estimation processing executed by a processor of an ECU of a vehicle.

[0021] Figure 9 is an action flowchart of vehicle control processing executed by a processor of an ECU of a vehicle. DETAILED DESCRIPTION

[0022] Hereinafter, a seat position estimation device, and a seat position estimation method and a seat position estimation computer program executed by the seat position estimation device will be described with reference to the drawings. The seat position estimation device estimates a position of a driver seat and a position of a steering device based on an image representing an interior of a vehicle in which a driver is seated, which is generated by a camera provided in the interior of the vehicle for monitoring the driver. To this end, the seat position estimation device determines a region representing the driver on the image by a background difference between the image representing the interior of the vehicle in which the driver is seated and a background image representing the interior of the vehicle when the driver is not seated. Further, the seat position estimation device generates a masking image in which the determined region is masked. Next, the seat position estimation device generates a difference image of each of the masking image and at least one reference image representing the interior of the vehicle in a predetermined position of the driver seat and a predetermined position of the steering device. Further, the seat position estimation device obtains estimation values of the position of the driver seat and the position of the steering device by inputting the generated difference images to a recognizer for estimating the position of the driver seat and the position of the steering device.

[0023] Hereinafter, an example of a vehicle control system in which the seat position estimation device is installed to monitor a state of a driver and perform a driving support corresponding to the state of the driver will be described.

[0024] Figure 1is an explanatory view of the positioning of the driver's seat and the positioning of the steering device that are estimation targets. In the present embodiment, the inclination of the backrest 11a of the driver's seat 11 that is adjustable in the direction indicated by the arrow 101 and the position in the depth direction of the seat surface 11b of the driver's seat 11 that is adjustable in the direction indicated by the arrow 102 become estimation targets of the positioning of the driver's seat. In addition, the position in the vertical direction of the steering device 12 that is adjustable in the direction indicated by the arrow 103 and the position in the depth direction of the steering device 12 that is adjustable in the direction indicated by the arrow 104 become estimation targets of the positioning of the steering device. Furthermore, it is not limited thereto, and the position in the height direction of the seat surface 11b of the driver's seat can also be an estimation target of the positioning of the driver's seat.

[0025] Figure 2 is a schematic configuration view of a vehicle control system including a seat positioning estimation device. In the present embodiment, the vehicle control system 1 that is mounted on the vehicle 10 and controls the vehicle 10 has the driver monitoring camera 2, the notification device 3, and the electronic control device (ECU) 4 that is one example of the seat positioning estimation device. The driver monitoring camera 2 and the notification device 3 and the ECU 4 are communicably connected to each other via an in-vehicle network in accordance with a communication standard such as a controller area network. Furthermore, the vehicle control system 1 can also have an out-of-vehicle camera (not illustrated) that captures an area of the periphery of the vehicle 10 to generate an image representing the area of the periphery. Alternatively, the vehicle control system 1 can also have a distance sensor (not illustrated) such as a LiDAR or a radar that measures the distance from the vehicle 10 to an object existing in the periphery of the vehicle 10. Furthermore, the vehicle control system 1 can also have a positioning device (not illustrated) such as a GPS receiver for positioning the position of the vehicle 10 based on a signal from a satellite. Furthermore, in addition, the vehicle control system 1 can also have a navigation device (not illustrated) for searching for a travel scheduled route up to a destination. Furthermore, in addition, the vehicle control system 1 can also have a storage device (not illustrated) that stores map information referred to in automatic driving control of the vehicle 10.

[0026] The driver monitoring camera 2 is an example of an imaging section, and has a two-dimensional detector composed of an array of photoelectric conversion elements having sensitivity to visible light or infrared light such as a CCD or a C-MOS, and an imaging optical system that images an image of a region that becomes a photographic subject on the two-dimensional detector. The driver monitoring camera 2 can also have a light source such as an infrared LED for illuminating the driver. Further, the driver monitoring camera 2 is disposed in the vehicle cabin in a manner such that the face of the driver seated on the driver's seat of the vehicle 10 is included in the photographic subject region, i.e., in a manner such that the face of the driver can be photographed. For example, the driver monitoring camera 2 is installed on the instrument panel or in the vicinity thereof, or on the steering device toward the driver. Further, the driver monitoring camera 2 photographs the photographic subject region at a predetermined photographic cycle (for example, 1 / 30 to 1 / 10 seconds), and generates an image (hereinafter referred to as a driver image) of the photographic subject region. The driver image obtained by the driver monitoring camera 2 is an example of an indoor image representing the inside of the vehicle cabin of the vehicle 10, and can be either a color image or a gray scale image. The driver monitoring camera 2 outputs the generated driver image to the ECU 4 via the in-vehicle network each time the driver image is generated.

[0027] The notification device 3 is a device disposed in the vehicle cabin of the vehicle 10 that performs a predetermined notification to the driver by light, sound, vibration, text display, or image display. For this purpose, the notification device 3 has, for example, at least any of a speaker, a light source, a vibrator, or a display device. Further, the notification device 3, upon receiving a notification indicating a warning to the driver from the ECU 4, notifies the driver of the warning by outputting sound from the speaker, emitting or blinking light from the light source, vibrating the vibrator, or displaying a warning message to the display device.

[0028] The ECU 4 performs driving control of the vehicle 10 or assists the driver in driving the vehicle 10 in accordance with the driving control level applied to the vehicle 10. Further, the ECU 4 monitors the driver on the basis of the driver image accepted from the driver monitoring camera 2, and detects an abnormality of the driver. Further, the ECU 4, in the case where it is determined that the driver has an abnormality, warns the driver or controls the vehicle 10 in a manner such that the vehicle 10 is brought to an emergency stop. Further, the ECU 4, at a predetermined timing, estimates the position of the driver's seat and the position of the steering device on the basis of the driver image. Further, the ECU 4 improves the detection accuracy of the face of the driver and the estimation accuracy of the state of the driver by reflecting the estimation result thereof to the detection conditions for detecting the face of the driver from the driver image.

[0029] Figure 3 is a hardware configuration diagram of the ECU 4. As shown in Figure 3As shown, the ECU 4 is provided with a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 can be configured as individual circuits, respectively, or can be integrally configured as one integrated circuit.

[0030] The communication interface 21 has an interface circuit for connecting the ECU 4 to an in-vehicle network. Also, the communication interface 21, whenever receiving a driver image from the driver monitoring camera 2, sends the received driver image to the processor 23. Further, in addition, the communication interface 21, upon receiving information indicating a notification of a warning to the driver, which is notified to the driver via the notification device 3, from the processor 23, outputs the information to the notification device 3.

[0031] The memory 22 is an example of a storage unit, such as a volatile semiconductor memory and a non-volatile semiconductor memory. Also, the memory 22 stores various algorithms and various data used in the seat positioning estimation processing performed by the processor 23 of the ECU 4. For example, the memory 22 stores various data and parameters utilized in estimation of positioning of the driver seat and the steering device, or detection of the driver, and the like. Among such data, for example, there are included a background image representing the interior of the vehicle 10 when the driver is not seated in the vehicle 10, and one or more reference images representing the interior of the vehicle 10 when the driver seat and the steering device are in predetermined positions. Further, the memory 22 stores a detection condition table representing a relationship of a detected condition and an estimation value of positioning of the driver's face. Further, in addition, the memory 22 stores parameters for specifying a determination condition (hereinafter, sometimes simply referred to as an abnormality determination condition) for determining whether or not the driver has generated an abnormality. Further, in addition, the memory 22 temporarily stores the driver image and various data generated in the course of the seat positioning estimation processing, and an estimation value of positioning obtained as a result of the positioning estimation processing. Further, in addition, the memory 22 stores various parameters and various data for performing driving control of the vehicle 10. Among such data, there are included an image generated by an outside camera, a ranging signal generated by a distance sensor, a position measurement signal representing a position of the vehicle 10 generated by a GPS receiver, a travel scheduled route generated by a navigation device, and map information.

[0032] The processor 23 has one or a plurality of CPUs (Central Processing Units) and peripheral circuits thereof. The processor 23 can also have other arithmetic circuits such as a logic operation unit, a numerical operation unit, or a graphic processing unit. Also, the processor 23 performs a driving control processing of the vehicle including the seat positioning estimation processing at a predetermined cycle.

[0033] Figure 4is a functional block diagram of the processor 23 related to the driving control processing of the vehicle including the seat position estimation processing. The processor 23 has a masking image generation section 31, a difference image generation section 32, an estimation section 33, a detection section 34, a posture detection section 35, an abnormality determination section 36, a warning processing section 37, and a vehicle control section 38. These sections of the processor 23 are, for example, functional modules realized by a computer program operating on the processor 23. Alternatively, these sections of the processor 23 can also be dedicated arithmetic circuits provided to the processor 23. Further, the processing performed by the masking image generation section 31, the difference image generation section 32, and the estimation section 33 among these sections of the processor 23 corresponds to the seat position estimation processing. Moreover, the masking image generation section 31, the difference image generation section 32, and the estimation section 33 perform the seat position estimation processing at predetermined timings with respect to the driver image accepted by the ECU 4 from the driver monitoring camera 2. Furthermore, the detection section 34 detects the face of the driver from the latest driver image at predetermined periods using the result of the seat position estimation processing. Moreover, the posture detection section 35, the abnormality determination section 36, the warning processing section 37, and the vehicle control section 38 determine whether or not the driver has generated an abnormality based on the detected face of the driver, and control the vehicle 10 or perform a warning to the driver based on the determination result.

[0034] The masking image generation section 31 detects a region (hereinafter sometimes referred to as a driver region) indicating the driver on the driver image by the background difference between the driver image received by the ECU 4 from the driver monitoring camera 2 at predetermined timings and the background image. Moreover, the masking image generation section 31 generates a masking image by masking the driver region on the driver image.

[0035] The predetermined timings are preferably timings at which the driver completes the setting of the position of the driver seat and the position of the steering device. The reason is that it is assumed that the positions of the driver seat and the steering device are not frequently changed after the driver completes the setting of the positions of the driver seat and the steering device. Therefore, the predetermined timings can be, for example, timings at which a certain time elapses from the ignition switch of the vehicle 10 becoming on. Alternatively, the predetermined timings can be timings at which the speed of the vehicle 10 initially exceeds a predetermined speed threshold (for example, 10 km / h) after the ignition switch of the vehicle 10 becomes on. Alternatively, the predetermined timings can be timings at which a predetermined period elapses from the last time the seat position estimation processing is performed. Further, in a case where the face of the driver cannot be detected from the driver image, there is a possibility that the detection conditions are inappropriate. Therefore, the predetermined timings can be timings at which the proportion of the driver images in which the face of the driver is not detected among a series of driver images in the latest predetermined period becomes a predetermined proportion or more.

[0036] In the masking image generation section 31, as the background difference, the absolute value of the luminance difference is calculated between the corresponding pixels of the driver image and the background image. Also, the masking image generation section 31 determines the set of pixels whose absolute value of the luminance difference is equal to or greater than a predetermined luminance difference threshold value as the driver region that represents the driver. Further, in the case where the driver image and the background image are represented by a color system that does not include a direct luminance component such as the RGB color system, the masking image generation section 31 converts the color system of the driver image and the background image to the HLS color system. Also, the masking image generation section 31 calculates the absolute value of the luminance difference for each corresponding pixel using the luminance component of each pixel of the driver image and the background image. In addition, a plurality of background images corresponding to the brightness of the vehicle interior can be stored in advance in the memory 22. In this case, the masking image generation section 31 reads out the background image associated with the brightness closest to the measured value of the light amount measured by the light amount sensor (not shown) mounted on the vehicle 10 from the memory 22, and uses it for the background difference.

[0037] The masking image generation section 31 overwrites the value of each pixel included in the driver region in the driver image with a predetermined value, thereby generating a masking image in which the driver region is masked.

[0038] Figure 5 is a diagram that explains the outline of the generation of the masking image. The driver 501 is represented on the driver image 500. Therefore, by the background difference between the driver image 500 and the background image 510 in which the driver is not represented, the driver region 502 in which the driver 501 is represented on the driver image 500 is determined. Therefore, by masking the driver region 502 by overwriting the value of each pixel within the driver region 502 of the driver image 500 with a predetermined value, the masking image 520 is generated.

[0039] Further, the masking image generation section 31 can also detect the driver region using a plurality of driver images obtained in time series before and after a predetermined timing. For example, the masking image generation section 31 detects the individual driver region of each driver image by performing the background difference between each of the plurality of driver images and the background image. Also, the masking image generation section 31 reconsiders the region that is the union or the intersection of the individual driver regions of each driver image as the driver region. The masking image generation section 31 generates the masking image by masking the driver region found in any of the series of driver images. Thereby, the masking image generation section 31 can more reliably mask the region in which the driver is highly likely to exist in the driver image.

[0040] The masking image generation section 31 sends the generated masking image to the difference image generation section 32.

[0041] The difference image generation section 32 generates a difference image of each of the reference images of the masking image and one or more reference images. Among the reference images, the interior of the vehicle 10 when the driver's seat and the steering device are set to predetermined positions is represented. Further, in a case where a plurality of reference images are prepared, for each reference image, a combination of positions of the driver's seat and positions of the steering device that are different from each other is represented. In particular, it is preferable that, among the plurality of reference images, two or more reference images set in such a manner that only an arbitrary one of the movable portions of the driver's seat and the steering device is made different in position be included. For example, it is preferable that two or more reference images in which the position of the seat surface of the driver's seat and the positions in the depth direction and the up-down direction of the steering device are fixed, and only the inclination of the backrest of the driver's seat is different be prepared. Alternatively, it is preferable that two or more reference images in which the position of the seat surface of the driver's seat and the inclination of the backrest and the position in the up-down direction of the steering device are fixed, and only the position in the depth direction of the steering device is different be prepared. Thereby, in each of the difference images, with respect to a particular movable portion, the difference between the position shown in the masking image and the position shown in the corresponding reference image becomes clear.

[0042] The difference image generation section 32 calculates, with respect to each of the one or more reference images, the absolute value of the difference in pixel value between the corresponding pixels of the reference image and the masking image. Further, the difference image generation section 32 generates a difference image in such a manner that, for each pixel, the absolute value of the difference in pixel value between the corresponding pixels becomes the value of the pixel. Further, in a case where the reference images and the masking image are color images, the difference image generation section 32 can generate a difference image by performing the above processing for each color component. Thereby, in the difference image, the difference between the positions of the driver's seat and the steering device shown in the reference image and the positions of the driver's seat and the steering device shown in the masking image is emphasized.

[0043] Alternatively, the difference image generation section 32 can apply a predetermined edge detection filter to each of the reference images and the masking image to generate an image representing the intensity of an edge. Further, the difference image generation section 32 can calculate, with respect to each reference image, the absolute value of the difference in pixel value between the corresponding pixels of an edge reference image generated based on the reference image and an edge masking image generated based on the masking image. In this case, the difference image generation section 32 generates a difference image in such a manner that, for each pixel, the absolute value of the difference in pixel value between the corresponding pixels becomes the value of the pixel. Further, in the difference image generation section 32, as the predetermined edge detection filter, for example, a Sobel filter, a Prewitt filter, or a Laplacian filter can be used.

[0044] The difference image generation section 32 supplies the difference image generated for each reference image to the estimation section 33.

[0045] The estimation section 33 inputs each of the difference images generated for each reference image to a first recognizer that estimates the position of the driver's seat and a second recognizer that estimates the position of the steering device. Thus, the estimation section 33 obtains individual estimation values of the position of the driver's seat and individual estimation values of the position of the steering device for each of the difference images. Further, in the estimation section 33, as the first recognizer and the second recognizer, for example, a deep neural network (DNN) having an architecture of a convolutional neural network (CNN) type having a plurality of convolutional layers can be used. Alternatively, in the estimation section 33, as the first recognizer and the second recognizer, a DNN having an architecture of a self attention network (SAN) type can be used. These recognizers are pre-learned in accordance with a predetermined learning method such as a back propagation through time method using a large number of teacher images representing the driver's seat or the steering device in various positions.

[0046] Further, it is assumed that there is a range representing the possibility of the steering device and a range representing the possibility of the driver's seat on the driver image. As described above, in the case where the driver monitoring camera 2 is disposed at or near the instrument panel, the steering device is located closer to the driver monitoring camera 2 than the driver's seat. Therefore, the range representing the steering device on the driver image is larger than the range representing the driver's seat. Therefore, the estimation section 33 inputs the entire difference image to the second recognizer that estimates the position of the steering device. On the other hand, the estimation section 33 can input only the range in the difference image where the possibility of the driver's seat is represented to the first recognizer that estimates the position of the driver's seat.

[0047] According to a modification, the estimation section 33 can input the entire difference image to both the first recognizer and the second recognizer. Alternatively, one recognizer can be pre-learned so as to estimate both the position of the driver's seat and the position of the steering device. In this case, the estimation section 33 can obtain individual estimation values of both the position of the driver's seat and the position of the steering device by inputting the difference image to the one recognizer.

[0048] The estimation section 33 calculates a statistical representative value, such as an average value or a median value, of the individual estimation values of the position of the driver's seat estimated from the respective difference images as the estimation value of the position of the driver's seat. Likewise, the estimation section 33 calculates a statistical representative value of the individual estimation values of the position of the steering device estimated from the respective difference images as the estimation value of the position of the steering device. Further, in the case where only one reference image is prepared, the individual estimation value of the position of the driver's seat calculated by the first recognizer with respect to the difference image obtained by the difference between the one reference image and the shielded image itself becomes the statistical representative value. That is, the individual estimation value itself becomes the estimation value of the position of the driver's seat. Likewise, the individual estimation value of the position of the steering device calculated by the second recognizer with respect to the difference image obtained by the difference between the one reference image and the shielded image itself becomes the estimation value of the position of the steering device.

[0049] Further, the allowable range of the position of the driver's seat can be set in advance for each reference image. The estimation section 33 selects the individual estimation value of the position of the driver's seat estimated from each difference image, which is included in the allowable range set with respect to the reference image used in the generation of the difference image. Also, the estimation section 33 can calculate a statistical representative value of the selected individual estimation values as the estimation value of the position of the driver's seat. Likewise, the allowable range of the position of the steering device can be set in advance for each reference image. The estimation section 33 selects the individual estimation value of the position of the steering device estimated from each difference image, which is included in the allowable range set with respect to the reference image used in the generation of the difference image. Also, the estimation section 33 can set a statistical representative value of the selected individual estimation values as the estimation value of the position of the steering device.

[0050] Figure 6is a diagram that explains the outline of the difference image generation and the positional estimation value calculation. In this example, n reference images 600-1 ~ 600-n (n is an integer of 2 or more) whose combinations of the position of the driver's seat and the position of the steering device are different from each other are prepared in advance. Also, for each of the reference images 600-1 ~ 600-n, a difference image from the masking image 601 is generated. That is, a difference image 602-i is generated from the reference image 600-i (i = 1, 2,..., n) and the masking image 601. The generated difference images 602-1 ~ 602-n are input to the 1st recognizer 610 and the 2nd recognizer 620 one by one. Further, as described above, it is also possible to input only the region 603 that indicates the possibility of the driver's seat existing on the difference image 602-i to the 1st recognizer 610, and to input the entire difference image 602-i to the 2nd recognizer 620. As a result, for each of the difference images 602-1 ~ 602-n, an individual estimation value of the position of the driver's seat and an individual estimation value of the position of the steering device are calculated.

[0051] Figure 7 is a diagram that explains the relationship of the allowable range of the position of the driver's seat set for each reference image, the individual estimation value of the position calculated by the 1st recognizer, and the finally estimated position of the driver's seat. In this example, the driver's seat whose inclination angle of the backrest is different from each other on the reference images 0 ~ 11 is set. Also, as for the estimation value of the inclination angle of the backrest of the driver's seat output by the 1st recognizer, it is output in the range of 10 degrees ~ 40 degrees in 5-degree units. Further, for the reference images 0 ~ 2, the allowable range of the inclination angle is set to the range of 10 degrees ~ 25 degrees. Also, for the reference images 3 ~ 5, the allowable range of the inclination angle is set to the range of 20 degrees ~ 35 degrees. Furthermore, for the reference images 6 ~ 8, the allowable range of the inclination angle is set to the range of 25 degrees ~ 40 degrees. Also, for the reference images 9 ~ 11, the allowable range of the inclination angle is set to the range of 15 degrees ~ 30 degrees.

[0052] In Figure 7 , each star 701 indicates the individual estimation value of the inclination angle of the driver's seat calculated by the 1st recognizer from the difference image calculated with respect to each of the reference images 0 ~ 11. In this example, with respect to the reference image 5 and the reference image 8, the individual estimation value of the inclination angle of the driver's seat deviates from the allowable range. Therefore, in the final estimation of the inclination angle of the backrest of the driver's seat, the individual estimation value calculated from the difference image corresponding to the reference image 5 and the reference image 8 is removed. Also, as a statistical representative value of the individual estimation values calculated from the difference images corresponding to the reference images 0 ~ 4, 6 ~ 7, 9 ~ 11, the estimation value of the inclination angle of the backrest of the driver's seat is calculated.

[0053] The estimation unit 33 stores the estimated value of the position of the driver's seat and the estimated value of the position of the steering device into the memory 22.

[0054] Figure 8 is an action flowchart of the seat position estimation processing executed by the processor 23. The processor 23 executes the seat position estimation processing in accordance with the following action flowchart at the above-mentioned predetermined timing.

[0055] The masking image generation unit 31 of the processor 23 detects the driver region from the driver image by background subtraction of the driver image and the background image (step S101). Further, the masking image generation unit 31 masks the driver region on the driver image, that is, rewrites the value of each pixel within the driver region to a predetermined value to generate a masking image (step S102).

[0056] The difference image generation unit 32 of the processor 23 generates a difference image of the masking image and each of the one or more reference images (step S103).

[0057] The estimation unit 33 of the processor 23 inputs each of the difference images generated with respect to each of the reference images to the first recognizer and the second recognizer. Thereby, the estimation unit 33 calculates an individual estimated value of the position of the driver's seat and an individual estimated value of the position of the steering device with respect to each of the difference images (step S104). Further, the estimation unit 33 calculates a statistical representative value of the individual estimated values of the position of the driver's seat estimated from the difference images as the estimated value of the position of the driver's seat (step S105). Similarly, the estimation unit 33 calculates a statistical representative value of the individual estimated values of the position of the steering device estimated from the difference images as the estimated value of the position of the steering device (step S106). After step S106, the processor 23 ends the seat position estimation processing.

[0058] The detection unit 34 detects a region indicating the face of the driver (hereinafter referred to as a face region) from the driver image received by the ECU 4 from the driver monitoring camera 2.

[0059] The detection unit 34 detects a candidate of the face region from the driver image, for example, by inputting the driver image to a recognizer that is previously learned in a manner of detecting the face of the driver from an image. In the detection unit 34, as such a recognizer, for example, a DNN having an architecture of a CNN type or a SAN type can be utilized. Alternatively, in the detection unit 34, as such a recognizer, a recognizer based on a machine learning method other than a DNN, such as a support vector machine or an AdaBoost recognizer, can be utilized. Such a recognizer uses a large number of teacher images indicating the face of the driver and is previously learned in accordance with a predetermined learning method such as a backpropagation method.

[0060] The detection section 34 determines whether the candidate of the face region output by the recognizer satisfies the detection condition, and in the case where the candidate of the face region satisfies the detection condition, sets the candidate as the face region. The detection condition is set to a range of the size of the face region on the driver image and a presumed region in which the face of the driver is supposed to be present on the driver image. That is, the detection section 34 determines that the candidate of the face region satisfies the detection condition in the case where the candidate of the face region output by the recognizer is included in the presumed region indicated in the detection condition and the size of the face region is within the range of the size indicated in the detection condition. The detection condition is set in accordance with the combination of the position of the driver seat and the position of the steering device. In the present embodiment, a detection condition table indicating the detection condition for each combination of the position of the driver seat and the position of the steering device is stored in advance in the memory 22. Specifically, the more the seat surface of the driver seat moves forward or the smaller the inclination angle of the backrest, the closer the position of the driver to the driver monitoring camera 2. Also, the closer the position of the driver to the driver monitoring camera 2, the larger the region indicating the face of the driver on the driver image. Further, the closer the position of the driver to the driver monitoring camera 2, the larger the amount of movement of the driver on the driver image accompanying the activity of the driver. Therefore, the more the seat surface of the driver seat moves forward or the smaller the inclination angle of the backrest, the larger the presumed region indicated in the detection condition is set to be. Also, in accordance with the position of the steering device, the region indicating the steering device on the driver image varies. Therefore, the presumed region indicated in the detection condition is set not to include the region indicating the steering device on the driver image corresponding to the position of the steering device. The detection section 34 refers to the detection condition table to determine the detection condition corresponding to the combination of the presumed value of the position of the driver seat and the presumed value of the position of the steering device calculated by the presumption section 33. Also, the detection section 34 uses the determined detection condition for detecting the face region.

[0061] The detection section 34 notifies the posture detection section 35 of information indicating the position and the range of the detected face region (for example, the coordinates of the upper left end and the coordinates of the lower right end of the face region).

[0062] The posture detection section 35 detects the posture of the driver from the face region detected from the driver image. In the present embodiment, in the posture detection section 35, the position of the face of the driver and the orientation of the face of the driver are detected as information indicating the posture of the driver.

[0063] The posture detection section 35 detects a plurality of feature points of the driver's face, such as the eye corners, the outer corners of the eyes, the tip of the nose, and the corners of the mouth, from the face region of the driver image. At this time, the posture detection section 35 detects the feature points of the face by inputting the face region to a recognizer that has been previously learned in a manner to detect the feature points of the face represented on an image. As such a recognizer, the posture detection section 35 can use, for example, a DNN having a CNN-type architecture, a support vector machine, or an AdaBoost recognizer. Furthermore, the recognizer for face region detection and the recognizer for feature point detection of the face can be integrally configured. In this case, the detection section 34 can input the driver image to the recognizer, whereby the face region and each of the feature points of the face can be detected, respectively. Alternatively, the posture detection section 35 can detect each of the feature points of the driver's face from the face region in accordance with template matching of a template representing the feature points of the face and a template of the face region, or other methods of detecting the feature points of the face.

[0064] The posture detection section 35 fits each of the detected feature points of the face to a three-dimensional face model representing the three-dimensional shape of the face. Furthermore, the posture detection section 35 detects the orientation of the face of the three-dimensional face model at the time of the best fit of each of the feature points to the three-dimensional face model as the orientation of the driver's face. Furthermore, the posture detection section 35 can detect the orientation of the driver's face from the driver image in accordance with other methods of determining the orientation of the face represented in the image.

[0065] Further, the posture detection section 35 detects the position of the center of gravity of the face region in the driver image as the position of the driver's face.

[0066] Furthermore, the posture detection section 35 can detect the driver's posture from the driver's line of sight. In this case, the posture detection section 35 detects an eye region representing the driver's eyes from the face region. At this time, the posture detection section 35 can detect the eye region with respect to each of the left and right eyes by the same method as the method of detecting the feature points of the face described above. Further, the posture detection section 35 detects the pupil center and the corneal reflection image (Purkinje image) of the light source of the driver monitoring camera 2 by template matching with respect to the template of either of the left and right eye regions. Furthermore, the posture detection section 35 can detect the driver's line of sight from the positional relationship between the pupil center and the Purkinje image. Furthermore, the posture detection section 35 determines that the detection of the line of sight has failed in a case where the eye region of either of the left and right eyes of the driver cannot be detected.

[0067] The posture detection section 35 notifies the abnormality determination section 36 of the detection result of the driver's posture with respect to the driver image, that is, the detection result of the position of the driver's face and the orientation of the face. Further, in a case where the detection of the driver's line of sight is performed, the posture detection section 35 notifies the abnormality determination section 36 of the detection result of the line of sight.

[0068] The abnormality determination section 36 determines that the driver has an abnormality when the driver's posture detected by the posture detection section 35 satisfies an abnormality determination condition. As described above, in the present embodiment, the driver's posture is indicated by the position of the driver's face and the orientation of the driver's face. Further, the abnormality determination condition is that the position of the driver's face or the orientation of the face is out of the normal range for a period of time that is equal to or longer than a threshold. Therefore, the abnormality determination section 36 determines whether the position of the face and the orientation of the face are included in the normal range that is set in advance every time the position of the face and the orientation of the face of the driver are notified from the posture detection section 35. Also, the abnormality determination section 36 determines that the driver has an abnormality when the position of the driver's face or the orientation of the face is out of the normal range for a period of time that is equal to or longer than a threshold. Further, in the case where the driver's line-of-sight direction is detected, the abnormality determination section 36 determines whether the line-of-sight direction is included in the normal range that is set in advance every time the line-of-sight direction of the driver is notified from the posture detection section 35. Also, the abnormality determination section 36 can determine that the driver has an abnormality when the line-of-sight direction of the driver is out of the normal range for a period of time that is equal to or longer than a threshold.

[0069] The abnormality determination section 36 notifies the warning processing section 37 and the vehicle control section 38 of the determination result of whether the driver has an abnormality.

[0070] The warning processing section 37 implements a predetermined warning process when the determination result that the driver has an abnormality is received from the abnormality determination section 36. For example, the warning processing section 37 causes the speaker included in the notification device 3 to emit a sound signal or a warning sound that requests the driver to take a driving posture. Alternatively, the warning processing section 37 causes the display included in the notification device 3 to display a warning message that requests the driver to take a driving posture. Alternatively, the warning processing section 37 causes the vibrator included in the notification device 3 to vibrate.

[0071] The warning processing section 37 stops the execution of the warning process when the determination result that the driver does not have an abnormality is received from the abnormality determination section 36 after the warning process that requests the driver to take a driving posture is implemented via the notification device 3.

[0072] The vehicle control section 38 controls the vehicle 10 in accordance with the level of driving control applied to the vehicle 10 until a determination result that the driver has generated an abnormality is received from the abnormality determination section 36. In a case where the level of driving control applied to the vehicle 10 is a level of driving control in which the driver does not participate in driving of the vehicle 10, the vehicle control section 38 controls the vehicle 10 in such a manner that the vehicle 10 travels along the own lane in which the vehicle 10 is traveling. To this end, the vehicle control section 38 detects a lane demarcation line that demarcates the own lane and an adjacent lane and a moving object such as another vehicle traveling in the vicinity of the vehicle 10 from an image generated by the outside camera. Also, the vehicle control section 38 estimates the position and posture of the vehicle 10 by collating the detected lane demarcation line and map information. Also, the vehicle control section 38 controls the vehicle 10 in such a manner that the vehicle 10 does not collide with each of the moving objects and travels along the own lane on the basis of the estimation result of the position and posture of the vehicle 10 and the detection result of each of the moving objects in the vicinity of the vehicle 10.

[0073] In addition, when a determination result that the driver has generated an abnormality is received from the abnormality determination section 36 for a certain period of time, the vehicle control section 38 controls the vehicle 10 in such a manner that the vehicle 10 is urgently stopped. Further, the vehicle control section 38 can also control the vehicle 10 in such a manner that the vehicle 10 is immediately urgently stopped when a determination result that the driver has generated an abnormality is received from the abnormality determination section 36. At this time, the vehicle control section 38 can also move the vehicle 10 to a shoulder after stopping the vehicle 10 on the basis of the estimation result of the position and posture of the vehicle 10, the detection result of each of the moving objects in the vicinity of the vehicle 10, and the map information.

[0074] Figure 9 This is an action flowchart of the vehicle control processing executed by the processor 23. The processor 23 can execute the vehicle control processing in accordance with the following action flowchart at a predetermined cycle.

[0075] The detection section 34 of the processor 23 detects a candidate of a face region from the latest driver image (step S201). In addition, the detection section 34 sets a detection condition on the basis of a combination of the estimated value of the position of the driver seat and the estimated value of the position of the steering device (step S202). Also, the detection section 34 determines whether or not the detected candidate of the face region satisfies the detection condition (step S203).

[0076] In a case where the candidate of the face region does not satisfy the detection condition (step S203 - "No"), the detection section 34 determines that the detection of the face of the driver has failed. Also, the processor 23 ends the vehicle control processing. On the other hand, in a case where the candidate of the face region satisfies the detection condition (step S203 - "Yes"), the detection section 34 determines that the detection of the face of the driver has succeeded. Also, the posture detection section 35 of the processor 23 detects the position of the face of the driver and the orientation of the face of the driver from the face region detected from the driver image (step S204). Further, as described above, the posture detection section 35 can also detect the line-of-sight direction of the driver from the face region.

[0077] The abnormality determination section 36 of the processor 23 determines whether the driver has generated an abnormality on the basis of the position of the face of the driver and the orientation of the face of the driver (step S205). At this time, the abnormality determination section 36 can determine whether the driver has generated an abnormality on the basis of the orientation of the face of the driver and the position of the face or the line-of-sight direction in a series of driver images of the time series in the recent predetermined time. In a case where the abnormality determination section 36 determines that the driver has generated an abnormality (step S205 - "Yes"), the warning processing section 37 of the processor 23 notifies the driver of a warning requesting the driver to take a driving posture via the notification device 3 (step S206). Further, the vehicle control section 38 of the processor 23 controls the vehicle 10 in such a manner that the vehicle 10 is urgently stopped (step S207).

[0078] On the other hand, in a case where the abnormality determination section 36 determines that the driver has not generated an abnormality in step S205 (step S205 - "No"), the vehicle control section 38 controls the vehicle 10 in accordance with the driving control level applied to the vehicle 10 (step S208).

[0079] After step S207 or step S208, the processor 23 ends the vehicle control processing.

[0080] As described above, the seat positioning estimation device generates a masked image in which a region representing the driver is masked, from an image representing the inside of the vehicle cabin of the vehicle in which the driver is riding. Further, the seat positioning estimation device generates a difference image of each reference image of at least one reference image representing the inside of the vehicle cabin in a predetermined positioning of the driver's seat and a predetermined positioning of the steering device and the masked image. Also, the seat positioning estimation device inputs each generated difference image to an identifier, thereby obtaining an estimation value of the positioning of the driver's seat and the positioning of the steering device. Thus, the seat positioning estimation device can estimate the positioning of the driver's seat from the image representing the inside of the vehicle cabin of the vehicle.

[0081] According to a modification, the occlusion image generation section 31 can determine the driver region by a method other than background subtraction. For example, the occlusion image generation section 31 detects a candidate region representing a driver by performing the same processing as that of the detection section 34 on each of a series of driver images obtained in time series. Further, the occlusion image generation section 31 obtains a region that is a set of the candidate regions detected from each of the series of driver images. The region that is the set is judged to represent the driver in each driver image, so the likelihood that it actually represents the driver is high. Therefore, the occlusion image generation section 31 can determine the region that is the set as the driver region.

[0082] In addition, depending on the vehicle, the driver monitoring camera 2 is sometimes installed at a position such that the positioning of the steering device does not affect the accuracy of detecting the face region from the driver image. In such a case, the seat positioning estimation device can not estimate the positioning of the steering device, but only estimate the positioning of the driver seat. In this case, each reference image can represent the driver seat at a different positioning from each other, regardless of the positioning of the steering device. In addition, in the estimation section 33, the processing related to the calculation of the individual estimation value of the positioning of the steering device by the second recognizer and the calculation of the estimation value of the positioning of the steering device based on each individual estimation value is omitted. According to this modification, the processing related to the estimation of the positioning of the steering device is omitted, so the computational load of the processor 23 is reduced.

[0083] In addition, the computer program that realizes the functions of the processor 23 of the ECU 4 according to the above-described embodiment or modification can be provided in the form of a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

[0084] As described above, those skilled in the art can make various modifications within the scope of the present application in conformity with the above-described embodiments.

Claims

1. A seat positioning estimation device, comprising: The storage unit stores one or more reference images of the vehicle interior when the driver's seat is in a predetermined position, wherein the position of the driver's seat in each of the one or more reference images is different from each other. The masking image generation unit determines an area representing the driver in a driver image generated by a camera unit installed in a manner that captures images of the vehicle interior during the period when the driver is riding in the vehicle, and masks the determined area to generate a masking image. The differential image generation unit generates one or more differential images by differentiating the masked image and each of the reference images of the one or more reference images; as well as The estimation unit inputs each of the more than one differential images to a recognizer that has been pre-learned in order to estimate the location of the driver's seat. It calculates an individual estimated value of the driver's seat location for each of the more than one differential images, and calculates a statistical representative value of the individual estimated value of each of the more than one differential images as the estimated value of the driver's seat location.

2. The seat positioning estimation device according to claim 1, wherein, For each of the more than one reference images, an allowable range for the positioning of the driver's seat is set. The estimation unit calculates a statistical representative value of the individual estimated values ​​of the driver's seat position obtained from the individual estimated values ​​of the driver's seat position obtained from each of the more than one differential images, which is included in the allowable range of the reference image corresponding to the differential image, as the estimated value of the driver's seat position.

3. The seat positioning estimation device according to claim 1, wherein, The differential image generation unit generates an edge-masked image by applying an edge detection filter to the masked image, and generates the one or more differential images by differentiating the edge-masked image with the edge reference images of one or more edge reference images obtained by applying the edge detection filter to each of the one or more reference images.

4. The seat positioning estimation device according to any one of claims 1 to 3, wherein, Each of the more than one reference images also represents a steering device of the vehicle with a different positioning. The estimation unit inputs each of the more than one differential images to a second recognizer that has been pre-learned in order to estimate the positioning of the steering device. It calculates an individual estimated value for the positioning of the steering device for each of the more than one differential images, and calculates a statistical representative value of the individual estimated value for the positioning of the steering device for each of the more than one differential images as an estimated value for the positioning of the steering device.

5. A method for estimating seat positioning, comprising: In a driver image generated during the driver's journey in the vehicle by a camera unit set up in a manner that captures images of the vehicle's interior, an area representing the driver is determined, and the determined area is masked to generate a masked image. One or more difference images are generated by differentiating the masked image and each of the reference images of one or more reference images of the vehicle interior when the driver's seat of the vehicle is in a predetermined position, wherein the position of the driver's seat in each of the one or more reference images is different from each other. as well as By inputting each of the more than one differential images into a pre-learned recognizer that infers the location of the driver's seat, an individual inferred value of the driver's seat location is obtained for each of the more than one differential images, and a statistical representative value of the individual inferred value of the more than one differential images is calculated as the inferred value of the driver's seat location.

6. A recording medium having a computer program for seat positioning estimation recorded thereon, the computer program for seat positioning estimation being used to cause a processor mounted in a vehicle to execute: In a driver image generated during the period when the driver is riding in the vehicle by a camera unit set up in a manner that captures images of the vehicle's interior, an area representing the driver is determined, and the determined area is masked to generate a masked image. One or more difference images are generated by differentiating the masked image and each of the reference images of one or more reference images of the vehicle interior when the driver's seat of the vehicle is in a predetermined position, wherein the position of the driver's seat in each of the one or more reference images is different from each other. as well as By inputting each of the more than one differential images into a pre-learned recognizer that infers the location of the driver's seat, an individual inferred value of the driver's seat location is obtained for each of the more than one differential images, and a statistical representative value of the individual inferred value of the more than one differential images is calculated as the inferred value of the driver's seat location.

Images