Image processing method and image processing system

Abstract

In the present invention, when it is determined from an image of the face of the driver of a vehicle (10) that the driver is looking to the side, a first portion (32, 33) and a second portion (34, 35) of the image (30) are stored, the first portion (32, 33) being set in advance in accordance with the orientation of the face of the driver and being estimated to display the eyes of the driver, and the second portion (34, 35) being set in advance in accordance with the orientation of the face of the driver and being estimated to display an ear of the driver.

Description

Image processing method and image processing system

[0001] The present invention relates to an image processing method and an image processing system.

[0002] An image acquisition device is known that detects the position of a pedestrian's head from an image of the area around a vehicle, sets a pedestrian area of ​​a predetermined shape in which the pedestrian's head is located at a specific position, removes the pedestrian's head from the pedestrian area, and saves an image of the pedestrian area from which the head has been removed as learning data (Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2017-224077

[0004] For example, if the above-mentioned conventional technology is applied to a case where a pedestrian is replaced with a driver and the driver's state is estimated from the direction of the driver's face, the image used for estimation will be saved with the driver's head removed, which creates a problem in that the validity of the estimation cannot be verified.

[0005] The problem to be solved by the present invention is to provide an image processing method and an image processing system that can appropriately store images used to determine the driver's state.

[0006] The present invention solves the above problem by saving, when it is determined from an image of the face of a vehicle that the driver has been looking away from the road, a portion of the image that is presumed to show the driver's eyes, which is set in advance according to the direction of the driver's face, and a portion that is presumed to show the driver's ears, which is set in advance according to the direction of the driver's face.

[0007] According to the present invention, the image used to determine the driver's state can be appropriately saved.

[0008] FIG. 1 is a block diagram showing an example of an embodiment of an image processing system according to the present invention. FIG. 2 is a diagram showing an example of determination of inattentive driving by the first determination unit of FIG. 1. FIG. 3 is a diagram showing another example of determination of inattentive driving by the first determination unit of FIG. 1. FIG. 4 is a diagram showing yet another example of determination of inattentive driving by the first determination unit of FIG. 1. FIG. 5 is a diagram showing an example of an image saved by the image processing system of FIG. 1. FIG. 6 is a diagram showing another example of an image saved by the image processing system of FIG. 1. FIG. 7 is a diagram showing yet another example of an image saved by the image processing system of FIG. 1. FIG. 8 is a diagram showing yet another example of an image saved by the image processing system of FIG. 1. FIG. 9 is a flowchart showing an example of a processing procedure in the image processing system of FIG. 1.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] [Configuration of Image Processing System] Figure 1 is a block diagram showing an example of an embodiment of an image processing system according to the present invention. The image processing system of this embodiment is a group of devices for saving images of a driver's face when monitoring the driver's condition from images of the driver's face, and in particular, saves a predetermined portion of the image of the driver's face when it is determined that the driver is looking away from the road. The images saved by the image processing system are used, for example, to determine whether the monitoring of the driver's condition has been performed appropriately, and when the monitoring of the driver's condition is performed automatically using a trained model, they can be used to train the model.

[0011] The vehicle is not particularly limited and travels under autonomous driving control or manual driving by a driver. Autonomous driving control refers to autonomously controlling the vehicle's driving operations using a vehicle control device, and driving operations include all driving operations such as acceleration, deceleration, starting, stopping, and steering. Furthermore, autonomously controlling driving operations refers to the vehicle's control device controlling the driving operations using the vehicle's devices. The control device controls these driving operations within a predetermined range, and driving operations that are not controlled by the control device are manually operated by the driver. On the other hand, when the driver manually drives, the control device does not autonomously control the driving operations, and the vehicle's driving operations are controlled by the driver's operation.

[0012] The term "driver" includes a driver who sits in the driver's seat and drives the vehicle, and a driver who operates the vehicle from a location away from the vehicle when the vehicle's driving operation is remotely controlled (for example, a driver who remotely operates the vehicle from a remote location such as a management center that centrally manages the vehicle's driving). The driver's state is not particularly limited, and includes a state of being distracted while driving, a state of being in poor health such as stomach pain, a state of talking with a passenger, etc. Inattentive driving refers to a driver driving with their eyes away from the front of the vehicle, and includes a driver driving while operating a mobile device, a driver sleeping, a driver driving while operating a navigation device, etc.

[0013] The image of the driver's face is not particularly limited as long as it includes at least a portion of the driver's face. However, in order to more accurately determine the driver's condition, it is preferable that the image includes at least one of the driver's right eye and left eye and at least one of the driver's right ear and left ear. Monitoring the driver's condition includes notifying the driver's condition, issuing a warning depending on the driver's condition, recording the driver's condition, etc. Note that the monitoring of the driver's condition may be performed by a device (or system) other than the image processing system of this embodiment.

[0014] As shown in Fig. 1, the image processing system 1 includes a vehicle 10 and a server 20. The vehicle 10 and the server 20 exchange information with each other via a network (not shown). The network refers to a telecommunications network such as the Internet, and the communication format is not particularly limited. The vehicle 10 also includes an imaging device 11 and a control device 12. The imaging device 11 and the control device 12 are connected by a controller area network (CAN) or other in-vehicle LAN, and can exchange information with each other.

[0015] The imaging device 11 is a device that captures an image of an object including the driver's face, and is a camera such as an infrared camera or a camera equipped with an imaging element such as a CCD. The object is an object present around the imaging device 11, such as a vehicle occupant, seat, or roof. The imaging device 11 is provided in a position where it can capture an image of the driver's face, such as the top of the windshield, a rearview mirror, or a steering wheel. Furthermore, when the driver remotely controls the vehicle 10, the imaging device 11 is provided in a position where it can capture an image of the driver's face, for example, on a device used by the driver for remote control.

[0016] The control device 12 controls the operation of the vehicle 10 by controlling and coordinating the various devices of the vehicle 10, thereby driving the vehicle 10 to a set destination. The control device 12 has a storage function that monitors the driver's condition when driving the vehicle 10 to a set destination and stores an image of the driver's face as necessary. The control device 12 is, for example, a computer, and includes a CPU (Central Processing Unit) that serves as a processor, a ROM (Read Only Memory) that stores programs, and a RAM (Random Access Memory) that functions as an accessible storage device. The CPU of the control device 12 is an operating circuit that executes the programs stored in the ROM to realize the storage function.

[0017] On the other hand, the server 20 is a device that receives and stores image data transmitted from the control device 12. Furthermore, the server 20 determines, as necessary, whether the monitoring of the driver's condition has been performed appropriately based on the images stored by the image processing system 1. The server 20 is, for example, a computer, and includes a CPU, ROM, and RAM, similar to the control device 12. The CPU of the server 20 is an operating circuit that executes a program stored in the ROM, stores images, and determines whether the monitoring is appropriate.

[0018] [Functions of the Control Device and Server] The program stored in the ROM of the control device 12 includes a first determination unit 13, an extraction unit 14, and a first storage unit 15, which are functional blocks for realizing the storage function. The server 20 also includes a second storage unit 21, which is a functional block for storing images received from the control device 12, and a second determination unit 22 for determining whether monitoring is appropriate. These functional blocks are extracted and shown in Figure 1 for convenience. Each functional block shown in Figure 1 will be described below.

[0019] The first determination unit 13 determines whether the driver of the vehicle 10 has been inattentive while driving based on an image of the driver's face. The first determination unit 13 acquires an image including at least one of the driver's right eye and left eye and at least one of the driver's right ear and left ear from the imaging device 11, and determines whether the driver has been inattentive while driving based on the positions of the eyes and ears in the image. For example, the first determination unit 13 detects the driver's eyes and ears from the image acquired from the imaging device 11, estimates the direction of the driver's face from the relative positions of the driver's eyes and ears, and performs a determination of inattentive driving based on the estimated direction of the face.

[0020] The orientation of the driver's face is estimated in each of the left-right direction (yaw direction of the vehicle) and the up-down direction (pitch direction of the vehicle). That is, the first determination unit 13 determines whether the driver's face is facing right or left, and also determines whether the driver's face is facing up or down.

[0021] For example, the first determination unit 13 compares the widths of the driver's left and right eyes (the horizontal lengths of the left and right eyes), and if the width of the right eye is longer than the width of the left eye, determines that the driver's face is facing left, and if the width of the right eye is shorter than the width of the left eye, determines that the driver's face is facing right. Alternatively or in addition, if only the driver's right ear is detected, the first determination unit 13 determines that the driver's face is facing left, and if only the driver's left ear is detected, the first determination unit 13 determines that the driver's face is facing right. Alternatively or in addition, if the distance between the eyes is shorter than the width of the right eye or the left eye, the first determination unit 13 determines that the driver's face is facing right or left.

[0022] The first determination unit 13 also compares the eye height position (the vertical position of the eyes) in the image with the ear height position (the vertical position of the ears) in the image, and if the eye height position is equal to or lower than the ear height position, determines that the driver's face is facing downward. On the other hand, if the eye height position is higher than the ear height position and the difference between the eye and ear height positions is equal to or greater than a predetermined value, the first determination unit 13 determines that the driver's face is facing upward. The predetermined value can be set appropriately within a range that can appropriately determine whether the driver's face is facing upward, and is, for example, 5 to 30% of the vertical length of the entire driver's face.

[0023] On the other hand, if the width of the right eye and the width of the left eye are approximately the same (for example, the ratio of the width of the left eye to the width of the right eye is within a range of 0.95 to 1.05), the eye height is higher than the ear height, and the difference between the eye and ear heights is less than a predetermined value, the first determination unit 13 determines that the driver's face is facing forward. Note that if the heights of the right eye and the left eye are different, the average of the right eye height and the left eye height is taken as the eye height. Similarly, if the heights of the right ear and the left ear are different, the average of the right ear height and the left ear height is taken as the ear height.

[0024] When it is determined that the driver's face is facing to the right, left, up, or down, if the state in which the driver's face is facing to the right, left, up, or down continues for a predetermined time or more, the first determination unit 13 determines that the driver is looking away from the road. On the other hand, when it is determined that the driver's face is facing to the right, left, up, or down, if the state in which the driver's face is facing to the right, left, up, or down does not continue for a predetermined time or more, the first determination unit 13 determines that the driver is not looking away from the road. The predetermined time can be set appropriately within a range in which the driver's looking away from the road can be appropriately determined, for example, 30 seconds to 1 minute.

[0025] Furthermore, if it is determined that the driver's face is not facing to the right, left, up, or down (i.e., the driver's face is facing forward), the first determination unit 13 determines that the driver is not looking away from the road. However, even in this case, the first determination unit 13 determines whether the driver's eyes are continuously closed for a predetermined period of time, and if it determines that the driver's eyes are continuously closed for more than the predetermined period of time, it determines that the driver is asleep. If the driver is asleep, the first determination unit 13 determines that the driver is looking away from the road. On the other hand, if it determines that the driver's eyes are continuously closed for less than the predetermined period of time, the first determination unit 13 determines that the driver is not asleep. In the following description, unless otherwise specified, it is assumed that the driver is not asleep.

[0026] 2A to 2C show a specific example of determining inattentive driving. When image 31 shown in FIG. 2A is acquired from imaging device 11, first determination unit 13 detects the driver's entire face (area 41), right eye (area 42), left eye (area 43), right ear (area 44), and left ear (area 45) from image 31. Image 31 includes both eyes and both ears of the driver, and the width W1 of the right eye and the width W2 of the left eye are approximately the same, and the height H1 of both eyes is higher than the height H2 of both ears. Therefore, when the difference D1 between the heights of both eyes and both ears is less than a predetermined value, first determination unit 13 determines that the driver's face displayed in image 31 is facing forward and that the driver is not inattentive driving.

[0027] When the image 32 shown in FIG. 2B is acquired from the imaging device 11, the first determination unit 13 detects the driver's entire face (area 41), right eye (area 42), left eye (area 43), and right ear (area 44) from the image 32. The image 32 includes both eyes and the right ear, but not the left ear. In addition, the width W1 of the right eye is longer than the width W2 of the left eye, and the height H1 of both eyes is higher than the height H2 of the right ear. Therefore, the first determination unit 13 determines that the driver's face displayed in the image 32 is facing leftward. If the state shown in the image 32 continues for a predetermined time or longer, the first determination unit 13 determines that the driver is looking away from the road.

[0028] When the image 33 shown in FIG. 2C is acquired from the imaging device 11, the first determination unit 13 detects the driver's entire face (area 41), right eye (area 42), left eye (area 43), right ear (area 44), and left ear (area 45) from the image 33. The image 33 includes both eyes and both ears of the driver, and the width W1 of the right eye and the width W2 of the left eye are approximately the same, and the height H1 of both eyes is lower than the height H2 of both ears. Therefore, the first determination unit 13 determines that the driver's face displayed in the image 33 is facing downward. If the state shown in the image 33 continues for a predetermined time or longer, the first determination unit 13 determines that the driver is looking away from the road.

[0029] The first determination unit 13 includes a first determination model that has been trained in advance to determine whether the driver is inattentively driving based on an image acquired from the imaging device 11, and performs a determination of inattentive driving using the first determination model. The first determination model is a trained model that has been trained in advance through machine learning so as to output an appropriate determination result regarding the driver's inattentive driving when an image of the driver's face is input. For example, the first determination model is a neural network that includes an input layer, an intermediate layer, and an output layer, each layer including at least one neuron. The input layer receives input data including the image acquired from the imaging device 11 and outputs the input data to the intermediate layer. The intermediate layer detects the driver's face, eyes, ears, etc. from the image input from the input layer and detects the positional relationship between the eyes and ears. The output layer performs a determination of inattentive driving based on the information input from the intermediate layer and outputs data including the determination result.

[0030] When determining inattentive driving using a trained model, the accuracy of the determination result depends on the degree of learning of the model. If the model is insufficiently trained, the first determination unit 13 may not be able to perform an accurate determination. Furthermore, when determining whether the driver is asleep, if the driver has narrow eyes (the vertical length of the left and right eyes is short), the first determination unit 13 may erroneously determine that the driver is asleep. To verify the validity of the determination result of the first determination unit 13, it is necessary to store images in which the first determination unit 13 determines that the driver is inattentive driving. However, if the images acquired from the imaging device 11 are stored as they are, this will consume the capacity of the storage medium of the control device 12 and will take a long time to transfer the image data to the server 20. Therefore, the control device 12 stores a predetermined portion of the image of the driver's face acquired from the imaging device 11 using the extraction unit 14 and the first storage unit 15.

[0031] When the first determination unit 13 determines that the driver is inattentive driving, the extraction unit 14 extracts a predetermined portion corresponding to the direction of the driver's face from the image acquired from the imaging device 11. On the other hand, when the first determination unit 13 determines that the driver is not inattentive driving, the extraction unit 14 does not execute the extraction process on the image acquired from the imaging device 11. Because the first determination unit 13 uses the positional relationship between the driver's eyes and ears to determine inattentive driving, the validity of the determination result can be verified as long as the portions of the driver's eyes and ears in the image acquired from the imaging device 11 are saved. Therefore, the predetermined portion corresponding to the direction of the driver's face includes a first portion that is pre-set according to the direction of the driver's face and is estimated to display the driver's eyes, and a second portion that is pre-set according to the direction of the driver's face and is estimated to display the driver's ears.

[0032] Because the positions of the first and second portions in the image depend on the installation position of the imaging device 11, the positions of the first and second portions are set in advance according to the orientation of the driver's face for each installation position of the imaging device 11. For example, for a driver's face facing right, a predetermined first portion corresponding to the right eye and left eye and a predetermined second portion corresponding to the left ear are set, and for a driver's face facing left, a predetermined first portion corresponding to the right eye and left eye and a predetermined second portion corresponding to the right ear are set. Furthermore, for a driver's face facing upward or downward, a predetermined first portion corresponding to the right eye and left eye and a predetermined second portion corresponding to the right ear are set.

[0033] As an example, for a driver's face facing left, such as in image 32 shown in Fig. 2B , the portions corresponding to regions 42 and 43 are set as the predetermined first portion, and the portion corresponding to region 44 is set as the predetermined second portion. As another example, for a driver's face facing downward, such as in image 33 shown in Fig. 2C , the portions corresponding to regions 42 and 43 are set as the predetermined first portion, and the portions corresponding to regions 44 and 45 are set as the predetermined second portion. When the first determination unit 13 determines that the driver has been looking away from the road, the extraction unit 14 extracts the predetermined first portion and the second portion from the image acquired from the imaging device 11 (or deletes portions other than the predetermined first portion and the second portion from the image acquired from the imaging device 11) to generate a new image with a reduced amount of information.

[0034] 3A and 3B are diagrams illustrating an image generated by the extraction unit 14 when the first determination unit 13 determines that the driver's face is facing left and that the driver is looking away while driving. For the face of the driver facing left, first portions 61 and 62 and a second portion 63 shown in FIGS. 3A and 3B are assumed to be set in advance.

[0035] In the example shown in FIG. 3A , an image 32 is acquired from the imaging device 11, and it is determined that the driver's face displayed in the image 32 is facing left. In this case, the extraction unit 14 extracts predetermined first portions 61 and 62 and a predetermined second portion 63 from the image 32, and generates a new image 51. The first portion 61 of the image 51 displays the right eye, the first portion 62 displays the left eye, and the second portion 63 displays the right ear. On the other hand, in the example shown in FIG. 3B , an image 31 is acquired from the imaging device 11, and it is determined that the driver's face displayed in the image 31 is facing left. In this case, the extraction unit 14 extracts the predetermined first portions 61 and 62 and the predetermined second portion 63 from the image 31, and generates a new image 52. The first portion 61 of the image 52 displays part of the left eye, but the first portion 62 does not display an eye, and the second portion 63 does not display an ear.

[0036] 4A and 4B are diagrams illustrating images generated by the extraction unit 14 when the first determination unit 13 determines that the driver's face is facing downward and that the driver is looking away while driving. For the driver's face facing downward, first portions 64, 65 and second portions 66, 67 shown in FIGS. 4A and 4B are assumed to be set in advance.

[0037] In the example shown in FIG. 4A , an image 33 is acquired from the imaging device 11, and it is determined that the driver's face displayed in the image 33 is facing downward. In this case, the extraction unit 14 extracts predetermined first portions 64 and 65 and predetermined second portions 66 and 67 from the image 33, and generates a new image 53. The first portion 64 of the image 53 displays the right eye, the first portion 65 displays the left eye, the second portion 66 displays the right ear, and the second portion 67 displays the left ear. On the other hand, in the example shown in FIG. 4B , an image 31 is acquired from the imaging device 11, and it is determined that the driver's face displayed in the image 31 is facing downward. In this case, the extraction unit 14 extracts the predetermined first portions 64 and 65 and the predetermined second portions 66 and 67 from the image 31, and generates a new image 54. The second portion 66 of the image 54 displays part of the right ear, but the first portions 64 and 65 do not display an eye, and the second portion 67 does not display an ear.

[0038] The first storage unit 15 stores the images generated by the extraction unit 14. The first storage unit 15 stores the images of the first and second parts extracted by the extraction unit 14 in a storage medium (e.g., a non-volatile memory) of the vehicle 10. Alternatively or in addition to this, the first storage unit 15 may transmit the images of the first and second parts extracted by the extraction unit 14 to the server 20. In this case, the second storage unit 21 of the server 20 receives and stores the images of the first and second parts from the first storage unit 15. On the other hand, if the first determination unit 13 determines that the driver is not inattentively driving, the first storage unit 15 does not store the images acquired from the imaging device 11 and does not transmit them to the server 20.

[0039] The images stored in at least one of the first storage unit 15 and the second storage unit 21 (hereinafter also simply referred to as stored images) are used to verify the validity of the determination result of the first determination unit 13. The validity of the determination result of the first determination unit 13 may be verified by a person (for example, a designer of the vehicle 10) or may be verified by the second determination unit 22 of the server 20.

[0040] The second determination unit 22 verifies the validity of the determination result of the first determination unit 13. Specifically, the second determination unit 22 determines whether the driver's eyes are displayed in the first portion of the stored image. The second determination unit 22 also determines whether the driver's ears are displayed in the second portion of the stored image. If the second determination unit 22 determines that the driver's eyes are displayed in the first portion and the driver's ears are displayed in the second portion, the second determination unit 22 determines that the inattentive driving determination result made by the first determination unit 13 is valid.

[0041] On the other hand, if it is determined that the driver's eyes are not displayed in the first portion, if it is determined that the driver's ears are not displayed in the second portion, or if it is determined that the driver's eyes are not displayed in the first portion and the driver's ears are not displayed in the second portion, the second determination unit 22 determines that the inattentive driving determination result by the first determination unit 13 is invalid. The second determination unit 22 may feed back the result of verifying the inattentive driving determination result by the first determination unit 13 to the first determination model of the first determination unit 13. When the first determination unit 13 receives the verification result of the determination result from the second determination unit 22, it re-learns and updates the first determination model based on the verification result.

[0042] For example, when the second determination unit 22 receives from the second storage unit 21 an image 51 shown in FIG. 3A together with a determination result that the driver's face is facing left and the driver is looking aside while driving, the second determination unit 22 determines that the determination result of inattentive driving by the first determination unit 13 is valid. This is because the right eye is displayed in the first portion 61 of the image 51, the left eye is displayed in the first portion 62, and the right ear is displayed in the second portion 63 of the image 51. On the other hand, when the second determination unit 22 receives from the second storage unit 21 an image 52 shown in FIG. 3B together with a determination result that the driver's face is facing left and the driver is looking aside while driving, the second determination unit 22 determines that the determination result of inattentive driving by the first determination unit 13 is invalid. This is because the first portion 62 of the image 52 does not display an eye, and the second portion 63 of the image 52 does not display an ear.

[0043] 4A is input from the second storage unit 21 together with the determination result that the driver's face is facing downward and the driver is looking aside while driving, the second determination unit 22 determines that the determination result of inattentive driving by the first determination unit 13 is valid. This is because the right eye is displayed in the first portion 64 of the image 53, the left eye is displayed in the first portion 65, the right ear is displayed in the second portion 66, and the left ear is displayed in the second portion 67. On the other hand, when the second determination unit 22 receives the image 54 shown in FIG. 4B from the second storage unit 21 together with the determination result that the driver's face is facing downward and the driver is looking aside while driving, the second determination unit 22 determines that the determination result of inattentive driving by the first determination unit 13 is invalid. This is because the first portions 64 and 65 of the image 54 do not display eyes, and the second portion 67 does not display ears.

[0044] The second determination unit 22 includes a second determination model trained in advance to determine whether the inattentive driving determination result made by the first determination unit 13 is valid or not, and verifies the inattentive driving determination result using the second determination model. The second determination model is a trained model (e.g., a neural network similar to the first determination model) trained in advance by machine learning so as to output an appropriate result regarding the validity of the determination result when the image and the determination result are input from the second storage unit 21. The input layer of the neural network receives input data including the image stored by the second storage unit 21 and the determination result made by the first determination unit 13, and outputs the input data to the intermediate layer. The intermediate layer detects whether eyes, ears, etc. are displayed in a predetermined portion of the image input from the input layer. The output layer verifies the validity of the inattentive driving determination result based on the information input from the intermediate layer, and outputs data including the verification result.

[0045] When the first determination unit 13 determines that the driver has engaged in inattentive driving, it may transmit the image acquired from the imaging device 11 to the second determination unit 22. In this case, the second determination unit 22, like the extraction unit 14, extracts a predetermined portion corresponding to the orientation of the driver's face from the image acquired from the first determination unit 13. The image of the predetermined portion (first portion and second portion) extracted by the second determination unit 22 is stored in the second storage unit 21 and used by the second determination unit 22 to verify the validity of the determination result of the first determination unit 13. The second determination unit 22 may also extract a predetermined portion corresponding to the orientation of the driver's face from the image received from the first determination unit 13 and verify the validity of the determination result of the first determination unit 13 from the extracted image. In this case, at least one of the image of the predetermined portion extracted by the second determination unit 22 and the verification result of the validity of the determination result of the first determination unit 13 is stored in the second storage unit 21.

[0046] If the first determination unit 13 determines that the driver has engaged in inattentive driving, it may detect the driver's right eye, left eye, right ear, and left ear from the image acquired from the imaging device 11, and the first storage unit 15 may store first coordinate information indicating the positions of the right eye, left eye, right ear, and left ear detected by the first determination unit 13 on the image. Furthermore, if the first determination unit 13 determines that the driver has engaged in inattentive driving, it may detect the driver's entire face (facial outline) from the image acquired from the imaging device 11 and store second coordinate information indicating the position of the driver's face on the image. The coordinate information may include, for example, information regarding the coordinates of a screen coordinate system relative to the camera coordinate system of the imaging device 11. This allows for more accurate verification of the inattentive driving determination result. Note that information regarding the positions of the right eye, left eye, right ear, and left ear that are not detected among the right eye, left eye, right ear, and left ear is not stored.

[0047] For example, as shown in FIG. 2B , when the driver's face, eyes, and ears are detected as rectangular regions, if it is determined that the driver has been looking away from the screen while driving, the first determination unit 13 detects information regarding the coordinates of a top left point P1 and a bottom right point P2 in the screen coordinate system of a region 41 corresponding to the face, and outputs the information to the first storage unit 15. Similarly, the first determination unit 13 detects information regarding the coordinates of a top left point P3 and a bottom right point P4 in the screen coordinate system of a region 42 corresponding to the right eye, and an top left point P5 and a bottom right point P6 in the screen coordinate system of a region 43 corresponding to the left eye, and outputs the information to the first storage unit 15. The first determination unit 13 also detects information regarding the coordinates of a top left point P7 and a bottom right point P8 in the screen coordinate system of a region 44 corresponding to the right ear, and outputs the information to the first storage unit 15. The first storage unit 15 stores the coordinate information output from the first determination unit 13 in a storage medium of the vehicle 10. Alternatively or in addition to this, the first storage unit 15 may transmit the coordinate information output from the first determination unit 13 to the server 20. The coordinate information transmitted to the server 20 is stored by the second storage unit 21.

[0048] 5, an example of the information processing procedure in the image processing system 1 will be described. The processing described below is executed at predetermined time intervals by a processor (CPU) included in the control device 12 and a processor (CPU) included in the server 20.

[0049] First, in step S1, the first determination unit 13 acquires an image of the driver from the imaging device 11. In the following step S2, the first determination unit 13 executes a determination of whether the driver is inattentive driving, and in step S3, the first determination unit 13 determines whether the driver is inattentive driving. If it is determined that the driver is not inattentive driving, the process proceeds to step S1. On the other hand, if it is determined that the driver is inattentive driving, the process proceeds to step S4. In step S4, the extraction unit 14 extracts a predetermined portion from the image acquired from the imaging device 11, and in step S5, the first storage unit 15 transmits the portion extracted by the extraction unit 14 to the server 20. In step S6, the second storage unit 21 receives image data from the control device 12, and in the following step S7, the second storage unit 21 stores the received image. Thereafter, the process proceeds to step S1, and the routine is repeatedly executed while the vehicle 10 is traveling.

[0050] According to the present embodiment, there is provided an image processing method executed by a control device 12 of a vehicle 10, in which, when the control device 12 determines from an image of the face of the driver of the vehicle 10 that the driver is looking away while driving, the control device 12 saves a first portion of the image that is pre-set according to the orientation of the face and that is estimated to display the eyes of the driver, and a second portion that is pre-set according to the orientation of the face and that is estimated to display the ears of the driver. This allows the image used to determine the driver's state to be saved appropriately.

[0051] In the image processing method of this embodiment, when the control device 12 determines that the driver has engaged in inattentive driving, the control device 12 detects the right eye, left eye, right ear, and left ear of the driver from the image and stores first coordinate information indicating the positions of the right eye, left eye, right ear, and left ear on the image, thereby enabling more accurate verification of the inattentive driving determination result.

[0052] In the image processing method of this embodiment, when the control device 12 determines that the driver has engaged in inattentive driving, the control device 12 detects the face of the driver from the image and stores second coordinate information indicating the position of the face on the image, thereby enabling more accurate verification of the inattentive driving determination result.

[0053] In the image processing method of this embodiment, the control device 12 does not store the image if it is determined that the driver is not inattentive driving, thereby making it possible to selectively store images necessary for verifying the determination result of inattentive driving.

[0054] In the image processing method of this embodiment, the control device 12 executes at least one of storing the first and second parts of the image in a storage medium of the vehicle and transmitting the first and second parts of the image to the server 20. This allows the image used to determine the driver's state to be appropriately stored.

[0055] Furthermore, according to the present embodiment, there is provided an image processing method executed by the server 20, in which, when the control device 12 of the vehicle 10 determines from an image of the face of the driver of the vehicle 10 that the driver is looking away while driving, the server 20 receives from the control device 12 and stores a first portion of the image, which is preset according to the direction of the face and is estimated to display the eyes of the driver, and a second portion of the image, which is preset according to the direction of the face and is estimated to display the ears of the driver. This makes it possible to appropriately store the image used to determine the driver's state.

[0056] In the image processing method of this embodiment, the server 20 determines whether the eyes are displayed in the first portion and whether the ears are displayed in the second portion, and determines that the inattentive driving judgment result is valid if it determines that the eyes are displayed in the first portion and the ears are displayed in the second portion, and determines that the inattentive driving judgment result is invalid if it determines that the eyes are not displayed in the first portion, if it determines that the ears are not displayed in the second portion, or if it determines that the eyes are not displayed in the first portion and the ears are not displayed in the second portion. This allows the verification result of the inattentive driving judgment to be fed back to the first judgment unit 13.

[0057] Furthermore, according to the present embodiment, in an image processing system 1 including a control device 12 of a vehicle 10 and a server 20, when the control device 12 determines from an image of the face of the driver of the vehicle 10 that the driver is looking away while driving, the control device 12 transmits to the server 20 a first portion of the image that is pre-set according to the direction of the face and that is estimated to show the driver's eyes, and a second portion that is pre-set according to the direction of the face and that is estimated to show the driver's ears, and the server 20 stores the images of the first and second portions received from the control device 12. This makes it possible to appropriately store the images used to determine the driver's state.

[0058] Furthermore, according to this embodiment, when it is determined from an image of the face of the driver of the vehicle 10 that the driver is looking away while driving, the control device 12 of the vehicle 10 stores a first portion of the image that is pre-set according to the direction of the face and that is estimated to show the eyes of the driver, and a second portion that is pre-set according to the direction of the face and that is estimated to show the ears of the driver. This allows the image used to determine the driver's state to be properly stored.

[0059] Furthermore, according to this embodiment, when the control device 12 of the vehicle 10 determines from an image of the face of the driver of the vehicle 10 that the driver is looking away while driving, the server 20 receives and stores from the control device 12 a first portion of the image that is pre-set according to the direction of the face and that is estimated to show the eyes of the driver, and a second portion that is pre-set according to the direction of the face and that is estimated to show the ears of the driver. This allows the image used to determine the driver's state to be properly stored.

[0060] REFERENCE SIGNS LIST 1...image processing system 10...vehicle, 11...imaging device, 12...control device, 13...first judgment unit, 14...extraction unit, 15...first storage unit 20...server, 21...second storage unit, 22...second judgment unit 31, 32, 33...image 41, 42, 43, 44, 45...area 51, 52, 53, 54...image 61, 62, 64, 65...first part, 63, 66, 67...second part D1, D2...difference, H1, H2...height position, P1, P2, P3, P4, P5, P6...point, W1, W2...width

Claims

1. In an image processing method performed by a vehicle control device, Image processing method for the control device, which, when it determines from an image of the driver's face that the driver of the vehicle has been driving while distracted, saves a first portion of the image that is presumed to show the driver's eyes, which is set in advance according to the direction of the face, and a second portion that is presumed to show the driver's ears, which is set in advance according to the direction of the face.

2. The control device is If it is determined that the driver engaged in the aforementioned distracted driving, the driver's right eye, left eye, right ear, and left ear are detected from the image. The image processing method according to claim 1, comprising storing first coordinate information indicating the positions of the right eye, the left eye, the right ear, and the left ear on the image.

3. The control device is If it is determined that the driver engaged in the aforementioned distracted driving, the driver's face is detected from the image, The image processing method according to claim 2, wherein a second coordinate information indicating the position of the face on the image is stored.

4. The image processing method according to any one of claims 1 to 3, wherein the control device determines that the driver is not driving while distracted, and therefore does not save the image.

5. The image processing method according to any one of claims 1 to 4, wherein the control device performs at least one of storing the first and second portions of the image in the vehicle's storage medium and transmitting the first and second portions of the image to a server.

6. In an image processing method performed by a server, The aforementioned server, If the vehicle's control device determines from the image of the driver's face that the driver of the vehicle was driving while distracted, the control device receives the image, From the aforementioned image, a first portion is extracted that is presumed to show the driver's eyes, which is set in advance according to the orientation of the face, and a second portion is extracted that is presumed to show the driver's ears, which is set in advance according to the orientation of the face. An image processing method for saving the first part and the second part.

7. The aforementioned server, Determine whether the eyes are displayed in the first part, and determine whether the ears are displayed in the second part. If it is determined that the eyes are displayed in the first part and the ears are displayed in the second part, then the determination result of distracted driving is deemed valid. The image processing method according to claim 6, wherein if it is determined that the eyes are not displayed in the first part, if it is determined that the ears are not displayed in the second part, or if it is determined that the eyes are not displayed in the first part and the ears are not displayed in the second part, the result of the distracted driving determination is determined to be invalid.

8. A vehicle control device that, if it is determined from an image of the driver's face that the driver was driving while distracted, saves a first portion of the image, which is presumed to show the driver's eyes and is set in advance according to the direction of the face, and a second portion, which is presumed to show the driver's ears and is set in advance according to the direction of the face.

9. In an image processing system comprising a vehicle control device and a server, If the control device determines from the image of the driver's face that the driver of the vehicle has been driving while distracted, it transmits to the server a first portion of the image, which is presumed to show the driver's eyes and is set in advance according to the direction of the face, and a second portion, which is presumed to show the driver's ears and is set in advance according to the direction of the face. The server is an image processing system that stores the first and second parts of the image received from the control device.