Detection apparatus, detection method, program, and detection system

Abstract

To reduce false detection of inattentive driving.SOLUTION: A detection apparatus is configured to: detect inattentive driving by a driver based on a deviation state in which a gaze point of the driver deviates from a region requiring attention to be monitored; and identify a high-speed movement region where an image moves at a speed higher than a high-speed movement determination threshold and if the gaze point is located in the high-speed movement region, make it less likely to determine that the driver is engaging in inattentive driving.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a detection device, a detection method, a program, and a detection system for detecting distracted driving by a driver. [Background technology]

[0002] Conventionally, various methods have been proposed for detecting distracted driving.

[0003] For example, there is a distracted driving detection device that uses saliency (Patent Document 1). This distracted driving detection device includes a saliency calculation means that calculates the saliency of the driver's line of sight based on pixel values, and a determination means that compares the degree to which the driver directs his or her gaze toward the salient part with a predetermined threshold based on the saliency calculated by the saliency calculation means to determine whether the driver is in a distracted state. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5966640 Summary of the Invention [Problem to be solved by the invention]

[0005] Humans have a tendency to be drawn to fast-moving objects. For example, when a vehicle is traveling at high speed, the closer the object, the faster the movement, and as a result, the driver's line of sight (point of view) tends to linger on the fast-moving object. Furthermore, when a human checks an object to be watched, they do not focus their gaze on the object for a long period of time, but rather move their gaze significantly from the point where their gaze is lingering to the object, and often focus their gaze on the object for a relatively short period of time. This is a human habit, but if distracted driving were to be determined based on whether or not the gaze is located on an object to be watched, there would be a problem of incorrect determination, since the gaze would often be located in a lingering area where there is no object to be watched for a long period of time, or in a point where there is an object to be watched for only a short period of time.

[0006] Therefore, an object of the present technology is to provide a technology for reducing erroneous determination of distracted driving. [Means for solving the problem]

[0007] The detection device detects distracted driving by the driver based on a deviation state in which the driver's gaze point deviates from a region requiring attention that should be monitored. The detection device also identifies a high-speed movement region where images move faster than a high-speed movement determination threshold, and makes it difficult to determine that the driver is driving distractedly when the gaze point is located in the high-speed movement region. [Effects of the Invention]

[0008] According to the present technology, distracted driving is determined taking into consideration the human tendency to be drawn to fast-moving objects, thereby reducing erroneous determinations of distracted driving. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining an outline of the embodiment. [Figure 2] FIG. 2 shows an example of a highly salient region in an image. [Figure 3] FIG. 3 is a diagram for explaining the optical flow. [Figure 4]FIG. 4 is a block diagram illustrating an example of a system configuration. [Figure 5] FIG. 5 is a processing flow diagram showing an example of the absentminded driving detection processing. [Figure 6] FIG. 6 is a diagram for explaining moving image data. [Figure 7] FIG. 7 is a process flow diagram according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0010] <Embodiment> Hereinafter, an embodiment will be described with reference to the drawings. FIG. 1 is a diagram for explaining an overview of the embodiment. The system according to this embodiment includes an in-vehicle device 10, a camera 20, and an eye tracker 30 mounted on a vehicle 1, and detects distracted driving by a driver 2. The camera 20 is an imaging device such as a drive recorder, and captures an image in the direction in which the vehicle 1 is moving forward. The eye tracker 30 is an eye-gaze measurement device that detects the line of sight of the driver 2 of the vehicle 1.

[0011] The in-vehicle device 10 acquires video data from the camera 20 and detects from the image an area where the driver should pay attention (hereinafter referred to as an attention-requiring area) and an area where the image (in other words, the captured subject) is moving at high speed (hereinafter referred to as a high-speed moving area) (FIG. 1: step S1). The video data is data captured by the camera 20. FIGS. 2 and 3 show examples of images constituting the video data of the vehicle 1.

[0012] The attention-requiring area is an area in which an object that the driver 2 should pay attention to is captured. The attention-requiring area mainly includes an area near the vehicle's forward path and an area where there is an obstacle or the like that may hinder vehicle travel and require attention. The area near the vehicle's forward path is calculated based on the vehicle's direction, the steering situation (steering angle), and the road conditions (curves, etc.) during travel. The area where there is an obstacle or the like that requires attention is calculated based on image recognition processing using a saliency map. Other examples of the obstacle that requires attention include an area where there is an object moving toward the area near the vehicle's forward path (an object in a moving state or at a position that is expected to come into contact with or approach another vehicle in the future). The in-vehicle device 10 may determine the attention-requiring area based on, for example, a saliency score, or may determine the attention-requiring area based on a model created by machine learning such as deep learning. For example, a saliency map is created based on the image data 200 of FIG. 2, and an area 201 with a saliency higher than a predetermined threshold is identified. The conspicuity score can be calculated using existing methods, for example, by integrating information such as color, brightness, edge direction, and movement.

[0013] A high-speed moving area is an area in a moving image where an object moving relatively fast is captured. The object moving relatively fast in a moving image is mainly an object that moves in conjunction with the movement of the vehicle, and an object that itself moves at high speed. The in-vehicle device 10 may calculate, for example, optical flow and determine the high-speed moving area based on the magnitude of the displacement vector of the captured object. The optical flow can be calculated using supervised learning such as FlowNet2, or simple learning such as SMURF. This calculation can be performed using existing methods such as unsupervised learning, gradient methods, and block matching. For example, optical flow is calculated for image data 200 in Figure 3, and displacement vectors, as indicated by arrows, are calculated based on the movement of objects (feature points in the image) present in the image per unit time. Then, using a predetermined high-speed movement threshold (set to an appropriate value through experiments, etc.), regions in the image (i.e., subjects) are stratified into high-speed moving objects and low-speed moving objects. Based on the distribution (density) of the stratification values ​​(information indicating whether the objects are high-speed moving or low-speed moving), high-speed moving regions (e.g., regions where the density of high-speed moving objects is equal to or greater than the region determination threshold) are determined. Through this processing, for example, regions in Figure 3 where objects close to the vehicle exist (region boundaries indicated by dashed lines), such as the left sidewall region 202 and road surface region 203, which are relatively close to the vehicle, are determined as high-speed moving regions. Furthermore, the right elevated road structure region 204, which is an object or scenery far from the vehicle, and the background region 205 are determined as low-speed moving regions. Additionally, a region where the object itself is moving at a relatively high speed is also considered a high-speed moving region.

[0014] In this embodiment, the in-vehicle device 10 acquires the gaze point of the driver 2 in the video data (FIG. 1: step S2). The eye tracker 30 outputs eye tracking data indicating the gaze direction of the driver 2, and the in-vehicle device 10 converts the eye tracking data into position data in the video captured by the camera 20. This conversion process is performed based on conversion data (e.g., a conversion formula and its parameter values) calculated in advance based on, for example, information about the camera's shooting area (information about the area relative to the vehicle it is installed in, calculated based on camera installation position data or obtained by analyzing actual images captured by the camera 20) and installation information about the eye tracker 30 (the relationship between the reference directions (multiple directions) of the gaze direction in the eye tracking data and the directions corresponding to the reference directions in the vehicle it is installed in). This conversion process may be performed by the eye tracker 30 (a controller (processor) included in the eye tracker 30) or by the in-vehicle device 10 (a controller (processor) included in the in-vehicle device 10).

[0015] Then, the in-vehicle device 10 determines whether or not the driver is driving aimlessly based on the position of the gaze point in the video (FIG. 1: step S3). The in-vehicle device 10 basically detects aimless driving based on the frequency, cumulative time, etc., that the driver 2 looks at the attention-requiring area in the video data. For example, in FIG. 2, the determination is made based on the frequency, cumulative time, etc., that the gaze point is located in area 201 near the vehicle path, which is the attention-requiring area. Furthermore, exceptionally, even if the gaze point of the driver 2 is outside the attention-requiring area, if it is in the high-speed movement area described above, the in-vehicle device 10 makes it difficult to determine or does not determine that the driver is driving aimlessly. For example, in FIG. 3, if the gaze point is in left side wall area 202 or road surface area 203, which are high-speed movement areas, the in-vehicle device 10 makes it difficult to determine or does not determine that the driver is driving aimlessly.

[0016] Even when the driver 2 is properly looking at the attention-requiring area, the gaze point of the driver 2 may remain in the high-speed movement area. In other words, if the driver 2 determines that the driver is driving aimlessly simply based on the frequency with which the driver 2 looks at the attention-requiring area, there is a possibility of false detection. According to this embodiment, it is difficult to determine that the driver is driving aimlessly when the gaze point is in the high-speed movement area, thereby reducing false detection of the driver driving aimlessly.

[0017] <Device configuration> 4 is a block diagram illustrating an example of the system configuration. The vehicle 1 is an automobile, and may be an autonomous driving vehicle. The vehicle 1 includes an in-vehicle device 10, a camera 20, an eye tracker 30, and a user interface (UI) 40. The in-vehicle device 10 also includes a processor 11 and a storage device 12. The in-vehicle device 10 is a computer, and may function as a drive recorder in cooperation with the camera 20, for example.

[0018] The processor 11 is an arithmetic processing device such as a CPU (Central Processing Unit). The processor 11 executes a program to perform each process according to the embodiment. The storage device 12 includes at least one of a main storage device such as a random access memory (RAM) or a read-only memory (ROM), and an auxiliary storage device such as a hard-disk drive (HDD), a solid-state drive (SSD), or a flash memory. The storage device 12 stores programs read by the processor 11 and information to be processed, and secures a working area for the processor 11. For example, the storage device 12 temporarily or permanently stores information used in the present embodiment, such as video data output by the camera 20 and eye tracking data output by the eye tracker 30.

[0019] The camera 20 is an imaging device that converts light into an electrical signal using an image sensor that uses, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and creates and outputs image data. The camera 20 outputs, for example, image data 200 shown in FIGS. 2 and 3. The image data 200 may be, for example, a series of images that form a moving image. The image data 200 corresponds to one still image among the frames. That is, the camera 20 repeats the process of creating and outputting the image data 200. The focal length of the camera 20 is not particularly limited, and the angle of view of the image is not limited to the example shown in the figure. The camera 20 may also be a device worn by the driver 2.

[0020] The eye tracker 30 is a non-wearable gaze measurement device installed in the vehicle cabin, or a wearable gaze measurement device worn by the driver 2. The eye tracker measures the gaze of the driver 2 using existing non-contact or contact methods. For example, a non-contact method measures gaze movement based on the Purkinje image formed by irradiating the pupil with infrared light and reflecting it off the cornea, and the position of the pupil. Contact methods such as the search coil method and electrooculogram (EOG) method may also be used. The eye tracker 30 or the in-vehicle device 10 estimates the position of the user's gaze point within a video. The estimated gaze direction is associated with coordinates within the video data by calculating conversion data, etc., based on the eye tracker 30, its specifications, and installation information (installation position, orientation, etc.).

[0021] The UI 40 is a user interface such as a speaker and a display. The display may be a display with a touch panel laminated thereon. The processor 11 displays and outputs audio information to the driver 2 of the vehicle via the UI 40 to warn the driver 2 about distracted driving.

[0022] <Distracted driving detection processing> 5 is a processing flow diagram showing an example of distracted driving detection processing. For example, when the ignition power of the vehicle 1 is turned on, when the vehicle 1 starts moving, or when the camera 20 and eye tracker 30 are activated, the in-vehicle device 10 starts the processing shown in FIG. 5 and repeatedly executes it at appropriate intervals that allow operation without malfunction. The camera 20 continuously outputs captured video data to the in-vehicle device 10, and the processor 11 of the in-vehicle device 10 stores the input video data in the storage device 12. The eye tracker 30 continuously outputs eye tracking data of the driver 2 to the in-vehicle device 10, and the processor 11 of the in-vehicle device 10 stores the input eye tracking data in the storage device 12.

[0023] As an initial process, the processor 11 of the in-vehicle device 10 resets the free-running timer (timer value = 0) and starts timing (step S11). After step S11, the processor 11 of the in-vehicle device 10 acquires image data output by the camera 20 (step S12). Specifically, image data 200 output by the camera 20, such as that shown in Figures 2 and 3, is sequentially stored in the storage device 12, and the processor 11 reads the image data 200 to be used for processing from the storage device 12 and uses it for processing. Figure 6 shows the structure of moving image data. Moving image data is data composed of still images captured at a predetermined interval (for example, 30 frames captured per second, Figure 6: 200A to 200E). Each still image data is stored in the storage device 12 along with data that can identify the time of capture, allowing the processor 11 to read any still image data required for processing.

[0024] After step S12, processor 11 creates a conspicuity map (step S13). Specifically, the latest still image (for convenience, still image 200E) is divided into grid-like microregions (size is set to an appropriate size based on experiments, etc.), the conspicuity score of each microregion is calculated using a known conspicuity calculation method, etc., and the calculated score is assigned to each microregion to create a two-dimensional conspicuity map. Then, an area formed by microregions whose conspicuity score is equal to or greater than a predetermined caution threshold (an appropriate value based on experiments, etc.) can be detected as an obstacle area where an obstacle, etc., is likely to exist.

[0025] After step S13, the processor 11 determines the steering angle of the vehicle, road conditions (by image recognition, etc.), The vehicle path is calculated based on the above-mentioned detection, and the image change processing is performed to detect the vehicle path on the image (step S14).

[0026] After step S14, processor 11 sets the attention-requiring area detected based on the prominence map created in step S13 and the surrounding area of ​​the vehicle path detected in step S14 (an appropriate range based on the vehicle path based on experiments, etc. is set as the surrounding area) as attention-requiring areas (step S15).

[0027] After step S15, processor 11 sets a high-speed movement region in the latest still image (step S16). Specifically, the optical flow of each region (singular points present in each region) in the image is calculated using the latest still image and a plurality of still images temporally adjacent to it. Then, for example, a region with a high density (a density equal to or greater than a region determination threshold set to an appropriate value based on experiments, etc.) stratified by displacement vectors equal to or greater than a high-speed movement determination threshold (set to an appropriate value based on experiments, etc.) is set as a high-speed movement region (e.g., region 202E). The optical flow is calculated by comparing still image 200E with a still image from a predetermined time earlier (an appropriate time set based on experiments, etc.), for example, still image 200B from three frames earlier.

[0028] After step S16, the processor 11 calculates the gaze point of the driver 2 based on the data from the eye tracker 30 (step S17).

[0029] After step S17, the processor 11 determines whether the calculated point of interest is located in the attention-requiring area (step S18).

[0030] In step S18, if it is determined that the gaze point is located in an area requiring attention (S18: YES), that is, if it is determined that the driver 2 is looking at a position to be gazed at, the processor 11 resets the timer (T=0) and transfers processing to step S12 (step S19).

[0031] In step S18, if it is determined that the fixation point is not located in the attention-requiring area (also referred to as a "deviation state") (S18: NO), the processor 11 determines whether the fixation point is located in a high-speed movement area (step S20).

[0032] In step S20, if it is determined that the gaze point is located in a high-speed movement area (S20: YES), the processor 11 determines whether the timer T exceeds the threshold value TL (a mindless driving threshold value when the gaze point is located in a high-speed movement area: set to an appropriate value through experiments, etc.) (step S21).

[0033] In step S21, if the timer T exceeds the threshold value TL (S21: YES), the processor 11 determines that the driver is driving aimlessly (step S23). After step S23, the processor 11 outputs a warning (display, sound, etc.) against the driver's driving aimlessly, and then ends the process (step S24).

[0034] Furthermore, in step S21, if the timer T is equal to or less than the threshold value TL (S21: NO), the processor 11 proceeds to step S12.

[0035] In step S20, if it is determined that the gaze point is not located in the high-speed movement area (S20: NO), the processor 11 determines whether the timer T exceeds a threshold value TM (a mindless driving threshold value when the gaze point is not located in the high-speed movement area: set to an appropriate value through experiments, etc.) (step S22). Note that the threshold value TM is a value smaller than the above-mentioned threshold value TL (TL>TM).

[0036] In step S22, if the timer T exceeds the threshold value TM, the processor 11 determines that the driving is aimless (step S23). After step S23, the processor 11 outputs a warning (display, sound, etc.) against the aimless driving, and then ends the processing (step S24). Also, in step S22, if the timer T is equal to or less than the threshold value TM, the processor 11 shifts the processing to step S12 (step S23).

[0037] With the above processing, when the driver 2 takes his / her eyes off the area around the vehicle's path or an obstacle, if his / her gaze position is in a high-speed movement area, a warning about careless driving will be issued when the time he / she has taken his / her eyes off exceeds the threshold TL time, and if his / her gaze position is not in a high-speed movement area, a warning about careless driving will be issued when the time he / she has taken his / her eyes off exceeds the threshold TM time (a time shorter than the threshold TL time).As a result, a warning about careless driving appropriate to the characteristics of each person will be issued.

[0038] <Effects> As described above, the gaze point of the driver 2 may stay in the high-speed movement area. As in the present embodiment, even if the gaze point is outside the attention-requiring area, if it is located in the high-speed movement area, it is difficult to determine that the driver is driving aimlessly (if it is not located in the high-speed movement area, it is easy to determine that the driver is driving aimlessly), thereby reducing false detection of aimless driving.

[0039] In this embodiment, if the driver 2 takes his / her eyes off the periphery of the vehicle's path or an obstacle, and if the gaze position is in a high-speed movement area, a warning about careless driving is issued when the time the driver has taken his / her eyes off exceeds a threshold time TL (a time longer than the area threshold time TM in a non-high-speed movement image). However, if the gaze position is in a high-speed movement area, the operation of not issuing a warning about careless driving can also have the effect of preventing unnecessary warnings. To realize such processing, as shown in Fig. 7, step S21 in the flowchart shown in Fig. 5 can be replaced with step S25, which resets the timer (T = 0) and then proceeds to step S12. In other words, this processing does not determine careless driving if the gaze position is in a high-speed movement area (the determination processing of step S22 is not performed).

[0040] <Other> The illustrated system and device configurations are merely examples and are not limited to the above examples. For example, at least some of the processing performed by the in-vehicle device 10 of the vehicle 1 may be shared and executed by multiple devices, or may be executed in parallel by multiple devices. The vehicle 1 may be a so-called autonomous driving vehicle.

[0041] When the vehicle 1 performs so-called automatic driving, in step S24 of FIG. 5, the processor 11 may perform driving control such as decelerating or stopping the vehicle 1.

[0042] Furthermore, the processing is not limited to the examples shown in Fig. 1 and Fig. 5. For example, the order of step S15 and steps S15 to S17 may be reversed, or other appropriate changes may be made so as to achieve equivalent processing and control.

[0043] The present invention also includes a computer program for executing the above-mentioned processing method, and a computer-readable recording medium on which the program is recorded. By having a computer read the program stored in the recording medium and executing the program, an apparatus equipped with the computer can realize the above-mentioned operations. A computer-readable recording medium is a recording medium that stores information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer. Among such recording media, those that can be removed from a computer include flexible disks, magneto-optical disks, optical disks, magnetic tapes, memory cards, etc. Also, Examples of recording media fixed to a computer include a hard disk drive, a ROM, etc. Alternatively, a method may be applied in which a computer program is stored in a storage medium (such as a hard disk drive) of a server, and the server distributes the computer program to an in-vehicle device or the like via a communication network environment. [Explanation of symbols]

[0044] 1: Vehicle, 10: In-vehicle device, 11: Processor, 12: Storage device, 20: Camera, 30: Eye tracker, 40: User interface (UI)

Claims

1. A detection device that detects distracted driving by a driver based on a deviation state in which the driver's gaze point deviates from a region requiring attention to be monitored, Identifying a high-speed moving area where the image moves faster than a high-speed moving determination threshold; When the gaze point is located in the high-speed movement area, it is difficult to determine that the vehicle is driving aimlessly. Detection device.

2. The attention-requiring region is set based on a saliency map. The detection device of claim 1 .

3. When the gaze point is located in the high-speed movement area, it is not determined that the driving is careless. The detection device according to claim 1 or 2.

4. When the gaze point is located in the high-speed movement area, the determination of absentminded driving is not performed. The detection device according to claim 1 or 2.

5. If the time during which the gaze point of the driver deviates from the attention-requiring area is equal to or greater than a determination threshold, the driver is determined to be driving absentmindedly; When the gaze point is located in the high-speed movement area, the determination threshold is set to a longer time than when the gaze point is located in an area other than the high-speed movement area. The detection device according to claim 1 or 2.

6. A detection method for detecting distracted driving by a driver based on a deviation state in which the driver's gaze point deviates from a region requiring attention to be monitored, comprising: Identifying a high-speed moving area where the image moves faster than a high-speed moving determination threshold; When the gaze point is located in the high-speed movement area, it is difficult to determine that the vehicle is driving aimlessly. Detection method.

7. A program for causing a computer to execute a process for detecting distracted driving by a driver based on a deviation state in which the driver's gaze point deviates from a region requiring attention to be monitored, Identifying a high-speed moving area where the image moves faster than a high-speed moving determination threshold; When the gaze point is located in the high-speed movement area, it is difficult to determine that the vehicle is driving aimlessly. program.

8. a camera that outputs video image data of the area in front of the vehicle; an eye tracker that outputs gaze point information regarding a gaze point of a driver of the vehicle; a detection device that detects the driver's distracted driving based on a gaze point position based on the gaze point information on a processing target image based on the moving image data; Equipped with The detection device is Detecting whether the driver is driving aimlessly based on a deviation state of the gaze point position from an attention-requiring area to be image-monitored on the processing target image; Identifying a high-speed movement area in the image to be processed where the image moves faster than a high-speed movement determination threshold; When the gaze point position is located in the high-speed movement area, it is difficult to determine that the vehicle is driving aimlessly. Detection system.

Images