Information output device

Abstract

To detect a tendency of inattentive driving with a simple configuration.SOLUTION: In an information processor 1, a visual saliency processing section 3 acquires a visual saliency map obtained by estimating degrees of visual saliency in an image in time series on the basis of the image obtained by picking up the outside from a moving body, a visual saliency peak detection section 4 detects at least one peak position in the visual saliency map in time series. An inattentive driving tendency determination section 5 sets an area G to be gazed in the image, and an inattentive driving warning section 6 outputs information that there is a tendency of inattentive driving when the peak position is deviated from the area G to be gazed continuously for a predetermined time or more.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information output device that outputs predetermined information based on an image of the outside taken from a moving object. [Background technology]

[0002] In order to reduce traffic accidents, detection of inattentiveness of drivers has been carried out. For example, Patent Document 1 describes a method in which visual feature points (e.g., a preceding vehicle 44, a traffic light 45) that are visually recognizable objects in front of the vehicle are detected based on an image G captured by an in-vehicle camera 1, and whether or not the driver is in an inattentive state is determined based on the visual feature points and the gaze point of the person being determined. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-224067 Summary of the Invention [Problem to be solved by the invention]

[0004] In the method described in Patent Document 1, the driver's line of sight, facial direction, posture, etc. are detected from the driver's image, and where the driver is looking is determined by comparing it with the driving image. Therefore, not only the driving image but also the video from the driver's side is required, which requires multiple cameras. In addition, a huge amount of calculation processing is required to compare the driver's line of sight with the driving image.

[0005] One example of a problem to be solved by the present invention is to detect a tendency to look aside with a simple configuration. [Means for solving the problem]

[0006] In order to solve the above problem, the invention described in claim 1 is characterized by comprising an acquisition unit that acquires visual saliency distribution information in a time series, which is obtained by estimating the level of visual saliency in an image based on an image of the outside taken from a moving body; a peak position detection unit that detects at least one peak position in the visual saliency distribution information in a time series; a gaze range setting unit that sets a range in the image that the driver of the moving body should gaze at; and an inattentive driving output unit that outputs information indicating a tendency to look away if the peak position is outside the gaze range for a predetermined period of time or more.

[0007] The invention described in claim 7 is an information output method executed by an information output device that outputs predetermined information based on an image of the outside taken from a moving body, and is characterized by including: an acquisition step of acquiring visual saliency distribution information in a time series, which is obtained by estimating the level of visual saliency in the image based on the image; a peak position detection step of detecting at least one peak position in the visual saliency distribution information in a time series; a gaze range setting step of setting a range in the image that the driver of the moving body should gaze at; and an inattentive driving output step of outputting information indicating a tendency to look away if the peak position is consistently outside the gaze range for more than a predetermined period of time.

[0008] The invention as set forth in claim 8 is characterized in that the information output method as set forth in claim 7 is executed by a computer.

[0009] The invention as set forth in claim 9 is characterized in that the information output program as set forth in claim 8 is stored. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a functional configuration diagram of an information output device according to a first embodiment of the present invention. [Figure 2] 2 is a block diagram illustrating the configuration of a visual saliency processing unit shown in FIG. 1. FIG. [Figure 3]1A is a diagram illustrating an example of an image input to a determination device, and FIG. 1B is a diagram illustrating an example of a visual saliency map estimated for FIG. 1A. [Figure 4] 2 is a flowchart illustrating a processing method of the visual saliency processing unit shown in FIG. 1 . [Figure 5] FIG. 2 is a diagram illustrating in detail an example of the configuration of a nonlinear mapping unit. [Figure 6] FIG. 2 is a diagram illustrating the configuration of an intermediate layer. [Figure 7] 10(a) and 10(b) are diagrams illustrating examples of convolution processing performed by a filter. [Figure 8] (a) is a diagram for explaining the processing of the first pooling unit, (b) is a diagram for explaining the processing of the second pooling unit, and (c) is a diagram for explaining the processing of the unpooling unit. [Figure 9] FIG. 10 is an explanatory diagram of a method for setting a gaze area. [Figure 10] FIG. 10 is an explanatory diagram of an aside-looking detection area. [Figure 11] FIG. 10 is an explanatory diagram of another inattentive looking detection area. [Figure 12] 2 is a flowchart of the operation of the information output device shown in FIG. [Figure 13] FIG. 10 is a configuration diagram of a system including an information processing device according to a second embodiment of the present invention. [Figure 14] FIG. 14 is a functional configuration diagram of the information processing device shown in FIG. [Figure 15] 15 is a flowchart of the operation of the information processing device shown in FIG. 14. DETAILED DESCRIPTION OF THE INVENTION

[0011] An information output device according to one embodiment of the present invention will be described below. In the information output device according to one embodiment of the present invention, an acquisition unit acquires visual saliency distribution information in a time series, which is obtained by estimating the level of visual saliency in an image captured from a moving object. A peak position detection unit detects at least one peak position in the visual saliency distribution information in a time series. A gaze range setting unit sets a range in the image that a driver of the moving object should gaze at. An inattentiveness output unit outputs information indicating a tendency for inattentiveness if the peak position is outside the gaze range for a predetermined period of time or more. This visual saliency distribution information indicates a statistical likelihood of attracting human gaze. Therefore, the peak of the visual saliency distribution information indicates a position within the information that is statistically most likely to attract human gaze. Therefore, by using the visual saliency distribution information, it is possible to detect a tendency for inattentiveness with a simple configuration without measuring the actual driver's gaze.

[0012] The gaze range setting unit may also set the gaze range based on the vanishing point of the image, which makes it possible to easily set the gaze range without detecting a vehicle ahead, for example.

[0013] Furthermore, the inattentive driving output unit may output information indicating a tendency for inattentive driving due to a fixed object if the peak position is located above or below the range to be focused on for a predetermined period of time or more. The area above the range to be focused on is generally an area where fixed objects such as buildings, traffic signals, signs, and streetlights are reflected, while the area below the range to be focused on is generally an area where road markings such as road signs are reflected. In other words, if the peak position falls within these ranges, it can be determined that the object that is causing inattentive driving due to inattentive driving is a fixed object.

[0014] The vehicle may also include a speed acquisition unit that acquires the moving speed of the moving object, and the gaze range setting unit may change the gaze range based on the moving speed. For example, it is known that the driver's field of vision narrows when driving at high speeds, and the gaze range can be set taking this into consideration. The speed acquisition unit may acquire information about speed from a vehicle speed sensor, an acceleration sensor, etc., or may determine the speed from a captured image.

[0015] The vehicle may further include a location acquisition unit that acquires location information of the moving body, and the gaze range setting unit may change the gaze range based on the location information. In this way, the gaze range can be set according to the location of the vehicle, such as a residential area, a main road, or a busy shopping district.

[0016] The acquisition unit may also include an input unit that converts the image into intermediate data that can be mapped, a nonlinear mapping unit that converts the intermediate data into mapped data, and an output unit that generates saliency estimation information indicating a saliency distribution based on the mapped data, and the nonlinear mapping unit may include a feature extraction unit that extracts features from the intermediate data and an upsampling unit that upsamples the data generated by the feature extraction unit. This allows visual saliency to be estimated with low computational cost. Furthermore, the visual saliency estimated in this manner reflects a contextual attention state.

[0017] In addition, an information output method according to one embodiment of the present invention includes: an acquisition step of acquiring visual saliency distribution information in a time series, the information being obtained by estimating the level of visual saliency in an image captured from a moving object; a peak position detection step of detecting at least one peak position in the visual saliency distribution information in a time series; a gaze range setting step of setting a range in the image that a driver of the moving object should gaze at; and an inattentiveness output step of outputting information indicating a tendency for inattentiveness if the peak position is outside the gaze range for a predetermined period of time or more. This visual saliency distribution information indicates a statistical likelihood of attracting human gaze. Therefore, the peak of the visual saliency distribution information indicates a position within the information that is statistically most likely to attract human gaze. Therefore, by using the visual saliency distribution information, it is possible to detect a tendency for inattentiveness with a simple configuration without measuring the actual driver's gaze.

[0018] Furthermore, the above-described information output method is executed by a computer, which makes it possible to detect the tendency of the driver to look away with a simple configuration without measuring the actual driver's line of sight using a computer.

[0019] The information output program may be stored in a computer-readable storage medium, which allows the program to be distributed as a standalone program rather than being incorporated into a device, and allows for easy version upgrades. [Example]

[0020] An information output device according to a first embodiment of the present invention will be described with reference to Figures 1 to 12. The information output device according to this embodiment is not limited to being installed in a mobile object such as an automobile, but may also be configured as a server device or the like installed in a business establishment, etc. In other words, analysis does not need to be performed in real time, and analysis may be performed after driving, etc.

[0021] As shown in Figure 1, the information output device 1 includes a driving image acquisition unit 2, a visual saliency processing unit 3, a visual saliency peak detection unit 4, an inattentive driving tendency determination unit 5, and an inattentive driving warning unit 6.

[0022] The traveling image acquisition unit 2 receives input of images (e.g., moving images) captured by, for example, a camera, and outputs the images as image data. The input moving images are output as image data broken down into time series, for example, frame by frame. Although still images may be input as images to be input to the traveling image acquisition unit 2, it is preferable to input them as an image group consisting of a plurality of still images in time series.

[0023] The images input to the traveling image acquisition unit 2 include, for example, images captured in the direction of travel of the vehicle. In other words, images are images captured continuously from the moving body. These images may be so-called panoramic images or images captured using multiple cameras, and may include images that include angles other than the traveling direction, such as 180° or 360° in the horizontal direction. Furthermore, the images input to the traveling image acquisition unit 2 are not limited to images captured by a camera, but may also be images read from a recording medium such as a hard disk drive or memory card.

[0024] The visual saliency processing unit 3 receives image data from the traveling image acquisition unit 2 and outputs a visual saliency map as visual saliency estimation information (described later). That is, the visual saliency processing unit 3 functions as an acquisition unit that acquires a visual saliency map (visual saliency distribution information) obtained by estimating the level of visual saliency in an image captured of the outside from a moving object.

[0025] FIG. 2 is a block diagram illustrating the configuration of the visual saliency processing unit 3. The visual saliency processing unit 3 according to this embodiment includes an input unit 310, a nonlinear mapping unit 320, an output unit 330, and a storage unit 390. The input unit 310 converts an image into intermediate data that can be subjected to mapping processing. The nonlinear mapping unit 320 converts the intermediate data into mapped data. The output unit 330 generates saliency estimation information indicating a saliency distribution based on the mapped data. The nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. The storage unit 390 stores image data input from the traveling image acquisition unit 2, filter coefficients (described later), and the like. These are described in detail below.

[0026] FIG. 3(a) is a diagram illustrating an example of an image input to the visual saliency processing unit 3, and FIG. 3(b) is a diagram illustrating an example of an image showing a visual saliency distribution estimated for FIG. 3(a). The visual saliency processing unit 3 according to this embodiment is a device that estimates the visual saliency of each part in an image. Visual saliency means, for example, how easily something stands out or how easily it attracts attention. Specifically, visual saliency is expressed as a probability or the like. Here, the magnitude of the probability corresponds to, for example, the probability that a person viewing the image will direct their gaze to that position.

[0027] 3(a) and 3(b) correspond to each other in position. In FIG. 3(a), the higher the visual saliency, the higher the brightness displayed in FIG. 3(b). The image showing the visual saliency distribution as shown in FIG. 3(b) is an example of a visual saliency map output by the output unit 330. In the example shown in this figure, visual saliency is visualized using brightness values ​​of 256 levels. An example of a visual saliency map output by the output unit 330 will be described in detail later.

[0028] FIG. 4 is a flowchart illustrating the operation of the visual saliency processing unit 3 according to this embodiment. The flowchart shown in FIG. 4 is part of an information output method executed by a computer, and includes an input step S115, a nonlinear mapping step S120, and an output step S130. In the input step S115, an image is converted into intermediate data that can be mapped. In the nonlinear mapping step S120, the intermediate data is converted into mapped data. In the output step S130, visual saliency estimation information (visual saliency distribution information) indicating a saliency distribution is generated based on the mapped data. Here, the nonlinear mapping step S120 includes a feature extraction step S121 that extracts features from the intermediate data, and an upsampling step S122 that upsamples the data generated in the feature extraction step S121.

[0029] Returning to FIG. 2, each component of the visual saliency processing unit 3 will be described. In input step S115, the input unit 310 acquires an image and converts it into intermediate data. The input unit 310 acquires image data from the driving image acquisition unit 2. The input unit 310 then converts the acquired image into intermediate data. The intermediate data is not particularly limited as long as it is data that can be accepted by the nonlinear mapping unit 320, but is, for example, a high-dimensional tensor. Furthermore, the intermediate data is, for example, data in which the brightness of the acquired image is normalized, or data in which each pixel of the acquired image is converted into a brightness gradient. In input step S115, the input unit 310 may further perform noise removal and resolution conversion on the image.

[0030] In the nonlinear mapping step S120, the nonlinear mapping unit 320 acquires intermediate data from the input unit 310. The nonlinear mapping unit 320 then converts the intermediate data into mapping data. Here, the mapping data is, for example, a high-dimensional tensor. The mapping process performed on the intermediate data by the nonlinear mapping unit 320 is, for example, a mapping process that can be controlled by parameters, and is preferably a process using a function, a functional, or a neural network.

[0031] Fig. 5 is a diagram illustrating a detailed configuration of the nonlinear mapping unit 320, and Fig. 6 is a diagram illustrating a configuration of the hidden layer 323. As described above, the nonlinear mapping unit 320 includes a feature extraction unit 321 and an upsampling unit 322. The feature extraction unit 321 performs the feature extraction step S121, and the upsampling unit 322 performs the upsampling step S122. In the example shown in this figure, at least one of the feature extraction unit 321 and the upsampling unit 322 is configured to include a neural network including a plurality of hidden layers 323. In the neural network, a plurality of hidden layers 323 are connected.

[0032] In particular, the neural network is preferably a convolutional neural network. Specifically, each of the multiple hidden layers 323 includes one or more convolutional layers 324. In the convolutional layers 324, input data is convolved by multiple filters 325, and activation processing is performed on the outputs of the multiple filters 325.

[0033] 5, feature extraction unit 321 is configured to include a neural network including a plurality of hidden layers 323, and a first pooling unit 326 is provided between the plurality of hidden layers 323. Furthermore, upsampling unit 322 is configured to include a neural network including a plurality of hidden layers 323, and an unpooling unit 328 is provided between the plurality of hidden layers 323. Furthermore, feature extraction unit 321 and upsampling unit 322 are connected to each other via a second pooling unit 327 that performs overlap pooling.

[0034] In the example shown in this figure, each intermediate layer 323 is made up of two or more convolutional layers 324. However, at least some of the intermediate layers 323 may be made up of only one convolutional layer 324. Adjacent intermediate layers 323 are separated by any of a first pooling unit 326, a second pooling unit 327, and an unpooling unit 328. Here, when an intermediate layer 323 includes two or more convolutional layers 324, it is preferable that the number of filters 325 in those convolutional layers 324 be equal to each other.

[0035] In this figure, an "A×B" hidden layer 323 is composed of B convolution layers 324, and each convolution layer 324 includes A convolution filters for each channel. Such a hidden layer 323 is also referred to as an "A×B hidden layer" below. For example, a 64×2 hidden layer 323 is composed of two convolution layers 324, and each convolution layer 324 includes 64 convolution filters for each channel.

[0036] In the example shown in the figure, the feature extraction unit 321 includes a 64×2 hidden layer 323, a 128×2 hidden layer 323, a 256×3 hidden layer 323, and a 512×3 hidden layer 323, in this order. The upsampling unit 322 includes a 512×3 hidden layer 323, a 256×3 hidden layer 323, a 128×2 hidden layer 323, and a 64×2 hidden layer 323, in this order. The second pooling unit 327 connects the two 512×3 hidden layers 323 to each other. The number of hidden layers 323 constituting the nonlinear mapping unit 320 is not particularly limited and can be determined, for example, according to the number of pixels in the image data.

[0037] Note that this diagram shows an example of the configuration of the nonlinear mapping unit 320, and the nonlinear mapping unit 320 may have other configurations. For example, a 64×1 intermediate layer 323 may be included instead of the 64×2 intermediate layer 323. Reducing the number of convolutional layers 324 included in the intermediate layer 323 may further reduce the computational cost. Also, for example, a 32×2 intermediate layer 323 may be included instead of the 64×2 intermediate layer 323. Reducing the number of channels in the intermediate layer 323 may further reduce the computational cost. Furthermore, both the number of convolutional layers 324 and the number of channels in the intermediate layer 323 may be reduced.

[0038] Here, in the plurality of intermediate layers 323 included in the feature extraction unit 321, it is preferable that the number of filters 325 increases every time it passes through the first pooling unit 326. Specifically, the first intermediate layer 323a and the second intermediate layer 323b are continuous with each other via the first pooling unit 326, and the second intermediate layer 323b is located at the subsequent stage of the first intermediate layer 323a. The first intermediate layer 323a is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N1, and the second intermediate layer 323b is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N2. At this time, it is preferable that N2 > N1 holds. More preferably, N2 = N1 × 2 holds.

[0039] Also, in the plurality of intermediate layers 323 included in the upsampling unit 322, it is preferable that the number of filters 325 decreases every time it passes through the unpooling unit 328. Specifically, the third intermediate layer 323c and the fourth intermediate layer 323d are continuous with each other via the unpooling unit 328, and the fourth intermediate layer 323d is located at the subsequent stage of the third intermediate layer 323c. The third intermediate layer 323c is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N3, and the fourth intermediate layer 323d is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N4. At this time, it is preferable that N4 < N3 holds. More preferably, N3 = N4 × 2 holds.

[0040] The feature extraction unit 321 extracts image features with multiple levels of abstraction, such as gradients and shapes, from the intermediate data acquired from the input unit 310 as channels for the intermediate layer 323. FIG. 6 illustrates an example of the configuration of a 64×2 intermediate layer 323. The processing in the intermediate layer 323 will be described with reference to this figure. In the example shown in this figure, the intermediate layer 323 is composed of a first convolutional layer 324a and a second convolutional layer 324b, and each convolutional layer 324 is equipped with 64 filters 325. In the first convolutional layer 324a, convolution processing using the filters 325 is performed on each channel of the data input to the intermediate layer 323. For example, if the image input to the input unit 310 is an RGB image, three channels h 0 i (i=1..3) are processed. In the example shown in the figure, the filter 325 is a 3x3 filter of 64 types, that is, a total of 64x3 types of filters. As a result of the convolution process, for each channel i, 64 results h 0 i,j (i=1..3,j=1..64) is obtained.

[0041] Next, the outputs of the multiple filters 325 are subjected to activation processing in the activation unit 329. Specifically, for the corresponding results j of all channels, activation processing is performed on the sum of each corresponding element. By this activation processing, the results h of 64 channels are 1 i (i=1..64), i.e., the output of the first convolution layer 324a, is obtained as the image feature. The activation process is not particularly limited, but a process using at least one of a hyperbolic function, a sigmoid function, and a rectified linear function is preferable.

[0042] Furthermore, the output data of the first convolution layer 324a is used as input data for the second convolution layer 324b, and the same processing as that of the first convolution layer 324a is performed in the second convolution layer 324b to obtain the result h of 64 channels. 2 i (i=1..64), that is, the output of the second convolutional layer 324b, is obtained as the image features. The output of the second convolutional layer 324b becomes the output data of this 64×2 hidden layer 323.

[0043] Here, the structure of the filter 325 is not particularly limited, but a 3x3 two-dimensional filter is preferable. Furthermore, the coefficients of each filter 325 can be set independently. In this embodiment, the coefficients of each filter 325 are stored in the memory unit 390, and the nonlinear mapping unit 320 can read and use them for processing. Here, the coefficients of the multiple filters 325 may be determined based on correction information generated and corrected using machine learning. For example, the correction information includes the coefficients of the multiple filters 325 as multiple correction parameters. The nonlinear mapping unit 320 can further use this correction information to convert the intermediate data into mapped data. The memory unit 390 may be provided in the visual saliency processing unit 3 or external to the visual saliency processing unit 3. Furthermore, the nonlinear mapping unit 320 may obtain the correction information from an external source via a communication network.

[0044] 7(a) and 7(b) are diagrams illustrating examples of convolution processing performed by the filter 325. Both of FIGS. 7(a) and 7(b) illustrate examples of 3×3 convolution. The example in FIG. 7(a) illustrates convolution processing using nearest neighbor elements. The example in FIG. 7(b) illustrates convolution processing using neighbor elements with a distance of two or more. Note that convolution processing using neighbor elements with a distance of three or more is also possible. It is preferable that the filter 325 performs convolution processing using neighbor elements with a distance of two or more. This is because it allows for the extraction of a wider range of features, thereby further improving the estimation accuracy of visual saliency.

[0045] The above has described the operation of the 64×2 hidden layer 323. The operations of the other hidden layers 323 (such as the 128×2 hidden layer 323, the 256×3 hidden layer 323, and the 512×3 hidden layer 323) are the same as the operation of the 64×2 hidden layer 323, except for the number of convolutional layers 324 and the number of channels. Furthermore, the operations of the hidden layer 323 in the feature extraction unit 321 and the hidden layer 323 in the upsampling unit 322 are also the same as those described above.

[0046] FIG. 8(a) is a diagram for explaining the processing of the first pooling unit 326, FIG. 8(b) is a diagram for explaining the processing of the second pooling unit 327, and FIG. 8(c) is a diagram for explaining the processing of the unpooling unit 328.

[0047] In the feature extraction unit 321, data output from the intermediate layer 323 is subjected to pooling processing for each channel in the first pooling unit 326, and then input to the next intermediate layer 323. The first pooling unit 326 performs, for example, non-overlapping pooling processing. FIG. 8(a) shows processing for associating four 2×2 elements 30 with one element 30 for a group of elements included in each channel. The first pooling unit 326 performs such association for all elements 30. Here, the four 2×2 elements 30 are selected so that they do not overlap with each other. In this example, the number of elements in each channel is reduced to one-fourth. Note that, as long as the number of elements is reduced in the first pooling unit 326, the number of elements 30 before and after the association is not particularly limited.

[0048] The data output from the feature extraction unit 321 is input to the upsampling unit 322 via the second pooling unit 327. The second pooling unit 327 performs overlap pooling on the output data from the feature extraction unit 321. FIG. 8(b) shows a process of associating four 2×2 elements 30 with one element 30 while overlapping some of the elements 30. That is, in repeated associations, some of the four 2×2 elements 30 in a certain association are also included in the four 2×2 elements 30 in the next association. The second pooling unit 327 shown in this figure does not reduce the number of elements. Note that the number of elements 30 before and after association in the second pooling unit 327 is not particularly limited.

[0049] The methods of processing performed by the first pooling unit 326 and the second pooling unit 327 are not particularly limited, but examples include matching in which the maximum value of four elements 30 is matched to one element 30 (max pooling) and matching in which the average value of four elements 30 is matched to one element 30 (average pooling).

[0050] The data output from the second pooling unit 327 is input to the hidden layer 323 in the upsampling unit 322. Then, the output data from the hidden layer 323 of the upsampling unit 322 undergoes unpooling processing for each channel in the unpooling unit 328, and is then input to the next hidden layer 323. FIG. 8(c) shows processing for expanding one element 30 into multiple elements 30. The method of expansion is not particularly limited, but an example is a method of duplicating one element 30 into four elements 30 (2 × 2).

[0051] The output data of the last hidden layer 323 of the upsampling unit 322 is output from the nonlinear mapping unit 320 as mapping data and input to the output unit 330. In the output step S130, the output unit 330 generates and outputs a visual saliency map by performing, for example, normalization or resolution conversion on the data acquired from the nonlinear mapping unit 320. The visual saliency map is, for example, an image (image data) that visualizes visual saliency using brightness values, as illustrated in FIG. 3(b). The visual saliency map may also be, for example, an image that is color-coded according to visual saliency, such as a heat map, or an image in which visual saliency regions with visual saliency higher than a predetermined standard are marked so as to be distinguishable from other positions. Furthermore, the visual saliency estimation information is not limited to map information displayed as an image or the like, but may also be a table or the like that lists information indicating visual saliency regions.

[0052] The visual saliency peak detection unit 4 detects peak positions (pixels) in the visual saliency map acquired by the visual saliency processing unit 3. In this embodiment, a peak is a pixel with high visual saliency where the pixel value is maximum (maximum brightness), and the position is expressed by coordinates. That is, the visual saliency peak detection unit 4 functions as a peak position detection unit that detects at least one peak position in the visual saliency map (visual saliency distribution information).

[0053] The inattentive driving tendency determination unit 5 determines whether the image input from the traveling image acquisition unit 2 has an inattentive driving tendency based on the peak position detected by the visual saliency peak detection unit 4. The inattentive driving tendency determination unit 5 first sets a gaze area (range to be gazed at) for the image input from the traveling image acquisition unit 2. The method for setting the gaze area will be described with reference to Fig. 9. That is, the inattentive driving tendency determination unit 5 functions as a gaze range setting unit that sets a range in the image to be gazed at by the driver of the moving object.

[0054] In the image P shown in FIG. 9, the gaze area G is set around the vanishing point V. That is, the gaze area G (the range to be gazed at) is set based on the vanishing point of the image. The size of this gaze area G (for example, 3 m wide and 2 m high) is set in advance, and the number of pixels of the set size can be calculated from the number of horizontal pixels, the number of vertical pixels, the horizontal angle of view, the vertical angle of view, the distance to the leading vehicle, the mounting height of the camera such as a drive recorder capturing the image, and the like. The vanishing point may be estimated from a white line or the like, or may be estimated using optical flow or the like. Furthermore, the distance to the leading vehicle does not need to be detected actually, and may be set virtually.

[0055] Next, an inattentiveness detection area is set in image P based on the set gaze area G (shaded area in FIG. 10). The inattentiveness detection area is divided into an upper area Iu, a lower area Id, a left side area Il, and a right side area Ir. These areas are separated by line segments connecting the vanishing point V and the vertices of gaze area G. That is, the upper area Iu and the left side area Il are separated by line segment L1 connecting the vanishing point V and the vertex Ga of gaze area G. The upper area Iu and the right side area Ir are separated by line segment L2 connecting the vanishing point V and the vertex Gd of gaze area G. The lower area Id and the left side area Il are separated by line segment L3 connecting the vanishing point V and the vertex Gb of gaze area G. The lower area Id and the right side area Ir are separated by line segment L4 connecting the vanishing point V and the vertex Gc of gaze area G.

[0056] The inattentive driving detection area is not limited to the division shown in Fig. 10. For example, it may be as shown in Fig. 11. In Fig. 11, the inattentive driving detection area is divided by line segments extending each side of the gaze area G. The method of Fig. 11 simplifies the shape, thereby reducing the amount of processing required to divide the inattentive driving detection area.

[0057] Next, the determination of the inattentive tendency by the inattentive tendency determination unit 5 will be described. If the peak position detected by the visual saliency peak detection unit 4 is continuously outside the gaze area G for a predetermined time or more, it is determined that there is an inattentive tendency. Here, the predetermined time can be, for example, 2 seconds, but may be changed as appropriate. In other words, the inattentive tendency determination unit 5 determines whether the peak position has been continuously outside the range to be gazed at for a predetermined time or more.

[0058] Furthermore, the inattentiveness tendency determination unit 5 may determine that there is a tendency for the driver to look inattentively due to fixed objects when the inattentiveness detection area is the upper area Iu or the lower area Id. This is because, in an image captured of the area ahead from the vehicle, fixed objects such as buildings, traffic signals, signs, and streetlights are generally reflected in the upper area Iu, and road markings such as road signs are generally reflected in the lower area Id. On the other hand, moving objects other than the driver's own vehicle, such as vehicles traveling in other driving lanes, may be reflected in the left side area Il and the right side area Ir, making it difficult to determine the object of inattentiveness (whether it is a fixed object or a moving object) depending on the area.

[0059] The inattentive warning unit 6 issues a warning or the like based on the determination result of the inattentive tendency determination unit 5. The method of issuing the warning may be by displaying on a display device or the like that is visible to the driver or the like, or may be output as a sound or vibration. That is, the inattentive warning unit 6 functions as an inattentive output unit that outputs information that there is a tendency to look inattentively. In this embodiment, the warning is issued as information that there is a tendency to look inattentively. However, as information that there is a tendency to look inattentively, in addition to information on inattentiveness detection such as a flag, information on the time, position, etc. at the time may also be output to a storage medium or outside the information output device 1 via communication or the like. Furthermore, based on the determination result of the inattentive tendency determination unit 5, the result may be output as information related to near misses.

[0060] Next, the operation (information output method) of the information output device 1 configured as described above will be described with reference to the flowchart in Fig. 12. This flowchart can be configured as a program executed by a computer functioning as the information output device 1, thereby making it an information processing program. This information output program is not limited to being stored in a memory or the like possessed by the information output device 1, but may also be stored in a storage medium such as a memory card or optical disk.

[0061] First, the inattentive driving warning unit 6 determines whether the inattentive driving warning switch (SW) is ON or OFF (step S101). The inattentive driving warning SW is a switch that switches whether or not the inattentive driving warning unit 6 issues a warning. The inattentive driving warning SW is provided in the inattentive driving warning unit 6, and the switching is controlled by the inattentive driving tendency determination unit 5.

[0062] When the inattentive driving warning SW is ON (step S101; SW=ON), the inattentive driving warning unit 6 compares the warning timer threshold (step S102). The warning timer is a timer that measures the period for which the inattentive driving warning unit 6 issues a warning, and the warning timer threshold is a threshold that determines the period for which the warning is to be issued. In other words, the inattentive driving warning unit 6 issues a warning for only the period determined by the warning timer threshold.

[0063] If the warning timer threshold has been exceeded (step S102; threshold exceeded), the inattentive driving warning unit 6 turns off the inattentive driving warning SW to stop the warning timer (step S103), and step S104, which will be described later, is executed. If the warning timer threshold has not been exceeded (step S102; threshold not exceeded), nothing is done and step S104, which will be described later, is executed.

[0064] On the other hand, if the inattentive driving warning switch is OFF or if the process proceeds from steps S102 or S103 described above, the driving image acquisition unit 2 acquires a driving image (step S104), and the visual saliency processing unit 3 performs visual saliency image processing (acquisition of a visual saliency map) (step S105).Then, the visual saliency peak detection unit 4 acquires (detects) a peak position based on the visual saliency map acquired by the visual saliency processing unit 3 in step S105 (step S106).

[0065] Next, the inattentive driving tendency determination unit 5 sets a gaze area G and compares the gaze area G with the peak position acquired by the visual saliency peak detection unit 4 (step S107). If the comparison result shows that the peak position is outside the gaze area G (step S107; outside the gaze area), the inattentive driving tendency determination unit 5 determines whether the dwell timer has started or is stopped (step S108). The dwell timer is a timer that measures the time that the peak position stays outside the gaze area G. Note that the gaze area G may be set when the image is acquired in step S104.

[0066] If the dwell timer is stopped (step S108; stopped), the inattentive driving tendency determination unit 5 starts the dwell timer (step S109). On the other hand, if the dwell timer has already started (step S108; after starting), the inattentive driving tendency determination unit 5 compares the dwell timer threshold value (step S110). The dwell timer threshold value is a threshold value for the time that the peak position remains outside the gaze area G, and is set to 2 seconds, as described above.

[0067] If the dwell timer exceeds the threshold (step S110; threshold exceeded), the inattentive tendency determination unit 5 turns on the inattentive warning switch of the inattentive tendency warning unit 6 and starts the warning timer (step S111). Then, the inattentive tendency determination unit 5 stops the dwell timer (step S112). In other words, since the time that the peak position stayed outside the gaze area G was equal to or greater than the threshold, the inattentive tendency determination unit 6 starts the warning.

[0068] On the other hand, if the residence timer does not exceed the threshold value (step S110; threshold not exceeded), the inattentive driving tendency determination unit 5 does nothing and returns to step S101.

[0069] Furthermore, if the comparison result in step S107 shows that the peak position is within the gaze area G (step S107; within the gaze area), the inattentive driving tendency determination unit 5 stops the dwell timer (step S112).

[0070] As is clear from the above description, step S105 functions as an acquisition step, step S106 functions as a peak position detection step, step S107 functions as a gaze range setting step, and steps S107 to S111 functions as an inattentive driving output step.

[0071] According to this embodiment, the information output device 1 acquires a visual saliency map in time series, which is obtained by estimating the level of visual saliency in an image captured from a moving object and then a visual saliency peak detection unit 4 detects at least one peak position in the visual saliency map in time series. The inattentiveness tendency determination unit 5 then sets a gaze area G in the image, and if the peak position is consistently outside the gaze area G for a predetermined period of time or more, the inattentiveness warning unit 6 outputs information indicating a tendency for inattentiveness. This visual saliency map indicates the statistical likelihood of human gaze focusing. Therefore, the peak of the visual saliency map indicates the position that is statistically most likely to attract human gaze. Therefore, by using the visual saliency map, it is possible to detect inattentiveness with a simple configuration without measuring the actual driver's gaze.

[0072] In addition, the inattentiveness tendency determination unit 5 sets the gaze area G based on the vanishing point V of the image. In this way, it becomes possible to easily set the gaze area G without detecting, for example, a vehicle ahead.

[0073] Furthermore, if the inattentive tendency determination unit 5 determines that the peak position is located above or below the gaze area G for a predetermined period of time or more, the inattentive tendency warning unit 6 may output information indicating that there is a tendency for the driver to look away due to a fixed object. The area above the gaze area G is generally an area in which fixed objects such as buildings, traffic signals, signs, and streetlights are reflected, while the area below the gaze area G is generally an area in which road markings such as road signs are reflected. In other words, if the range includes the peak position, it is possible to identify that the object causing the driver to look away due to inattentiveness is a fixed object.

[0074] The visual saliency processing unit 3 includes an input unit 310 that converts an image into intermediate data that can be mapped, a nonlinear mapping unit 320 that converts the intermediate data into mapped data, and an output unit 330 that generates saliency estimation information indicating a saliency distribution based on the mapped data. The nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. This allows visual saliency to be estimated with low computational cost. Furthermore, the visual saliency estimated in this manner reflects the contextual attention state.

[0075] The gaze area G does not have to be set to a fixed range. For example, it may be changed according to the moving speed of the moving object. For example, it is known that the driver's field of vision narrows when traveling at high speed. Therefore, for example, the inattentiveness tendency determination unit 5 may acquire the vehicle speed from a speed sensor or the like mounted on the vehicle, and narrow the range of the gaze area G as the speed increases. Furthermore, since the appropriate inter-vehicle distance also changes according to the moving speed, the range of the gaze area G calculated by the calculation method described with reference to FIG. 9 may also be changed. The vehicle speed is not limited to being obtained from a speed sensor, but may also be obtained from an acceleration sensor or captured images.

[0076] Furthermore, the gaze area G may be changed depending on the driving position and situation of the vehicle, etc. In situations where attention to the surroundings is required, the gaze area G needs to be widened. For example, the range to be watched varies depending on the driving position, such as residential areas, main roads, and busy areas. In residential areas, there are few pedestrians, but attention must be paid to pedestrians suddenly running out into the street, so the gaze area G cannot be narrowed. On the other hand, on main roads, the driving speed increases, and the field of view becomes narrower as described above.

[0077] To give specific examples, school routes, parks, and areas near schools are considered to be at risk of children running out into the street. There are considered to be many pedestrians near train stations, schools, event locations, and tourist spots. There are considered to be many cyclists near bicycle parking lots and schools. There are considered to be many drunk people near entertainment districts. In locations such as these, attention to the surroundings is required, so the gaze area G can be widened and the area determined to be prone to inattentive driving can be narrowed. On the other hand, when driving on highways or in areas with low traffic volume and population density, driving speeds tend to be high, so the gaze area G can be narrowed and the area determined to be prone to inattentive driving can be widened.

[0078] Furthermore, the gaze area G may be changed depending on the time of day, events, etc. For example, during commuting hours, when attention to the surroundings is required, the gaze area G may be made wider than during normal times, narrowing the area where it is determined that the driver is likely to be looking away. Similarly, the gaze area G may be made wider from twilight to nighttime, narrowing the area where it is determined that the driver is likely to be looking away. On the other hand, the gaze area G may be narrowed in the middle of the night, widening the area where it is determined that the driver is likely to be looking away.

[0079] Furthermore, the gaze area G may be changed according to event information. For example, an event may be held in a place or at a time when there is a lot of foot traffic, so the gaze area G may be made wider than usual to loosen the judgment of the tendency to look away.

[0080] The inattentiveness tendency determination unit 5 acquires such location information from a means capable of determining the current position and the area in which the vehicle is traveling, such as a GPS receiver or map data, and associates the information with image data, thereby changing the range of the gaze area G. The information output device 1 may acquire time information internally or externally. Event information may be acquired from an external site, etc. Furthermore, a change may be determined using a combination of the position, time, and date, or any one of them.

[0081] Furthermore, the dwell timer threshold may be shortened when driving at high speeds, because even a short period of inattentive driving can be dangerous when driving at high speeds. [Example]

[0082] Next, an information processing device according to a second embodiment of the present invention will be described with reference to Figures 13 to 15. Note that the same parts as those in the first embodiment described above are given the same reference numerals and descriptions thereof will be omitted.

[0083] 13 shows a typical system configuration example of this embodiment. The system according to this embodiment includes an information processing device 1A and a server device 10. The information processing device 1A according to this embodiment is mounted on a vehicle V. The information processing device 1A and the server device 10 are capable of communicating with each other via a network N such as the Internet.

[0084] An information processing device 1A according to this embodiment is shown in Fig. 14. The information processing device 1A includes a driving image acquisition unit 2, a visual saliency processing unit 3, a visual saliency peak detection unit 4, an inattentive driving tendency determination unit 5A, and an output unit 7.

[0085] The running image acquisition unit 2, visual saliency processing unit 3, and visual saliency peak detection unit 4 are the same as those in the first embodiment. In addition to determining the tendency to look indirectly, the inattentiveness tendency determination unit 5A determines whether the inattentive object (peak position) when it is determined that there is a tendency to look indirectly is a habitual object. That is, the inattentiveness tendency determination unit 5A functions as a determination unit that determines whether the object corresponding to the peak position is a habitual object when the peak position has been out of the gaze area G (range to be gazed at) for a predetermined period of time or more.

[0086] A constant object refers to a fixed object such as a building, traffic signal, sign, street lamp, or road paint, as described in the first embodiment, and is an object that can be constantly observed from the video shooting location (a building or other object that is always present in that position).

[0087] In addition to determining whether the peak position is in the upper area Iu or the lower area Id, as explained in the first embodiment, the determination of a resident object is also made when the peak position is in the left side area Il or the right side area Ir.

[0088] If the peak position is in the upper area Iu or the lower area Id, it is possible to determine whether the object is a resident object based on the area alone. On the other hand, if the peak position is in the left side area Il or the right side area Ir, it is not possible to determine whether the object is a resident object based on the area alone, so object recognition is used to make the determination. Object recognition (also called object detection) can be performed using any known algorithm, and the specific method is not particularly limited.

[0089] Furthermore, in addition to object recognition, relative speed may also be used to determine whether a target object is a resident object. This involves calculating the relative speed between the vehicle speed and the frame-to-frame movement speed of the inattentive target, and then using this relative speed to determine whether the inattentive target object is a resident object. Here, the frame-to-frame movement speed of the inattentive target can be calculated by calculating the frame-to-frame movement speed of the peak position. If the calculated relative speed is equal to or greater than a predetermined threshold, the target object can be determined to be fixed in a certain position (a resident object).

[0090] As described above, a constantly present object is an object that is fixed at a certain position and can be constantly observed from the video shooting location (it is always present at that position, such as a building), so by making a judgment using this embodiment, a position where an object that may cause a driver to look away (peak position) is judged to be a constantly present object can be considered to be a position where a driver may always look away.

[0091] If the inattentive tendency determination unit 5 determines that the inattentive object is a constantly present object, the output unit 7 transmits the determination result to the server device 10. Alternatively, the output unit 7 may always transmit the presence or absence of a constantly present object as the determination result. At this time, if the determination is made in real time, the time of determination may be added, and if the determination is made from an image saved on a memory card or the like, the time and date the image was captured may be added. By adding time information, it is possible to extract points where inattentiveness is likely to occur depending on the time of day. For example, it is possible to determine points where inattentiveness is likely to occur due to buildings that are only visible during the day, points where inattentiveness is likely to occur due to lighting, points where inattentiveness is likely to occur due to events such as fireworks, etc. Information about the image capture point may also be added.

[0092] Server device 10 tally the determination results transmitted from information processing device 1A. For example, by tallying the determination results transmitted from information processing device 1A (vehicle) including location information, it is possible to extract points where inattentive driving is likely to occur. Furthermore, by including time information, it is possible to extract points where inattentive driving is likely to occur depending on the time of day.

[0093] The operation of the information processing device 1A according to this embodiment will be described with reference to the flowchart of Fig. 15. In Fig. 15, steps S104 to S110 and S112 are the same as those in Fig. 12. Also, steps S101 to S103 in Fig. 12 have been omitted because a warning is not essential in this embodiment.

[0094] 15, if the residence timer exceeds the threshold (step S110; exceeded threshold), the inattentiveness tendency determination unit 5 determines whether the vehicle is a resident object and transmits the determination result (step S111A). In step S111A, the inattentiveness tendency determination unit 5 determines whether the vehicle is a resident object by the above-mentioned object recognition or the like and transmits the determination result to the server device 10.

[0095] Although the steps relating to the warning are omitted in the flowchart shown in Fig. 15, a warning may also be given. In the case of giving a warning, the inattentive driving warning unit 6 shown in Fig. 1 may also be provided, steps S101 to S103 may also be executed, and step S111 in Fig. 12 may be executed before, after, or in parallel with step S111A.

[0096] Furthermore, in this embodiment, the information processing device 1A on the vehicle side determines whether a vehicle is a constantly present object, but the determination of whether a vehicle is a constantly present object may be performed on the server device 10 side. That is, the server device 10 may acquire a driving image or the like, determine whether a vehicle is a constantly present object for the image, and output the determination result to a storage device within the server device 10 or another server device or the like.

[0097] According to this embodiment, the information processing device 1A acquires a visual saliency map in time series by the visual saliency processing unit 3, which estimates the level of visual saliency within an image captured from a moving object. The visual saliency peak detection unit 4 detects at least one peak position in the visual saliency map in time series. The inattentiveness tendency determination unit 5 then sets a gaze area G in the image and, if the peak position is outside the gaze area G for a predetermined period of time or more, determines whether the peak position corresponds to a resident object. This makes it possible to determine whether an inattentive object detected based on the visual saliency map is a resident object that can be constantly observed from the video capture location (such as a building that is always present in that location) or a moving object. Therefore, it is possible to identify the inattentive object as at least a resident object.

[0098] Furthermore, the inattentiveness tendency determination unit 5 may determine whether the peak position is a resident object by object recognition when the peak position is located to the left or right of the gaze area G. By doing so, when the peak position is located to the left or right of the range to be gazed at, the peak position corresponds to either a resident object or a moving object, and therefore, by performing object recognition, it is possible to accurately determine whether the peak position is a resident object such as a building or a moving object such as a car.

[0099] Furthermore, the inattentiveness tendency determination unit 5 may acquire the moving speed of the moving object, and if the peak position is off to the left or right of the gaze area G, calculate the relative speed of the object indicated by the peak position based on the moving speed, and determine whether it is a resident object based on the relative speed. In this way, it is possible to determine whether the object is a resident object based on the relative speed, and the processing load during the determination can be reduced.

[0100] The device also includes an output unit 7 that outputs the determination result. In this way, it becomes possible to transmit the result of the determination made in the vehicle to the server device 10 or the like and tally it up.

[0101] Furthermore, the present invention is not limited to the above-described embodiments. In other words, a person skilled in the art can implement various modifications in accordance with conventionally known knowledge without departing from the gist of the present invention. As long as such modifications still include the information processing device of the present invention, they are of course included in the scope of the present invention. [Explanation of symbols]

[0102] 1. Information output device 2. Driving image acquisition unit 3 Visual saliency processing section (acquisition section) 4. Visual saliency peak detection unit (peak position detection unit) 5. Inattentiveness tendency determination unit (gaze range setting unit, inattentiveness output unit, speed acquisition unit, position acquisition unit, determination unit) 6. Inattentiveness Warning Section 7 Output section

Claims

1. an acquisition unit that acquires, in time series, visual saliency distribution information obtained by estimating the level of visual saliency within an image captured from a moving object; a peak position detection unit that detects at least one peak position in the visual saliency distribution information in time series; a gaze range setting unit that sets a range in the image that the driver of the moving object should gaze at; an inattentive driving output unit that outputs information indicating a tendency for the driver to look away if the peak position is continuously outside the range to be looked at for a predetermined period of time or more; An information output device comprising:

2. 2. The information output device according to claim 1, wherein the gaze range setting unit sets the gaze range based on a vanishing point of the image.

3. The information output device according to claim 1 or 2, characterized in that, if the peak position is located above or below the range to be watched for a predetermined period of time or more, the inattentive driving output unit outputs information indicating that there is a tendency for the driver to look away due to fixed objects on the ground.

4. a speed acquisition unit that acquires a moving speed of the moving object; 4. The information output device according to claim 1, wherein the gaze range setting unit changes the gaze range based on the moving speed.

5. a position acquisition unit that acquires position information of the moving object, 4. The information output device according to claim 1, wherein the gaze range setting unit changes the gaze range based on the position information.

6. The acquisition unit an input unit for converting the image into intermediate data that can be subjected to mapping processing; a nonlinear mapping unit that converts the intermediate data into mapping data; an output unit that generates saliency estimation information indicating a saliency distribution based on the mapping data, the nonlinear mapping unit includes a feature extraction unit that extracts features from the intermediate data, and an upsampling unit that upsamples the data generated by the feature extraction unit.

6. The information output device according to claim 1, wherein the information output device is a display device.

7. An information processing method executed by an information processing device that outputs predetermined information based on an image of the outside taken from a moving object, an acquisition step of acquiring, in time series, visual saliency distribution information obtained by estimating the level of visual saliency within the image based on the image; a peak position detection step of detecting at least one peak position in the visual saliency distribution information in time series; a gaze range setting step of setting a range in the image that the driver of the moving object should gaze at; an inattentive driving output step of outputting information indicating a tendency for inattentive driving when the peak position is continuously outside the range to be watched for a predetermined period of time or more; An information output method comprising:

8. An information output program for causing a computer to execute the information output method according to claim 7.

9. 9. A computer-readable storage medium storing the information output program according to claim 8.

Images