Risk information output device

Abstract

To accurately determine risk and output information on the risk.SOLUTION: In an information processor 1, a visual saliency calculation section 3 acquires a visual saliency map obtained by estimating degrees of visual saliency in an image for each entrance road from the image for each entrance road, which is a road for entering an intersection, and a sight line coordinate setting section 4 sets coordinates of ideal sight line in the image for each exit road, which is a road for exiting after entering the intersection, for the visual saliency map. A vector error calculation section 5 calculates a visual attention concentration degree Ps for each exit road in the image on the basis of the visual saliency map and the ideal sight line, and an output section 6 outputs information on risk at the intersection on the basis of the visual attention concentration degree Ps calculated for each exit road.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a risk information output device that outputs risk information based on an image of the outside taken from a mobile body. [Background technology]

[0002] For example, it has been proposed to reduce the risk of traffic accidents by providing drivers with information on locations where there is a high risk of traffic accidents (accident risk locations). In this case, the setting of accident risk locations is done by taking into consideration both estimations that depend on physical attributes of the traffic environment such as traffic volume and natural phenomena such as time and weather, and locations where accidents have actually occurred.

[0003] Patent Document 1 describes a method for estimating the degree to which a driving environment is likely to cause eye fatigue from an image captured in the direction in which the vehicle is traveling, by acquiring an image captured in the direction in which the vehicle is traveling, estimating a position in the captured image at which the driver will naturally gaze, estimating a position in the captured image at which the driver should gaze when driving the vehicle, and estimating the visual load based on the positional relationship between the position at which the driver will naturally gaze and the position at which the driver should gaze. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5482737 Summary of the Invention [Problem to be solved by the invention]

[0005] It is also possible to determine whether or not there is a risk in the scene depicted in the video based on the visual load extracted using the method described in Patent Document 1. However, the method described in Patent Document 1 cannot reflect the contextual attention state, in which the gaze tends to unconsciously focus on objects such as signs and pedestrians contained in the image, and therefore there may be a discrepancy with the actual visual state of the driver, leaving room for improvement in the accuracy of the determination. Furthermore, in the case of a point where there are approaches from multiple directions, such as an intersection, it is insufficient to determine the risk based on only a single direction.

[0006] One example of the problem to be solved by the present invention is that it is characterized by accurately determining risk and outputting information about the risk. [Means for solving the problem]

[0007] In order to solve the above problem, the invention described in claim 1 is characterized by comprising: an acquisition unit that acquires visual saliency distribution information for each entry road, which is a road when entering an intersection, from an image of the entry road by estimating the level of visual saliency in the image; a gaze position setting unit that sets a reference gaze position in the image for the visual saliency distribution information for each exit road, which is a road used to exit the intersection after entering; a visual attention concentration calculation unit that calculates a visual attention concentration level for each exit road in the image based on the visual saliency distribution information and the gaze position; and an output unit that outputs information regarding the risk at the intersection based on the safety index calculated for each exit road.

[0008] The invention described in claim 6 is characterized by comprising an acquisition unit that acquires visual saliency distribution information obtained by estimating the level of visual saliency in an image from an image of entering a curve on a road; a gaze position setting unit that sets reference gaze positions in the image in the curvature direction of the curve and in a direction different from the curvature direction of the curve for the visual saliency distribution information; a visual attention concentration calculation unit that calculates a degree of visual attention concentration in the curvature direction and a direction different from the curvature direction in the image based on the visual saliency distribution information and the gaze position; and an output unit that outputs information regarding risks at the curve based on the degree of visual attention concentration calculated for each of the exit routes.

[0009] The invention described in claim 7 is a risk information output method executed by a risk information output device that outputs risk information about an intersection, and is characterized by including: an acquisition step of acquiring visual saliency distribution information for each entry road, which is a road used to enter the intersection, by estimating the level of visual saliency within the image from the image of each entry road; a gaze position setting step of setting a reference gaze position in the image for each exit road, which is a road used to exit the intersection after entering, for the visual saliency distribution information; a visual attention concentration degree calculation step of calculating a visual attention concentration degree for each exit road in the image based on the visual saliency distribution information and the gaze position; and an output step of outputting information related to the risk at the intersection based on the visual attention concentration degree calculated for each exit road.

[0010] The invention as set forth in claim 8 is characterized in that the risk information output method as set forth in claim 7 is executed by a computer.

[0011] The invention as set forth in claim 9 is characterized in that the risk information output program as set forth in claim 8 is stored.

[0012] The invention described in claim 10 is a risk information output method executed by a risk information output device that outputs risk information about a curve on a road, and is characterized by including: an acquisition step of acquiring visual saliency distribution information obtained by estimating the level of visual saliency in an image from an image of entering the curve; a gaze position setting step of setting reference gaze positions in the image in the curvature direction of the curve and in a direction different from the curvature direction of the curve, for the visual saliency distribution information; a visual attention concentration calculation step of calculating a visual attention concentration level in the image in the curvature direction and in a direction different from the curvature direction, based on the visual saliency distribution information and the gaze position; and an output step of outputting information related to the risk at the curve based on the calculated visual attention concentration level.

[0013] The invention as set forth in claim 11 is characterized in that the risk information output method as set forth in claim 10 is executed by a computer.

[0014] The invention as set forth in claim 12 is characterized in that the risk information output program as set forth in claim 11 is stored. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a functional configuration diagram of an information processing device according to a first embodiment of the present invention. [Figure 2] 2 is a block diagram illustrating the configuration of a visual saliency calculation unit shown in FIG. 1. FIG. [Figure 3] 1A is a diagram illustrating an example of an image input to a determination device, and FIG. 1B is a diagram illustrating an example of a visual saliency map estimated for FIG. 1A. [Figure 4] 2 is a flowchart illustrating a processing method of the visual saliency calculation unit shown in FIG. 1; [Figure 5] FIG. 2 is a diagram illustrating in detail an example of the configuration of a nonlinear mapping unit. [Figure 6] FIG. 2 is a diagram illustrating the configuration of an intermediate layer. [Figure 7]10(a) and 10(b) are diagrams illustrating examples of convolution processing performed by a filter. [Figure 8] (a) is a diagram for explaining the processing of the first pooling unit, (b) is a diagram for explaining the processing of the second pooling unit, and (c) is a diagram for explaining the processing of the unpooling unit. [Figure 9] FIG. 10 is an explanatory diagram of a vector error. [Figure 10] 2 is an example of an image input to the image input unit shown in FIG. 1 and a visual saliency map obtained from the image. [Figure 11] 10 is a graph showing an example of temporal changes in visual attention concentration level. [Figure 12] 2 is a flowchart of the operation of the information processing device shown in FIG. [Figure 13] FIG. 10 is a diagram showing an example of an intersection targeted by an information processing device according to a second embodiment of the present invention. [Figure 14] FIG. 14 is a diagram showing calculations of the visual attention concentration level for the intersection shown in FIG. 13 by setting an ideal line of sight. [Figure 15] 15 is a graph showing temporal changes in the visual attention concentration level shown in FIG. 14. [Figure 16] 16 is a graph showing the results of calculating the ratio of the visual attention concentration level shown in FIG. 15 when turning right or left and when going straight. [Figure 17] 10 is a flowchart of the operation of an information processing device according to a second exemplary embodiment of the present invention. [Figure 18] This is an example of a curve that is the target of a modification of the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] A risk information output device according to one embodiment of the present invention will be described below. In this risk information output device, an acquisition unit acquires visual saliency distribution information for each entry road, which is a road used when entering an intersection, by estimating the level of visual saliency within the image from the image of each entry road. A gaze position setting unit sets a reference gaze position in the image for each exit road, which is a road used to exit the intersection after entering, based on the visual saliency distribution information. A visual attention concentration calculation unit calculates a visual attention concentration level for each exit road in the image based on the visual saliency distribution information and the gaze position. An output unit outputs risk information for the intersection based on the visual attention concentration level calculated for each exit road. In this manner, it is possible to evaluate the risk of a target intersection and output risk-related information.

[0017] The output unit may also output risk information based on the ratio of the degree of visual attention concentration on a straight exit route to the safety index of a right or left turn exit route, among the exit routes. In this way, it is possible to evaluate whether attention is more likely to be focused when going straight or when turning right or left, and output the evaluation result.

[0018] The output unit may also output risk information based on a temporal change in the degree of visual attention concentration, thereby detecting, for example, a sudden change in the degree of visual attention concentration and outputting the result as risk-related information.

[0019] Furthermore, when a route leading to an exit route with a low calculated degree of visual attention concentration overlaps with a route leading to an exit route with a low calculated degree of visual attention concentration when the exit route is used as the entrance route, the output unit may output risk information for the route. In this way, for example, information on high-risk routes can be output for the routes at the intersection.

[0020] The acquisition unit may also include an input unit that converts the image into intermediate data that can be mapped, a nonlinear mapping unit that converts the intermediate data into mapped data, and an output unit that generates saliency estimation information indicating a saliency distribution based on the mapped data, and the nonlinear mapping unit may include a feature extraction unit that extracts features from the intermediate data and an upsampling unit that upsamples the data generated by the feature extraction unit. This allows visual saliency to be estimated with low computational cost. Furthermore, the visual saliency estimated in this manner reflects a contextual attention state.

[0021] In another embodiment of the risk information output device of the present invention, an acquisition unit acquires visual saliency distribution information obtained by estimating the level of visual saliency in an image of a vehicle entering a curve on a road, and a gaze position setting unit sets reference gaze positions in the image in the curvature direction of the curve and in a direction different from the curvature direction of the curve, respectively, for the visual saliency distribution information. A visual attention concentration calculation unit calculates visual attention concentration levels in the curvature direction and a direction different from the curvature direction in the image based on the visual saliency distribution information and the gaze position. An output unit outputs information about the risk at the curve based on the visual attention concentration levels calculated for each exit path. This makes it possible to evaluate the risk of a target curve and output risk-related information.

[0022] In one embodiment of the risk information output method, in an acquisition step, visual saliency distribution information is acquired for each entry road, which is a road used when entering an intersection, by estimating the level of visual saliency within the image from the image of each entry road; in a gaze position setting step, a reference gaze position in the image is set for each exit road, which is a road used to exit the intersection after entering, for the visual saliency distribution information; in a visual attention concentration degree calculation step, a visual attention concentration degree for each exit road in the image is calculated based on the visual saliency distribution information and the gaze position; and in an output step, risk information for the intersection is output based on the visual attention concentration degree calculated for each exit road. In this manner, it is possible to evaluate the risk of a target intersection and output risk-related information.

[0023] Furthermore, the risk information output method described above is executed by a computer, which makes it possible to use the computer to evaluate the risk of a target intersection and output information related to the risk.

[0024] The risk information output program may be stored in a computer-readable storage medium, which allows the program to be distributed as a standalone program rather than being incorporated into a device, and allows for easy version upgrades.

[0025] In another embodiment of the risk information output method of the present invention, an acquisition step acquires visual saliency distribution information obtained by estimating the level of visual saliency in an image of entering a curve on a road, from the image; a gaze position setting step sets reference gaze positions in the image in the curvature direction of the curve and in a direction different from the curvature direction of the curve, respectively, for the visual saliency distribution information; a visual attention concentration calculation step calculates visual attention concentration levels in the curvature direction and a direction different from the curvature direction in the image based on the visual saliency distribution information and the gaze position; and an output step outputs information about the risk at the curve based on the visual attention concentration levels calculated for each exit path. In this way, it is possible to evaluate the risk of a target curve and output information about the risk.

[0026] Furthermore, the risk information output method described above is executed by a computer, which makes it possible to use the computer to evaluate the risk of a target curve and output information relating to the risk.

[0027] The risk information output program may be stored in a computer-readable storage medium, which allows the program to be distributed as a standalone program rather than being incorporated into a device, and allows for easy version upgrades. [Example]

[0028] An information processing device according to a first embodiment of the present invention will be described with reference to Figures 1 to 12. The information processing device according to this embodiment is not limited to being installed in a mobile object such as an automobile, but may also be configured as a server device or the like installed in a business establishment, etc. In other words, analysis does not need to be performed in real time, and analysis may be performed after driving, etc.

[0029] As shown in FIG. 1, the information processing device 1 includes an image input unit 2, a visual saliency calculation unit 3, a gaze coordinate setting unit 4, a vector error calculation unit 5, and an output unit 6.

[0030] The image input unit 2 receives an image (e.g., a moving image) captured by a camera or the like, and outputs the image as image data. The input moving image is output as image data broken down into a time series, such as for each frame. Although still images may be input as images to the image input unit 2, it is preferable to input them as an image group consisting of a plurality of still images in time series.

[0031] The images input to the image input unit 2 include, for example, images captured in the direction of travel of the vehicle. In other words, images are taken of the outside world continuously from a moving body. These images may be so-called panoramic images or images acquired using multiple cameras, and may include images that include angles other than the direction of travel, such as 180° or 360° in the horizontal direction. Furthermore, the images input to the image input unit 2 are not limited to images captured by a camera, and may also be images read from a recording medium such as a hard disk drive or a memory card.

[0032] The visual saliency calculation unit 3 receives image data from the image input unit 2 and outputs a visual saliency map as visual saliency estimation information (described later). That is, the visual saliency calculation unit 3 functions as an acquisition unit that acquires a visual saliency map (visual saliency distribution information) obtained by estimating the level of visual saliency based on an image of the outside captured from a moving object.

[0033] FIG. 2 is a block diagram illustrating an example of the configuration of the visual saliency calculation unit 3. The visual saliency calculation unit 3 according to this embodiment includes an input unit 310, a nonlinear mapping unit 320, an output unit 330, and a storage unit 390. The input unit 310 converts an image into intermediate data that can be subjected to mapping processing. The nonlinear mapping unit 320 converts the intermediate data into mapped data. The output unit 330 generates saliency estimation information indicating a saliency distribution based on the mapped data. The nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. The storage unit 390 stores image data input from the image input unit 2, filter coefficients (described later), and the like. These are described in detail below.

[0034] FIG. 3(a) is a diagram illustrating an example of an image input to the visual saliency calculation unit 3, and FIG. 3(b) is a diagram illustrating an example of an image showing a visual saliency distribution estimated for FIG. 3(a). The visual saliency calculation unit 3 according to this embodiment is a device that estimates the visual saliency of each part in an image. Visual saliency means, for example, how easily something stands out or how easily it attracts attention. Specifically, visual saliency is expressed as a probability or the like. Here, the magnitude of the probability corresponds to, for example, the probability that a person viewing the image will direct their gaze to that position.

[0035] 3(a) and 3(b) correspond to each other in position. In FIG. 3(a), the higher the visual saliency, the higher the brightness displayed in FIG. 3(b). The image showing the visual saliency distribution as shown in FIG. 3(b) is an example of a visual saliency map output by the output unit 330. In the example shown in this figure, visual saliency is visualized using brightness values ​​of 256 levels. An example of a visual saliency map output by the output unit 330 will be described in detail later.

[0036] FIG. 4 is a flowchart illustrating the operation of the visual saliency calculation unit 3 according to this embodiment. The flowchart shown in FIG. 4 is part of an information processing method executed by a computer, and includes an input step S110, a nonlinear mapping step S120, and an output step S130. In the input step S110, an image is converted into intermediate data that can be mapped. In the nonlinear mapping step S120, the intermediate data is converted into mapped data. In the output step S130, visual saliency estimation information (visual saliency distribution information) indicating a saliency distribution is generated based on the mapped data. Here, the nonlinear mapping step S120 includes a feature extraction step S121 that extracts features from the intermediate data, and an upsampling step S122 that upsamples the data generated in the feature extraction step S121.

[0037] Returning to FIG. 2, each component of the visual saliency calculation unit 3 will be described. In input step S110, the input unit 310 acquires an image and converts it into intermediate data. The input unit 310 acquires image data from the image input unit 2. The input unit 310 then converts the acquired image into intermediate data. The intermediate data is not particularly limited as long as it is data that can be accepted by the nonlinear mapping unit 320, and is, for example, a high-dimensional tensor. Furthermore, the intermediate data is, for example, data in which the brightness of the acquired image is normalized, or data in which each pixel of the acquired image is converted into a brightness gradient. In input step S110, the input unit 310 may further perform noise removal, resolution conversion, etc. on the image.

[0038] In the nonlinear mapping step S120, the nonlinear mapping unit 320 acquires intermediate data from the input unit 310. The nonlinear mapping unit 320 then converts the intermediate data into mapping data. Here, the mapping data is, for example, a high-dimensional tensor. The mapping process performed on the intermediate data by the nonlinear mapping unit 320 is, for example, a mapping process that can be controlled by parameters, and is preferably a process using a function, a functional, or a neural network.

[0039] Fig. 5 is a diagram illustrating a detailed configuration of the nonlinear mapping unit 320, and Fig. 6 is a diagram illustrating a configuration of the hidden layer 323. As described above, the nonlinear mapping unit 320 includes a feature extraction unit 321 and an upsampling unit 322. The feature extraction unit 321 performs the feature extraction step S121, and the upsampling unit 322 performs the upsampling step S122. In the example shown in this figure, at least one of the feature extraction unit 321 and the upsampling unit 322 is configured to include a neural network including a plurality of hidden layers 323. In the neural network, a plurality of hidden layers 323 are connected.

[0040] In particular, the neural network is preferably a convolutional neural network. Specifically, each of the multiple hidden layers 323 includes one or more convolutional layers 324. In the convolutional layers 324, input data is convolved by multiple filters 325, and activation processing is performed on the outputs of the multiple filters 325.

[0041] 5, feature extraction unit 321 is configured to include a neural network including a plurality of hidden layers 323, and a first pooling unit 326 is provided between the plurality of hidden layers 323. Furthermore, upsampling unit 322 is configured to include a neural network including a plurality of hidden layers 323, and an unpooling unit 328 is provided between the plurality of hidden layers 323. Furthermore, feature extraction unit 321 and upsampling unit 322 are connected to each other via a second pooling unit 327 that performs overlap pooling.

[0042] In the example shown in this figure, each intermediate layer 323 is made up of two or more convolutional layers 324. However, at least some of the intermediate layers 323 may be made up of only one convolutional layer 324. Adjacent intermediate layers 323 are separated by any of a first pooling unit 326, a second pooling unit 327, and an unpooling unit 328. Here, when an intermediate layer 323 includes two or more convolutional layers 324, it is preferable that the number of filters 325 in those convolutional layers 324 be equal to each other.

[0043] In this figure, an "A×B" hidden layer 323 is composed of B convolution layers 324, and each convolution layer 324 includes A convolution filters for each channel. Such a hidden layer 323 is also referred to as an "A×B hidden layer" below. For example, a 64×2 hidden layer 323 is composed of two convolution layers 324, and each convolution layer 324 includes 64 convolution filters for each channel.

[0044] In the example shown in the figure, the feature extraction unit 321 includes a 64×2 hidden layer 323, a 128×2 hidden layer 323, a 256×3 hidden layer 323, and a 512×3 hidden layer 323, in this order. The upsampling unit 322 includes a 512×3 hidden layer 323, a 256×3 hidden layer 323, a 128×2 hidden layer 323, and a 64×2 hidden layer 323, in this order. The second pooling unit 327 connects the two 512×3 hidden layers 323 to each other. The number of hidden layers 323 constituting the nonlinear mapping unit 320 is not particularly limited and can be determined, for example, according to the number of pixels in the image data.

[0045] Note that this diagram shows an example of the configuration of the nonlinear mapping unit 320, and the nonlinear mapping unit 320 may have other configurations. For example, a 64×1 intermediate layer 323 may be included instead of the 64×2 intermediate layer 323. Reducing the number of convolutional layers 324 included in the intermediate layer 323 may further reduce the computational cost. Also, for example, a 32×2 intermediate layer 323 may be included instead of the 64×2 intermediate layer 323. Reducing the number of channels in the intermediate layer 323 may further reduce the computational cost. Furthermore, both the number of convolutional layers 324 and the number of channels in the intermediate layer 323 may be reduced.

[0046] Here, in the multiple intermediate layers 323 included in the feature extraction unit 321, it is preferable that the number of filters 325 increases each time the data passes through the first pooling unit 326. Specifically, the first intermediate layer 323a and the second intermediate layer 323b are connected to each other via the first pooling unit 326, and the second intermediate layer 323b is located after the first intermediate layer 323a. The first intermediate layer 323a is configured with convolutional layers 324 in which the number of filters 325 for each channel is N1, and the second intermediate layer 323b is configured with convolutional layers 324 in which the number of filters 325 for each channel is N2. In this case, it is preferable that N2 > N1. It is more preferable that N2 = N1 × 2.

[0047] Also, in the plurality of intermediate layers 323 included in the upsampling unit 322, it is preferable that the number of filters 325 decreases every time it passes through the unpooling unit 328. Specifically, the third intermediate layer 323c and the fourth intermediate layer 323d are continuous with each other via the unpooling unit 328, and the fourth intermediate layer 323d is located after the third intermediate layer 323c. The third intermediate layer 323c is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N3, and the fourth intermediate layer 323d is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N4. At this time, it is preferable that N4 < N3 holds. More preferably, N3 = N4 × 2 holds.

[0048] In the feature extraction unit 321, image features having a plurality of levels of abstraction such as gradients and shapes are extracted as channels of the intermediate layer 323 from the intermediate data acquired from the input unit 310. FIG. 6 illustrates the configuration of the 64×2 intermediate layer 323. Referring to this figure, the processing in the intermediate layer 323 will be described. In the example of this figure, the intermediate layer 323 is composed of a first convolutional layer 324a and a second convolutional layer 324b, and each convolutional layer 324 includes 64 filters 325. In the first convolutional layer 324a, convolution processing using the filter 325 is performed on each channel of the data input to the intermediate layer 323. For example, when the image input to the input unit 310 is an RGB image, processing is performed on each of the three channels h 0 i (i = 1..3). Also, in the example of this figure, the filter 325 is a 64 types of 3×3 filters, that is, a total of 64×3 types of filters. As a result of the convolution processing, for each channel i, 64 results h 0 i,j (i = 1..3, j = 1..64) are obtained.

[0049] Next, activation processing is performed on the outputs of the plurality of filters 325 in the activation unit 329. Specifically, activation processing is performed on the sum of corresponding elements for the corresponding results j of all channels. By this activation processing, results h of 64 channels 1i (i=1..64), i.e., the output of the first convolution layer 324a, is obtained as the image feature. The activation process is not particularly limited, but a process using at least one of a hyperbolic function, a sigmoid function, and a rectified linear function is preferable.

[0050] Furthermore, the output data of the first convolution layer 324a is used as input data for the second convolution layer 324b, and the same processing as that of the first convolution layer 324a is performed in the second convolution layer 324b to obtain the result h of 64 channels. 2 i (i=1..64), that is, the output of the second convolutional layer 324b, is obtained as the image features. The output of the second convolutional layer 324b becomes the output data of this 64×2 hidden layer 323.

[0051] Here, the structure of the filter 325 is not particularly limited, but a 3x3 two-dimensional filter is preferable. Furthermore, the coefficients of each filter 325 can be set independently. In this embodiment, the coefficients of each filter 325 are stored in the memory unit 390, and the nonlinear mapping unit 320 can read and use them for processing. Here, the coefficients of the multiple filters 325 may be determined based on correction information generated and corrected using machine learning. For example, the correction information includes the coefficients of the multiple filters 325 as multiple correction parameters. The nonlinear mapping unit 320 can further use this correction information to convert the intermediate data into mapped data. The memory unit 390 may be provided in the visual saliency calculation unit 3 or external to the visual saliency calculation unit 3. Furthermore, the nonlinear mapping unit 320 may obtain the correction information from an external source via a communication network.

[0052] 7(a) and 7(b) are diagrams illustrating examples of convolution processing performed by the filter 325. Both of FIGS. 7(a) and 7(b) illustrate examples of 3×3 convolution. The example in FIG. 7(a) illustrates convolution processing using nearest neighbor elements. The example in FIG. 7(b) illustrates convolution processing using neighbor elements with a distance of two or more. Note that convolution processing using neighbor elements with a distance of three or more is also possible. It is preferable that the filter 325 performs convolution processing using neighbor elements with a distance of two or more. This is because it allows for the extraction of a wider range of features, thereby further improving the estimation accuracy of visual saliency.

[0053] The above has described the operation of the 64×2 hidden layer 323. The operations of the other hidden layers 323 (such as the 128×2 hidden layer 323, the 256×3 hidden layer 323, and the 512×3 hidden layer 323) are the same as the operation of the 64×2 hidden layer 323, except for the number of convolutional layers 324 and the number of channels. Furthermore, the operations of the hidden layer 323 in the feature extraction unit 321 and the hidden layer 323 in the upsampling unit 322 are also the same as those described above.

[0054] FIG. 8(a) is a diagram for explaining the processing of the first pooling unit 326, FIG. 8(b) is a diagram for explaining the processing of the second pooling unit 327, and FIG. 8(c) is a diagram for explaining the processing of the unpooling unit 328.

[0055] In the feature extraction unit 321, data output from the intermediate layer 323 is subjected to pooling processing for each channel in the first pooling unit 326, and then input to the next intermediate layer 323. The first pooling unit 326 performs, for example, non-overlapping pooling processing. FIG. 8(a) shows processing for associating four 2×2 elements 30 with one element 30 for a group of elements included in each channel. The first pooling unit 326 performs such association for all elements 30. Here, the four 2×2 elements 30 are selected so that they do not overlap with each other. In this example, the number of elements in each channel is reduced to one-fourth. Note that, as long as the number of elements is reduced in the first pooling unit 326, the number of elements 30 before and after the association is not particularly limited.

[0056] The data output from the feature extraction unit 321 is input to the upsampling unit 322 via the second pooling unit 327. The second pooling unit 327 performs overlap pooling on the output data from the feature extraction unit 321. FIG. 8(b) shows a process of associating four 2×2 elements 30 with one element 30 while overlapping some of the elements 30. That is, in repeated associations, some of the four 2×2 elements 30 in a certain association are also included in the four 2×2 elements 30 in the next association. The second pooling unit 327 shown in this figure does not reduce the number of elements. Note that the number of elements 30 before and after association in the second pooling unit 327 is not particularly limited.

[0057] The methods of processing performed by the first pooling unit 326 and the second pooling unit 327 are not particularly limited, but examples include matching in which the maximum value of four elements 30 is matched to one element 30 (max pooling) and matching in which the average value of four elements 30 is matched to one element 30 (average pooling).

[0058] The data output from the second pooling unit 327 is input to the hidden layer 323 in the upsampling unit 322. Then, the output data from the hidden layer 323 of the upsampling unit 322 undergoes unpooling processing for each channel in the unpooling unit 328, and is then input to the next hidden layer 323. FIG. 8(c) shows processing for expanding one element 30 into multiple elements 30. The method of expansion is not particularly limited, but an example is a method of duplicating one element 30 into four elements 30 (2 × 2).

[0059] The output data of the last hidden layer 323 of the upsampling unit 322 is output from the nonlinear mapping unit 320 as mapping data and input to the output unit 330. In the output step S130, the output unit 330 generates and outputs a visual saliency map by performing, for example, normalization or resolution conversion on the data acquired from the nonlinear mapping unit 320. The visual saliency map is, for example, an image (image data) that visualizes visual saliency using brightness values, as illustrated in FIG. 3(b). The visual saliency map may also be, for example, an image that is color-coded according to visual saliency, such as a heat map, or an image in which visual saliency regions with visual saliency higher than a predetermined standard are marked so as to be distinguishable from other positions. Furthermore, the visual saliency estimation information is not limited to map information displayed as an image or the like, but may also be a table or the like that lists information indicating visual saliency regions.

[0060] The gaze coordinate setting unit 4 sets an ideal gaze, which will be described later, on a visual saliency map. The ideal gaze is the gaze that a driver of a vehicle would direct along the direction of travel in an ideal traffic environment with no obstacles or other traffic participants. It is handled as an (x, y) coordinate in the image data and on the visual saliency map. In this embodiment, the ideal gaze is a fixed value, but it may be handled as a function of the speed or road friction coefficient that affects the stopping distance of a moving object, or may be determined using set route information. In other words, the gaze coordinate setting unit 4 functions as a gaze position setting unit that sets an ideal gaze (reference gaze position) in an image according to a predetermined rule.

[0061] The vector error calculation unit 4 calculates a vector error based on the visual saliency map output by the visual saliency calculation unit 3 and the ideal gaze set by the gaze coordinate setting unit 5 for the visual saliency map and the image, and calculates a visual attention concentration level Ps (described below) indicating the degree of visual attention concentration based on the vector error. That is, the vector error calculation unit 4 functions as a visual attention concentration level calculation unit that calculates the degree of visual attention concentration in an image based on the visual saliency distribution information and the gaze position.

[0062] Here, the vector error in this embodiment will be described with reference to FIG. 9. FIG. 9 shows an example of a visual saliency map. This visual saliency map is shown with 256 gradation brightness values ​​of H pixels x V pixels, and similarly to FIG. 3, pixels with higher visual saliency are displayed with higher brightness. In FIG. 9, the coordinates (x, y) of the ideal line of sight are (x, y)=(x im ,y im ), the vector error with the pixel at any coordinate (k, m) in the visual saliency map is calculated. When the coordinate with high brightness in the visual saliency map is far from the coordinate of the ideal gaze, it means that the position to be gazed at and the position that is actually easy to gaze at are far apart, and it can be said that the image is likely to distract visual attention. On the other hand, when the coordinate with high brightness is close to the coordinate of the ideal gaze, it means that the position to be gazed at and the position that is actually easy to gaze at are close, and it can be said that the image is likely to focus visual attention on the position to be gazed at.

[0063] Next, we will explain how to calculate the visual attention concentration level Ps in the vector error calculation unit 4. In this embodiment, the visual attention concentration level Ps is calculated by the following equation (1).

number

[0064] In equation (1), V vc is the pixel depth (brightness value), f w is the weighting function, d err indicates the vector error. This weighting function is, for example, V vcis a function that sets a weight based on the distance from the pixel showing the value of α to the coordinate of the ideal gaze. α is a coefficient that makes the visual attention concentration level Ps equal to 1 when the coordinate of the bright spot and the coordinate of the ideal gaze match in a visual saliency map (reference heat map) of one bright spot.

[0065] That is, the vector error calculation unit 5 (visual attention concentration calculation unit) calculates the degree of visual attention concentration based on the value of each pixel that constitutes the visual saliency map (visual saliency distribution information) and the vector error between the position of each pixel and the coordinate position of the ideal gaze (reference gaze position).

[0066] The visual attention concentration level Ps obtained in this way is the reciprocal of the weighted sum of the vector error of the coordinates of all pixels from the coordinates of the ideal gaze set on the visual saliency map and the relationship between the luminance value. This visual attention concentration level Ps is calculated to be a low value when the distribution of high luminance on the visual saliency map is far from the coordinates of the ideal gaze. In other words, the visual attention concentration level Ps can be said to be the concentration level relative to the ideal gaze.

[0067] Fig. 10 shows an example of an image input to the image input unit 2 and a visual saliency map obtained from that image. Fig. 10(a) is the input image, and (b) is the visual saliency map. In Fig. 10, if the coordinates of the ideal gaze are set on the road, for example, on a truck traveling ahead, the visual attention concentration level Ps in that case can be calculated.

[0068] The output unit 6 outputs information relating to the risk of the scene shown by the image for which the visual attention concentration level Ps was calculated based on the visual attention concentration level Ps calculated by the vector error calculation unit 5. For example, the information relating to risk may be such that a predetermined threshold is set for the visual attention concentration level Ps, and information indicating that the scene is high risk is output when the calculated visual attention concentration level Ps is equal to or lower than the threshold. For example, when the visual attention concentration level Ps calculated in FIG. 10 is equal to or lower than the threshold, it may be determined that the scene is high risk, and information indicating that there is risk (or high risk) may be output.

[0069] Furthermore, the output unit 6 may output information relating to risk based on the change over time in the visual attention concentration level Ps calculated by the vector error calculation unit 5. Fig. 11 shows an example of the change over time in the visual attention concentration level Ps. Fig. 11 shows the change in the visual attention concentration level Ps in a 12-second video. In Fig. 11, the visual attention concentration level Ps changes suddenly between approximately 6.5 seconds and approximately 7 seconds. This occurs, for example, when another vehicle cuts in front of the vehicle.

[0070] 11, a scene may be determined to be high risk by comparing the rate of change per short period of time or the change value of the visual attention concentration level Ps with a predetermined threshold, and information indicating the presence of risk (or high risk) may be output. Alternatively, the presence or absence of risk (high or low) may be determined based on a change pattern, for example, when the visual attention concentration level Ps temporarily drops and then rises.

[0071] Next, the operation (information processing method) of the information processing device 1 configured as described above will be described with reference to the flowchart in Fig. 12. This flowchart can be configured as a program executed by a computer that functions as the information processing device 1, thereby creating an information processing program. This information processing program is not limited to being stored in a memory or the like that the information processing device 1 has, but may also be stored in a storage medium such as a memory card or an optical disk.

[0072] First, the image input unit 2 outputs the input image as image data to the visual saliency calculation unit 3 (step S11). In this step, the image data input to the image input unit 2 is decomposed into a time series of image frames, etc., and input to the visual saliency calculation unit 3. In this step, image processing such as noise removal and geometric transformation may also be performed.

[0073] Next, the visual saliency calculation unit 3 acquires a visual saliency map (step S12). The visual saliency calculation unit 3 outputs the visual saliency map as shown in Fig. 3(b) in time series using the method described above.

[0074] Meanwhile, in parallel with step S12, the line-of-sight coordinate setting unit 4 sets the coordinates of the ideal line of sight (step S13). These coordinates are set to a fixed position such as forward gaze, as described above.

[0075] Next, the vector error calculation unit 5 calculates the visual attention concentration level Ps from the visual saliency map and the ideal gaze (step S14). That is, as described above, the vector error between the coordinates of the ideal gaze and the coordinates of the visual saliency map is calculated, and the visual attention concentration level Ps is calculated using equation (1) based on the vector error and the value of each pixel.

[0076] Next, the output unit 6 outputs the risk information (step S15). In this step, as described above, the risk information is output based on one calculated visual attention concentration level Ps or its change over time.

[0077] As is clear from the above description, step S12 functions as an acquisition step, step S13 functions as a gaze position setting step, and step S14 functions as a visual attention concentration level calculation step.

[0078] According to this embodiment, the information processing device 1 acquires a visual saliency map obtained by estimating the level of visual saliency in an image captured from a moving object using the visual saliency calculation unit 3, and the gaze coordinate setting unit 4 sets the coordinates of the ideal gaze at a predetermined fixed position. The vector error calculation unit 5 then calculates the visual attention concentration level Ps for the image based on the visual saliency map and the ideal gaze. By using the visual saliency map in this way, it is possible to reflect the contextual attention state, in which the gaze tends to unconsciously focus on objects such as signs and pedestrians included in the image. This makes it possible to accurately calculate indicators related to safety and risk.

[0079] Furthermore, the vector error calculation unit 5 calculates the visual attention concentration level Ps based on the value of each pixel constituting the visual saliency map and the vector error between the position of each pixel and the coordinate position of the ideal gaze. In this way, a value according to the difference between the position of high visual saliency and the ideal gaze is calculated as the visual attention concentration level Ps. Therefore, for example, the value of the visual attention concentration level Ps can be changed according to the distance between the position of high visual saliency and the ideal gaze.

[0080] The system also includes an output unit 6 that outputs risk information at the location indicated by the image based on the temporal change in the visual concentration level Ps. This makes it possible to output, for example, a location where the temporal change in the visual concentration level Ps is large as an accident risk location, etc. The risk information may also be output as information related to near misses based on the temporal change in the visual concentration level Ps.

[0081] The visual saliency calculation unit 3 includes an input unit 310 that converts an image into intermediate data that can be mapped, a nonlinear mapping unit 320 that converts the intermediate data into mapped data, and an output unit 330 that generates saliency estimation information indicating a saliency distribution based on the mapped data. The nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. This allows visual saliency to be estimated with low computational cost. Furthermore, the visual saliency estimated in this manner reflects the contextual attention state. [Example]

[0082] Next, a risk information output device according to a second embodiment of the present invention will be described with reference to Figures 13 to 16. Note that the same parts as those in the first embodiment described above will be given the same reference numerals and descriptions thereof will be omitted.

[0083] This embodiment has the same block configuration as that shown in Fig. 1. That is, the information processing device 1 functions as a risk information output device according to this embodiment. The image input from the image input unit 2 is an image of entering an intersection, and the method of risk determination in the output unit 6 is different.

[0084] An example of an intersection for which risk information is output in this embodiment is shown in Fig. 13. Fig. 13 shows an intersection that forms a four-way intersection (crossroads). Images of the intersection direction (travel direction) when entering this intersection from directions A, B, and C are shown. In other words, the image shown in Fig. 13 is an image when directions A, B, and C are each considered to be approach roads that are roads when entering the intersection.

[0085] For the image shown in Fig. 13, a visual saliency map is obtained as described in the first embodiment. Then, for the image, an ideal gaze is set for each traveling direction (straight ahead, right turn, and left turn), and a visual attention concentration level Ps is calculated for each ideal gaze (Fig. 14). That is, for each exit road that is the road to exit after entering an intersection, an ideal gaze (reference gaze position) in the image is set, and the vector error calculation unit 5 calculates the visual attention concentration level Ps for each ideal gaze.

[0086] FIG. 15 shows the temporal change in visual attention concentration level Ps when entering an intersection from each approach road. This temporal change is based on the calculation results of vector error calculation unit 5. In the graph of FIG. 15, the vertical axis represents visual attention concentration level Ps and the horizontal axis represents time, with the thick line representing going straight, the thin line representing a left turn, and the dashed line representing a right turn when the ideal line of sight is set for each direction of travel. FIG. 15(a) shows the case of entering from direction A, FIG. 15(b) shows the case of entering from direction B, and FIG. 15(c) shows the case of entering from direction C.

[0087] According to Figure 15, the visual concentration level Ps tends to decrease when approaching an intersection, but it can also drop sharply just before the intersection, as shown in Figure 15(b). Also, according to Figure 15, the visual concentration level Ps tends to be lower when the driver looks straight ahead to go straight, compared to when the driver turns his or her eyes to the left or right to turn right or left.

[0088] Next, the output unit 6 calculates the ratio of the visual attention concentration level Ps in the right or left direction to the visual attention concentration level Ps in the straight-ahead direction using the change over time in the visual attention concentration level Ps calculated in Fig. 15. The change in the calculated ratio is shown in Fig. 16. In the graph of Fig. 16, the vertical axis shows the ratio and the horizontal axis shows time, with the thick line showing the left turn / straight-ahead ratio (L / C) and the thin line showing the right turn / straight-ahead ratio (R / C). Fig. 16(a) shows the case of entering from direction A, Fig. 16(b) shows the case of entering from direction B, and Fig. 16(c) shows the case of entering from direction C. For example, in Fig. 16(a), AL is PS LA (A direction left visual attention concentration level) / PS CA (Directional A visual attention concentration level) AR is PS RA (A direction right visual attention concentration level) / PS CA (Directional visual attention concentration in direction A). BL , I BR ,I in Fig. 16(c) CL , I CR The meaning is the same, but the direction of approach is different.

[0089] According to Figure 16, I AL and I AR When the visual attention concentration ratio is smaller than 1, it can be said that the intersection is one where the driver's concentration (= visual attention concentration Ps) is lower when looking to turn right or left than when looking to go straight. Conversely, when the ratio is larger than 1, it can be said that the intersection is one where the driver's concentration is lower when looking to go straight.

[0090] Therefore, the output unit 6 can determine the risk state of the target intersection based on the temporal changes in the visual attention concentration level Ps and the ratio of the visual attention concentration levels Ps as described above, and output the determination result as information regarding the risk.

[0091] Next, the operation (information processing method) of the information processing device 1 according to this embodiment will be described with reference to the flowchart of FIG.

[0092] First, the image input unit 2 outputs the input image as image data to the visual saliency calculation unit 3 (step S21). In this step, as shown in FIG. 13, images of vehicles approaching the intersection from each approach road are acquired. At this time, it is advisable to acquire not only the images but also location information and time information at the same time. The location information makes it possible to determine the direction from which the vehicle is approaching, and the time information makes it possible to analyze the vehicle by time of day, such as morning, noon, or night, and also to calculate the speed of the vehicle approaching the intersection.

[0093] Next, the visual saliency calculation unit 3 acquires a visual saliency map (step S22). The visual saliency map is acquired based on images from each entry road. Meanwhile, in parallel with step S22, the gaze coordinate setting unit 4 sets the coordinates of the ideal gaze (step S23). The ideal gaze in step S23 is set for each traveling direction (exit road), as shown in FIG. 14.

[0094] Next, the vector error calculation unit 5 calculates the visual attention concentration level Ps from the visual saliency map and the ideal line of sight (step S24). In this step, the visual attention concentration level Ps is calculated for each exit route for which the ideal line of sight is set, as shown in Fig. 14. In addition, the visual attention concentration level Ps is calculated in time series, as shown in Fig. 15.

[0095] Next, the output unit 6 determines the risk of the intersection based on the visual attention concentration level Ps calculated in step S24 (step S25). For example, if the visual attention concentration level Ps changes suddenly as the intersection approaches and there are approaches from multiple directions, the intersection is determined to be a risky (or high-risk) intersection. Alternatively, if the ratio is less than 1 as the intersection approaches and there are approaches from multiple directions, the intersection is determined to be a risky (or high-risk) intersection. Alternatively, the intersection may be determined to be a risky (or high-risk) intersection if both of these conditions are met. Furthermore, the intersection may be determined to be a risky (or high-risk) intersection if the above conditions are met multiple times. "Multiple times" means that this flowchart has been executed multiple times based on images of the same intersection taken at different times (or by different vehicles).

[0096] Then, the output unit 6 outputs the determination result of step S25 as risk information (step S26).

[0097] As is clear from the above description, step S22 functions as an acquisition step, step S23 functions as a gaze position setting step, step S24 functions as a visual attention concentration level calculation step, and steps S25 and S26 function as output steps.

[0098] According to this embodiment, the information processing device 1 acquires a visual saliency map for each entry road, which is a road used when entering an intersection, by estimating the level of visual saliency within the image, and the gaze coordinate setting unit 4 sets the coordinates of the ideal gaze in the image for each exit road, which is a road used to exit the intersection after entering, in the visual saliency map.The vector error calculation unit 5 then calculates a visual attention concentration level Ps for each exit road in the image based on the visual saliency map and the ideal gaze, and the output unit 6 outputs risk information for the intersection based on the visual attention concentration level Ps calculated for each exit road.In this manner, it is possible to evaluate the risk of a target intersection and output risk information.

[0099] Furthermore, the output unit 6 outputs risk information based on the ratio of the visual attention concentration level Ps of the exit route going straight to the visual attention concentration level Ps of the exit route turning right or left. In this way, it is possible to evaluate which direction the driver's attention is more likely to be directed to, going straight or turning right or left, and output the evaluation result.

[0100] Furthermore, the output unit 6 may output risk information based on a temporal change in the visual concentration level Ps. In this way, it is possible to detect, for example, a sudden change in the visual concentration level Ps and output risk information.

[0101] In the second embodiment, for example, in FIG. 14, when entering an intersection from direction A, the visual attention concentration level Ps for turning right (toward direction B) decreases. When entering an intersection from direction B, the visual attention concentration level Ps for turning right or left (toward direction A or C) decreases. When entering an intersection from direction C, the visual attention concentration level Ps for turning right decreases.

[0102] In this case, for example, a route turning right from direction A and a route turning left from direction B are both routes in which the visual attention concentration level Ps is lower than other routes, and furthermore, when the entrance road and exit road are interchanged, they become the same route. Therefore, at this intersection, information regarding risk, such as "risky (or high risk)" may be output.

[0103] Although the second embodiment has been described with reference to an intersection, this concept can also be applied to a curve on a road. This will be described with reference to FIG.

[0104] FIG. 18 shows an example of a curved road. This road can be driven on a left curve from direction D (bottom of the figure) or on a right curve from direction E (left side of the figure). Here, for example, when entering the curve from direction D, an ideal gaze is set not only to the left, which is the direction in which the road curves, but also to the direction in which the road would have been if it had been going straight (direction D'), and the visual attention concentration level Ps is calculated for each. Similarly, when entering the curve from direction E, an ideal gaze is set not only to the right, which is the direction in which the road curves, but also to the direction in which the road would have been going straight (direction E'), and the visual attention concentration level Ps is calculated for each.

[0105] Then, based on the calculated visual attention concentration level Ps, risk can be determined based on time series changes, ratios, etc., in the same way as at intersections.

[0106] In addition, when the curvature of the curve is large as shown in Figure 18, the virtual ideal line of sight may be set not only in the straight-ahead direction but also in the direction opposite to the curvature of the curve. In Figure 18, if the vehicle is entering from direction D, the ideal line of sight may be set not only in direction D' but also in direction E' to calculate the visual attention concentration level Ps. In other words, the ideal line of sight may be set in a direction different from the curvature of the curve.

[0107] That is, the visual saliency calculation unit 3 obtains a visual saliency map obtained by estimating the level of visual saliency in an image taken when entering a curve on a road, and the gaze coordinate setting unit 4 sets the coordinates of the ideal gaze in the image in the curve direction and in a direction different from the curve direction for the visual saliency map.The vector error calculation unit 5 then calculates the visual attention concentration level Ps in the curve direction and in a direction different from the curve direction in the image based on the visual saliency map and the ideal gaze, and the output unit 6 outputs risk information for the curve based on the visual attention concentration level Ps calculated for each exit route.

[0108] In this way, it is possible to evaluate the risk of the target curve and output information relating to the risk.

[0109] Furthermore, the present invention is not limited to the above-described embodiments. In other words, a person skilled in the art can implement various modifications in accordance with conventionally known knowledge without departing from the gist of the present invention. As long as such modifications still include the information processing device of the present invention, they are of course included in the scope of the present invention. [Explanation of symbols]

[0110] 1. Information processing device (risk information output device) 2 Image input section 3. Visual saliency calculation unit (acquisition unit) 4 Line-of-sight coordinate setting section (line-of-sight position setting section) 5. Vector error calculation unit (visual attention concentration calculation unit)

Claims

1. an acquisition unit that acquires visual saliency distribution information obtained by estimating the level of visual saliency in an image captured of an intersection from an approach road that is a road when a mobile object enters the intersection; a gaze position setting unit that sets a reference gaze position, which is a gaze position of a driver of the moving object in the image, for each exit road that is a road that is used to exit the intersection after entering the intersection, with respect to the visual saliency distribution information; a visual attention concentration degree calculation unit that calculates a visual attention concentration degree for each of the exit paths in the image based on the visual saliency distribution information and the gaze position; an output unit that outputs information about a risk at the intersection based on the degree of visual attention concentration calculated for each of the exit routes; A risk information output device comprising:

2. The risk information output device described in claim 1, characterized in that the output unit outputs information regarding the risk based on the ratio of the visual attention concentration level for a straight exit route to the visual attention concentration level for an exit route that turns right or left among the exit routes.

3. 2. The risk information output device according to claim 1, wherein the output unit outputs information related to the risk based on a change over time in the degree of concentration of visual attention.

4. The risk information output device of claim 1, characterized in that if a route leading to an exit route with a lower calculated level of visual attention concentration among the exit routes for one of the entrance routes at the intersection overlaps with a route leading to an exit route with a lower calculated level of visual attention concentration when the exit route is used as the entrance route, the output unit outputs the risk information for that route.

5. The acquisition unit an input unit for converting the image into intermediate data that can be subjected to mapping processing; a nonlinear mapping unit that converts the intermediate data into mapping data; an output unit that generates saliency estimation information indicating a saliency distribution based on the mapping data, the nonlinear mapping unit includes a feature extraction unit that extracts features from the intermediate data, and an upsampling unit that upsamples the data generated by the feature extraction unit.

5. The risk information output device according to claim 1, wherein the risk information output device is a device for outputting risk information to a user.

6. A risk information output method executed by a risk information output device that outputs risk information about an intersection, an acquisition step of acquiring visual saliency distribution information obtained by estimating the level of visual saliency in an image captured of the intersection from an approach road, which is a road used when a mobile object enters the intersection; a gaze position setting step of setting a reference gaze position, which is a gaze position of a driver of the moving object in the image, for each exit road that is a road that is used to exit the intersection after entering the intersection, with respect to the visual saliency distribution information; a visual attention concentration calculation step of calculating a visual attention concentration level for each of the exit paths in the image based on the visual saliency distribution information and the gaze position; an output step of outputting information about a risk at the intersection based on the degree of visual attention concentration calculated for each of the exit routes; A risk information output method comprising:

7. A risk information output program that causes a computer to execute the risk information output method according to claim 6.

8. A computer-readable storage medium storing the risk information output program according to claim 7.

Images