Determination device

Abstract

To detect an object that can be missed visually.SOLUTION: A determination device 1 generates a visual saliency map obtained by estimating the level of visual saliency from image data which visual saliency extraction means 3 obtained by taking an image of the outside from such a mobile body as a vehicle. The type of an object to be detected is set in object region detection means 4, and the set type of the object is detected from the image data. Oversight determination means 5 determines an oversight of the object detected by the object region detection means 4 on the basis of the visual saliency map.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a determination device for determining the possibility of overlooking an object based on an image captured from a moving object. [Background technology]

[0002] Conventionally, various methods for calling attention to a driver of a moving body, such as a vehicle, have been proposed. For example, Patent Document 1 describes a road sign visibility determination system including a visibility estimation unit, a visibility completion time calculation unit, and a visibility determination unit.

[0003] To explain Patent Document 1 in detail, a visibility estimation unit calculates the complexity of a road sign based on the contents of the road sign, and estimates the visibility of the road sign according to the calculated complexity of the road sign. Next, a visibility completion time calculation unit calculates the distance from the driver to the road sign, and calculates the visibility completion time required for the driver to visually recognize the road sign using the vehicle speed, the visibility of the road sign, and the calculated distance from the driver to the road sign. Then, a visibility determination unit calculates the gaze time during which the driver continuously gazes at the road sign based on vehicle position information, road sign information, and gaze information including the driver's gaze direction, and determines whether the driver recognizes the contents of the road sign based on the gaze time and the visibility completion time. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-111469 A Summary of the Invention [Problem to be solved by the invention]

[0005] The road sign visibility judgment system described in Patent Document 1 targets road signs and does not take into consideration other vehicles, pedestrians, and other moving objects. Since there are usually other moving objects (cars, motorcycles, bicycles, pedestrians, etc.) on the road, it is desirable to draw attention to overlooking these as well. In addition, the road sign visibility judgment system described in Patent Document 1 needs to detect the driver's line of sight, and therefore equipment for detecting the line of sight must be installed inside the vehicle.

[0006] One example of a problem to be solved by the present invention is the detection of an object that may be visually overlooked. [Means for solving the problem]

[0007] In order to solve the above problem, the system is characterized by comprising a generation unit that generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image captured of the outside from a moving body, a setting unit that sets the type of object to be detected, an object detection unit that detects objects of the set type from the image, and a determination unit that makes a determination of the possibility of an object being overlooked for an object detected by the object detection unit based on the sum of the visual saliency values ​​within the object's area in the visual saliency distribution information divided by the area of ​​the object.

[0008] The invention described in claim 7 is characterized in that it comprises a generation unit that generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image of the outside taken from a moving body, a setting unit that sets the type of object to be detected, an object detection unit that detects objects of the type set in the setting unit in an area including the imaging range of the image, and a judgment unit that makes a judgment on the possibility of overlooking an object detected by the object detection unit based on a value obtained by dividing the sum of the visual saliency values ​​within the object area in the visual saliency distribution information by the area of ​​the object area.

[0009] The invention described in claim 8 is a determination method executed by a determination device that performs an oversight possibility determination based on an image captured from a moving object, and estimates the level of visual saliency from the image. The method includes a generating process for generating visual saliency distribution information obtained by the above process; a setting process for setting the type of object to be detected; an object detection process for detecting objects of the set type from the image; and a determining process for determining the likelihood of an object detected in the object detection process being overlooked based on a value obtained by dividing the sum of the visual saliency values ​​within an object area in the visual saliency distribution information by an area of ​​the object area.

[0010] The invention as recited in claim 9 is characterized in that the determination method as recited in claim 8 is executed by a computer.

[0011] The invention as recited in claim 10 is characterized in that the determination program as recited in claim 9 is stored.

[0012] The invention described in claim 11 is a determination method executed by a determination device that performs an oversight possibility determination based on an image captured of the outside from a moving body, and is characterized in that it includes a generation process of generating visual saliency distribution information obtained by estimating the level of visual saliency from the image, a setting process of setting the type of object to be detected, an object detection process of detecting objects of the type set in the setting process in an area including the imaging range of the image, and a determination process of performing an oversight possibility determination for objects detected in the object detection process based on a value obtained by dividing the sum of the visual saliency values ​​within the object area in the visual saliency distribution information by the area of ​​the area.

[0013] The invention as recited in claim 12 is characterized in that the determination method as recited in claim 11 is executed by a computer.

[0014] The invention as recited in claim 13 is characterized in that the judgment program as recited in claim 12 is stored. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is a functional configuration diagram of a determination device according to an embodiment of the present invention. [Diagram 2]2 is a block diagram illustrating a configuration of a visual saliency extraction unit shown in FIG. 1. [Diagram 3] FIG. 1A is a diagram illustrating an example of an image to be input to a determination device, and FIG. 1B is a diagram illustrating an example of a visual saliency map estimated for FIG. [Figure 4] 2 is a flowchart illustrating a processing method of the visual saliency extraction means shown in FIG. 1; [Diagram 5] FIG. 2 is a diagram illustrating in detail an example of the configuration of a nonlinear mapping unit. [Figure 6] FIG. 2 is a diagram illustrating a configuration of an intermediate layer. [Figure 7] 13A and 13B are diagrams illustrating an example of convolution processing performed in a filter. [Figure 8] FIG. 1A is a diagram for explaining the processing of a first pooling unit, FIG. 1B is a diagram for explaining the processing of a second pooling unit, and FIG. 1C is a diagram for explaining the processing of an unpooling unit. [Figure 9] 2 is a flowchart of the operation of a determination unit shown in FIG. 1. [Figure 10] 5 is a diagram illustrating an example of region information output from an object region detection means; FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, a determination device according to an embodiment of the present invention will be described. In the determination device according to an embodiment of the present invention, a generation unit generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image captured from a moving body of the outside. Meanwhile, a type of object to be detected is set in a setting unit, and an object detection unit detects an object of the set type from the image. Then, the determination unit performs an oversight possibility determination based on the visual saliency distribution information for the object detected by the object detection unit. In this way, it is possible to determine the oversight possibility by combining the visual saliency distribution information and the object detection. Therefore, it is possible to detect an object that may be visually overlooked. In addition, since the oversight possibility can be determined only from an image captured from the outside of a moving body, it can be determined from an image of a drive recorder, for example, and gaze detection is not required.

[0017] In addition, the determination unit may compare the visual saliency distribution information with the object detected by the object detection unit to determine the possibility of overlooking the object. In this way, the possibility of overlooking the object can be determined by comparing the distribution of the visual saliency distribution information with the captured image.

[0018] The determination unit may also determine that, with respect to the objects detected by the object detection unit, objects that do not overlap with areas determined to have high visual saliency are likely to be overlooked. In this way, it is possible to determine that objects located in parts of the image that do not have high visual saliency are likely to be overlooked.

[0019] The generation unit may include an input unit that converts the image into intermediate data that can be mapped, a nonlinear mapping unit that converts the intermediate data into mapped data, and an output unit that generates saliency estimation information indicating a saliency distribution based on the mapped data, and the nonlinear mapping unit may include a feature extraction unit that extracts features from the intermediate data, and an upsampling unit that upsamples the data generated by the feature extraction unit. In this way, visual saliency can be estimated with low calculation cost.

[0020] The present invention may further include a notification unit that notifies the driver of the result of the determination by the determination unit, thereby making it possible to warn the driver of the possibility of overlooking the obstacle.

[0021] In addition, in a determination device according to another embodiment of the present invention, a generation unit generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image captured from a moving body of the outside. Meanwhile, a type of object to be detected is set in a setting unit, and an object detection unit detects an object of the type set in a setting unit of an area including an imaging range of the image. Then, the determination unit performs an oversight possibility determination for the object detected by the object detection unit based on the visual saliency distribution information. In this way, it is possible to determine the oversight possibility by combining the visual saliency distribution information and object detection. Therefore, it is possible to detect an object that may be visually overlooked. In addition, object detection does not have to be based on an image, and object detection results of other sensors such as LiDAR (Light Detection and Ranging) can be used.

[0022] In addition, in the determination method according to one embodiment of the present invention, a generation step generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image captured from a moving body of the outside. Meanwhile, a setting step sets the type of object to be detected, and an object of the set type is detected from the image in an object detection step. Then, in a determination step, an oversight possibility determination is performed for the object detected in the object detection step based on the visual saliency distribution information. In this way, the oversight possibility can be determined by combining the visual saliency distribution information and the object detection. Therefore, an object that may be visually overlooked can be detected. In addition, since the oversight possibility can be determined only from an image captured from the outside of the moving body, it can be determined from an image of a drive recorder, for example, and gaze detection is not required.

[0023] In addition, the above-mentioned determination method is executed by a computer. In this way, it is possible to determine the possibility of overlooking by combining visual saliency distribution information and object detection using a computer. Therefore, it is possible to detect an object that may be visually overlooked.

[0024] The above-mentioned determination program may be stored in a computer-readable storage medium. In this way, the program can be distributed as a standalone program in addition to being incorporated into a device, and version upgrades can be easily performed.

[0025] In addition, in a determination method according to another embodiment of the present invention, a generation step generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image captured from a moving body of the outside. Meanwhile, a setting step sets the type of object to be detected, and an object of the type set in a setting section of an area including an imaging range of the image is detected in an object detection step. Then, in a determination step, a possibility of overlooking is determined for the object detected in the object detection step based on the visual saliency distribution information. In this way, it is possible to determine the possibility of overlooking by combining the visual saliency distribution information and object detection. Therefore, it is possible to detect an object that may be visually overlooked. Furthermore, object detection does not have to be based on an image, and object detection results of other sensors such as a lidar can be used.

[0026] In addition, the above-mentioned determination method is executed by a computer. In this way, it is possible to determine the possibility of overlooking by combining visual saliency distribution information and object detection using a computer. Therefore, it is possible to detect an object that may be visually overlooked.

[0027] The above-mentioned determination program may be stored in a computer-readable storage medium. In this way, the program can be distributed as a standalone program in addition to being incorporated into a device, and version upgrades can be easily performed. EXAMPLES

[0028] A determination device according to an embodiment of the present invention will be described with reference to Figs. 1 to 10. The determination device according to this embodiment is installed in a mobile body such as an automobile. However, it is not limited to mounting all of the components shown in Fig. 1 on the mobile body. As long as at least information presentation means 6, which will be described later, is installed on the mobile body, the other means may be configured as, for example, a server device, etc., and communication may be performed between the server device and the mobile body.

[0029] As shown in FIG. 1, the determination device 1 includes an input unit 2, a visual saliency extraction unit 3, an object region detection unit 4, an oversight determination unit 5, and an information presentation unit 6.

[0030] The input means 2 receives an image (still image or moving image) captured by, for example, a camera, and outputs the image as image data. If the input image is a moving image, it outputs the image data broken down into a time series, for example, for each frame. The image input to the input means 2 may be, for example, an image captured in the direction of travel of the vehicle, but may also be an image that includes a horizontal direction other than the travel direction, such as 180° or 360°, such as a so-called panoramic image. Moreover, the images input to the input means 2 are not limited to images captured by a camera, and may also be images read from a recording medium, such as a hard disk drive or a memory card.

[0031] The visual saliency extraction means 3 receives image data from the input means 2 and outputs a visual saliency map as visual saliency estimation information described later. That is, the visual saliency extraction means 3 functions as a generation unit that generates a visual saliency map (visual saliency distribution information) obtained by estimating the level of visual saliency based on an image captured of the outside from a moving body.

[0032] FIG. 2 is a block diagram illustrating the configuration of the visual saliency extraction means 3. The visual saliency extraction means 3 according to this embodiment includes an input unit 310, a nonlinear mapping unit 320, an output unit 330, and a storage unit 390. The input unit 310 converts an image into intermediate data that can be subjected to mapping processing. The nonlinear mapping unit 320 converts the intermediate data into mapping data. The output unit 330 generates saliency estimation information indicating a saliency distribution based on the mapping data. The nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. The storage unit 390 holds image data input from the input unit 2, coefficients of a filter to be described later, and the like. This will be described in detail below.

[0033] FIG. 3(a) is a diagram illustrating an example of an image input to the visual saliency extraction means 3, and FIG. 3(b) is a diagram illustrating an example of an image showing a visual saliency distribution estimated for FIG. 3(a). The visual saliency extraction means 3 according to this embodiment is a device that estimates the visual saliency of each part in an image. Visual saliency means, for example, how conspicuous an area is or how easily it attracts gazes. Specifically, visual saliency is expressed as a probability or the like. Here, the magnitude of the probability corresponds to, for example, the probability that the gaze of a person who sees the image will be directed to that position.

[0034] 3(a) and 3(b) correspond to each other in position. In FIG. 3(a), the higher the visual saliency is, the higher the luminance is displayed in FIG. 3(b). The image showing the visual saliency distribution as shown in FIG. 3(b) is an example of a visual saliency map output by the output unit 330. In the example of this figure, the visual saliency is visualized with 256 gradations of luminance values. An example of the visual saliency map output by the output unit 330 will be described in detail later.

[0035] FIG. 4 is a flowchart illustrating the operation of the visual saliency extraction means 3 according to this embodiment. The flowchart shown in FIG. 4 is a part of a determination method executed by a computer, and includes an input step S110, a nonlinear mapping step S120, and an output step S130. In the input step S110, an image is converted into intermediate data that can be mapped. In the nonlinear mapping step S120, the intermediate data is converted into mapped data. In the output step S130, visual saliency estimation information indicating a saliency distribution is generated based on the mapped data. Here, the nonlinear mapping step S120 includes a feature extraction step S121 in which features are extracted from the intermediate data, and an upsampling step S122 in which upsampling is performed on the data generated in the feature extraction step S121.

[0036] Returning to FIG. 2, each component of the visual saliency extraction means 3 will be described. In an input step S110, the input unit 310 acquires an image and converts it into intermediate data. The input unit 310 acquires image data from the input means 2. The input unit 310 then converts the acquired image into intermediate data. The intermediate data is not particularly limited as long as it is data that the nonlinear mapping unit 320 can accept, and is, for example, a high-dimensional tensor. In addition, the intermediate data is, for example, data in which the luminance of the acquired image is normalized, or data in which each pixel of the acquired image is converted into a luminance gradient. In the input step S110, the input unit 310 may further perform noise removal, resolution conversion, etc. on the image.

[0037] In the nonlinear mapping step S120, the nonlinear mapping unit 320 acquires intermediate data from the input unit 310. Then, the nonlinear mapping unit 320 converts the intermediate data into mapping data. Here, the mapping data is, for example, a high-dimensional tensor. The mapping process applied to the intermediate data by the nonlinear mapping unit 320 is, for example, a mapping process controllable by parameters or the like, and is preferably a process using a function, a functional, or a neural network.

[0038] Fig. 5 is a diagram illustrating a detailed configuration of the nonlinear mapping unit 320, and Fig. 6 is a diagram illustrating a configuration of the intermediate layer 323. As described above, the nonlinear mapping unit 320 includes a feature extraction unit 321 and an upsampling unit 322. The feature extraction unit 321 performs a feature extraction step S121, and the upsampling unit 322 performs an upsampling step S122. In the example of this figure, at least one of the feature extraction unit 321 and the upsampling unit 322 is configured to include a neural network including a plurality of intermediate layers 323. In the neural network, a plurality of intermediate layers 323 are connected.

[0039] In particular, the neural network is preferably a convolutional neural network. Specifically, each of the multiple intermediate layers 323 includes one or more convolutional layers 324. In the convolutional layer 324, input data is convolved by multiple filters 325, and activation processing is performed on the outputs of the multiple filters 325.

[0040] 5, feature extraction unit 321 is configured to include a neural network including a plurality of hidden layers 323, and includes a first pooling unit 326 between the plurality of hidden layers 323. Upsampling unit 322 is configured to include a neural network including a plurality of hidden layers 323, and includes an unpooling unit 328 between the plurality of hidden layers 323. Furthermore, feature extraction unit 321 and upsampling unit 322 are connected to each other via a second pooling unit 327 that performs overlap pooling.

[0041] In the example shown in the figure, each intermediate layer 323 is composed of two or more convolution layers 324. However, at least some of the intermediate layers 323 may be composed of only one convolution layer 324. Adjacent intermediate layers 323 are separated by any of a first pooling unit 326, a second pooling unit 327, and an unpooling unit 328. Here, when an intermediate layer 323 includes two or more convolution layers 324, it is preferable that the numbers of filters 325 in those convolution layers 324 are equal to each other.

[0042] In this figure, the intermediate layer 323 marked with "A×B" means that it is composed of B convolution layers 324, and each convolution layer 324 includes A convolution filters for each channel. Such an intermediate layer 323 is also referred to as an "A×B intermediate layer" below. For example, a 64×2 intermediate layer 323 means that it is composed of two convolution layers 324, and each convolution layer 324 includes 64 convolution filters for each channel.

[0043] In the example of this figure, the feature extraction unit 321 includes a 64×2 intermediate layer 323, a 128×2 intermediate layer 323, a 256×3 intermediate layer 323, and a 512×3 intermediate layer 323 in this order. The upsampling unit 322 includes a 512×3 intermediate layer 323, a 256×3 intermediate layer 323, a 128×2 intermediate layer 323, and a 64×2 intermediate layer 323 in this order. The second pooling unit 327 connects the two 512×3 intermediate layers 323 to each other. The number of intermediate layers 323 constituting the nonlinear mapping unit 320 is not particularly limited, and can be determined according to the number of pixels of the image data, for example.

[0044] Note that this figure is an example of the configuration of the nonlinear mapping unit 320, and the nonlinear mapping unit 320 may have another configuration. For example, a 64×1 intermediate layer 323 may be included instead of the 64×2 intermediate layer 323. By reducing the number of convolution layers 324 included in the intermediate layer 323, the calculation cost may be further reduced. Also, for example, a 32×2 intermediate layer 323 may be included instead of the 64×2 intermediate layer 323. By reducing the number of channels of the intermediate layer 323, the calculation cost may be further reduced. Furthermore, both the number of convolution layers 324 and the number of channels in the intermediate layer 323 may be reduced.

[0045] Here, in the plurality of intermediate layers 323 included in the feature extraction unit 321, it is preferable that the number of filters 325 increases every time passing through the first pooling unit 326. Specifically, the first intermediate layer 323a and the second intermediate layer 323b are continuous with each other via the first pooling unit 326, and the second intermediate layer 323b is located at the subsequent stage of the first intermediate layer 323a. The first intermediate layer 323a is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N1, and the second intermediate layer 323b is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N2. At this time, it is preferable that N2 > N1 holds. More preferably, N2 = N1 × 2 holds.

[0046] Also, in the plurality of intermediate layers 323 included in the upsampling unit 322, it is preferable that the number of filters 325 decreases every time passing through the unpooling unit 328. Specifically, the third intermediate layer 323c and the fourth intermediate layer 323d are continuous with each other via the unpooling unit 328, and the fourth intermediate layer 323d is located at the subsequent stage of the third intermediate layer 323c. The third intermediate layer 323c is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N3, and the fourth intermediate layer 323d is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N4. At this time, it is preferable that N4 < N3 holds. More preferably, N3 = N4 × 2 holds.

[0047] The feature extraction unit 321 extracts image features with multiple abstraction levels, such as gradients and shapes, from the intermediate data acquired from the input unit 310 as channels of the intermediate layer 323. FIG. 6 illustrates the configuration of the 64×2 intermediate layer 323. The processing in the intermediate layer 323 will be described with reference to this figure. In the example of this figure, the intermediate layer 323 is composed of a first convolution layer 324a and a second convolution layer 324b, and each convolution layer 324 has 64 filters 325. In the first convolution layer 324a, convolution processing using the filters 325 is performed on each channel of the data input to the intermediate layer 323. For example, when the image input to the input unit 310 is an RGB image, processing is performed on each of the three channels h0i (i=1..3). Also, in the example of this figure, the filters 325 are 64 types of 3×3 filters, that is, a total of 64×3 types of filters. As a result of the convolution process, 64 results h0i,j (i=1..3, j=1..64) are obtained for each channel i.

[0048] Next, the activation unit 329 performs activation processing on the outputs of the multiple filters 325. Specifically, activation processing is performed on the sum of the corresponding elements for the corresponding results j of all channels. By this activation processing, the results h1i (i=1..64) of 64 channels, that is, the output of the first convolution layer 324a, are obtained as image features. The activation processing is not particularly limited, but it is preferable to use at least one of a hyperbolic function, a sigmoid function, and a normalized linear function.

[0049] Furthermore, the output data of the first convolution layer 324a is used as input data for the second convolution layer 324b, and the same processing as the first convolution layer 324a is performed in the second convolution layer 324b, and the result of 64 channels h2i (i=1..64), that is, the output of the second convolution layer 324b, is obtained as an image feature. The output of the second convolution layer 324b becomes the output data of this 64×2 intermediate layer 323.

[0050] Here, the structure of the filter 325 is not particularly limited, but is preferably a 3×3 two-dimensional filter. Moreover, the coefficients of each filter 325 can be set independently. In this embodiment, the coefficients of each filter 325 are held in the storage unit 390, and the nonlinear mapping unit 320 can read them and use them for processing. Here, the coefficients of the multiple filters 325 may be determined based on correction information generated and corrected using machine learning. For example, the correction information includes the coefficients of the multiple filters 325 as multiple correction parameters. The nonlinear mapping unit 320 can further use this correction information to convert the intermediate data into mapping data. The storage unit 390 may be provided in the visual saliency extraction means 3, or may be provided outside the visual saliency extraction means 3. Moreover, the nonlinear mapping unit 320 may acquire the correction information from the outside via a communication network.

[0051] 7(a) and 7(b) are diagrams showing examples of convolution processing performed by the filter 325. In both of FIG. 7(a) and FIG. 7(b), an example of 3×3 convolution is shown. The example of FIG. 7(a) is convolution processing using nearest neighbor elements. The example of FIG. 7(b) is convolution processing using neighbor elements with a distance of two or more. Note that convolution processing using neighbor elements with a distance of three or more is also possible. It is preferable that the filter 325 performs convolution processing using neighbor elements with a distance of two or more. This is because it is possible to extract a wider range of features and further improve the estimation accuracy of visual saliency.

[0052] The above describes the operation of the 64×2 intermediate layer 323. The operations of the other intermediate layers 323 (such as the 128×2 intermediate layer 323, the 256×3 intermediate layer 323, and the 512×3 intermediate layer 323) are the same as the operation of the 64×2 intermediate layer 323, except for the number of convolution layers 324 and the number of channels. In addition, the operations of the intermediate layer 323 in the feature extraction unit 321 and the operation of the intermediate layer 323 in the upsampling unit 322 are also the same as those described above.

[0053] FIG. 8(a) is a diagram for explaining the processing of the first pooling unit 326, FIG. 8(b) is a diagram for explaining the processing of the second pooling unit 327, and FIG. 8(c) is a diagram for explaining the processing of the unpooling unit 328.

[0054] In the feature extraction unit 321, the data output from the intermediate layer 323 is subjected to pooling processing for each channel in the first pooling unit 326, and then input to the next intermediate layer 323. In the first pooling unit 326, for example, non-overlapping pooling processing is performed. FIG. 8(a) shows a process of associating four 2×2 elements 30 with one element 30 for an element group included in each channel. In the first pooling unit 326, such association is performed for all elements 30. Here, the four 2×2 elements 30 are selected so that they do not overlap with each other. In this example, the number of elements in each channel is reduced to one-fourth. Note that, as long as the number of elements is reduced in the first pooling unit 326, the number of elements 30 before and after the association is not particularly limited.

[0055] The data output from the feature extraction unit 321 is input to the upsampling unit 322 via the second pooling unit 327. In the second pooling unit 327, overlap pooling is performed on the output data from the feature extraction unit 321. FIG. 8(b) shows a process of associating four 2×2 elements 30 with one element 30 while overlapping some of the elements 30. That is, in repeated association, some of the four 2×2 elements 30 in a certain association are also included in the four 2×2 elements 30 in the next association. The number of elements is not reduced in the second pooling unit 327 as shown in this figure. Note that the number of elements 30 before and after the association in the second pooling unit 327 is not particularly limited.

[0056] The methods of processing performed by the first pooling unit 326 and the second pooling unit 327 are not particularly limited, but examples include matching in which the maximum value of four elements 30 is matched to one element 30 (max pooling) and matching in which the average value of four elements 30 is matched to one element 30 (average pooling).

[0057] The data output from the second pooling unit 327 is input to the intermediate layer 323 in the upsampling unit 322. Then, the output data from the intermediate layer 323 of the upsampling unit 322 is subjected to an unpooling process for each channel in the unpooling unit 328, and then input to the next intermediate layer 323. Fig. 8(c) shows a process of expanding one element 30 to multiple elements 30. The method of expansion is not particularly limited, but a method of duplicating one element 30 to four elements 30 (2 x 2) can be given as an example.

[0058] The output data of the last intermediate layer 323 of the upsampling unit 322 is output from the nonlinear mapping unit 320 as mapping data and input to the output unit 330. In the output step S130, the output unit 330 generates and outputs a visual saliency map by performing, for example, normalization, resolution conversion, etc. on the data acquired from the nonlinear mapping unit 320. The visual saliency map is, for example, an image (image data) in which visual saliency is visualized by brightness values, as exemplified in FIG. 3(b). In addition, the visual saliency map may be, for example, an image that is colored according to visual saliency, such as a heat map, or an image in which visual saliency regions having visual saliency higher than a predetermined standard are marked so as to be distinguishable from other positions. Furthermore, the visual saliency estimation information is not limited to map information shown as an image or the like, and may be a table or the like that lists information indicating visual saliency regions.

[0059] The object region detection means 4 detects (detects) a specified object in the image data based on the image data input from the input means 2 and object designation information that designates the type of object to be detected. The object detection method performed by the object region detection means 4 may be a well-known method such as SSD (Single Shot Multiple Detector), and is not particularly limited. The object designation information may be, for example, a label of the object to be detected. The object designation information may also be held as internal information of the object region detection means 4. Labels of objects designated in the object designation information include, for example, moving bodies such as automobiles, motorcycles, bicycles, and pedestrians, but may also include road markings and road signs.

[0060] The object region detection means 4 outputs the detection result as region information indicating a region on the image (detection region). The region indicated by this region information does not have to follow the shape of the object, and may be, for example, a rectangular or circular region that includes the object. That is, the object region detection means 4 functions as a setting unit that sets the type of object to be detected, and also functions as an object detection unit that detects the set type of object from the image.

[0061] The oversight determination means 5 compares the visual saliency map output by the visual saliency extraction means 3 with the object region information detected by the object region detection means 4 to determine whether an object may be overlooked based on predetermined criteria information, and outputs the determination result as overlooked object information. The criteria information is information used as a criterion for determining whether an object has been overlooked, and may be, for example, a specific threshold value (scalar value or vector value). This criteria information may also be held as internal information of the oversight determination means 5. The oversight object information output by the oversight determination means 5 may be, for example, rectangular region information including the overlooked object or information indicating pixel coordinates. In other words, the oversight determination means 5 functions as a determination unit that performs an oversight possibility determination based on the visual saliency map (visual saliency distribution information) for the object detected by the object region detection means 4 (object detection unit).

[0062] The information presentation means 6 presents the object oversight information output by the oversight determination means 5. The information presentation means can be configured with a display device that displays the object oversight information. This display device is preferably installed in a position that is easily visible to the driver, such as a head-up display or in a meter.

[0063] Next, the operation (determination method) of the determination device 1 having the above-mentioned configuration will be described with reference to the flowchart in Fig. 9. Moreover, this flowchart can be configured as a program executed by a computer functioning as the determination device 1 to form a determination program. Moreover, this determination program is not limited to being stored in a memory or the like possessed by the determination device 1, but may also be stored in a storage medium such as a memory card or an optical disk.

[0064] First, the input means 2 outputs the input image as image data to the visual saliency extraction means 3 and the object region detection means 4 (step S210). In this step, in the case of a moving image, the image data input to the input means 2 is decomposed into a time series and input to the visual saliency extraction means 3 and the object region detection means 4. In this step, image processing such as noise removal and geometric transformation may also be performed.

[0065] Next, the visual saliency extraction means 3 extracts a visual saliency map (step S220). The visual saliency extraction means 3 outputs the visual saliency map as shown in FIG. 3(b) by the above-mentioned method.

[0066] In parallel with step S220, the object region detection means 4 outputs region information (step S230). The object region detection means 4 detects the region of an object present in the image data input from the input means 2 based on the object designation information and outputs the region information. An example of region information is shown in FIG. 10. FIG. 10 shows the image data shown in FIG. 3(a) to which region information has been added. As shown in FIG. 10, the region information is made up of region sections 41, 42 indicating the region including the detected object, and label names 43, 44 indicating the type, name, etc. of the detected object.

[0067] Region 41 is shown by a rectangular frame as illustrated. In Fig. 10, the frame is shown so as to include a "dog" detected as an object. Region 42 is also shown by a rectangular frame as illustrated. In Fig. 10, the frame is shown so as to include a "car" detected as an object. Note that the shape of regions 41 and 42 is not limited to a rectangle and may be a circle, an ellipse, or the like.

[0068] Label name 43 is shown adjacent to region 41 as illustrated. In Fig. 10, the label name of the detected object is shown as "Dog". Label name 44 is shown adjacent to region 42 as illustrated. In Fig. 10, the label name of the detected object is shown as "Car".

[0069] Returning to the description of Fig. 9, next, an oversight determination is performed (step S240). In the oversight determination, the oversight determination means 5 selects an object that may be visually overlooked from among the objects included in the region information based on the visual saliency map output by the visual saliency extraction means 3 and the region information output by the object region detection means 4, based on the determination criterion information, and outputs the selected object as oversight determination information.

[0070] For example, in the image illustrated in FIG. 3, if a "dog" and a "car" are detected in FIG. 3(a), and the high brightness part located in the center of the right side in the visual saliency map in FIG. 3(b) is judged to have high visual saliency exceeding the judgment reference information, the "dog" is an object located in the overlapping area when the area judged to have high visual saliency is superimposed on FIG. 3(a), and it is judged to be highly likely to be overlooked. On the other hand, the "car" is an object located in a non-overlapping area when the area judged to have high visual saliency is superimposed on FIG. 3(a), and it is judged to be highly likely to be overlooked. In other words, the oversight judgment means 5 (judgment unit) judges that, for the objects detected by the object area detection means 4 (object detection unit), objects that do not overlap with the area judged to have high visual saliency in the visual saliency map (visual saliency distribution information) are highly likely to be overlooked.

[0071] The following describes the details of the determination method used by the above-mentioned oversight determination means 5. For each object detected by the object region detection means 4, if the average visual saliency is equal to or less than a threshold value obtained as the determination criterion information, it is determined that there is a possibility of oversight using the following formula (1).

number

[0072] In equation (1), area(obj i ) is the area of ​​object i, (x,y)∈obj i are the coordinates of all pixels in object i, sal(x, y) is the visual saliency value of coordinates (x, y), and Th is the threshold value.

[0073] Then, the overlooked object is presented (step S250). In this embodiment, for example, the area and label name corresponding to the object that may be overlooked may be displayed in a color or font that stands out more than the area and label names corresponding to other objects, such as red. For example, if the "car" shown in FIG. 10 is determined to be an object that may be overlooked, the area 41 and label name 43 are displayed in red. Alternatively, for example, the area 41 and label name 43 may be displayed only for the object that may be overlooked.

[0074] According to this embodiment, the determination device 1 generates a visual saliency map obtained by the visual saliency extraction means 3 estimating the level of visual saliency from image data of the outside taken from a moving body such as a vehicle. Meanwhile, the object type to be detected is set in the object region detection means 4, and objects of the set type are detected from the image data. Then, the oversight determination means 5 performs oversight determination for the object detected by the object region detection means 4 based on the visual saliency map. In this way, oversight can be determined by combining the visual saliency map and object detection. Therefore, it is possible to detect objects that may be visually overlooked. In addition, since oversight can be determined only from images taken of the outside from a moving body, it can be determined from images taken by, for example, a drive recorder or an in-vehicle camera for ADAS (advanced driving system), and line of sight detection, etc. are not required.

[0075] In addition, the oversight determination means 5 performs oversight determination by comparing the visual saliency map with the object detected by the object region detection means 4. In this manner, oversight can be determined by comparing the visual saliency map with the captured image.

[0076] Moreover, the oversight determination means 5 determines that, with regard to the objects detected by the object region detection means 4, objects that do not overlap with regions determined to have high visual saliency are likely to be overlooked. In this way, it is possible to determine that objects located in parts of the image that do not have high visual saliency are likely to be overlooked.

[0077] Moreover, the visual saliency extraction means 3 includes an input unit 310 that converts an image into intermediate data that can be mapped, a nonlinear mapping unit 320 that converts the intermediate data into mapped data, and an output unit 330 that generates saliency estimation information indicating a saliency distribution based on the mapped data, and the nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. In this manner, visual saliency can be estimated with low calculation cost.

[0078] Also, there is provided an information presentation means 6 for presenting the result of the determination made by the oversight determination means 5. In this way, the result of the determination can be presented to the driver to warn him of the possibility of oversight.

[0079] In the above-described embodiment, the object region detection means 4 does not have to detect objects based on image data from the input means 2. For example, the results of detection by another sensor such as a lidar may be used. In this case, it is preferable that the object detection range of the other sensor is the same as the imaging range of the image data, and it is necessary to include at least the imaging range of the image data.

[0080] Furthermore, the present invention is not limited to the above-mentioned embodiment. In other words, a person skilled in the art can implement the present invention by modifying it in various ways in accordance with the conventionally known knowledge without departing from the gist of the present invention. As long as the determination device of the present invention is still included in such modifications, it is of course included in the scope of the present invention. [Explanation of symbols]

[0081] 1 Judgment device 2. Input Method 3 Visual saliency extraction means (generation unit) 4. Object region detection means (setting unit, object detection unit) 5. Oversight judgment means (judgment unit) 6 Information presentation means (presentation part)

Claims

1. a generation unit that generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image captured from a moving object of the outside; A setting unit for setting a type of object to be detected; an object detection unit that detects an object of a set type from the image; a determination unit that performs a likelihood of overlooking an object detected by the object detection unit based on a value obtained by dividing a sum of the visual saliency values ​​within a region of the object in the visual saliency distribution information by an area of ​​the region; A determination device comprising:

2. The determination device according to claim 1 , wherein the determination unit compares the value with a predetermined threshold value to determine the possibility of oversight.

3. 3 . The apparatus according to claim 1 , wherein the determination unit compares the object detected by the object detection unit with the visual saliency distribution information to determine a possibility of overlooking the object.

4. The determination device according to claim 3 , characterized in that the determination unit determines that, with respect to an object detected by the object detection unit, an object that does not overlap with an area determined to have high visual saliency in the visual saliency distribution information is likely to be overlooked.

5. The generation unit is an input unit for converting the image into intermediate data that can be subjected to mapping processing; a nonlinear mapping unit that converts the intermediate data into mapping data; an output unit that generates saliency estimation information indicating a saliency distribution based on the mapping data, the nonlinear mapping unit includes a feature extraction unit that extracts features from the intermediate data, and an upsampling unit that upsamples data generated by the feature extraction unit.

5. The determination device according to claim 1, wherein the determination device is a device for determining whether or not a vehicle is in a vehicle.

6. 6. The determination device according to claim 1, further comprising a presentation unit that presents a result of the determination made by the determination unit.

7. a generation unit that generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image captured from a moving object of the outside; A setting unit for setting a type of object to be detected; an object detection unit that detects an object of a type set in the setting unit in an area including an imaging range of the image; a determination unit that performs a likelihood of overlooking an object detected by the object detection unit based on a value obtained by dividing a sum of the visual saliency values ​​within a region of the object in the visual saliency distribution information by an area of ​​the region; A determination device comprising:

8. A method for determining a possibility of overlooking an object based on an image captured from a moving object, comprising: A generating step of generating visual saliency distribution information obtained by estimating the level of visual saliency from the image; A setting step for setting a type of object to be detected; an object detection step of detecting an object of a set type from the image; a determination step of determining a possibility of overlooking an object detected in the object detection step based on a value obtained by dividing a sum of the visual saliency values ​​within a region of the object in the visual saliency distribution information by an area of ​​the region; A method for determining whether or not a particular

9. A determination program for causing a computer to execute the determination method according to claim 8.

10. A computer-readable storage medium storing the determination program according to claim 9.

11. A method for determining a possibility of overlooking an object based on an image captured from a moving object, comprising: A generating step of generating visual saliency distribution information obtained by estimating the level of visual saliency from the image; A setting step for setting a type of object to be detected; an object detection step of detecting an object of the type set in the setting step in an area including an imaging range of the image; a determination step of determining a possibility of overlooking an object detected in the object detection step based on a value obtained by dividing a sum of the visual saliency values ​​within a region of the object in the visual saliency distribution information by an area of ​​the region; A method for determining whether or not a particular

12. A determination program for causing a computer to execute the determination method according to claim 11.

13. A computer-readable storage medium storing the determination program according to claim 12.

Images