Processing device, arithmetic device, processing method, and program

Abstract

To achieve saliency estimation at a small calculation cost.SOLUTION: A processing device 10 includes an input section 110, a nonlinear mapping section 120, and an output section 130. The input section 110 converts an image into intermediate data which can be mapped. The nonlinear mapping section 120 converts the intermediate data into mapping data. The output section 130 generates saliency estimation information indicating a saliency distribution on the basis of the mapping data. The nonlinear mapping section 120 includes a feature extraction section 121 for extracting features from the intermediate data and an up-sample section 122 for up-sampling the data generated by the feature extraction section 121.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a processing device, an arithmetic device, a processing method, and a program. [Background technology]

[0002] A technology for automatically detecting salient regions in an image has been proposed. Such technology is becoming increasingly important in relation to autonomous driving technology.

[0003] Patent Document 1 describes a method of setting a plurality of candidate regions, calculating the saliency of each of the plurality of candidate regions, and integrating the saliency of the plurality of candidate regions to generate saliency information for an input image.

[0004] Non-Patent Document 1 also describes that an input image is processed using a DCN (Dilated Convolutional Network), which is a type of convolutional network, and an LSTM (Long Short-Term Memory network), which is a type of time-series network. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-4480 A [Non-patent literature]

[0006] [Non-Patent Document 1] M. Cornia, et. al., "Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model", arXive:1611.09571v2, 17 Mar. 2017. Summary of the Invention [Problem to be solved by the invention]

[0007] However, the techniques of Patent Document 1 and Non-Patent Document 1 have a problem in that the calculation cost required for the saliency estimation operation is high.

[0008] One example of a problem to be solved by the present invention is to realize saliency estimation with low calculation cost. [Means for solving the problem]

[0009] The first invention is an input unit that converts the image into intermediate data that can be mapped; a nonlinear mapping unit that converts the intermediate data into mapping data; an output unit that generates saliency estimation information indicating a saliency distribution based on the mapping data, The nonlinear mapping unit is a processing device including a feature extraction unit that extracts features from the intermediate data, and an upsampling unit that upsamples the data generated by the feature extraction unit.

[0010] The second invention is: 3. A computing device for generating the correction information used in the processing device according to claim 2, an error calculation unit that calculates an error between the saliency distribution indicated by the saliency estimation information and the saliency distribution indicated by the saliency measurement information, using the saliency estimation information generated for the teacher image and saliency measurement information indicating a saliency distribution actually measured for the teacher image; a correction unit that calculates the correction information based on the error.

[0011] The third invention is A computer-implemented processing method comprising: an input step of converting the image into intermediate data that can be processed by mapping; a nonlinear mapping step of converting the intermediate data into mapping data; and an output step of generating saliency estimation information indicating a saliency distribution based on the mapping data, The nonlinear mapping step is a processing method including a feature extraction step of extracting features from the intermediate data, and an upsampling step of upsampling the data generated in the feature extraction step.

[0012] The fourth invention is A program that causes a computer to execute each step of the processing method according to the third invention. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram illustrating a configuration of a processing device according to a first embodiment. [Figure 2] 1A is a diagram illustrating an example of an image to be input to the processing device, and FIG. 1B is a diagram illustrating an example of an image showing a saliency distribution estimated for FIG. 1A. [Figure 3] 4 is a flowchart illustrating a processing method according to the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating in detail an example of the configuration of a nonlinear mapping unit. [Figure 5] FIG. 2 is a diagram illustrating the configuration of an intermediate layer. [Figure 6] 10(a) and 10(b) are diagrams illustrating examples of convolution processing performed by a filter. [Figure 7] (a) is a diagram for explaining the processing of the first pooling unit, (b) is a diagram for explaining the processing of the second pooling unit, and (c) is a diagram for explaining the processing of the unpooling unit. [Figure 8] FIG. 10 is a diagram illustrating a computer for realizing the processing device. [Figure 9] FIG. 10 is a diagram illustrating the configuration of a processing apparatus according to a second embodiment. [Figure 10] 10 is a flowchart illustrating a learning operation according to a second embodiment. [Figure 11] 10 is a diagram illustrating the configuration and usage environment of a computing device according to a third embodiment. FIG. [Figure 12] FIG. 10 is a diagram illustrating the configuration of a processing apparatus according to a fourth embodiment. [Figure 13] 10A and 10B are diagrams illustrating examples of images indicated by synthesis information generated by a synthesis unit. [Figure 14] FIG. 10 is a diagram illustrating the configuration of a processing apparatus according to a fifth embodiment. [Figure 15] (a) is an input image in Example 1A and Comparative Example 1A, (b) is a diagram showing the results of measuring the saliency for (a) using an eye tracker, (c) is a diagram showing the estimation results of saliency in Comparative Example 1A, and (d) is a diagram showing the estimation results of saliency in Example 1A. [Figure 16] 10(a) to 10(c) are diagrams showing the estimation results of saliency in Example 3A, Example 3B, and Example 3C, respectively. [Figure 17] (a) is an input image in Example 4A and Comparative Example 4A, (b) is saliency estimation information generated based on (a) in Example 4A, (c) is a diagram showing the object detection result in Comparative Example 4A, and (d) is a diagram showing the object detection result in Example 4A. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0015] In the following description, each component of the processing device 10 and the arithmetic device 20 is represented as a functional block, rather than as a hardware configuration, unless otherwise specified. Each component of the processing device 10 and the arithmetic device 20 is realized by any combination of hardware and software, centered on the CPU of any computer, memory, a program loaded into the memory, a storage medium such as a hard disk for storing the program, and a network connection interface. There are various variations in the realization method and device.

[0016] (First embodiment) FIG. 1 is a block diagram illustrating the configuration of a processing device 10 according to the first embodiment. The processing device 10 according to this embodiment includes an input unit 110, a nonlinear mapping unit 120, and an output unit 130. The input unit 110 converts an image into intermediate data that can be subjected to mapping processing. The nonlinear mapping unit 120 converts the intermediate data into mapped data. The output unit 130 generates saliency estimation information indicating a saliency distribution based on the mapped data. The nonlinear mapping unit 120 further includes a feature extraction unit 121 that extracts features from the intermediate data, and an upsampling unit 122 that upsamples the data generated by the feature extraction unit 121. These will be described in detail below.

[0017] FIG. 2(a) is a diagram illustrating an example of an image input to the processing device 10, and FIG. 2(b) is a diagram illustrating an example of an image showing a saliency distribution estimated for FIG. 2(a). The processing device 10 according to this embodiment is a device that estimates the saliency of each part in an image. Saliency means, for example, how easily it stands out or how easily it attracts attention. Specifically, saliency is expressed as a probability or the like. Here, the magnitude of the probability corresponds to, for example, the probability that a person viewing the image will direct their gaze to that position.

[0018] 2(a) and 2(b) correspond to each other in position. In FIG. 2(a), the higher the saliency, the higher the brightness displayed in FIG. 2(b). An image showing the saliency distribution as shown in FIG. 2(b) is an example of saliency estimation information output by the output unit 130. In the example of this figure, saliency is visualized using brightness values ​​of 256 levels. An example of saliency estimation information output by the output unit 130 will be described in detail later.

[0019] The estimation result of the saliency distribution can be used in various fields, such as predicting the gaze of traffic participants such as drivers and pedestrians, preventing oversight of traffic participants, evaluating the appearance of content such as advertising media, gaze guidance, digitizing the know-how of athletes and skilled technicians, understanding the visual cognition of living organisms, etc. Furthermore, the processing device 10 and processing method according to this embodiment can be applied to the mobility field such as autonomous driving, advanced driver assistance systems (ADAS), and road traffic systems, the entertainment field such as virtual reality (VR), augmented reality (AR), and games, the content field such as documents, video content, and signage, and the medical field such as image diagnosis, surgery support, and nursing care services.

[0020] FIG. 3 is a flowchart illustrating a processing method according to the first embodiment. The processing method according to this embodiment is a processing method executed by a computer and includes an input step S110, a nonlinear mapping step S120, and an output step S130. In the input step S110, an image is converted into intermediate data that can be subjected to mapping processing. In the nonlinear mapping step S120, the intermediate data is converted into mapped data. In the output step S130, saliency estimation information indicating a saliency distribution is generated based on the mapped data. Here, the nonlinear mapping step S120 includes a feature extraction step S121 that extracts features from the intermediate data, and an upsampling step S122 that upsamples the data generated in the feature extraction step S121. The processing method according to this embodiment is realized by a processing device 10 according to this embodiment.

[0021] Returning to FIG. 1, each component of the processing device 10 will be described. In an input step S110, the input unit 110 acquires an image and converts it into intermediate data. The input unit 110 acquires, for example, an image (image data) from a device external to the processing device 10. The external device is, for example, an imaging device such as a camera. Alternatively, the image may be stored in a storage unit 190 provided inside or outside the processing device 10, and the input unit 110 may read and acquire the image from the storage unit 190. The input unit 110 then converts the acquired image into intermediate data. The intermediate data is not particularly limited as long as it is data that can be accepted by the nonlinear mapping unit 120, but is, for example, a high-dimensional tensor. The intermediate data is, for example, data in which the brightness of the acquired image is normalized, or data in which each pixel of the acquired image is converted into a brightness gradient. In the input step S110, the input unit 110 may further perform noise removal, resolution conversion, etc. on the image.

[0022] In the nonlinear mapping step S120, the nonlinear mapping unit 120 acquires intermediate data from the input unit 110. The nonlinear mapping unit 120 then converts the intermediate data into mapping data. Here, the mapping data is, for example, a high-dimensional tensor. The mapping process performed on the intermediate data by the nonlinear mapping unit 120 is, for example, a mapping process that can be controlled by parameters, and is preferably a process using a function, a functional, or a neural network.

[0023] Fig. 4 is a diagram illustrating a detailed configuration of the nonlinear mapping unit 120, and Fig. 5 is a diagram illustrating a configuration of the intermediate layer 123. As described above, the nonlinear mapping unit 120 includes a feature extraction unit 121 and an upsampling unit 122. The feature extraction unit 121 performs a feature extraction step S121, and the upsampling unit 122 performs an upsampling step S122. In the example shown in the figure, at least one of the feature extraction unit 121 and the upsampling unit 122 includes a neural network including a plurality of intermediate layers 123. In the neural network, a plurality of intermediate layers 123 are connected.

[0024] In particular, the neural network is preferably a convolutional neural network. Specifically, each of the multiple intermediate layers 123 includes one or more convolutional layers 124. In the convolutional layers 124, input data is convolved by multiple filters 125, and activation processing is performed on the outputs of the multiple filters 125.

[0025] In the example of Fig. 4, feature extraction unit 121 is configured to include a neural network including a plurality of hidden layers 123, and a first pooling unit 126 is provided between the plurality of hidden layers 123. Furthermore, upsampling unit 122 is configured to include a neural network including a plurality of hidden layers 123, and an unpooling unit 128 is provided between the plurality of hidden layers 123. Furthermore, feature extraction unit 121 and upsampling unit 122 are connected to each other via a second pooling unit 127 that performs overlap pooling.

[0026] In the example shown in the figure, each intermediate layer 123 is made up of two or more convolutional layers 124. However, at least some of the intermediate layers 123 may be made up of only one convolutional layer 124. Adjacent intermediate layers 123 are separated by any of a first pooling unit 126, a second pooling unit 127, and an unpooling unit 128. Here, when an intermediate layer 123 includes two or more convolutional layers 124, it is preferable that the numbers of filters 125 in those convolutional layers 124 are equal to each other.

[0027] In this figure, an "A×B" hidden layer 123 is composed of B convolution layers 124, and each convolution layer 124 includes A convolution filters for each channel. Such a hidden layer 123 is also referred to as an "A×B hidden layer" below. For example, a 64×2 hidden layer 123 is composed of two convolution layers 124, and each convolution layer 124 includes 64 convolution filters for each channel.

[0028] In the example shown in the figure, the feature extraction unit 121 includes a 64×2 hidden layer 123, a 128×2 hidden layer 123, a 256×3 hidden layer 123, and a 512×3 hidden layer 123, in this order. The upsampling unit 122 includes a 512×3 hidden layer 123, a 256×3 hidden layer 123, a 128×2 hidden layer 123, and a 64×2 hidden layer 123, in this order. The second pooling unit 127 connects the two 512×3 hidden layers 123 to each other. The number of hidden layers 123 constituting the nonlinear mapping unit 120 is not particularly limited and can be determined, for example, according to the number of pixels in the image data.

[0029] Note that this diagram is an example of the configuration of the nonlinear mapping unit 120, and the nonlinear mapping unit 120 may have other configurations. For example, a 64×1 intermediate layer 123 may be included instead of the 64×2 intermediate layer 123. Reducing the number of convolutional layers 124 included in the intermediate layer 123 may further reduce the computational cost. Also, for example, a 32×2 intermediate layer 123 may be included instead of the 64×2 intermediate layer 123. Reducing the number of channels in the intermediate layer 123 may further reduce the computational cost. Furthermore, both the number of convolutional layers 124 and the number of channels in the intermediate layer 123 may be reduced.

[0030] Here, in the multiple intermediate layers 123 included in the feature extraction unit 121, it is preferable that the number of filters 125 increases each time the first pooling unit 126 is passed through. Specifically, the first intermediate layer 123a and the second intermediate layer 123b are connected to each other via the first pooling unit 126, and the second intermediate layer 123b is located after the first intermediate layer 123a. The first intermediate layer 123a is configured with a convolutional layer 124 in which the number of filters 125 for each channel is N1, and the second intermediate layer 123b is configured with a convolutional layer 124 in which the number of filters 125 for each channel is N2. In this case, it is preferable that N2>N1. It is more preferable that N2=N1×2.

[0031] Also, in the plurality of intermediate layers 123 included in the upsampling unit 122, it is preferable that the number of filters 125 decreases every time passing through the unpooling unit 128. Specifically, the third intermediate layer 123c and the fourth intermediate layer 123d are continuous with each other via the unpooling unit 128, and the fourth intermediate layer 123d is located at the subsequent stage of the third intermediate layer 123c. The third intermediate layer 123c is composed of a convolutional layer 124 in which the number of filters 125 for each channel is N3, and the fourth intermediate layer 123d is composed of a convolutional layer 124 in which the number of filters 125 for each channel is N4. At this time, it is preferable that N4 < N3 holds. More preferably, N3 = N4 × 2 holds.

[0032] In the feature extraction unit 121, image features having a plurality of levels of abstraction, such as gradients and shapes, are extracted from the intermediate data acquired from the input unit 110 as channels of the intermediate layer 123. FIG. 5 illustrates the configuration of the 64×2 intermediate layer 123. Referring to this figure, the processing in the intermediate layer 123 will be described. In the example of this figure, the intermediate layer 123 is composed of a first convolutional layer 124a and a second convolutional layer 124b, and each convolutional layer 124 includes 64 filters 125. In the first convolutional layer 124a, convolution processing using the filter 125 is performed on each channel of the data input to the intermediate layer 123. For example, when the image input to the input unit 110 is an RGB image, processing is performed on each of the three channels h 0 i (i = 1..3). Also, in the example of this figure, the filter 125 is a 64 types of 3×3 filters, that is, a total of 64×3 types of filters. As a result of the convolution processing, for each channel i, 64 results h 0 i,j (i = 1..3, j = 1..64) are obtained.

[0033] Next, activation processing is performed on the outputs of the plurality of filters 125 in the activation unit 129. Specifically, activation processing is performed on the sum of corresponding elements for the corresponding results j of all channels. By this activation processing, results h of 64 channels 1i (i=1..64), i.e., the output of the first convolutional layer 124a, is obtained as the image feature. The activation process is not particularly limited, but a process using at least one of a hyperbolic function, a sigmoid function, and a rectified linear function is preferable.

[0034] Furthermore, the output data of the first convolution layer 124a is used as input data for the second convolution layer 124b, and the second convolution layer 124b performs the same processing as the first convolution layer 124a, and the result of 64 channels h 2 i (i=1..64), that is, the output of the second convolutional layer 124b, is obtained as the image features. The output of the second convolutional layer 124b becomes the output data of this 64×2 hidden layer 123.

[0035] Here, the structure of the filter 125 is not particularly limited, but is preferably a 3x3 two-dimensional filter. Furthermore, the coefficients of each filter 125 can be set independently. In this embodiment, the coefficients of each filter 125 are stored in the memory unit 190, and the nonlinear mapping unit 120 can read and use them for processing. Here, the coefficients of the multiple filters 125 may be determined based on correction information generated and corrected using machine learning. For example, the correction information includes the coefficients of the multiple filters 125 as multiple correction parameters. The nonlinear mapping unit 120 can further use this correction information to convert intermediate data into mapped data. The memory unit 190 may be provided in the processing device 10 or external to the processing device 10. Furthermore, the nonlinear mapping unit 120 may obtain the correction information from an external source via a communication network.

[0036] 6(a) and 6(b) are diagrams showing examples of convolution processing performed by the filter 125. Both of FIGS. 6(a) and 6(b) show examples of 3×3 convolution. The example in FIG. 6(a) is convolution processing using nearest neighbor elements. The example in FIG. 6(b) is convolution processing using neighbor elements with a distance of two or more. Note that convolution processing using neighbor elements with a distance of three or more is also possible. It is preferable that the filter 125 performs convolution processing using neighbor elements with a distance of two or more. This is because it allows for the extraction of a wider range of features and further improves the accuracy of saliency estimation.

[0037] The above has described the operation of the 64×2 hidden layer 123. The operations of the other hidden layers 123 (such as the 128×2 hidden layer 123, the 256×3 hidden layer 123, and the 512×3 hidden layer 123) are the same as the operation of the 64×2 hidden layer 123, except for the number of convolutional layers 124 and the number of channels. Furthermore, the operations of the hidden layer 123 in the feature extraction unit 121 and the hidden layer 123 in the upsampling unit 122 are also the same as those described above.

[0038] FIG. 7(a) is a diagram for explaining the processing of the first pooling unit 126, FIG. 7(b) is a diagram for explaining the processing of the second pooling unit 127, and FIG. 7(c) is a diagram for explaining the processing of the unpooling unit 128.

[0039] In the feature extraction unit 121, data output from the intermediate layer 123 is subjected to pooling processing for each channel in the first pooling unit 126, and then input to the next intermediate layer 123. The first pooling unit 126 performs, for example, non-overlapping pooling processing. FIG. 7(a) shows processing for associating four 2×2 elements 30 with one element 30 for a group of elements included in each channel. The first pooling unit 126 performs such association for all elements 30. Here, the four 2×2 elements 30 are selected so that they do not overlap with each other. In this example, the number of elements in each channel is reduced to one-fourth. Note that, as long as the number of elements is reduced in the first pooling unit 126, the number of elements 30 before and after the association is not particularly limited.

[0040] The data output from the feature extraction unit 121 is input to the upsampling unit 122 via the second pooling unit 127. The second pooling unit 127 performs overlap pooling on the output data from the feature extraction unit 121. FIG. 7(b) shows a process of associating four 2×2 elements 30 with one element 30 while overlapping some of the elements 30. That is, in repeated associations, some of the four 2×2 elements 30 in a certain association are also included in the four 2×2 elements 30 in the next association. The second pooling unit 127 shown in this figure does not reduce the number of elements. Note that the number of elements 30 before and after association in the second pooling unit 127 is not particularly limited.

[0041] The processing methods performed by the first pooling unit 126 and the second pooling unit 127 are not particularly limited, but examples include matching the maximum value of four elements 30 to one element 30 (max pooling) and matching the average value of four elements 30 to one element 30 (average pooling).

[0042] The data output from the second pooling unit 127 is input to the hidden layer 123 in the upsampling unit 122. Then, the output data from the hidden layer 123 of the upsampling unit 122 undergoes unpooling processing for each channel in the unpooling unit 128, and is then input to the next hidden layer 123. FIG. 7(c) shows processing for expanding one element 30 into multiple elements 30. The method of expansion is not particularly limited, but an example is a method of duplicating one element 30 into four elements 30 (2 × 2).

[0043] The output data of the final hidden layer 123 of the upsampling unit 122 is output as mapping data from the nonlinear mapping unit 120 and input to the output unit 130. In the output step S130, the output unit 130 generates and outputs saliency estimation information by performing, for example, normalization or resolution conversion on the data acquired from the nonlinear mapping unit 120. The saliency estimation information is, for example, an image (image data) in which saliency is visualized using brightness values, as shown in FIG. 2(b). The saliency estimation information may also be, for example, an image color-coded according to saliency, such as a heat map, or an image in which salient regions with a saliency higher than a predetermined standard are marked so as to be distinguishable from other regions. Furthermore, the saliency estimation information is not limited to an image, but may also be a table listing information indicating salient regions.

[0044] The saliency estimation information output from the output unit 130 may be subjected to various computer vision processes such as image segmentation, object recognition, and image classification within the processing device 10 or outside the processing device 10.

[0045] Each functional component of the processing device 10 may be realized by hardware that realizes the functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a case where each functional component of the processing device 10 is realized by a combination of hardware and software will be further described.

[0046] 8 is a diagram illustrating a computer 1000 for realizing the processing device 10. The computer 1000 is any computer. For example, the computer 1000 is an SoC (System On Chip), a Personal Computer (PC), a server machine, a tablet terminal, or a smartphone. The computer 1000 may be a dedicated computer designed to realize the processing device 10, or may be a general-purpose computer.

[0047] The computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path through which the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 transmit and receive data to and from each other. However, the method of interconnecting the processor 1040 and other components is not limited to bus connection. The processor 1040 may be any of various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device implemented using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device implemented using a hard disk, a solid state drive (SSD), a memory card, a read-only memory (ROM), or the like.

[0048] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0049] The network interface 1120 is an interface for connecting the computer 1000 to a network. This communication network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The network interface 1120 may be connected to the network wirelessly or by wire.

[0050] The storage device 1080 stores program modules that realize the various functional components of the processing device 10. The processor 1040 reads these program modules into the memory 1060 and executes them to realize the functions corresponding to the respective program modules.

[0051] Furthermore, when the storage unit 190 is provided inside the processing device 10, the storage unit 190 is realized using a storage device 1080, for example.

[0052] As described above, according to this embodiment, the nonlinear mapping unit 120 includes the feature extraction unit 121 that extracts features from the intermediate data, and the upsampling unit 122 that upsamples the data generated by the feature extraction unit 121. Therefore, saliency can be estimated with low calculation cost.

[0053] (Second embodiment) 9 is a diagram illustrating the configuration of a processing device 10 according to a second embodiment. The processing device 10 according to this embodiment is the same as the processing device 10 according to the first embodiment, except that it further includes an error calculation unit 140 and a correction unit 150. The error calculation unit 140 uses saliency estimation information generated for an image and saliency measurement information indicating a saliency distribution actually measured for the image to calculate an error between the saliency distribution indicated by the saliency estimation information and the saliency distribution indicated by the saliency measurement information. The correction unit 150 then corrects the correction information based on the calculated error.

[0054] The processing device 10 according to this embodiment performs an estimation operation and a learning operation. In the estimation operation, saliency estimation information for an input image is generated and output. The estimation operation is the same as that described in the first embodiment. In particular, in this embodiment, the nonlinear mapping unit 120 converts intermediate data into mapping data using correction information. Meanwhile, in the learning operation, machine learning is performed using a teacher image and saliency measurement information for the teacher image, and correction information is generated or modified (updated). The correction information is information used by the nonlinear mapping unit 120, and includes, for example, multiple correction parameters.

[0055] In this embodiment, the nonlinear mapping unit 120 converts the intermediate data into mapping data using the correction information. The correction information is information that has been generated and / or modified using machine learning. Specifically, as described in the first embodiment, the nonlinear mapping unit 120 includes multiple filters 125, and the coefficients of the multiple filters 125 are determined based on the correction information. For example, the correction information includes the coefficients of the multiple filters 125 as multiple correction parameters.

[0056] The processing device 10 according to this embodiment is realized by, for example, a computer 1000 as shown in Fig. 8. The storage device 1080 of the computer 1000 further stores program modules that realize an error calculation unit 140 and a correction unit 150.

[0057] FIG. 10 is a flowchart illustrating a learning operation according to the second embodiment. The learning operation will be described in detail below. For the learning operation, a teacher image and saliency measurement information for the teacher image are prepared. For example, the teacher image and the saliency measurement information are associated with each other and stored in the storage unit 190. The input unit 110 and the error calculation unit 140 can read out and use this information from the storage unit 190.

[0058] The teacher image is any image, such as a photograph. The saliency measurement information is generated based on the results of measuring, for example, the gaze of a person when looking at the teacher image using an eye tracker. The saliency measurement information can have the same form as the saliency estimation information. That is, the saliency measurement information may be an image that visualizes saliency using brightness values, or may be an image that is color-coded according to saliency, such as a heat map.

[0059] In the learning operation, the input step S110, the nonlinear mapping step S120, and the output step S130 are performed in the same manner as the input step S110, the nonlinear mapping step S120, and the output step S130 according to the first embodiment. However, the image acquired by the input unit 110 in the input step S110 is a teacher image. Furthermore, in the nonlinear mapping step S120, the nonlinear mapping unit 120 reads correction information from the storage unit 190. Then, the correction information is used to convert intermediate data into mapping data. Note that the nonlinear mapping unit 120 may directly acquire the correction information from the modification unit 150 instead of reading it from the storage unit 190. Furthermore, in the initial state, the correction parameters included in the correction information can be set to any values.

[0060] Next, in error calculation step S140, the error calculation unit 140 acquires saliency estimation information from the output unit 130. The error calculation unit 140 also acquires measured saliency information associated with the teacher image that is the source of the saliency estimation information. The error calculation unit 140 then calculates the error between the acquired saliency estimation information and the measured saliency information. The method for calculating the error is not particularly limited, but it is preferable to calculate at least one of the L1 distance, L2 distance (Euclidean distance, mean square error), Kullback-Leibler distance, Jensen-Shannon distance, and Pearson correlation coefficient, for example.

[0061] Specifically, the Euclidean distance is calculated by the following formula (1), the Kullback-Leibler distance is calculated by the following formula (2), and the Jensen-Shannon distance is calculated by the following formula (3).i indicates the estimation result (value based on saliency estimation information), and q i indicates the true value (value based on actual saliency measurement information).

[0062]

number

number

number

[0063] Next, in a correction step S150, the correction unit 150 acquires the error from the error calculation unit 140 and corrects the correction parameters so as to reduce the error. Then, the correction parameters held in the storage unit 190 are replaced with the corrected correction parameters. Here, the method for correcting the correction parameters is not particularly limited, but it is preferable to use at least one of the least squares method, quadratic programming, stochastic gradient descent (SGD), adaptive moment estimation (ADAM), and the calculus of variations, for example.

[0064] Here, there are many correction parameters to be corrected, and in order to efficiently determine their values ​​and estimate saliency with high accuracy, it is preferable to use statistical learning (machine learning) using a large amount of training data. Therefore, in the learning operation, it is preferable that machine learning be performed by cooperation of the nonlinear mapping unit 120, the error calculation unit 140, and the correction unit 150.

[0065] Note that instead of replacing the correction parameters stored in the storage unit 190 with the corrected correction parameters, the correction unit 150 may output the corrected correction parameters directly to the nonlinear mapping unit 120. In the next nonlinear mapping step S120, the nonlinear mapping unit 120 performs processing using the corrected correction parameters.

[0066] Note that one or more pieces of saliency measurement information may be associated with one teacher image. When multiple pieces of saliency measurement information are associated with one teacher image, the multiple pieces of saliency measurement information are based on different measurement results. Then, the error calculation unit 140 calculates the error between the saliency estimation information and each piece of saliency measurement information. Furthermore, the modification unit 150 modifies the correction parameters, for example, so that the sum of all errors becomes smaller.

[0067] The learning operation may be performed for multiple pairs of training images and saliency measurement information. By repeating the learning operation, the accuracy of saliency estimation is further improved.

[0068] The timing at which the learning operation is performed is not particularly limited. For example, the processing device 10 can accept a user's operation to start the learning operation. Then, based on the operation to start the learning operation, the processing device 10 can start the learning operation. Furthermore, the processing device 10 can end the learning operation based on a user's end operation or a predetermined end condition. Examples of the end condition include satisfying a predetermined number of repetitions of the learning operation or the error being equal to or less than a predetermined reference value.

[0069] As described above, according to this embodiment, similar to the first embodiment, the nonlinear mapping unit 120 includes the feature extraction unit 121 that extracts features from the intermediate data, and the upsampling unit 122 that upsamples the data generated by the feature extraction unit 121. Therefore, saliency can be estimated with low calculation cost.

[0070] Additionally, according to this embodiment, the processing device 10 includes an error calculation unit 140 and a correction unit 150. Therefore, using the correction information corrected by the learning operation, saliency estimation with higher accuracy is realized.

[0071] (Third embodiment) FIG. 11 is a diagram illustrating the configuration and usage environment of a calculation device 20 according to the third embodiment. The calculation device 20 according to this embodiment is a device that generates correction information used in the processing device 10. The calculation device 20 includes an error calculation unit 240 and a correction unit 250. The error calculation unit 240 uses saliency estimation information generated for a teacher image and saliency measurement information indicating a saliency distribution actually measured for the teacher image to calculate an error between the saliency distribution indicated by the saliency estimation information and the saliency distribution indicated by the saliency measurement information. The correction unit 250 calculates the correction information based on the error.

[0072] The processing device 10 according to this embodiment is similar to the processing device 10 according to the first embodiment. The processing device 10 according to this embodiment includes an input unit 110, a nonlinear mapping unit 120, and an output unit 130. The processing device 10 according to this embodiment may not include the error calculation unit 140 and the correction unit 150 described in the second embodiment. The input unit 110 according to this embodiment is the same as the input unit 110 according to at least one of the first and second embodiments, the nonlinear mapping unit 120 according to this embodiment is the same as the nonlinear mapping unit 120 according to at least one of the first and second embodiments, and the output unit 130 according to this embodiment is the same as the output unit 130 according to at least one of the first and second embodiments. The operation of the error calculation unit 240 according to this embodiment is the same as the operation of the error calculation unit 140 according to the second embodiment, and the operation of the correction unit 250 according to this embodiment is the same as the operation of the correction unit 150 according to the second embodiment. The processing device 10 and the arithmetic device 20 cooperate to perform the learning operation and estimation operation described in the second embodiment. Furthermore, the processing device 10 and the arithmetic device 20 may be physically separate from each other and may be connected to each other via, for example, a communication network.

[0073] In the learning operation according to this embodiment, it is preferable that the nonlinear mapping section 120, the error calculation section 240, and the correction section 250 cooperate to perform machine learning.

[0074] The output unit 130 may temporarily store the generated saliency estimation information in the storage unit 190, and the error calculation unit 240 may read out and use the saliency estimation information stored in the storage unit 190.

[0075] The arithmetic device 20 according to this embodiment is realized by, for example, a computer 1000 as shown in Fig. 8. A storage device 1080 of the computer 1000 stores program modules that realize the respective functional components of the arithmetic device 20.

[0076] In the example shown in the figure, the storage unit 190 is provided separately from the processing device 10 and the arithmetic device 20, but this is not limited to this example, and the storage unit 190 may be provided in the processing device 10 or in the arithmetic device 20. When the storage unit 190 is provided inside the arithmetic device 20, for example, the storage unit 190 is realized using a storage device 1080 of the computer 1000 that realizes the arithmetic device 20. Furthermore, the storage unit 190 may be realized by cooperation between the storage device 1080 of the computer 1000 that realizes the processing device 10 and the storage device 1080 of the computer 1000 that realizes the arithmetic device 20.

[0077] As described above, according to this embodiment, similar to the first embodiment, the nonlinear mapping unit 120 includes the feature extraction unit 121 that extracts features from the intermediate data, and the upsampling unit 122 that upsamples the data generated by the feature extraction unit 121. Therefore, saliency can be estimated with low calculation cost.

[0078] Additionally, according to this embodiment, the arithmetic device 20 includes an error calculation unit 240 and a correction unit 250. Therefore, using the correction information corrected by the learning operation, saliency estimation with higher accuracy is realized.

[0079] (Fourth embodiment) 12 is a diagram illustrating the configuration of a processing device 10 according to a fourth embodiment. The processing device 10 according to this embodiment is the same as the processing device 10 according to at least one of the first and second embodiments, except that it further includes a synthesis unit 160 and a display unit 180.

[0080] The synthesis unit 160 generates synthesis information by synthesizing the saliency distribution indicated by the saliency estimation information with the image (input image) input to the input unit 110. Specifically, the synthesis unit 160 acquires the saliency estimation information from the output unit 130, and acquires the input image from, for example, the storage unit 190. Then, the synthesis unit 160 outputs synthesis information that indicates both the input image and the saliency distribution. The synthesis information is output to, for example, the display unit 180 provided in the processing device 10. Furthermore, the synthesis information output from the synthesis unit 160 may be held in the storage unit 190 or acquired by an external device.

[0081] FIG. 13 is a diagram illustrating an example of an image indicated by the synthesis information generated by the synthesis unit 160. In the example of this figure, the synthesis information is an image in which the input image and a heat map indicating saliency are superimposed. The format of the synthesis information is not particularly limited. For example, the synthesis information may be an image in which a saliency region is enclosed by a circle or a square in the input image. The synthesis method is also not particularly limited, and examples include alpha blending.

[0082] The processing device 10 according to this embodiment can be implemented in, for example, a mobile terminal (smartphone, tablet, etc.) equipped with an imaging device such as a camera. In this case, while taking an image with the mobile terminal, it is possible to extract important objects with high salience on the spot and visualize them with good visibility.

[0083] The processing device 10 according to this embodiment is realized, for example, by a computer 1000 as shown in Fig. 8. The storage device 1080 of the computer 1000 further stores a program module that realizes a synthesis unit 160. The display unit 180 is, for example, a display device connected to an input / output interface 1100.

[0084] As described above, according to this embodiment, similar to the first embodiment, the nonlinear mapping unit 120 includes the feature extraction unit 121 that extracts features from the intermediate data, and the upsampling unit 122 that upsamples the data generated by the feature extraction unit 121. Therefore, saliency can be estimated with low calculation cost.

[0085] In addition, according to this embodiment, the processing device 10 further includes a synthesis unit 160. Therefore, the saliency at each position in the image can be visualized with good visibility.

[0086] (Fifth embodiment) 14 is a diagram illustrating the configuration of a processing device 10 according to a fifth embodiment. The processing device 10 according to this embodiment is the same as the processing device 10 according to at least one of the first, second, and fourth embodiments, except that it further includes a mask image generation unit 170, a region extraction unit 172, and an object detection unit 174.

[0087] The mask image generation unit 170 acquires the saliency estimation information from the output unit 130 and generates a mask image. Specifically, the mask image generation unit 170 generates a mask image in which, in the saliency distribution indicated by the saliency estimation information, regions where the saliency is lower than a predetermined criterion are designated as masked regions, and regions where the saliency is equal to or higher than the predetermined criterion are designated as non-masked regions. That is, the mask image generation unit 170 binarizes the saliency distribution. Here, the criterion is set in advance and stored in the storage unit 190, and the mask image generation unit 170 can read and use it.

[0088] The region extraction unit 172 acquires an input image and a mask image. Then, by applying the mask image to the input image, it extracts a highly salient region from the input image. For example, the region extraction unit 172 can extract a highly salient region from the input image by performing a logical operation on the input image and the mask image.

[0089] Then, the object detection unit 174 detects an object from the region extracted by the region extraction unit 172. The object detection method is not particularly limited, but an example is a method using a Single Shot Multibox Detector (SSD). In the processing device 10 according to this embodiment, highly salient regions are extracted in advance, and object detection is performed only in the extracted regions, thereby suppressing erroneous detection.

[0090] The processing device 10 according to this embodiment is mounted on a moving body such as an automobile, etc. The object detection result by the object detection unit 174 can be used for automatic driving and driving assistance.

[0091] The processing device 10 according to this embodiment is realized by, for example, a computer 1000 as shown in Fig. 8. The storage device 1080 of the computer 1000 further stores program modules that realize a mask image generation unit 170, a region extraction unit 172, and an object detection unit 174.

[0092] As described above, according to this embodiment, similar to the first embodiment, the nonlinear mapping unit 120 includes the feature extraction unit 121 that extracts features from the intermediate data, and the upsampling unit 122 that upsamples the data generated by the feature extraction unit 121. Therefore, saliency can be estimated with low calculation cost.

[0093] Additionally, according to this embodiment, the processing device 10 further includes a mask image generating unit 170, a region extracting unit 172, and an object detecting unit 174. Therefore, highly accurate object detection can be performed in the input image. [Example]

[0094] The above-described embodiment will be described in detail below with reference to examples, but the embodiment is not limited to the description of these examples.

[0095] Example 1 The saliency of an image was estimated using the processing method according to the first embodiment, and this was designated Example 1A. Specifically, in Example 1A, the saliency was estimated using a processing device having the configuration shown in Figures 1, 4, 5, 6(a), and 7(a) to 7(c). In addition, the saliency of the same image was estimated using the method described in Non-Patent Document 1, and this was designated Comparative Example 1A.

[0096] In the method of Non-Patent Document 1, image features are extracted from an input image using a dilated convolutional network, which is a type of convolutional neural network. Then, an attentive convolutional LSTM, which is a type of time-series network, is used to selectively focus on only highly salient features from the image features.

[0097] In contrast to this, in Example A1, a type of convolutional neural network is used as an upsampling unit to upsample (increase the resolution of) highly salient features from image features, thereby improving the accuracy of saliency extraction.

[0098] Table 1 shows the calculation costs of Comparative Example 1A and Example 1A. In addition, Table 1 also shows the ratio of the calculation costs when the calculation cost of Comparative Example 1A is set to 100%. In this table, the unit M is 10 6 is.

[0099] [Table 1]

[0100] In Comparative Example 1A, the neural network layer of the Attentive Convolutional LSTM was deep, which increased the computational cost of the estimation operation. In contrast, in Example 1A, the computational cost was effectively reduced by using an upsampling unit based on a convolutional neural network. In Comparative Example 1A and Example 1A, the intermediate data size was set to 320 x 240, and the computational cost of Comparative Example 1A was calculated based on publicly available software (http: / / saliency.mit.edu / results_cat2000.html).

[0101] Fig. 15(a) is an input image in Example 1A and Comparative Example 1A. Fig. 15(b) is a diagram showing the results of measuring the saliency of Fig. 15(a) using an eye tracker. Fig. 15(c) is a diagram showing the results of saliency estimation in Comparative Example 1A. Fig. 15(d) is a diagram showing the results of saliency estimation in Example 1A. In Figs. 15(b) to 15(d), areas with higher saliency are shown with higher brightness.

[0102] In the actual measurement results of Fig. 15(b), the difference was particularly noticeable in the portion indicated by the arrow in Fig. 15(a). Furthermore, in Example 1A, an estimated result closer to the actual measurement result than in Comparative Example 1A was obtained.

[0103] As described above, in Example 1A, saliency could be estimated with high accuracy at a lower calculation cost than in Comparative Example 1A.

[0104] Example 2 The saliency of the image was estimated using the same processing method as in Example 1A, resulting in Example 2A. The saliency of the same image was also estimated using the method described in Patent Document 1, resulting in Comparative Example 2A.

[0105] In the method of Patent Document 1, a local saliency acquisition unit uses a type of convolutional neural network to estimate local saliency on a pixel-by-pixel basis for each local region of an input image, and a global saliency acquisition unit estimates global saliency using the local saliency and feature vectors extracted from the object candidate region image.

[0106] In contrast, in Example A2, a type of convolutional neural network is used to extract image features from the entire image region of an input image. Then, the type of convolutional neural network is used as an upsampling unit to upsample (increase the resolution of) highly salient features from the image features. This improves the accuracy of saliency extraction.

[0107] Table 2 shows the calculation costs of Comparative Example 2A and Example 2A. In addition, Table 2 also shows the ratio of the calculation costs when the calculation cost of Comparative Example 2A is set to 100%. In this table, the unit M is 10 6 is.

[0108] [Table 2]

[0109] In Comparative Example 2A, the number of channels, filter size, and input size of the convolutional neural networks constituting the local saliency acquisition unit and global saliency acquisition unit were large, resulting in a high computational cost for the estimation operation. In contrast, in Example 2A, the feature extraction unit and upsampling unit could be configured using a convolutional neural network configuration with a small number of channels and filter size, effectively reducing the computational cost for the estimation operation. The computational cost for Comparative Example 2A was calculated based on Figure 1 and Table 1 of Patent Document 1. In Comparative Example 2A and Example 2A, the intermediate data sizes were 51 x 51 and 320 x 240, respectively.

[0110] As described above, in Example 2A, saliency could be estimated with lower calculation cost than in Comparative Example 2A.

[0111] Example 3 In Examples 3A, 3B, and 3C, the learning operation described in the second embodiment was performed, and the estimation operation was performed using the calculated correction parameters. The loss function (error) calculated in the learning operation was the Euclidean distance in Example 3A, the Kullback-Leibler distance in Example 3B, and the Jensen-Shannon distance in Example 3C. Other conditions were the same for Examples 3A, 3B, and 3C. Note that Example 3B corresponds to Example 1A and Example 2A.

[0112] 16(a) to 16(c) are diagrams showing the estimation results of saliency in Example 3A, Example 3B, and Example 3C, respectively. In FIGS. 16(a) to 16(c), areas with higher saliency are shown with higher brightness. In all Examples, estimation results closer to the actual measurement results than in Comparative Example 1A were obtained.

[0113] Example 4 In Example 4A, object detection was performed from the input image using the processing method described in the fifth embodiment. In Comparative Example 4A, object detection was performed directly from the input image. In Example 4A and Comparative Example 4A, object detection was performed using SSD.

[0114] FIG. 17(a) is an input image in Example 4A and Comparative Example 4A. FIG. 17(a) is specifically an image acquired by a drive recorder. FIG. 17(b) is saliency estimation information generated based on FIG. 17(a) in Example 4A. In this figure, areas with higher saliency are shown with higher brightness. FIG. 17(c) is a diagram showing the object detection result in Comparative Example 4A. FIG. 17(d) is a diagram showing the object detection result in Example 4A. In this figure, the black-filled area corresponds to the mask area.

[0115] In the detection results of Comparative Example 4A shown in Figure 17(c), objects were detected in areas A, B, C, and D, each enclosed by a square. In the detection results of Example 4A shown in Figure 17(d), objects were detected in areas E and F, each enclosed by a square. Area A is the area where an airplane was detected, area B is the area where a train was detected, areas C and E are the areas where a car was detected, and areas D and F are the areas where a bus was detected.

[0116] As can be seen from Fig. 17(c), in Comparative Example 4A, cars and buses were correctly detected, but road areas were detected as airplanes and trains, resulting in false positives. In contrast, as can be seen from Fig. 17(d), in Example 4A, airplanes and trains were not detected, and false positives were suppressed.

[0117] As described above, the object detection accuracy was higher in Example 4A than in Comparative Example 4A.

[0118] Although the embodiments and examples have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

[0119] Below, examples of reference forms are added. 1-1. An input unit that converts images into intermediate data that can be mapped; a nonlinear mapping unit that converts the intermediate data into mapping data; an output unit that generates saliency estimation information indicating a saliency distribution based on the mapping data, The nonlinear mapping unit is a processing device including a feature extraction unit that extracts features from the intermediate data, and an upsampling unit that upsamples the data generated by the feature extraction unit. 1-2. In the processing apparatus described in 1-1, the nonlinear mapping unit further uses correction information to convert the intermediate data into the mapped data; A processing device, wherein the correction information is information corrected using machine learning. 1-3. In the processing apparatus according to 1-2, an error calculation unit that calculates an error between the saliency distribution indicated by the saliency estimation information and the saliency distribution indicated by the saliency measurement information, using the saliency estimation information generated for the image and saliency measurement information indicating a saliency distribution actually measured for the image; a correction unit that corrects the correction information based on the error. 1-4. In the processing apparatus according to 1-2. or 1-3., the nonlinear mapping unit includes a plurality of filters; A processing device in which coefficients of the plurality of filters are determined based on the correction information. 1-5. In the processing apparatus according to any one of 1-1 to 1-3, A processing device in which at least one of the feature extraction unit and the upsampling unit includes a neural network including a plurality of intermediate layers. 1-6. In the processing apparatus according to 1-5, each of the plurality of hidden layers includes one or more convolutional layers; In the convolution layer, input data is convolved using a plurality of filters, and activation processing is performed on the outputs of the plurality of filters. 1-7. In the processing apparatus according to 1-6, The feature extraction unit The neural network includes the plurality of intermediate layers, A processing device comprising a first pooling unit between the plurality of intermediate layers. 1-8. In the processing device according to 1-7, the first intermediate layer and the second intermediate layer located after the first intermediate layer are continuous with each other via the first pooling unit, the first hidden layer is configured with the convolution layer in which the number of filters for each channel is N1, and the second hidden layer is configured with the convolution layer in which the number of filters for each channel is N2; A processing device for which N2>N1 holds. 1-9. In the processing apparatus according to any one of 1-6. to 1-8., the upsampling unit is configured to include the neural network including the plurality of intermediate layers, and a processing apparatus provided with an unpooling unit between the plurality of intermediate layers. 1-10. In the processing apparatus according to 1-9., the third intermediate layer and the fourth intermediate layer located after the third intermediate layer are continuously connected to each other via the unpooling unit, the third intermediate layer is composed of the convolutional layer in which the number of the filters for each channel is N3, and the fourth intermediate layer is composed of the convolutional layer in which the number of the filters for each channel is N4, and a processing apparatus in which N4 < N3 holds. 1-11. In the processing apparatus according to any one of 1-5. to 1-10., a processing apparatus in which the feature extraction unit and the upsampling unit are connected to each other via a second pooling unit that performs overlapping pooling. 1-12. An arithmetic unit for generating the correction information used in the processing apparatus according to 1-2., an error calculation unit that calculates an error between the saliency distribution indicated by the saliency estimation information generated for the teacher image and the saliency distribution indicated by the saliency measurement information indicating the saliency distribution actually measured for the teacher image, using the saliency estimation information and the saliency measurement information, and a correction unit that calculates the correction information based on the error. 2-1. A processing method executed by a computer, including an input step of converting an image into intermediate data capable of mapping processing, a non-linear mapping step of converting the intermediate data into mapping data, and an output step of generating saliency estimation information indicating a saliency distribution based on the mapping data. The nonlinear mapping step includes a feature extraction step of extracting features from the intermediate data, and an upsampling step of upsampling data generated in the feature extraction step. 2-2. In the processing method described in 2-1, In the nonlinear mapping step, the intermediate data is converted into the mapped data using further correction information; A processing method in which the correction information is information corrected using machine learning. 2-3. In the processing method described in 2-2, an error calculation step of calculating an error between the saliency distribution indicated by the saliency estimation information and the saliency distribution indicated by the saliency measurement information, using the saliency estimation information generated for the image and saliency measurement information indicating a saliency distribution actually measured for the image; and a correcting step of correcting the correction information based on the error. 2-4. In the processing method described in 2-2. or 2-3., The nonlinear mapping step involves processing using a plurality of filters; The processing method, wherein the coefficients of the plurality of filters are determined based on the correction information. 2-5. In the processing method according to any one of 2-1 to 2-3, A processing method in which at least one of the feature extraction step and the upsampling step is performed using a neural network including multiple hidden layers. 2-6. In the processing method described in 2-5., each of the plurality of hidden layers includes one or more convolutional layers; In the convolution layer, input data is convolved using a plurality of filters, and activation processing is performed on the outputs of the plurality of filters. 2-7. In the processing method described in 2-6, In the feature extraction step, Processing is performed using the neural network including the plurality of hidden layers; A processing method further including a first pooling step during the processing of the plurality of intermediate layers. 2-8. In the processing method according to 2-7., The processing of the first intermediate layer and the processing of the second intermediate layer located after the first intermediate layer are continuous with each other via the first pooling step, The first intermediate layer is composed of the convolutional layer where the number of the filters for each channel is N1, and the second intermediate layer is composed of the convolutional layer where the number of the filters for each channel is N2, A processing method in which N2 > N1 holds. 2-9. In the processing method according to any one of 2-6. to 2-8., In the upsampling step, Processing is performed using the neural network including the plurality of intermediate layers, A processing method including an unpooling step during the processing of the plurality of intermediate layers. 2-10. In the processing method according to 2-9., The processing of the third intermediate layer and the processing of the fourth intermediate layer located after the third intermediate layer are continuous with each other via the unpooling step, The third intermediate layer is composed of the convolutional layer where the number of the filters for each channel is N3, and the fourth intermediate layer is composed of the convolutional layer where the number of the filters for each channel is N4, A processing method in which N4 < N3 holds. 2-11. In the processing method according to any one of 2-5. to 2-10., A processing method in which the feature extraction step and the upsampling step are continuous with each other via a second pooling step that performs overlapping pooling. 2-12. An arithmetic method for generating the correction information used in the processing method according to 2-2., an error calculation step of calculating an error between the saliency distribution indicated by the saliency estimation information and the saliency distribution indicated by the saliency measurement information, using the saliency estimation information generated for the teacher image and saliency measurement information indicating a saliency distribution actually measured for the teacher image; and a correction step of calculating the correction information based on the error. 3-1. A program that causes a computer to execute each step of the processing method described in any one of 2-1 to 2-11. 3-2. A program that causes a computer to execute each step of the calculation method described in 2-12. [Explanation of symbols]

[0120] 10 Processing equipment 20 Arithmetic unit 30 elements 110 Input section 120 Nonlinear Mapping Section 121 Feature Extraction Unit 122 Upsampling section 123 Middle Class 124 convolutional layers 125 filters 126 First Pooling Section 127 Second Pooling Section 128 Unpooling Department 129 Activation part 130 Output section 140 Error calculation section 150 Correction section 160 Synthesis Section 170 Mask image generation unit 172 Region extraction part 174 Object detection unit 180 Display section 190 Memory section 240 Error calculation section 250 Correction Unit 1000 calculator 1020 Bus 1040 processor 1060 memory 1080 storage device 1100 Input / Output Interface 1120 Network Interface

Claims

1. an input unit that converts the image into intermediate data that can be mapped; a nonlinear mapping unit that converts the intermediate data into mapping data; an output unit that generates saliency estimation information indicating a saliency distribution based on the mapping data, The nonlinear mapping unit a feature extraction unit that extracts features from the intermediate data; an upsampling unit that upsamples the data generated by the feature extraction unit; a second pooling unit connected to the feature extraction unit and the upsampling unit, the second pooling unit performing overlap pooling; The saliency estimation information indicates the probability that the gaze of a person viewing the image will be directed to each position in the image. Processing equipment.

2. 2. The processing device according to claim 1, the nonlinear mapping unit further uses correction information to convert the intermediate data into the mapped data; A processing device, wherein the correction information is information corrected using machine learning.

3. 3. The processing device according to claim 2, an error calculation unit that calculates an error between the saliency distribution indicated by the saliency estimation information and the saliency distribution indicated by the saliency measurement information, using the saliency estimation information generated for the image and saliency measurement information indicating a saliency distribution actually measured for the image; a correction unit that corrects the correction information based on the error.

4. 4. The processing apparatus according to claim 2, wherein: the nonlinear mapping unit includes a plurality of filters; A processing device in which coefficients of the plurality of filters are determined based on the correction information.

5. 4. The processing apparatus according to claim 1, A processing device in which at least one of the feature extraction unit and the upsampling unit includes a neural network including a plurality of intermediate layers.

6. 6. The processing apparatus according to claim 5, each of the plurality of hidden layers includes one or more convolutional layers; In the convolution layer, input data is convolved using a plurality of filters, and activation processing is performed on the outputs of the plurality of filters.

7. 7. The processing apparatus according to claim 6, The feature extraction unit The neural network includes the plurality of intermediate layers, A processing device comprising a first pooling unit between the plurality of intermediate layers.

8. 8. The processing device according to claim 7, the first intermediate layer and the second intermediate layer located after the first intermediate layer are continuous with each other via the first pooling unit, The first hidden layer has a number of filters for each channel of N １ and the second hidden layer is configured such that the number of filters for each channel is N ２ and the convolution layer is N ２ >N １ A processing device in which

9. 9. The processing apparatus according to claim 6, The upsampling unit The neural network includes the plurality of intermediate layers, The processing apparatus includes an unpooling section between the plurality of intermediate layers.

10. 10. The processing device according to claim 9, the third intermediate layer and the fourth intermediate layer located after the third intermediate layer are continuous with each other via the un-pooling portion, The third hidden layer has a number of filters for each channel of N ３ and the fourth hidden layer is configured such that the number of filters for each channel is N ４ and the convolution layer is N ４ <N ３ A processing device in which

11. 3. A calculation device for generating the correction information used in the processing device according to claim 2, an error calculation unit that calculates an error between the saliency distribution indicated by the saliency estimation information and the saliency distribution indicated by the saliency measurement information, using the saliency estimation information generated for the teacher image and saliency measurement information indicating a saliency distribution actually measured for the teacher image; a correction unit that calculates the correction information based on the error.

12. A computer-implemented processing method comprising: an input step of converting the image into intermediate data that can be processed by mapping; a nonlinear mapping step of converting the intermediate data into mapping data; and an output step of generating saliency estimation information indicating a saliency distribution based on the mapping data, The nonlinear mapping step a feature extraction step of extracting features from the intermediate data; an upsampling step of upsampling the data generated in the feature extraction step; a second pooling step between the feature extraction step and the upsampling step, wherein the second pooling step performs overlap pooling; The saliency estimation information indicates the probability that the gaze of a person viewing the image will be directed to each position in the image. Processing method.

13. A program that causes a computer to execute each step of the processing method according to claim 12.

Images