Image recognition evaluation program product, image recognition evaluation method, evaluation device, and evaluation system

Abstract

The image recognition evaluation program is executed by an evaluation device that evaluates the recognition accuracy of an image recognition device that performs image segmentation, and causes the evaluation device to perform the following: image processing on an input image input to the image recognition device, generation of a plurality of processed input images, input of the generated plurality of processed input images to the image recognition device, image segmentation by the image recognition device, thereby acquiring a plurality of output images classified by category, and calculation of a variance value of the output images based on the acquired plurality of output images.

Description

Technical Field

[0001] This invention relates to an image recognition evaluation program, an image recognition evaluation method, an evaluation device, and an evaluation system. Background Technology

[0002] As an image recognition technique, semantic segmentation using a Fully Convolutional Network (FCN) is known (see, for example, Non-Patent Document 1). Semantic segmentation performs category classification (inference) on a pixel-by-pixel basis on a digital image that is input as an input image. That is, semantic segmentation classifies each pixel of the digital image into categories, and as the inference result, it segments the digital image into multiple image regions of different categories by labeling each classified pixel with its category, and outputs the resulting image.

[0003] Furthermore, as a technique for evaluating image recognition accuracy, a method called Bayesian SegNet is known (see, for example, Non-Patent Document 2). In Bayesian SegNet, a method called DropOut is used to randomly oscillate the internal state of the Network, and the fluctuation of the inference result is calculated. Moreover, if the calculated inference result fluctuates significantly, it is determined to have low reliability (recognition accuracy), and if the calculated inference result does not fluctuate, it is determined to have high reliability (recognition accuracy).

[0004] Existing technical documents

[0005] Non-patent literature

[0006] Non-patent literature 1: Hengshuang Zhao, et al. "Pyramid scene parsing network" IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2017

[0007] Non-patent document 2: Alex Kendall, et al. "Bayesian SegNet: Model Uncertaintyin Deep Convolutional Encoder-Decoder Architectures for Scene Understanding" arXiv:1511.02680v2 [cs.CV], 10 Oct 2016 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] In Non-Patent Document 2, the network structure needs to be modified because the internal state of the network is randomly oscillating. Here, the network being evaluated is a so-called Black Box Network, where the network structure is black-boxed. In this case, Non-Patent Document 2 presupposes a modification to the network structure, but on the other hand, it is impossible to modify the Black Box Network. Therefore, the method of Non-Patent Document 2 cannot be applied to Black Box Networks, making it difficult to evaluate the network recognition accuracy.

[0010] The purpose of this invention is to provide an image recognition evaluation program, image recognition evaluation method, evaluation device, and evaluation system that can evaluate the recognition accuracy of an image recognition device even when the image recognition device is black-boxed.

[0011] Technical means to solve the problem

[0012] One approach is an image recognition evaluation procedure, which is executed by an evaluation device that evaluates the recognition accuracy of an image recognition device that performs image segmentation. The evaluation device performs the following actions: processing an input image input to the image recognition device to generate multiple processed input images; inputting the generated multiple processed input images back to the image recognition device; obtaining multiple output images classified by category due to image segmentation performed by the image recognition device; and calculating the variance value of the output images based on the obtained multiple output images.

[0013] One approach is an image recognition evaluation method, which is performed by an evaluation device that evaluates the recognition accuracy of an image recognition device that performs image segmentation. The image recognition evaluation method is performed as follows: processing an input image input to the image recognition device to generate multiple processed input images; inputting the multiple processed input images to the image recognition device; performing image segmentation based on the image recognition device; obtaining multiple output images classified by category; and calculating the variance value of the output images based on the obtained multiple output images.

[0014] One type of evaluation device evaluates the recognition accuracy of an image recognition device performing image segmentation. It includes: an input / output unit that inputs an input image to the image recognition device and acquires an output image generated by the image recognition device; and a control unit that processes the input image to the image recognition device to generate multiple processed input images, inputs the generated multiple processed input images to the image recognition device, performs image segmentation by the image recognition device, acquires multiple output images classified by category, and calculates the variance value of the acquired multiple output images.

[0015] An evaluation system of one method includes: the evaluation device described above; and the image recognition device, which performs image segmentation on a plurality of processing input images input from the evaluation device, and outputs a plurality of output images classified by category to the evaluation device. Attached Figure Description

[0016] Figure 1 This is a diagram showing an overview of the evaluation system for the implementation method.

[0017] Figure 2 This is a diagram that shows the general function of the evaluation system in the implementation method during the evaluation process.

[0018] Figure 3 This is a diagram illustrating an example of an input image, a processed input image, and an output image.

[0019] Figure 4 This is a diagram representing an example of a variance image obtained by superimposing the input and output images.

[0020] Figure 5 This is a diagram illustrating an example of the processing related to the evaluation of an image recognition device. Detailed Implementation

[0021] The embodiments of this application will be described in detail with reference to the accompanying drawings. In the following description, the same reference numerals are sometimes used to denote the same constituent elements. Furthermore, repeated descriptions are sometimes omitted. Additionally, in describing the embodiments of this application, descriptions and illustrations of non-closely related content are also omitted.

[0022] (Implementation Method)

[0023] Figure 1 This is a diagram showing an overview of the evaluation system for the implementation method. Figure 2This diagram illustrates the general function of the evaluation system in the embodiment during evaluation. Evaluation system 1 is a system for evaluating the accuracy of image recognition based on image recognition device 5, and is configured to include image recognition device 5, which is the object of evaluation, and evaluation device 6 for evaluating image recognition device 5. In evaluation system 1, image recognition device 5 and evaluation device 6 are bidirectionally connected in a manner capable of communicating data. Furthermore, in this embodiment, evaluation system 1 is configured as a separate structure where image recognition device 5 and evaluation device 6 are independent, but this structure is not particularly limited. Evaluation system 1 may also be configured as a single device integrating image recognition device 5 and evaluation device 6.

[0024] Image recognition device 5 identifies objects contained in the input image I and outputs the recognition result as output image O. Image recognition device 5 inputs a photographic image captured by an imaging device such as a camera as input image I. Furthermore, although detailed later, during evaluation, the processed input image Ia generated in evaluation device 6 is input to image recognition device 5.

[0025] Image recognition device 5 performs image segmentation on input image I. Image segmentation refers to the classification of segmented image regions of a digital image, also known as category inference (category classification). That is, image segmentation involves determining which category a segmented region of a digital image belongs to by labeling it with an identifier (category) to identify the category indicated by the image region, thereby segmenting the image region into multiple categories. Image recognition device 5 outputs the image after image segmentation (category inference) of input image I as output image O.

[0026] Image recognition device 5 is installed in, for example, a vehicle-mounted recognition camera. The vehicle-mounted recognition camera captures images of the vehicle's driving status in real time at a predetermined frame rate and inputs the captured images to image recognition device 5. Image recognition device 5 acquires the captured images input at the predetermined frame rate as input image I. Image recognition device 5 classifies the object categories contained in input image I and outputs the classified image as output image O at the predetermined frame rate. Furthermore, image recognition device 5 is not limited to being mounted on a vehicle-mounted recognition camera and can also be installed in other devices.

[0027] The image recognition device 5 includes a control unit 11, a storage unit 12, and an image recognition unit 13. The storage unit 12 stores programs and data. Additionally, the storage unit 12 can also be used as an operating area to temporarily store the processing results of the control unit 11. The storage unit 12 can include any storage device such as a semiconductor storage device or a magnetic storage device. Furthermore, the storage unit 12 can also include multiple storage devices. Additionally, the storage unit 12 can also include a combination of a portable storage medium such as a memory card and a storage medium reading device.

[0028] The control unit 11 coordinates the operation of the image recognition device 5 to achieve various functions. The control unit 11 includes, for example, an integrated circuit such as a CPU (Central Processing Unit). Specifically, the control unit 11 executes commands contained in a program stored in the storage unit 12, and achieves various functions by controlling the image recognition unit 13, etc. For example, the control unit 11 performs image recognition based on the image recognition unit 13 by executing a program related to image recognition.

[0029] The image recognition unit 13 includes an integrated circuit such as a GPU (Graphics Processing Unit). The image recognition unit 13 performs image segmentation, for example, using semantic segmentation. Semantic segmentation infers the category of each pixel in the input image I, and by labeling each classified pixel with its category, it segments the input image I into regions according to multiple categories. When the input image I is input, the image recognition unit 13 performs image segmentation, thereby outputting an image O after classifying each pixel of the input image I into its category.

[0030] The image recognition unit 13 performs image segmentation using neural networks (hereinafter referred to as networks) such as FCNs (Fully Convolutional Networks) composed entirely of convolutional layers. The image recognition unit 13 uses a learned network, for example, a black-box network whose learning process is unclear. The image recognition unit 13 has an encoder 22 and a decoder 23.

[0031] Encoder 22 performs encoding processing on the input image I. Encoding processing involves simultaneously generating a feature map that extracts features from the input image I and performing downsampling (also known as pooling) to reduce the resolution of the feature map. Specifically, in the encoding processing, the input image I is processed in convolutional and pooling layers. In the convolutional layers, kernels (filters) used to extract features from the input image I move across the input image I with a specified stride. Then, in the convolutional layers, convolutional computations are performed based on the weights of the convolutional layers to extract the features from the input image I, generating a feature map with extracted features. The generated feature map is generated with a number of channels corresponding to the number of channels in the kernel. In the pooling layers, the feature map with extracted features is reduced in size, resulting in a low-resolution feature map. In the encoding processing, by repeatedly performing the processing in the convolutional layers and the pooling layers, a feature map with downsampled features is generated.

[0032] Decoder 23 performs decoding processing on the encoded feature map. Decoding processing involves upsampling (also known as unpooling) to increase the resolution of the feature map. Specifically, decoding processing operates on the feature map in deconvolution layers and unpooling layers. In the unpooling layer, the low-resolution feature map containing the feature quantities is amplified to generate a high-resolution feature map. In the deconvolution layer, deconvolution calculations are performed based on the weights of the deconvolution layer to restore the feature quantities contained in the feature map, generating a feature map that restores the feature quantities. Then, in the decoding process, by repeatedly executing the processing in the unpooling layer and the processing in the deconvolution layer, an upsampled and region-segmented image, i.e., the output image O, is generated. The output image O is upsampled until it has the same resolution as the input image I input to the image recognition unit 7.

[0033] As shown above, the image recognition unit 13 performs encoding and decoding processing on the input image I, and performs image segmentation of the input image I by performing category inference (category classification) on a pixel-by-pixel basis. Then, the image recognition unit 13 outputs an image O that has been segmented into regions according to categories from the input image I.

[0034] Evaluation device 6 evaluates the recognition accuracy of image recognition device 5. Evaluation device 6 processes the input image I input to image recognition device 5 and evaluates the recognition accuracy based on the output image O output from image recognition device 5.

[0035] The evaluation device 6 includes a control unit 15, a storage unit 16, and an input / output unit 17. Furthermore, the storage unit 16 has almost the same structure as the storage unit 12 of the image recognition device 5, therefore its description is omitted.

[0036] The input / output unit 17 is an interface for inputting and outputting various data with the image recognition device 5. It inputs a processed input image Ia as the processed input image I to the image recognition device 5 and acquires the output image O generated by the image recognition device 5.

[0037] The control unit 15 coordinates the operation of the evaluation device 6 to achieve various functions. The control unit 15 includes an integrated circuit such as a CPU (Central Processing Unit). Specifically, the control unit 15 executes commands contained in a program stored in the storage unit 16 and achieves various functions by controlling the input / output unit 17, etc. The control unit 15 executes, for example, an image recognition evaluation program P related to the evaluation of the image recognition device 5, acquires an output image O from the image recognition device 5, and evaluates the recognition accuracy of the image recognition device 5 based on the acquired output image O. Furthermore, the control unit 15 processes the input image I input to the image recognition device 5 by executing the image recognition evaluation program P to generate a processed input image Ia.

[0038] like Figure 2 As shown, when the evaluation device 6 acquires the input image I, the evaluation system 1 processes the input image I to generate a processed input image Ia, and inputs the generated processed input image Ia to the image recognition unit 13. The image recognition unit 13 performs image segmentation of the processed input image Ia by performing encoding and decoding processing on the processed input image Ia. Then, the image recognition unit 13 outputs an image O to the evaluation device 6, which has been segmented into regions according to categories. The evaluation device 6 acquires the output image O and generates a variance image V for evaluating the image recognition device 5 based on the acquired output image O.

[0039] Furthermore, when the image recognition device 5 and the evaluation device 6 are a single integrated device, the control unit 11 and the control unit 15 can be the same control unit, and the storage unit 12 and the storage unit 16 can also be the same storage unit.

[0040] Next, refer to Figure 3 and Figure 4 The input image I, the processed input image Ia, the output image O, and the variance image V are described. Figure 3 This is a diagram illustrating an example of an input image, a processed input image, and an output image. Figure 4 This is a diagram representing an example of a variance image obtained by superimposing the input and output images.

[0041] Input image I is a digital image composed of multiple pixels. For example, input image I is an image with a resolution corresponding to the number of pixels of an imaging device such as a camera. That is, input image I is a high-resolution original image that has not undergone upsampling processing to increase the number of pixels or downsampling processing to decrease the number of pixels.

[0042] Processing the input image Ia is the image obtained after processing the input image I. In Figure 3 The following illustrations illustrate image processing examples 1 to 3, which serve as examples of processing input image Ia. Examples of image processing techniques include Burmester noise reduction, Gaussian noise reduction, gamma conversion, white balance adjustment, and blurring. In Image Processing Example 1, the input image Ia is an image that has undergone gamma conversion processing on input image I. In Image Processing Example 2, the input image Ia is an image that has undergone Gaussian noise reduction processing on input image I. In Image Processing Example 3, the input image Ia is an image that has undergone white balance adjustment processing on input image I.

[0043] The output image O is segmented into regions by category. Categories include, for example, objects contained in the input image I, such as people, vehicles, roads, buildings, etc. The output image O is categorized for each object at the pixel level; by labeling the category of each pixel unit, regions are segmented according to each category. Figure 3 In this context, images are categorized, for example, into categories such as people, vehicles, roads, and the sky. Additionally, as the output image O, there exists an output image O corresponding to the processed input image Ia. Figure 3 The diagram illustrates output images Example 1 to Example 3 corresponding to the processing input image Ia of Image Processing Examples 1 to 3. Output image O of Output Image Example 1 is the output image corresponding to the processing input image Ia of Image Processing Example 1. Output image O of Output Image Example 2 is the output image corresponding to the processing input image Ia of Image Processing Example 2. Output image O of Output Image Example 3 is the output image corresponding to the processing input image Ia of Image Processing Example 3. Figure 3 In the examples shown, output image O, from example 1 to example 3, represents a reduced recognition accuracy. Furthermore, Figure 3 The output image O is one example, but is not specifically limited to this category. Furthermore, the output image O has the same resolution as the input image I.

[0044] Figure 4 The upper image shown is the image where the input image I coincides with the output image O, and the lower image is the variance image V based on the input image I and the output image O. The variance image V is generated using multiple output images O, which are generated by processing the input image I to produce multiple processed input images Ia, and then inputting these processed input images Ia into the image recognition device 5. Alternatively, when generating the variance image V, multiple output images O corresponding to the multiple processed input images Ia generated by changing the type of image processing can also be used. Furthermore, when generating the variance image V, multiple output images O corresponding to the multiple processed input images Ia generated by randomly performing image processing without changing the type of image processing can also be used.

[0045] Specifically, the variance image V is an image that visualizes the variance values ​​of each pixel based on multiple output images O. In the variance image V, white image areas represent areas with low variance values, and black image areas represent areas with high variance values. That is, if the categories in the specified pixels of the multiple output images O are dispersed, the variance values ​​in the specified pixels of the variance image V are set to be higher, resulting in black image areas. On the other hand, if the categories in the specified pixels of the multiple output images O are not dispersed, the variance values ​​in the specified pixels of the variance image V are set to be lower, resulting in white image areas. Thus, the variance image V is an image in which the variance value is set for each pixel.

[0046] Next, refer to Figure 5 The processing related to the evaluation of the image recognition device 5 based on the evaluation device 6 will be explained. Figure 5 This is a diagram illustrating an example of the processing related to the evaluation of an image recognition device.

[0047] First, the input image I input to the image recognition device 5 is input to the evaluation device 6 (step S1). Then, the control unit 15 of the evaluation device 6 performs image processing on the input image I to generate multiple processed input images Ia (step S2). In step S2, multiple processed input images Ia can be generated by performing a predetermined type of image processing on the input image I multiple times; multiple processed input images Ia can also be generated by performing different types of image processing; or multiple processed input images Ia can be generated by performing both. Furthermore, when image processing is performed on the input image I, the image processing of the input image I is performed with a processing degree within a preset disturbance range. Here, the disturbance range is the range within which objects reflected in the input image I can be recognized even after image processing.

[0048] Next, the evaluation device 6 inputs the generated multiple processing input images Ia to the image recognition device 5 (step S3). When the processing input image Ia is input, the image recognition unit 13 performs encoding processing on the processing input image Ia (step S4). By performing the encoding processing, the image recognition unit 13 generates a feature map containing the downsampled feature quantities. The image recognition unit 13 performs decoding processing on the feature map containing the downsampled feature quantities (step S5). By performing the decoding processing, the image recognition unit 13 restores the feature map containing the feature quantities while upsampling to achieve the same resolution as the processing input image Ia. Then, the image recognition unit 13 performs category inference by dividing the image into regions by category in pixel units (step S6). The image recognition unit 13 generates an output image O, and by outputting the generated output image O to the evaluation device 6, the evaluation device 6 acquires the output image O as the result of the category inference (step S7). Steps S4 to S6 are executed multiple times depending on the number of processing input images Ia, and in step S7, multiple output images O corresponding to the multiple processing input images Ia are acquired.

[0049] Next, the evaluation device 6 calculates the variance value of the multiple output images O based on the acquired output images O (step S8). In step S8, the variance value of the category in each pixel is calculated using the multiple output images O. Afterwards, the evaluation device 6 generates and acquires a variance image V based on the variance value of the category in each pixel (step S9).

[0050] Next, the evaluation device 6 determines whether the variance value of the output image O is greater than a preset threshold (step S10). Here, the threshold is a value used to determine whether the inference based on the category classification of the image recognition device 5 is in a point inference state. A point inference state refers to a state where the image recognition device 5 performs low-robust learning, and therefore, peak (sensitive) inferences are made during the inference process. Specifically, a point inference state means that during the learning process of the image recognition device 5, when only images of the front of the object are used, the image recognition device 5 can only use images of the front of the object to make inferences, and it is difficult to make inferences about the object's state using images of the back of the object. In addition, in step S10, specifically, it is determined whether the variance value of the category of the output image O is greater than the preset threshold, and whether the inference is in a point inference state is determined according to the category.

[0051] If the variance of the output image O (category) is greater than the threshold (step S10: Yes), the evaluation device 6 determines that the image recognition device 5 is in the point estimation state (step S11). On the other hand, if the variance of the output image O (category) is less than the threshold (step S10: No), the evaluation device 6 determines that the image recognition device 5 is not in the point estimation state (step S12).

[0052] As described above, in the evaluation of the image recognition device 5 in this embodiment, by performing image processing on the input image I to perturb it, and then inputting the perturbed input image I, i.e., the processed input image Ia, into the image recognition device 5, the variance value of the output image O can be calculated. Therefore, even if the image recognition device is black-boxed, the input image I can be perturbed, and by performing an evaluation based on the variance value, the recognition accuracy of the image recognition device 5 can be appropriately evaluated.

[0053] Furthermore, in the evaluation of the image recognition device 5 in the embodiment, since the variance value of the category in each pixel of the output image O can be calculated, the recognition accuracy of the image recognition device 5, expressed in category units, can be appropriately evaluated.

[0054] Furthermore, in the evaluation of the image recognition device 5 in the embodiment, by comparing the variance value of the output image O with a preset threshold, it is possible to appropriately determine whether the image recognition device 5 is in a point estimation state.

[0055] Furthermore, in the evaluation of the image recognition device 5 in the embodiment, various image processing techniques such as Burmester noise processing, Gaussian noise processing, gamma conversion processing, white balance processing, and blurring processing can be used. Therefore, since various perturbations can be applied to the input image I, various evaluations of the recognition accuracy of the image recognition device 5 can be performed.

[0056] Furthermore, although the image recognition device 5 performs image segmentation using semantic segmentation in this embodiment, it is not limited to this. Other neural networks can also be used as the network for image recognition.

[0057] Explanation of reference numerals in the attached figures

[0058] 1 Evaluation System

[0059] 5 Image recognition devices

[0060] 6 Evaluation Devices

[0061] 11 Control Department

[0062] 12 Storage Units

[0063] 13 Image Recognition Department

[0064] 15 Control Department

[0065] 16 Storage Units

[0066] 17 Input / Output Section

[0067] 22 encoder

[0068] 23 decoders

[0069] P Image Recognition Evaluation Program

[0070] I input image

[0071] Ia Processing Input Image

[0072] O Output Image

[0073] V variance image

Claims

1. An image recognition evaluation program product, executed by an evaluation device that evaluates the recognition accuracy of an image recognition device performing image segmentation, wherein, The evaluation device shall perform the following: The input image input to the image recognition device is processed to generate multiple processed input images. The generated multiple processing input images are input to the image recognition device, which performs image segmentation to obtain multiple output images classified by category. Based on the acquired multiple output images, the variance value of the output images is calculated. The variance value of the output image is the variance value of the category that establishes a correspondence between each pixel of the output image. A threshold is preset to determine whether the inference based on the category classification of the image recognition device is in a point inference state. The evaluation device is also made to perform: Based on the calculated variance value of the output image and the threshold, it is determined whether it is in a point estimation state; as well as Determine whether the variance value of each category of the output image is greater than the threshold, and determine whether the inference is in a point inference state for each category. The point estimation state refers to the state of low robustness learning during the learning process of the image recognition device.

2. The image recognition evaluation program product as described in claim 1, wherein, The image processing includes at least one of the following: Berlin noise processing, Gaussian noise processing, gamma conversion processing, white balance processing, and blur processing.

3. An image recognition evaluation method, executed by an evaluation device that evaluates the recognition accuracy of an image recognition device performing image segmentation, wherein, The image recognition evaluation method is performed as follows: The input image input to the image recognition device is processed to generate multiple processed input images. The generated multiple processing input images are input to the image recognition device, and image segmentation based on the image recognition device is performed to obtain multiple output images classified by category. Based on the acquired multiple output images, the variance value of the output images is calculated. The variance value of the output image is the variance value of the category that establishes a correspondence between each pixel of the output image. A threshold is preset to determine whether the inference based on the category classification of the image recognition device is in a point inference state. The evaluation device is also made to perform: Based on the calculated variance value of the output image and the threshold, it is determined whether it is in a point estimation state; as well as Determine whether the variance value of each category of the output image is greater than the threshold, and determine whether the inference is in a point inference state for each category. The point estimation state refers to the state of low robustness learning during the learning process of the image recognition device.

4. An evaluation device for evaluating the recognition accuracy of an image recognition device performing image segmentation, wherein, have: The input / output unit inputs an input image to the image recognition device and acquires an output image generated by the image recognition device. as well as The control unit processes the input image input to the image recognition device to generate multiple processed input images. These multiple processed input images are then input to the image recognition device, which performs image segmentation to obtain multiple output images categorized by class. Based on the obtained output images, the control unit calculates the variance value of each output image. This variance value is the variance value of the category corresponding to each pixel of the output image. The control unit pre-sets a threshold for determining whether the inference based on the category classification of the image recognition device is in a point inference state. The control unit also causes the evaluation device to perform: Based on the calculated variance value of the output image and the threshold, it is determined whether the image is in a point estimation state, and Determine whether the variance value of each category of the output image is greater than the threshold, and determine whether the inference is in a point inference state for each category. The point estimation state refers to the state of low robustness learning during the learning process of the image recognition device.

5. An evaluation system, wherein, have: The evaluation device as described in claim 4; and The image recognition device performs image segmentation on multiple processing input images input from the evaluation device, and outputs multiple output images classified by category to the evaluation device.

Images